Learn what data silos are (definition); why they pose the biggest challenge to your big data strategy, pipelines and dreams; and how to break them down.
First things first, if you’re searching for articles on the topic “data silos” and what they mean, congratulations—you’ve almost attained the state of self-realization.🧘🏽 Why?
Because let’s face it—one of the biggest challenges facing data teams or those interested in making data-driven decisions is the lack of easy access to data. You would think that this problem would have been sorted by now, right? Well, turns out it’s not.
Just like Tom Cruise keeps on coming back to sort out the baddies of the world in the wildly popular MI movie franchise, data access problems keep on cropping up as well. 🤷
Data workers spend 90% of their working week (around 36 hours) on data-related activities such as searching, preparation and analytics (with searching for and preparing data being the most common activities).The State of Data Science and Analytics report by IDC
That’s the bad news. Is there good news when it comes to data silos? Read on to find out!
(By the way, we at Atlan are working on solving the problems of data discovery and access for data teams globally. If you’re impressed by the audaciousness of this plug and want to know more, please visit our website here.)
Now buckle up for the long drive ahead—let’s dive deeper into the concept of data silos. For starters (second starters that is)…
What are data silos, what does siloed data mean and how do you define them?
Much like the term suggests, data silos are literally situations when data exists in silos or in isolation.
When data silos exist, it means that some sets of people in an organization are able to access certain data or its sources and others aren’t.
And sometimes people aren’t able to get enough information on the context of data, or its metadata—including a whole bunch of its attributes such as lineage, meaning, business glossary, changes made or source.
The biggest problem with data silos is that there is no transparency or democracy in this equation. And the problem keeps on getting compounded when you add the scale and implications of using data in an enterprise.
Are data silos a new problem?
- Eight years ago, in 2012, McKinsey reported that “employees spend 1.8 hours every day—9.3 hours per week, on average—searching and gathering information.”
So why are we worried about data silos today?
Data silos are a more recognizable and pressing problem now given the scale, volume, veracity, applications and implications of big data.
- Companies are investing heavily in big data to gain a competitive edge—they were expected to invest $114 billion in big data in 2018, up from $31 billion in 2013.
And the problem of data silos cuts across the lines of teams or departments. Let’s take one of the most popular and sparkly teams in companies—marketing.
Most pioneering marketing teams today use data to make business decisions (also referred to as programmatic marketing or data-driven marketing in places).
Now if someone in marketing wants to optimize their ad spend, they will probably look at impressions generated, CTR and CPC.
Which is great—all the right boxes get ticked, right? Wrong—this data is incomplete.
This data only shows the marketer how to optimize the ads—but ideally they’d want to look at sales data and understand if they are even getting in the right kind of leads and traffic. Else the company will just end up bumping ad budgets but never seeing any increase in actual sales and customers.
Yeah, yeah. I know what you’re thinking—that this is not a data problem, it’s a mindset problem. But that’s the thing—data silos are not just about sharing the data, they are also about sharing the love, sorry… mindset.
Now, I know I’m jumping the gun here, considering that we still need to go over the challenges posed by data silos. So I’m just going to leave a crumb here, or fold the page a little if you will, so we remember to come back and understand the concept of—creating a healthy data culture and democratizing your data.
Now that we know what data silos are, let’s go on and understand what causes them in the first place. And that will tell us whether this mission is possible at all!
Causes of data silos—the baddies in the world of data
That’s like asking what causes the sun to shine or the birds to chirp. No, really!
The truth is that no matter which way you slice and dice the facts, data silos occur due to one of the two following reasons, or usually a combination:
- People… also known as humans of data in the Atlan universe 🧑🤝🧑
- Technology… associated with software, vendors and platforms 👨💻
When it comes to people, culture and organizational structures are the biggest challenges. Many times, departments are siloed and far removed from each other, especially in larger enterprises.
And while this is the natural order in most cases, and even seen as necessary for companies to function, the problem is that information ends up being tied to a particular department or team and doesn’t flow naturally from one desktop to another.
Simply because no one knows who else could use or is looking for that data. Multiple layers of hierarchy and management just make the problem worse.
And considering we are human, and data means power, a competitive mindset could just be getting in the way of data democratization.
Now, when you have all these different humans of data using data for different use cases and purposes, you are bound to have an overload of the tools and technologies they use.
For instance, maybe your Sales team uses Salesforce, but your Marketing team uses Hubspot. Yet, Salesforce might contain valuable information that your marketers could use.
Just think about it for an itty-bitty minute. How many apps and tools does your company use? (Interested in how people use apps at work? Check out this useful nugget).
And just like humans don’t talk to each other, tools don’t talk to each other. They could, but they are bound by their people and processes.
See how Atlan can help your tools and humans talk to each other by creating one easy-to-manage data ecosystem. Read more here!
And yes—you might say that your company is moving to or has already moved to the cloud—Google storage, Azure or AWS—which is great. But this is also making things very cloudy.
Do you know what’s in your cloud data lake and who has access to it?
TL;DR: The problem is that humans don’t talk to other human beings and tools don’t talk to other tools. They don’t exchange data or its context (metadata).
<Writer inserts “We don't talk anymore song” song here as the theme song>
So, what do you think? Can you solve these problems? But more importantly, do we even want to solve the problem of data silos?
Are data silos a big enough problem to solve? What do data silos cost you?
The short answer—time, effort and money. Can you put a price on that for your enterprise?
Now for the (slightly) long answer.
Silos significantly hamper workplace collaboration and productivity
This one is a no-brainer. If you’ve ever tried to hunt down your latest sales numbers or ask someone to explain what a particular table in your data set customer_2020_US means, you know what I mean.
When teams do not share data, they end up duplicating work and effort and getting demotivated in the process.
Less data sharing = wasted storage + inconsistent data
If your data lives in a million places, it will take up extra space—just like the gazillion printed copies of last year’s sustainability report.
The cloud may give you unlimited storage, but it can also give you unlimited chaos if not managed at the organizational level. And duplicate data will only threaten the accuracy of your data and create many sources of the un-truth.
Inability to see the big data picture
Have you ever heard of the Indian fable of the three blind men and an elephant?
“It is a story of a group of blind men, who have never come across an elephant before and who learn and conceptualize what the elephant is like by touching it. Each blind man feels a different part of the elephant’s body, but only one part, such as the side or the tusk. They then describe the elephant based on their limited experience…the moral of the parable is that humans have a tendency to claim absolute truth based on their limited, subjective experience as they ignore other people’s limited, subjective experiences which may be equally true.”
Data can be like that. The big elephant in the room. Data silos mean that you will never get a comprehensive view of data and the best way to use it for your use case.
Data silos are often associated with specific industries such as healthcare or marketing.
But the truth is that any function or organization that uses data as a moat may very well be dealing with data silos. (You can read this excellent article on data silos and customer centricity to understand how data affects customer success.)
Can we break down these data silos?
Yes—in these 2 steps of moderate difficulty levels. 🙂 #MissionPossible
1. Clean up your data culture
Remember the bookmark we added to the beginning of this blog, the one about creating a healthy data culture? Well, it’s important to educate your teammates about the importance of sharing and collaborating on data and becoming data literate.
2. Create one source of truth for your data
And if you’re a CDO or CAO, or any type of leader in an organization, even better. You can set up the data ecosystem in your company to be open and accessible, without compromising on data security and governance and compliance.
Think one tool to rule them all.
How can Atlan help you remove data silos?
But first… a little bit about us
As data team ourselves in a previous avatar, we too have dealt with the problem of data silos, incomplete context and missing information.
That too while working on large, impactful projects such as building the national data platform for the world’s largest democracy and global SDG tracking for the United Nations. You can read more here.
That’s why we have built a solution to help data teams around the world democratize their data lake initiatives, accelerate digital transformation and governance, and reduce time to insight by 60X.
All this in minutes, not months!
And now, here’s what we can do to help you with your silos problem
With Atlan, you can finally make your data and humans work better together:
- Automagically create a living catalog of all your data assets and knowledge: Make data and its metadata instantly discoverable via an intuitive Amazon Marketplace-like interface and Google-esque search functionalities. Try it out to believe it! And just as easily monitor your data asset’s usage and adoption across your applications with fine-grained access controls.
- Easily understand what your data means with one profile view: Everything you need to understand and trust your data and its quality in a single place—shareable via a single URL! Replete with a data dictionary, crowdsourced metadata & ratings, end-to-end data lineage, revision history, README editor, Data support desk and so much more.
- Make your tools talk to each other: Let your data live where you want it to and connect with the tools you choose. Make it easy for anyone to use the data they need, whenever they need it, and in whichever format they desire. No more tool overload! All this in a beautiful data UX designed for teamwork and collaboration. Read more here.
Too good to be true? Read how we successfully helped a Fortune500 go from data silos to data democratization. Get the full story here.
And that’s it, folks! Would love to hear from you—how do you manage data silos? Do share by leaving a comment below.