Data Governance at Elastic: Explained

Why the Right Data Catalog is Key to Their Success

Starting as a search engine built by Shay Banon in 2000 to help his wife search through a growing list of recipes during culinary school, Elastic has since grown into a legendary technology company, powering search solutions, helping applications run smoothly, and protecting against cyber threats for the world’s most respected organizations.

With a rich history in open source, and an offering that’s built on a foundation of simplicity, speed, scale, and relevance, and powered by data, Elastic has high expectations from their Data & Analytics team.

Responsible for that team is Takashi Ueki, Director of Enterprise Data & Analytics at Elastic, who joined Atlan at the 2023 Gartner Data & Analytics Summit, and led an Atlan Masterclass, to share how Elastic aligns its data governance practice with their culture, how they evaluated the Active Metadata Management market, and present and future use cases made possible by active metadata.

Democratizing Trusted Data Across Elastic

Joining Elastic in 2021, Takashi began by evaluating the state of the data practice. He found a number of opportunities for improvement, including multiple sources of truth that resulted in disconnected reporting, a BI platform strategy that was misaligned with organizational needs, and varying definitions making it difficult to accurately and consistently report.

“We lacked trust in our data, and we didn’t know what data to use. So I came in and said ‘Alright, we need a kind of North star. What’s our mission?’,” Takashi shared. “And I know this is going to look very cliche, but when I say ‘We want to democratize trusted data across Elastic,’ what that means is enabling the availability and reliability of our data, how we consume it, how we govern it, how we define it, and how we deliver on it.”

Key to how Takashi and his team support their stakeholders is close alignment with Elastic’s culture and operating model. “Even before COVID, Elastic was distributed and remote. It’s important to realize that when we think about how we operate as a company. You’ll see that in our source code, and you kind of think of that as a kind of culture statement,” Takashi explained.

Elastic is distributed by nature, and its team comes from diverse backgrounds. More than diversity in experience, vast differences exist in geography and culture, making it even more important that anything Takashi’s team creates and services is relevant and personalized to a spectrum of needs, expectations, and skill sets.

Aligning Data Governance with Culture

This cultural context is crucial to how Takashi’s team formulates their data governance strategy. “It means that whatever and however we deliver, it needs to meet people where they work. It needs to be embedded in a seamless part of the first of their day-to-day experience because we are a distributed company. And that is driving across three different areas. We drive transparency, we drive accountability and engagement,” Takashi explained.

Elastic’s data governance strategy interprets transparency as helping people understand where data is and who owns it. Accountability means ownership and stewardship, and that their stakeholders are accountable for the data they use.

Engagement, their final area of focus, means building a trusting, productive relationship with their stakeholders. “We need partnership from our business, from our IT partners. Data Governance isn’t a central dictate that we push upon,” Takashi shared. “We help with the framework, how to think about governance, how to drive that accountability and transparency to build and grow and to improve the governance of the quality of our data across the board.”

Choosing Active Metadata Management

With the goals of transparency, accountability, and engagement in mind, Elastic’s data team began to consider technology that could deliver fit-for-purpose experiences to their diverse stakeholders, make clear available data and capabilities, and serve as a collaboration layer as they built a partnership with all corners of the business.

“We recognized there was going to be a need for something to help enable us,” Takashi explained. “So we said ‘We’re going to go look at different data catalogs.’ We looked at some of the legacy players, and we also looked at some of the newer entrants in this space. Atlan being one of them.”

Elastic moved forward with proofs of concept with four vendors, finding that the core capabilities of each met their expectations, but that it was critical that the technology they chose aligned with their unique culture. “If we’re going to drive this program, it has to have a clean UI. It has to be something that people are going to want to use, and that’s going to be easy to adopt. And it has to integrate with our current stack,” Takashi explained.

Supporting Elastic’s remote work model meant that Takashi’s team would need to pay careful consideration to how their users would adopt a product. While collaboration might be simple in an office, where confirming a data asset’s definition or seeking approval for a change means walking to a colleague’s cubicle, collaborating with a remote workforce demands a thoughtfully constructed user experience.

“When we’re working remote and in a distributed manner, our workplace, our desktop is effectively the applications that we use. If I add another application that’s not integrated seamlessly, it just becomes more clutter on your desktop and it makes that experience more cumbersome,” Takashi explained.

Considering their data governance principles of transparency, accountability, and engagement, and combining their high standard for supporting a remote workforce, Atlan became a clear choice to drive their program forward. “When we looked at it from that perspective, it was really the only one that truly hit all those forms,” Takashi shared.

And with the importance of concepts like partnership and cultural alignment may not appear on an RFP, Elastic found Atlan to be the right partner as they grew their data governance practice. “It was more ‘What are you trying to accomplish as a company?’ And really being a partner in the space,” Takashi explained. “It’s not just ‘Here’s a data catalog.’ But ‘What’s your ultimate goal and objective?’ And helping to make sure we’re able to deliver.”

Crucial Context for Business Users

Among the most important set of stakeholders for Elastic’s data team are business users. Sharing a use case any data leader could relate to, Takashi elaborated, “You have this new dashboard, you send it to an executive, and they say ‘What am I supposed to be using this for? What are the terms related to? If I have any questions, who should I be reaching out to?”

Prior to Atlan’s arrival, crucial context was delivered via Slack or emails, with the data team sending a document about the dashboard, if one existed. Business users would then click away to read the document, then back to the dashboard. “It’s not the greatest experience,” Takashi shared.

Now, using Atlan’s Chrome Plug-in, relevant context is available in dashboards directly, avoiding pivoting into new applications or documents, or costly back-and-forth with subject matter experts. “This way, especially for executive business users where they’re not actively going to be in a data catalog, it puts the information at their fingertips, where they work,” Takashi explained.

An Early Warning System for Data Analysts

Another common pain point for data teams are pipeline breakages. Before adopting Atlan, Data Analysts were responsible for communicating these breakages downstream, typically using Slack channels that didn’t always include all affected stakeholders.

In the event of a pipeline breakage, Elastic’s data team can now use Atlan to identify all assets impacted downstream, from BigQuery to dependent Tableau dashboards, then trigger a warning for their users. “Instead of the experience of ‘I opened up a Tableau dashboard and I see something wrong’ and people chasing it, it’s ‘I opened a dashboard, I see there’s a warning on it. There was a pipeline breakage. I know something’s wrong.’,” Takashi shared.

Proactively addressing breakages and errors, as they happen, advances Elastic’s principle of transparency, building even more trust with stakeholders. And by utilizing Atlan’s close partnership with dbt and integration with Jira, Takashi’s team is beginning to automate these alerts.

“We’re going to expand to some of the data monitoring we’re doing in dbt, with the native integration in Atlan,” Takashi explained. “Monitoring triggers automated warnings, provides those alerts within slack, and then will be able to trigger and automate tickets in Jira. So people can start to follow along as the issue is getting resolved.”

Trusted Information for Architects and Engineers

Finally, Atlan is becoming a crucial part of Elastic’s data architecture toolkit, making more informed decisions, better onboarding users, and importantly, driving efficiency. Using Atlan’s popularity metrics, Elastic is identifying the most and least used assets in their data warehouse, and the cost to execute queries, and are using that information to optimize the performance and cost of their data estate.

But for new users, popularity metrics are a guide to understanding what data is relevant and appropriate. “Let’s say you’re new to the company and you’re a business analyst. You jump in and say ‘I need to use our bookings data,’ and you search within Atlan and realize ‘We have 20 different bookings tables in BigQuery, which one am I supposed to use?’,” Takashi explained. “It gives them that information at the fingertips of users to know what other people are using within the company. They’re able to self-serve on what fields and tables others are using so people are speaking from the same starting point.”

Present and Future Success with Atlan

Since the beginning of their Atlan journey, spending roughly 4 weeks to integrate key data sources and invite a core set of users, Elastic’s data team has made significant strides across enriching data assets, defining terms, and utilizing automated lineage. “We’ve been on this journey for 9 or 10 months, and we’ve made a lot of progress,” Takashi shared.

In this time, Elastic has integrated to key systems like BigQuery, dbt, Tableau, and Fivetran, has enabled seamless login with Okta, and uses Slack to share data assets and track data issues. Of particular note to Elastic’s team is growing engagement from end users. “We’ve had 572 Chrome Extension views, and we really want to drive that. It’s how we’re going to get more people engaged with Atlan,” Takashi explained.

With a foundation of success in place, Elastic’s future ambitions are clear. Their team’s next frontier with Atlan involves driving ownership and accountability, driving adoption through personalization, and maximizing the return on investment on critical data technology.

The concept of ownership has been a key factor in Elastic’s data governance strategy from the function’s very beginnings, with Takashi and his team identifying data owners, and confirming their agreement to steward their data. But over time, the need for more programmatic, data-driven strategies for ownership became clear. 

“How do we move the needle on ownership? What we’re thinking about over the next year is driving ownership through formalized data contracts,” Takashi shared. “We’re making sure to hold folks accountable for the quality of their data. It’s true ownership. With a contract in place, there’s expectations from a central governance perspective, and we’re setting up ways to hold them accountable.” For instance, Elastic intends to use Atlan to keep a close eye on data quality for critical fields, and how those metrics improve or decline over time, using data to drive accountability with each data owner.

Secondly, by personalizing their outreach, training, and experiences around Atlan, the Elastic team will drive more meaningful adoption of Atlan. “We’re developing ways of marketing Atlan to users to strip it down to what’s relevant to them,” Takashi explained. “For business users, we might say ‘Atlan is our Tableau catalog’ where they can know what reports and dashboards there are. For a new executive, we might say ‘Atlan is our metrics catalog, you can click through this one stop shop.’”

Last is a keen eye toward making the most of Elastic’s technology, integrating Atlan more deeply into a data estate that includes BigQuery, dbt, Fivetran, and Tableau. “We’re working on how we make all of these component parts work together. We want to make it a seamless experience, and maximize the ROI on our enterprise data.”

Raising Expectations

But while their by-the-numbers success has been substantial, and their short-to-mid-term priorities are clear, Elastic’s data team has far higher ambitions than they anticipated, with Atlan elevating their expectations of an Active Metadata Management platform and partner in success.

“The biggest takeaway for us, as we started this journey and as we think about the initial requirements that we were looking for in this space, the expectations that we thought we needed, has evolved and changed. It’s not just that we need things that you know you would expect from a data catalog. What we can get out of a tool like Atlan has kind of raised the bar of our expectations.” Takashi explained. 

Header photo: charlesdeluvio on Unsplash