Examples of Quantification from the Atlan Community
Survey results from Vendr’s SaaS Trends Report (Q3 2023) confirm what many data teams have already experienced: It’s getting a lot harder to invest in new technologies and approaches. Purchases are down 37% year-over-year, and scrutiny over the cost and benefit of investments mean that more than half the time data teams spend considering the right technology is spent on negotiation, alone.
With increasing scrutiny on the cost of existing technology and teams, and higher standards for investing in new technologies and resources, data teams are often asked to communicate the value of their programs and proposed initiatives. Where technical benefits were once persuasive enough to close gaps and inefficiencies in process and technology, data teams must focus on additional planes of value, and begin to quantify the operational benefits, and business value, of their work.
Data leaders throughout the Atlan Community have already begun this crucial change, moving beyond investment arguments that include modernity, scalability, speed, and user experience. In the below examples, you’ll see meticulous measurement of the progress of their programs, the operational benefits yielded by data teams and consumers, and the business value their data helped to unlock.
Finding, Measuring, and Communicating Value
The common thread of all value measurement is baseline analysis. Whether data teams are starting an enrichment program, automating a key workflow, or delivering new capabilities or data to their consumers, taking a moment to understand the “before state” makes it possible to measure your impact as your work progresses, then completes.
In the Atlan Community, we see three levels of value and associated Metrics and KPIs that data teams can and should be measuring:
- Program Value – Holding data teams and stakeholders to account on the pace, completeness, and quality of their work.
- Operational Value – Understanding the operational benefits and quantified efficiencies of new programs, processes, and technologies.
- Business Value – Moving beyond the “data as a service” model, and into one of partnership with the business, understanding the impact of data and analytics outputs on business initiatives.
1: Program Value
The first level of measurement for data teams is that of their programs. From understanding the satisfaction of their end users to tracking stakeholder contributions, there are numerous “low-hanging fruit” metrics that should be recorded and communicated to stakeholders and leaders.
Defined by progress against your team’s goals, and feedback from your end users, data teams should pay careful attention to the relative improvements they make through new technologies and processes, consistently speak with and survey data consumers, and set measurable goals for their stakeholders to commit to.
Tracking Stakeholder Contribution
Swapfiets, the world’s first bicycle subscription service, launched their Data Governance program by offering a self-service, modern data catalog on Atlan. Key to their success is a regular collaborating and reporting cadence, with six cross-functional Data Governance Committees composed of data analysts, data owners, and data stewards across their business domains.
In that monthly meeting, we focus on three governance topics, which are ownership, documentation, and quality. For documentation and ownership, Atlan is really handy. It’s also handy for quality to find out where in the data lifecycle quality issues arise. During the meeting, we write out action points and commit to tasks like verifying more terms.”
Lisa Smits, Data & Analytics Team Manager
Swapfiets now maintains 125 defined metrics across these six business domains, and with monthly committee meetings, has created a mechanism for accountability and attribution, and a platform for regularly reporting their progress toward enrichment goals.
Quantifying Experience & Satisfaction
Using experiential or NPS surveys, or a proxy metric like Slack messages, tickets, or escalations, are all potentially effective ways to quantify user satisfaction. Crucial to communicating a clear before-and-after is collecting a baseline measurement prior to releasing new projects or experiences.
For example, measuring the number of data questions a Data Engineering team receives prior to the release of a glossary, then three months into its adoption, paints a clear picture of changing user behaviors that can be easily communicated with stakeholders.
Chargebee, a market-leading solution for recurring revenue management, uses Atlan to create reusable data products, and to meet a growing internal appetite for data. Before the introduction of self-service, Chargebee’s Data Engineering team was responsible for responding to requests for data. Also responsible for delivering data to Chargebee’s customers, their Data Engineers often had to de-prioritize internal requests.
This was a huge challenge. Internal data requests were pushed to the back of the queues as customers were always a priority. This meant that we were not meeting SLAs and there was unhappiness among our stakeholders and lots of escalation, which led to unhappiness within the team as well.”
Lloyd Lamington, Business Solutions Manager, Chargebee
Prior to implementing Atlan, Lloyd and his team carefully scoped their “before state”, logging the number of data requests received per quarter, how many of them were repetitive or asked for raw data, and the number of escalations when SLAs were missed due to de-prioritization.
With a baseline established, Chargebee’s data team were able to quantify significant improvements after launching Atlan to support self-service, and that zero escalation requests were received since the rollout.
2: Operational Value
Beyond tracking user satisfaction and stakeholder contribution, significant opportunity exists for data teams to save cost, and drive efficiencies in the way that they, and their end-users, operate.
Crucial to accurate measurement, and a compelling narrative to communicate to leadership and stakeholders, is establishing a baseline measurement, or “before state” to compare improvements in process and technology against. Tools for establishing these baseline measurements may include:
- Financial Analysis – Keeping a thorough record of the cost of user licenses, compute, and data storage.
- Effort Scoping – Identifying the responsible team members, and the effort, cost, and time required of competing approaches or technologies.
- Data Team Surveys – Determining how much time is spent on servicing requests, repetitive tasks, and diagnosing breakages.
- Data Consumer Surveys – Determining how much time is spent on data discovery, or how long their requests take from submission to closure.
- Analyzing Systems – Calculating average project completion times or the volume of tickets in tools like Jira, or measuring the average number of Slack messages and the time their underlying issues take to complete.
With a baseline established, any improvements to teams, processes, and technology can and should be measured against it, then communicated to stakeholders and leadership.
Measuring Cost Savings
Familiar to many data teams are potential cost saving opportunities, such as optimizing data warehouse assets or deprecating unused dashboards, removing unutilized data pipelines, or optimizing expensive queries.
Mistertemp, a leader in recruitment and temporary staffing based in France, adopted Atlan to improve the navigability and usability of their newly implemented modern data stack.
Prior to implementing Atlan, Mistertemp’s policy of connecting with source data, then determining what was useful after the fact, was leading to significant technical debt, and reduced the navigability of their data estate.
All those quick decisions created a lot of assets in Snowflake that basically without a business use were never really touched or never really documented or never really connected to our BI tool or any other tool. So they just stayed there being synchronized, costing us money.”
David Milosevic, Head of Data and Analytics, Mistertemp
Using automated lineage and popularity metrics, David and his team determined that of their 30,000 data assets in Snowflake, just one-third were used in the preceding 12 months. These assets were then deprecated, significantly improving the navigability of their data estate, and reducing unnecessary cost and effort to maintain them.
Operational Value for Data Teams
Measuring Time-to-value for Technical Implementations
There are myriad options for data teams implementing new technologies and projects, from hand-coding, to open-source tools, to an array of SaaS solutions. But key to quantifying the value of these efforts is careful analysis of the effort involved in each of these alternatives.
In the case of replacing or modernizing existing technologies, data teams should take care to inventory the technical and experiential shortcomings of their legacy technology, and carefully scope the effort expended on their previous approach, drawing a clear comparison between new and old.
Porto, a Brazil-based insurance and banking leader with 13 million clients, sought to replace a legacy data catalog to maximize the ROI of their data stack, and improve data literacy across their organization.
We used to have a data catalog. We had Informatica EDC. The motivation to look for a different solution came from some difficulties in implementing certain features like lineage, a business glossary. It was kind of frustrating, and it came to a point where we felt like the technology was kind of blocking us from reaching where we wanted to go.”
Danrlei Alves, Senior Data Governance Analyst, Porto
Choosing Atlan to replace their legacy catalog, Porto’s team quickly integrated key data sources, then used custom metadata fields as a migration target for their existing documentation, metadata, and relationships between tables, columns, schemas, and databases.
In just six weeks, Porto’s data team launched their new catalog, with content at parity with their legacy solution. But by drawing on experience with their previous solution, and keeping a specific inventory of the improvements they desired, Porto’s team were able to draw a stark comparison between the two solutions, achieving in six weeks what once took them more than 2 years, communicating significant improvements in time-to-value.
In the case of one-time projects, data teams must spend time estimating the effort involved in each proposed approach, inclusive of the number of team members to allocate, time spent for each of these team members across the duration of the project, and the estimated completion date.
Tide, a UK-based digital bank with nearly 500,000 small business customers, sought to improve their compliance with GDPR’s Right to Erasure. Ensuring all personal data was deleted upon request first meant aligning on what constituted personal data, then determining where that data lived and how it moved through their architecture. With a sophisticated data estate, the manual effort involved was extensive.
People would have to go into the databases and try to translate my list of personal data elements. There were 31 elements to find in our databases, and we have more than 100 schemas, each with between 10 to 20 tables. So it would be a lot of work to identify it. If we were very diligent and did it for every schema, then it would probably be half a day for each schema. So half a day, 100 times.”
Michal Szymanski, Data Governance Manager, Tide
Using rules-based bulk automation in Atlan, the Tide team was able to identify, tag, and secure this personal data in mere hours, an impressive feat in and of itself. But by first scoping the baseline manual effort needed to complete the process, Tide’s data team were able to report nearly 50 days of effort saved.
Measuring Data Service Effort
For many data practitioners, a constant back-and-forth answering simple questions from data consumers is a significant drag on productivity. Despite the fact that this problem is easy to understand, data teams should take care to measure the impact these requests have on their productivity.
By surveying team members to understand the amount of time they spend on these questions, and by including a measurable metric like Slack messages received or tickets processed, data teams can create a clear baseline of effort, against which they can quantify productivity gains.
Octane, a speciality lender offering financing for powersports vehicles, uses Atlan to document their data assets, then make them available via self-service. Prior to introducing Atlan, questions about Octane’s data were answered in a shared Slack channel, where their Data Engineering team was responsible for responding.
Mindful of the potential efficiency benefits of introducing self-service with Atlan, Octane’s Data Product Manager, Alex Bendix, analyzed the amount of time their team spent on these questions.
Each engineer was answering questions 10 to 20% of their time. And then you multiply that over a certain number of employees across the team, that’s hundreds of hours per month that you’re losing in terms of productivity.”
Alex Bendix, Data Product Manager
By meticulously quantifying the amount of time their team spent answering questions about data, and measuring the number of Slack messages the team was receiving after implementing Atlan, Alex and his team were able to report that they were now saving 200 hours of Data Engineering effort per month.
Measuring Efficiencies by Eliminating Repetitive Tasks
Similar to measuring the effort expended answering questions, data teams should also take care to measure the effort they expend on recurring responsibilities, setting a baseline of effort to measure productivity gains from solutions such as automation.
Beyond their documented success accelerating implementation time, Porto’s Data Governance team has also quantified recurring savings from automating previously manual tasks.
Danrlei and his team started by identifying repetitive work, including assigning owners, documenting assets, and securing personal data, then created a framework to ensure that only the data assets that demanded their personal attention were surfaced to the Data Governance team.
If we consider everything we’re doing now with Atlan compared to before we had Atlan, we are saving 40% in efficiency, in terms of time and expensive operational tasks for everything related to governance. This is a 40% reduction of five people’s time. We’re using the time savings to focus on optimizing our processes and upleveling the type of work we are doing.”
Danrlei Alves, Senior Data Governance Analyst, Porto
With these automations running against a data estate of over 1 million assets, Danrlei and his team were quickly able to identify the reduced volume of governance tasks they were now responsible for, then report this significant improvement.
Measuring Time-to-resolution
Finally, productivity gains from accelerating “fire drill” tasks like Root Cause Analysis, or difficult investigatory tasks like Impact Analysis, also benefit from a clear accounting of efficiency gains. Thankfully, with these workstreams often managed on tools like Jira, measuring reductions in time-to-resolution can be as simple as analyzing time and effort spent on these tasks before and after improvements.
Takealot, a South African eCommerce and Retail leader, uses Atlan to improve technical understanding of their data estate, and drive business self-service. Prior to Atlan, each time Takealot’s data team were informed of a bug, they would conduct traditional sprint planning, identifying the time needed to investigate and resolve the issue.
By analyzing these resolution workstreams, their team determined they were spending half of their time investigating the problem and determining the location of a breakage, setting a baseline metric to improve against. Then, by utilizing automated column-level lineage on Atlan, the Takealot team could more quickly investigate the root cause of data issues, cutting their issue resolution time in half.
Instead of trawling through all the code, you can quickly follow lineage backwards and check it at every point to see what’s happening. Before, it could take a week or two weeks depending on how difficult a bug was to manage, with 50% of that time being investigating what the problem was and where it’s broken before actually applying the fix and getting it into production. I’d say we’ve probably halved that time.”
Group BI Manager, Takealot
Operational Value for Data Consumers
Measuring Friction in Data Discovery
Whether issues with navigability, strict access controls, or limited context, there are myriad reasons data consumers have a difficult time discovering and understanding the data available to them. In order to understand where to make improvements, and to measure the benefit of these improvements, data teams should research the effort their end-users expend on these activities.
Nasdaq, responsible for their eponymous stock exchange in New York City, and an array of technology and financial service products, uses Atlan to evangelize their data strategy and improve their governance practice.
Key to their implementation process was time spent surveying Nasdaq’s data consumers, driving better understanding of how the data team could improve their process of discovering and applying data.
Nearly 100 users responded to the survey, with 75% of respondents reporting spending time trying to understand data. Respondents who spent six hours or more hours pers week on data, spent two of those hours trying to understand the context around what they already had access to.
Think about that! A third of their time every week is spent just trying to understand what is there. Imagine if we could bring a product in that helps reduce that effort and really enables them to get right to the heart of the problem — to drive data products from insights into the business. And that is what we’re trying to get to.”
Michael Weiss, Senior Product Manager, Nasdaq
With this survey data in-hand, and a mechanism for improving the discovery experience for their data consumers, Michael and his team have a “North Star” metric to measure their success, potentially representing hundreds of thousands of dollars in savings in data consumer effort, alone.
Measuring Time to Insight
For data teams that have a service-driven relationship with their data consumers and customers, baseline measurement of the time processes take to complete, like building dashboards or completing queries, makes it simple to communicate the effect of process improvements and new technologies.
Infillion, a Digital Media Platform serving the Fortune 50 and more, uses Atlan to stitch together a complex data estate, and provide data-driven insights to their customers.
A crucial part of their Business Intelligence team’s responsibilities are supporting account teams, whose customers expect frequent, accurate reporting on marketing campaign performance. Previously processed manually, these requests would take as many as three weeks to complete, with one week of waiting for Data Engineering bandwidth, and another two to produce a report.
Using Atlan Scheduled Queries, Daniel’s team now spends 30 minutes or less responding to requests for data, eliminating the need for scarce Data Engineering resources, and reporting a 3-week time savings for their business partners and customers on each request.
It’s allowed us to fully automate reporting without any engineering resources and become self-sufficient, and it’s definitely removed a big bottleneck in our process. A three-week waiting period is just not scalable. So this definitely cut that down to a half an hour, maybe less, to set up.”
Daniel Chon, Director of Business Intelligence, Infillion
Measuring Effort
Finally, on a more sophisticated level, a detailed inventory of the tasks data consumers undertake, combined with an analysis of the effort expended and average personnel cost for each of these tasks, is a labor-intensive, but powerful tool for measuring the specific impact of a data team’s efforts.
Zip, a digital financial service company and buy-now-pay-later pioneer, uses Atlan to help their data consumers quickly, reliably, and easily find, understand, and use the capabilities of their modern data stack.
Leroy Kahn, Zip’s Data Management Lead, developed a model to estimate the business value yielded from each activity his colleagues would perform on Atlan, and a process to track the volume of these activities as part of their business case for a modern data catalog.
Through user surveys, and the opportunities to improve that they inventoried across tasks involving finding, sharing, understanding, and using Zip’s data, Leroy determined an average amount of time saved for each activity or event type for Atlan users. Based on average salary, each minute of time saved was worth $1 (AUD) in efficiency gains, which could then be multiplied by the time saved per event in Atlan.
By analyzing usage metrics, Leroy and his team were able to account for each activity performed on Atlan during a 30-day period, comparing it against the time and dollar savings, projecting $390,000 AUD in efficiency gains.
3: Business Value
While measuring the performance of data team programs, and carefully recording the value of operational improvements is critical, there are a limited number of opportunities to make these improvements. In order to continue communicating the value of their work, data teams must strive to understand what data is used for, and to what effect. This necessitates moving beyond a model where requests are serviced, and toward a model of partnership with business users.
While data teams might have grown accustomed to building new capabilities or servicing requests that are then picked up by an analyst or data-savvy business user, moving beyond this paradigm means understanding the purpose of these requests, and following up on their impact.
Data teams can begin putting this into practice by recording a business goal each time a request is made. From a simple inventory of which teams or colleagues made these requests, to more complex records of projects like reducing customer churn or identifying cross-sell or up-sell opportunities, data teams can begin comparing their work against achievement of business objectives, and following up for specific anecdotes of what impact these insights made.
A fast-growing pet care company is among the best examples of business value measurement in the Atlan Community. After migrating to a new telecommunications system, their call center team was underperforming, leading to diminished customer engagement, and loss of revenue.
While data was available from their new system, its format was unique to telecommunications systems, dubbed a Call Detail Record (CDR), making it difficult for their Data Analysts to investigate the problem and propose a solution.
I’ve worked in situations like this before, and if we didn’t have Atlan it would have taken six months. We’d be answering questions like ‘What do you mean by event code?’ or ‘What’s the difference between extension number and regroup number?’.”
Data Governance Lead
Using Atlan, a member of their team with Telecommunications experience worked to relate Snowflake assets with CDR data, and translated, then documented what each of the data points meant. Now able to understand the context around this data in a self-service manner, their Data Analysts quickly performed their analysis in PowerBI, and identified three core problems for under-performance.
Taking an Operational Value lens to this achievement, their data team could report that they were able to support this analysis in 30 days with Atlan, rather than 6 months. But by maintaining a close relationship with their counterparts in the business, and by keeping informed about how data analysis drove a resolution, their data team can now report that they supported a 10% increase in bookings through their call center, a significant increase in revenue, and a mission-critical achievement.
While fastidious reporting on Program Value and Operational Value are crucial, data teams who build strong relationships with their business counterparts, and have the curiosity and persistence to understand how their work is used downstream, can tie their work to the most important problems their organizations are facing.
Key Takeaways
Across Program, Operational, and Business value, the best examples of quantification in the Atlan Community have a number of key commonalities.
- Curiosity about, and commitment to, the value of their teams, programs, processes, and technology.
- Consistent collection of quantitative and qualitative data prior to investment or changes to processes or technology.
- A mechanism for capturing changes in behavior, sentiment, or effort, consistently measured against a baseline and communicated to stakeholders and leaders.
Photo by Patrick Perkins on Unsplash
