Financial and Technology Heavyweight Reducing Time Spent on Data Discovery by One-third with Atlan
When one thinks of Nasdaq, their eponymous exchange in New York City likely comes to mind. It’s the world’s second-largest, with $18 trillion in market capitalization; home to the world’s most prestigious technology companies.
Michael Weiss, Product Manager at Nasdaq, joined Atlan at the 2023 Gartner Data & Analytics Summit to share how Nasdaq is so much more than its flagship exchange, and how a company that has run on data for over five decades is becoming a leader in data technology, as well.
“It might be a surprise just how big Nasdaq is,” Michael explained. “Nasdaq operates over 30 exchanges across a variety of asset classes across the globe. We’re also home to over 4,100 publicly listed companies globally. In the U.S., we’re home to 73% of all publicly listed tech companies, and the four largest by market cap. Nasdaq is truly the home of technology.”
A Decade of Cloud Experience
Considering their position in the cautious and often slow-moving financial sector, it might come as a surprise to many that Nasdaq began their cloud journey with AWS as far back as 2012. Initially, they built internal products focusing on regulatory reporting, then revenue management, supported by the cloud. “From there, we went to data warehousing and analytics. Ever since then, we’ve been systematically moving toward the cloud,” Michael shared.
With more than a decade of experience operating in AWS, Nasdaq is beginning to move the bulk of its critical workloads to the cloud, and is defining how the next generation of capital markets will operate.
Despite this deep experience, Nasdaq sees ample opportunity to improve. First, trading system data is challenging in size, and in complexity, with as many as 140 billion events per day processed in the U.S., alone. “When you look at trading system data, it’s important to know it’s optimized for [operational] performance, not for analytics,” Michael explained.
Nasdaq’s process for preparing and presenting data was another focus, with their legacy ETL tools unable to keep up when both the types of data, and the demand for data, scaled significantly. “They were a bit rigid,” Michael shared. “They didn’t really adjust to where we were trying to get to at the time.”
Between the complexity of their data, legacy tools that were slow to grow to Nasdaq’s ambitions. A team that had its hands full maintaining their technical landscape, Nasdaq’s data team realized an opportunity to better support their business partners. “If you were a product owner or business user who wanted to get a new set of insights into your data, it could take quite some time to have someone build it out on the existing platform,” Michael shared. “We weren’t really enabling our business users to get what they needed.”
As a result, parallel teams began to emerge, each with a unique approach to creating data solutions. “If you were a business unit, you could go to one of four teams to try and get an answer. As a matter of fact, they would go to all four teams to see who would win first,” Michael explained.
It would take a compelling vision and strategy, backed by a more modern data stack and team structure, to better support Nasdaq’s growing internal demand for data.
A Data Leader’s Data Stack
“It was only about 18 months ago that we decided to take a step back and re-evaluate our tech stack,” Michael explained. And in that short time, Nasdaq has made several key investments.
First was an implementation of dbt, accelerating their team’s ability to build data models. “We really wanted a tool that was flexible, and could really enable people who had SQL skills already to take advantage of that technology,” Michael shared. Nasdaq continued to operate on AWS, embracing Redshift as their primary data store, alongside a variety of AWS services such as S3 as their data lake, Glue for data integration, and QuickSight for business intelligence.
This new stack also included Monte Carlo, adopted for data observability. “A lot of our data monitoring at the time was very reactive. The pipeline might succeed but we wouldn’t know if the data was right, wrong, or indifferent,” Michael shared. “With a tool like Monte Carlo, we’re able to catch some of that earlier on.”
But the final piece of the puzzle was Atlan. Atlan was adopted as Nasdaq’s window to their modernizing data stack, and as a vessel for a data governance practice that was quickly maturing.
This is how we’re really going to take ourselves to the next level.
Michael Weiss, Product Manager
Evolving Teams and Processes
Beyond taking a leap forward in technical maturity, Nasdaq’s data team understood that their team structure and operating model would need to make a similar leap if they were to meet, then exceed business needs. “You can bring in new tools and to keep the same methodology and the same cultural approaches you don’t really solve the core problem,” Michael shared. “How do you bring a data-first culture to the entire organization?”
Nasdaq long had a centralized structure for reporting; appropriate for their legacy technology and responsibilities, but a model that struggled to enable their business counterparts to use new tools and technologies.
“We looked at the skill sets within different teams to try to figure out the best way to bring forth change at a more fundamental organizational level,” Michael explained.
This approach led to carefully focusing Nasdaq’s data organization into more specific roles. First is the Platform Team, responsible for maintaining core technology and reducing cognitive load on analysts and consumers. Next is the Economic Research team, focused on data science and serving as a center of expertise for business units conducting deeper analysis.
While a less-centralized model was a better fit for the growing number of data consumers, it required a more mature engagement and onboarding model as users began to exploit new capabilities and data.
“We started looking at how we fundamentally embed data into each one of our core businesses so they can self-serve,” Michael shared. “So we came up with an engagement model where we’d go to one of our organizations. We do a survey to start. We were trying to get an assessment of where they were with their skills, and then from there we would actually embed with them a set of data initiatives they wanted to pursue and get them up and running.”
This new process, considering the unique needs of Nasdaq’s business units, and guiding them through requirements, joint development of business models, the creation of a metrics layer, and enablement to transition responsibility, meant that rather than acting as a service center, Nasdaq’s central data team could focus on larger initiatives, coaching and consulting when necessary.
We didn’t want to just say, ‘Hey, here’s the platform. Have fun, good luck. Come back within six months and let us know what you think of it.’ We found that this enabling really has helped drive success getting teams on board.
Michael Weiss, Product Manager
Choosing Atlan for Active Metadata Management
With core data technology in place, a team structure that matched Nasdaq’s data service strategy, and an onboarding and engagement model to ensure business partners were well-equipped to use new capabilities, Michael’s team turned to active metadata management as a path for education, collaboration, and discovery.
“This is not the first time we’ve launched new technology at Nasdaq for data. I’ve been at Nasdaq for 10 years. This is the third or fourth time we did it. Each time you learn a little something different,” Michael shared. “When we launched this one of the key things that our executive team brought up was ‘What is going to be different this time?’ ‘What other things are you going to bring in the mix to reduce friction between data producers and consumers?’”
Crucial to Nasdaq’s executive team was the ability for users to understand what data and tools they have at their disposal, and active metadata management became a crucial technology for answering this question. Without a platform like Atlan, data and capabilities would remain undiscoverable, reducing the potential return on investment Michael had hoped for at the beginning of Nasdaq’s journey.
“To get key stakeholders to buy into everything, it was critical we had a plan in place. We once found ourselves focused on BI and ETL but forgot about how to reduce friction between producers and consumers of data. It was front-of-mind for all stakeholders that we planned to resolve this in any new data approach. Without this the rest of our model would be DOA,” Michael explained.
Among Nasdaq’s stringent requirements for active metadata management was a SaaS model – avoiding the need to manage their own infrastructure – as well as the ability to carefully model access points across users and roles, and a proven record of security aligning with Nasdaq’s position as a highly regulated business entity.
Perhaps most important was cultural alignment, a deciding factor for why Atlan proved to be the best fit for Nasdaq.
It’s not just the technology, but the organization behind it. We really wanted a partner who would come in and help us really understand the power and benefit of this tool. We found that.
Michael Weiss, Product Manager
Scoping the Data Service Opportunity
An important part of Nasdaq’s path to implementing Atlan was surveying data consumers to better understand how they could improve the discovery and application of data.
Nearly 100 end users responded to the survey, and what Michael’s team found was intriguing. 75% of respondents reported spending time trying to understand the context around data. And power users, who spend six or more hours per week on data, spent two of those hours trying to understand the context around what they already have access to.
Think about that! A third of their time every week is spent just trying to understand what is there. Imagine if we could bring a product in that helps reduce that effort and really enables them to get right to the heart of the problem — to drive data products from insights into the business. And that is what we’re trying to get to.
Michael Weiss, Product Manager
These findings were consistent across Nasdaq’s diverse data and capabilities. Whether raw data from their matching engine, or prepared metrics and visuals, significant time was spent on collecting context, slowing delivery on key insights and data products.
Seeking a beyond-the-numbers understanding of stakeholder sentiment, the survey asked for qualitative feedback, leading to the realization that siloed and lost knowledge were key problems to address. Without strong ownership of data assets, or consistent documentation, Michael’s stakeholders found it difficult to pick up where another member of Nasdaq’s organization left off.
“It was not uncommon that you would have someone working in one part of the organization, knowing one part of something. Then a data product owner would come in, and doesn’t understand why decisions were made, or even what was available, so they’re back to square one,” Michael explained.
Driving Common Understanding of Nasdaq’s Data
“Some of the early results have been really astonishing,” Michael shared. Despite being early in their Atlan adoption journey, a follow-up survey shows signs that Nasdaq’s choice of Atlan for active metadata management is yielding positive results.
“The one comment that keeps coming up is ‘This is like having Google for our data,’” Michael explained. “Letting them start with column names and baseline descriptions has enabled our users to really understand what’s in our lakes and warehouses already, and then start asking the right questions.”
Having made the right technology choices, and with a team structure that supports their business units unique data needs, the introduction of Atlan into the Nasdaq ecosystem has catalyzed a common understanding of their data and the tools at their disposal. And with a growing array of data, insights, and capabilities at consumers’ fingertips in Atlan, confidence in Michael and his team’s data strategy is improving.
“Even though we are early into this journey there is already a bit of a shift in the sentiment of our team. While we still have a lot to do, this journey is already proving valuable and has been critical for the executive team having confidence we can continue to deliver on our promise.”
