Making the Most of Existing Data with Atlan
The Active Metadata Pioneers series features Atlan customers who have recently completed a thorough evaluation of the Active Metadata Management market. Paying forward what you’ve learned to the next data leader is the true spirit of the Atlan community! So they’re here to share their hard-earned perspective on an evolving market, what makes up their modern data stack, innovative use cases for metadata, and more.
In this installment of the series, we meet Robert Bayliss, Technical Manager, Data Platform from Racing and Wagering Western Australia, who shares how a modern data catalog will be key to ensuring their data is visible, complete, accurate, and timely for the business partners they serve.
This interview has been edited for brevity and clarity.
Would you mind describing RWWA and how your data team supports the organization?
Working for Racing and Wagering Western Australia (RWWA) is incredibly unique, given we’re the controlling authority for thoroughbred, harness, and greyhound racing across our State, while we also have the responsibility for off-course TAB wagering. We’re here to provide a sustainable future to the Western Australia racing industry – an industry which is worth $1.3 billion for the Western Australia economy annually. However, racing is worth much more than its economic injection, its social benefits are immeasurable, especially in regional towns, fostering community connection from Albany to Kununurra and everywhere in between.
Given how vast RWWA is, technology plays a major role in our day-to-day operations. Our data platform team is a part of a larger technology team, with our core duties and responsibilities focused on ensuring the data platform provides accurate, timely, complete, and trustworthy data to the entire RWWA team.
Could you tell us a bit about yourself, your background, and what drew you to Data & Analytics?
I began my career building data-driven applications, and really enjoyed making apps that enabled people to do things they couldn’t do before. I moved into database development, solution design, ICT project management, and then, leading an applications team.
Lastly, I’ve moved to RWWA where I lead the data platform team and implement our data strategy. Ultimately, all my roles have been focused on enabling people and teams to achieve things that they couldn’t do before. I believe data to be a great enabler that anyone can use to improve the way they do things.
What does your data stack look like?
With our source data spread across a diverse technology landscape including Oracle, SQL Server and bespoke APIs, we had to harness a number of technologies to achieve our goals.This includes AWS DMS for data extraction, S3 for intermediate storage and Snowflake for Business Intelligence and reporting. Both Python-based Lambas and DBT scripts orchestrated through Airflow complete the transformation stage in our ELT process.
Why search for an Active Metadata Management solution?
We have a lot of data that can help our teams make timely and critical decisions, so the need to source an Active Metadata Management Solution was to help increase the visibility and accessibility of our data to everyone in the organization, from those that work closely with our data engineers to those that support the animal welfare programs.
These stakeholders need access to accurate data, and the data catalog was the tool to do that. The fundamental question was “How do we surface our data, and how do we make it accurate and timely so that people can rely on it?”
Why was Atlan a good fit? Did anything stand out during your evaluation process?
To be honest, Atlan was the one tool that made our data engineering team excited and also resonated extremely well with the end users. It made sense from a usability perspective. It was easy to visualize, it flowed and the simplicity of the experience meant not too many clicks to get to the information.
We were really focused on the fact that our engineering team was excited by it, and liked the fact that we can see lineage and expose our data sets, show how it’s all connected, and also surface all of our data quality metrics, as well.
What do you intend on creating with Atlan? Do you have an idea of what use cases you’ll build, and the value you’ll drive?
Data accessibility, as well as surfacing data observability metrics, are really important to us. It’s no use having access to data if you are unsure of the completeness, accuracy, or timeliness. Atlan is helping us to achieve this by surfacing our data quality rules and observability metrics alongside our data assets.
Additionally, having a way to surface data classifications and key glossary terms is a huge benefit to all of our teams, to help bring a common understanding and agreement on terminology.
It’s really about making sure that we’re leveraging our assets. We’ve got large volumes of data but it needs to be in the hands of those who can add value with it.
Anything else to share?
For me, it’s always about the why. It’s, “Why are we actually doing anything?” The reason for Atlan was always about how we get the most out of our data. That was the issue, then we worked out what we could use to make that happen.
Photo by Philippe Oursel on Unsplash
