Leading Global Communications Agency Improves Discoverability, Documentation, and Root Cause and Impact Analysis
Brought together by multinational technology company Konica Minolta, Indicia Worldwide is the only global agency that combines creative, data and technology talent with production and procurement expertise to realize marketing ROI for some of the world’s biggest brands.
Data enables Indicia Worldwide’s clients to communicate with potential and current customers, and ensures the most effective messages reach them at the perfect moment. Supporting that data-driven mission is Graham Lannigan, Head of Data Platform at Indicia Worldwide.
“My role is looking after the data platform that we use for our own teams, and for the work we do with clients,” Graham shared. “Our data maturity has really grown over the years, and that led to the idea that we need to be looking at our own data platform and capabilities, and using that as a launchpad for backing up the services we provide to clients, and enabling us to build and power a load of products we can offer to clients and run ourselves.”
Building a Data Platform
Indicia Worldwide is the product of multiple organizations acquired by Konica Minolta, including Charterhouse, a creative production agency, Ergo, a print management and marketing automation agency, and the marketing agency Indicia. Consistent, past and present, in each organization is a service model that stresses individual attention and care to each client. But despite this bespoke model, Graham and his team noticed patterns emerging in how each client was serviced.
“When we joined up the traditional agency model with the global print production expertise, we realized that while we work across 40 different markets, we had a lot of common tasks and common problems,” Graham explained. “There were a lot of times we were doing the same thing over and over, and sometimes using different technology.”
Having identified these patterns of repetitive work, Graham and his team have been building a suite of data capabilities and products that Indicia Worldwide can either offer to clients, or use internally to support the custom services they provide.
The last couple years in my role as Head of Data Platform has been about identifying the right technology partners, the right problems we’re trying to solve with our data platform, and fitting those two together. It’s finding the right blend to make sure that we have the capabilities we need, our data science team has the tools to get their job done, our data engineering team has the tools they need, and that all of this comes together to deliver the right data to the right people.”
Graham Lannigan, Head of Data Platform
Supporting this transformation begins with a diverse and specialized data team, beginning with Data Science and Analytics, supporting internal dashboarding and modeling needs, and ensuring clients can closely measure the value derived from their marketing spend. Engineering teams include Product Engineering, managing user-facing products and web applications, and Data Engineering, responsible for Indicia’s Data Platform and ensuring downstream teams like Data Science have accessible, high-quality data.
And powering this increasingly specialized team is a fast-modernizing data stack, underpinned by AWS as their primary cloud platform, Matillion for ETL, Amazon Sagemaker for Data Science, and Power BI for business intelligence. But central to their success in recent years has been a commitment to Snowflake, beginning in 2020.
“The core of our data platform is Snowflake. We started with some client workloads and we’ve slowly re-platformed our data capabilities onto Snowflake. Our team has a really strong SQL background, so Snowflake was a really nice fit,” Graham shared. Plus, with our global model and some of the things we do with analytics like supporting rich client data, Snowflake has a lot of really nice capabilities around data sharing and data monetization that really fit with our ethos as a company.”
With a specialized team, and a growing modern data platform based on Snowflake at its center, Graham’s team began the search for a technology that could make these capabilities even easier to utilize.
“Part of my focus now is working on how we close the gap between the data in Snowflake and what the team are doing in Sagemaker and our own product offering, reducing data movement, and making sure the barriers to them doing cool stuff are removed as much as possible. We recognized the need for understanding what our data is, where we’re holding it, and sharing a load of knowledge, and that’s where Atlan came in,” Graham explained.
Finding a Philosophical Fit in Atlan
Historically, Graham’s team would use its wiki to store documentation for each of their projects, including details around data structures and models. And while this documentation was helpful as each project began, its accuracy would wane as change requests were made and technology evolved, and as more documentation was added, it became more and more difficult to navigate.
“You would spend a lot of time tracking down the people who had done some changes and trying to ask them if the documentation was still representative of reality or not,” Graham shared.
And while Indicia Worldwide’s engineering teams were committed to making its documentation work well, including a custom-built tool to crawl databases and synchronize information back to the wiki to update documentation, the scale and complexity of customer data made the tool unworkable.
“We have some clients who have 40 million customers and build tens upon tens of billions of rows of data. Our Atlan instance right now has something like 500,000 assets in it,” Graham explained. “There’s a lot to keep an eye on, so when we tried to link this data up with our own tools, we found that we just weren’t keeping up.”
Beginning their search for a solution that would improve data discovery and documentation, Graham and his team considered options from cloud providers and open-source alternatives, but found they lacked a user interface that made their data assets neatly accessible. They also considered legacy data catalog solutions, but were discouraged by high cost of ownership, and found they didn’t align with Indicia’s burgeoning cloud-first mentality.
Atlan’s whole philosophy just fit really well with us as an organization, where we were, and what we were looking to do. We brought Atlan in, started playing around with it, crawling some of our assets, and using some of the capabilities. And I think very quickly the platform made a case for itself because everyone I showed it to was like ‘Wow. This is brilliant. This would be really helpful.’”
Graham Lannigan, Head of Data Platform
Early Value through Better Visibility
Beginning by integrating Atlan with Snowflake, Graham’s goal was to dramatically increase visibility into their data estate. With this newfound visibility, he and his team first noticed an opportunity to optimize their storage costs, using Atlan’s usage and popularity metrics to identify tables that had not been accessed over a set period of time. Using custom properties, Graham and his team tag these assets with a life stage, review them with the teams that created them to determine their necessity, and decide whether to retain or deprecate them.
“The reporting in Atlan has been really helpful for this,” Graham shared. “We’re starting to get a better handle on (questions like) ‘Why keep data if we don’t need to?’ and ‘Why have assets if we’re not going to use them?’”
Moving to enrich these assets, Graham has also used this newfound visibility to proactively ask subject matter experts to add documentation as their assets change.
It just gives this view of what’s going on in our platform on a day-to-day basis that would have been really hard to put together otherwise. And the fact that we can then tag someone on Teams or share the link to the asset in Atlan and ask ‘What’s this thing?’ or ask ‘Please add some documentation to this.’ It’s amazingly helpful.”
Graham Lannigan, Head of Data Platform
And while answering a question from his colleagues once meant moving system by system, looking up outdated documentation, or asking expert after expert for the information he needed, Graham can now use Atlan to quickly find the information he needs, so he can focus on building and maturing their data platform.
“Personally, it’s been absolutely massive in terms of my being able to answer questions or being able to understand what’s going on,” Graham explained. “When we have a question about some data or how it’s being used, now my first stop is Atlan. I won’t log into Snowflake to have a look for an object. I’ll search for it in the asset view, and if I need to, I’ll go into the lineage tab and I’ll explore how the data got there.”
Extending this value beyond saving time in his own day, Graham is beginning to encourage Indicia Worldwide’s data consumers to self-serve for answers on data. Today, when teams are building a new data product and have difficulty finding the right data, Graham services these questions personally, sending a link to the right assets in Atlan. But tomorrow, he sees dozens of data consumers empowered to find this information, or enrich it, by themselves.
“It’s a massive time-saver not just for me, but the rest of the team. The next thing I’m trying to instill is that rather than pinging me a message and asking for help, just go to Atlan. Try and search for what you think it might be called, and if you find it and it’s not documented, add some documentation to tie it up so the next person can find it more easily,” Graham explained.
Improving Engineering Practices
Extending past the value of better servicing internal clients, Graham’s team is finding new ways to improve the practices of their engineering teams.
Beginning with the common problem of dashboards and data pipelines breaking when new functionality is deployed or when change requests are completed, Indicia Worldwide’s engineers can now use Atlan’s automated lineage for root cause analysis, significantly accelerating their time to resolution.
One of the great things is that rather than everyone trying to scramble around and remember what they changed and where, we can literally just pop into the lineage view in Atlan and trace that back to say ‘Here’s the dashboard, there’s the dataset it’s getting the data from, that’s the table, and that’s where it’s ingested.’ We can track that all the way back and find where the change was made.”
Graham Lannigan, Head of Data Platform
Going forward, Graham and his team intend to use Atlan’s automated lineage for proactive impact analysis, identifying the downstream assets potentially impacted by changes before those changes are even made.
“The nicer way is doing all of this upfront. Rather than waiting until you have a problem, preempt it by doing the impact analysis upfront. That’s the bit where I see massive value for our teams,” Graham shared.
Finally, Graham and his team are working to integrate Atlan into their CI/CD process. Jenkins, Indicia Worldwide’s CI/CD orchestration tool, would send a notification to Atlan using its open API when assets are created, which would then send a reminder to the asset owner to complete documentation.
“Where we’ve got people taking a feature branch and checking that into Dev and maybe doing a pull request with an approval step, that now goes through our environments and gets deployed. But what if we could put an approval step in there that says ‘Have you documented this in Atlan?’,” Graham explained.
Governance on the Horizon
While early in their journey, Graham and his team see a bright future for Atlan, beyond the value they’ve already yielded with better visibility into their data estate, faster answers to data consumers, and more effective impact and root cause analysis.
As a marketing agency to the world’s best consumer brands, Indicia Worldwide sometimes finds itself the steward of personally identifiable information, and intends to use Atlan to understand what PII assets they hold, how old they are, how regularly used they are, and to ensure that only those authorized to access them, can.
And by using Atlan’s native integration with Snowflake, and benefitting from the partnership between the two organizations, Indicia Worldwide is optimistic about better activating Snowflake’s native tagging and masking capabilities with Atlan’s user interface, lineage, and automation capabilities.
To do any of this, we need that pane of glass to be able to look at the data and understand what’s going on within our data estate, what data we hold, what its characteristics are, and how it’s being used. Atlan gives us that in one tool, which is incredible. So the more Atlan is able to integrate natively with things like Snowflake’s object tagging capabilities, the more powerful Atlan gets, and it becomes more and more important in our stack.”
Graham Lannigan, Head of Data Platform
Photo by Bank Phrom on Unsplash
