I just spoke at Zero Gravity by Incorta, a modern cloud data pipeline event. I’ve been working with data pipelines for years, so it was amazing sharing about my experiences taming the chaos in today’s modern data stack!
In case you missed my talk or want to dive into the details, here’s my slide deck from this talk. If you have any questions, please drop a comment or reach out to me on Twitter or LinkedIn!
Today, metadata is everywhere. Every component of the modern data stack and every user interaction on it generates metadata. Apart from traditional forms like technical metadata (e.g. schemas) and business metadata (e.g. taxonomy, glossary), our data systems now create entirely new forms of metadata.
All these new forms of metadata are being created by living data systems, sometimes in real-time. This has led to an explosion in the size and scale of metadata. However, this has made it difficult for centralized data engineering and reporting teams to fully leverage metadata that holds the key to the elusive promised land — a single source of truth.
Teams today struggle to keep pace with “service requests” and to ramp up on domain expertise needed. The net result is lost opportunity. But by effectively collecting metadata, a team can unify context about all their tools, processes, and data.
This session covers how can data teams leverage metadata to power better collaboration, enabling organizations to accelerate onboarding and build data products outside the central team. I will cover active metadata and how its application through embedded collaboration, open API interface, powered by knowledge graphs, programmable-intelligence bots, data process automation, reverse metadata, can help meet enterprise governance standards.
Curious to learn more? Here’s more information on everything I referenced during my talk:
- The half-a-decade backstory that drives everything we do at Atlan
- What is active metadata, and why does it matter?
- The future of data catalogs
- My thoughts on the modern data culture stack
- Why we failed to set up a data catalog 3x
- What we should expect from the modern data stack this year
