Almost two years ago, we laid out our idea of modern metadata for the modern data stack, the Data Catalog 3.0.
Since then, it’s been a dream seeing our vision come to life. Active metadata has become a category and central talking point in the data world. Gartner, Forrester, and G2 alike made major moves to scrap old ideas of data cataloging and introduce new ones around active metadata and modern cataloging.
This transformation was driven by real product developments, ones that turned this abstract dream into something that data teams could actually use.
In 2022, our engineering team shipped over 200 new features, including a brand new version of our product. These improvements proved the power of active metadata and even got us ranked as a Leader in the Forrester Wave, Q2 2022!
Before we take on the next year of new features, let’s take a few minutes to look back on an amazing year. Here are 15 of our favorite features from 2022.
TL;DR: 15 updates in Atlan you should know about
- From siloed systems to embedded context
- From dispersed, manual tools to automated 360° context
- Atlan Playbooks: the Zapier for metadata
- Reduce manual documentation with Trident suggestions
- Manage assets and cut costs with asset popularity and usage metrics
- Get complete visibility and column-level lineage with our Fivetran Metadata API integration
- Make metrics and models first-class citizens with our dbt Semantic Layer integration
- Get a birds-eye view of your data estate with our Reporting Center
- From generic tools to personalized experiences
- From closed, siloed tools to open systems
From siloed tools to embedded context
We have always been driven by one core idea — diversity is the only reality. Data teams are one of the most diverse teams today, from the tools they use and the people they work with to their own use cases and personas. This diversity is both a strength and a weakness.
Data teams work across a diverse set of tools — BI tools like Looker, collaboration tools like GitHub, communication tools like Slack, ticketing systems like Jira, and more. Too often, these turn into siloed data systems. If you’re in a dashboard, you want to know if you can trust it then and there, rather than switching and searching across other tools.
In 2022, we built diversity into the core of our product. By activating metadata, Atlan sends crucial context back into the workflows and tools where the humans of data work every day.
Enhance collaboration and capture knowledge with our Slack integration
With our Slack integration, launched in January 2022, we set the stage for extensive collaboration in the world of metadata. Atlan’s deep integration with Slack goes beyond just sending messages — it’s about sharing and capturing important knowledge across your organization.
Atlan’s Slack integration includes:
- Rich preview links: Share assets with your team on Slack and see detailed previews.
- Workflow alerts: Get alerts in Slack whenever an Atlan workflow fails or succeeds.
- Ask a question from Atlan: Ask a question by clicking on the Slack icon for any asset. That Slack conversation will also be documented automatically in Atlan for future reference.
- Document threads directly from Slack to Atlan: Easily share Slack conversations to specific assets in Atlan.
- Search terms and queries directly from Slack: Search for terms and queries in Atlan without leaving Slack.
Stay in flow with our Atlan Chrome Extension
Active metadata makes it possible for metadata to flow effortlessly and quickly across the entire data stack, embedding enriched context and information in every tool in the data stack.
Imagine you’re looking at a dashboard and want to know who owns it and what a metric means. How would you go about it? Well, you would probably have to figure out who the owner was. Then you would most likely take a screen grab of the dashboard, email or Slack it to that owner, and ask your questions. Not anymore!
Now you can skip all those steps by simply clicking on the Atlan icon in your browser. It will open the Atlan Chrome extension right in your dashboard, giving you all the context you need right when you need it. Atlan makes this possible in the tools where your data teams and data consumers spend most of their time — Looker, Tableau Online, Snowsight, Mode, Power BI, BigQuery, and Salesforce, with more tools on the way.
Solve issues faster with our Jira integration
Tracking issues and solving problems just got a lot easier with our Jira integration. For many data teams, getting to the “root cause” of broken dashboards or inconsistent numbers can be a long, drawn-out process. Not anymore!
With Atlan’s Jira integration, users can now quickly open a new Jira ticket or manage an existing ticket directly from any Atlan asset. Just click the Atlan sidebar.
Minimize risk during data creation with our GitHub integration
When a data engineer makes changes to a dbt model, they might unknowingly be affecting tens or hundreds of people, tables, or dashboards. How can we minimize this risk while also providing transparency for the data engineer and data consumer?
With our integration between Atlan and GitHub, data engineers can now access all the context they need to minimize the risk for data consumers.
Here’s an example of what this looks like: say that you’re a data engineer, and you’ve created a pull request. When the GitHub action runs, it will automatically create a list of all downstream assets that will be affected by this request — before you make the change. From there, you can reach out to users in advance, or research and test the assets to see how they might be affected.
From dispersed, manual tools to automated 360° context
Just as data teams today use diverse tools, their data ecosystem is made up of diverse data assets. These are no longer just tables made up of rows and columns. Data assets are tests, metrics, pipelines, code, queries, dashboards, and more — all scattered across different data tools and systems.
Today, when you onboard an engineer, you just share a link to a GitHub repository. This has the code, documentation, history, collaborators, and everything else you need to know about the code, all in one place.
However, it takes a lot of time and manual effort to set up and bring documentation into traditional data catalogs today — and just as soon as you’re done, they’re out of date. At Atlan, we wanted to make data assets just as easy to share and understand as software on GitHub. Along with activating metadata back into daily workflows, our workspace gives users a 360-degree view of every asset by automatically pulling in and stitching together metadata from every tool in your data stack.
Atlan Playbooks: the Zapier for metadata
One of the common questions we get from data teams is, “How can we automate our metadata?” While other teams like marketing or sales can easily do action-based automation at scale with tools like Zapier or Salesforce, data teams often don’t have the bandwidth to code custom automations for each diverse use case. Why can’t there be a Zapier for metadata?
That is why we have developed the first low-code/no-code metadata automation for data teams. With Atlan’s Playbooks, users can now create rule-based automations at scale. These can be used to drive endless use cases across your organization. Here are a few examples:
- Deprecate assets: Mark any assets that haven’t been queried in the last 30 days as deprecated.
- Add/change ownership: For Salesforce assets that are missing an owner, add RevOps as the owner.
- Report failed assets: Post a list on Slack of every table and dashboard with a “failed” Airflow status.
- Protect sensitive data: Attach GDPR custom metadata on all assets tagged as “PII”.
- Flag upstream alerts: Notify downstream owners when an upstream asset is tagged with a warning or announcement.
Reduce manual documentation with Trident suggestions
Traditional metadata management can be a tiresome, manual process that requires a lot of human intervention from data leaders and stewards.
That’s why we introduced Trident, which provides automated suggestions for new descriptions, owners, terms, and classifications. Trident makes metadata enrichment easier so that data teams can spend more time on meaningful projects and less time writing descriptions and classifying data. Since its launch, one-third of all description updates on Atlan have been made with Trident!
P.S. We recently gave a preview of the potential of Trident by combining it with GPT-3 to create Trident AI — increasingly intelligent automated descriptions, business terms, READMEs, and more.
Manage your data and cut costs with asset popularity and usage metrics
One of the biggest challenges facing data teams today is managing their data and reducing data infrastructure costs. That’s why we introduced usage and popularity metrics.
Data teams can now analyze their Snowflake data assets without leaving Atlan. With these new metrics, organizations can identify their most used assets, find people with the most context on assets, identify unused assets, and more. These metrics include:
- Asset query logs: The number of times a data asset has been queried (i.e. asset popularity), when it was last updated, etc.
- User query logs: A data asset’s top users and most recent users, who last queried it, etc.
- Cost and performance optimization logs: Which queries are slowest, most expensive, etc.
Get complete visibility and column-level lineage with our Fivetran Metadata API integration
Atlan’s deep partnership and integration with Fivetran creates the first end-to-end view of the modern data stack. By leveraging Fivetran’s new Metadata API, Atlan now builds column-level lineage from your Fivetran sources all the way to BI tools like Looker and Tableau — giving you complete visibility across your modern data stack with powerful root cause and impact analysis.
We’re excited to partner with Atlan to bring better metadata management to our customers. Data governance is a significant struggle for data teams today.
Fraser Harris, VP of Product at Fivetran
Together, Fivetran and Atlan are solving this problem by helping data teams gain much-needed visibility on where data came from, who accessed it, and what changes have occurred in the pipeline.
Make metrics and models first-class citizens with our dbt Semantic Layer integration
Another big partnership from this year was Atlan + dbt. With our new Semantic Layer integration, dbt metrics and models are now first-class citizens in Atlan.
Atlan’s deep integration with the Semantic Layer brings dbt’s rich metrics and models into the rest of the data stack, allowing data teams to create repeatable, metadata properties — like table owners and verified tags — and build documentation standards.
dbt’s metrics are also now a part of Atlan’s column-level lineage, spanning from source systems and data storage to transformation and BI. With increased context, visibility, and self-service, this integration helps diverse data teams work together better.
Get a birds-eye view of your data estate with our Reporting Center
At Atlan, we’ve seen a pattern of questions arise as our customers gather more metadata. Our users want to understand how their data landscape and its metadata are evolving.
That’s why we introduced the Reporting Center, which provides a bird’s-eye view of all the metadata updates in Atlan through visualizations to help you track adoption, curation, and metadata management. In short, Reporting Center gives you data on your metadata. It doesn’t get more meta than this.
With the Reporting Center users can now:
- Monitor their assets
- Track glossary terms and categories
- Enforce governance by tracking personas, purposes, classifications and requests
- Understand queries and query runs from insights in Atlan.
From generic tools to personalized experiences
Another challenge with traditional data catalogs and metadata solutions is they treat every user the same, delivering a generic experience across the organization.
The problem with that approach is, context means different things to different people. For example, a data engineer looks at a table and they’re interested in transformations, data quality, and pipeline stability. Meanwhile, an analyst may care about column names and descriptions, frequency distributions, and popular queries ran against that table. Finally, a business user might just see a metric and wonder “What does this metric mean?” and “Can I trust it?”
We wanted to change that narrative and build a platform that delivers a personalized experience for each user across your organization.
Create a Netflix-like data experience with Personas and Purposes
If Netflix can create a personalized viewing experience for their users, we thought, why can’t Atlan do the same thing for data teams? That is exactly what we did with the introduction of personas and purposes.
Personas and purposes give admins the ability to control access to assets, policies, and preferences tailored to the role of each individual. Personas provide asset-level controls for each team, while purposes group assets together for cross-team projects.
For example, the marketing persona may give all marketing people access to certain Snowflake tables and Tableau assets. With a purpose, some members of the marketing team may also get access to other marketing campaign data and dashboards that are relevant to campaigns they are running this year.
We’ve seen this having all kinds of interesting implementations inside organizations. For instance, many data teams using Atlan have started thinking about implementing paradigms like the data mesh, which at its core has this concept of a domain. Using personas and purposes, you can create extremely personalized experiences by domain, which then allows your data mesh to start being successful.
Curate user experiences with custom metadata
Custom metadata is one of the most valuable features in Atlan. Along with creating their own metadata (like a data freshness score or data quality metrics), admins can select which custom metadata should be visible to each persona while restricting visibility to those that wouldn’t find this information relevant.
For example, it is likely that a finance persona won’t need access to dbt transformations and Airflow DAGs, but they may need to dig into lineage to troubleshoot an odd Salesforce report or query a Snowflake table to determine ARR.
With custom metadata, Atlan provides the ability to curate each person’s unique experience, helping them work more efficiently with the right information and ensuring they don’t get overwhelmed by the entire data estate.
Power personalized active metadata with an all-new Atlan user interface
At Atlan, we’ve always been committed to helping data teams work together better. We started out as a data team ourselves and failed three times in trying to implement a data catalog. So we decided to build Atlan to be the kind of platform and partner that we wished we had back then.
After over a year of work, we launched a brand new version of Atlan last year. From a complete redesign of the Atlan user interface to powerful active metadata and personalization features, this update made it easier than ever for data teams to build trust and democratize data with metadata.
From closed, siloed tools to open systems
When you have a diverse set of users, using a diverse set of tools and assets, all across industries, data stacks, and more, there will be a diversity in use cases. Every company’s use case is different, and one size doesn’t fit all. That’s why we care about building open and extensible platforms that allow data teams to solve their own problems in their own ways.
Drive endless integrations and use cases with a fully open API architecture
Providing a fully open API architecture allows every action to be programmatically driven in Atlan. We now open all our metadata change events through AWS Eventbridge and a Kafka queue, Lamda webhooks, and more.
At the core of the Atlan platform, everything is open by default — and this allows data teams across any organization to build for and solve a wide variety of use cases. Here are a couple that we have seen organizations start to implement.
Programmatic data archival and deletion
One of our customers, a media company, purchases a lot of external data. The data has an expiration date, so they set up manual processes to regularly manage and delete their expired data. This sometimes led to data not being deleted, resulting in legal and compliance risks. This process also forced the data team to regularly spend significant time on manual work, resulting in lost productivity and team output.
Now, using the Atlan API, a notification is triggered 30 days before the contractual end date for the data. This creates a Jira ticket, letting the appropriate parties know that the data can then be deleted or archived. Through this new programmatic metadata workflow, the company was not only able to eliminate legal risk but also save significant time and productivity.
Active data trust management
Another customer has integrated Atlan with a data quality tool and power active data trust management. They use our open APIs to bring in anomalies that are detected by the data quality tool, and then downstream BI assets and potentially impacted business metrics are automatically flagged. For example, if a table feeding a KPI has a data quality issue, Atlan automatically creates an Announcement for the KPI in the Atlan business glossary.
Build custom event-based automations with our AWS EventBridge integration
Many teams across organizations, from data observability to incident management, are building automations that can trigger actions based on events. But there’s never been anything available for metadata management to create these types of event-driven use cases for metadata… until now.
Data teams can now use Atlan to create production-grade, event-driven automations for the world of metadata. This leverages our integration between Atlan and AWS EventBridge, an AWS service that creates an event and lets users consume and build use cases on it, to create Atlan metadata events into an EventBridge account.
Here are some examples of how this might work in Atlan:
- Ownership alerts: Get notifications in Slack when there is a change in the ownership of an asset.
- Propagation and classification: If someone marks a field as PII in an upstream data source, automatically create a masking policy to change all related fields downstream.
- And countless more use cases, such as notifications around schema changes on assets in Atlan for data engineering, login/logout events for security teams, or triggering a Fivetran enrichment event to kick off an Atlan workflow.
—
💡 To learn more about the latest features in Atlan, check out shipped.atlan.com or our December product roundup. You can also subscribe to our Product Updates newsletter for all updates.
💡 To check out Atlan for yourself, visit our product tour.
💡 Ready to start using these new features? Reach out to our Sales team or your Customer Success Manager to find out how.
