Improving Developer Productivity and Delivery Velocity with Atlan
The Active Metadata Pioneers series features Atlan customers who have recently completed a thorough evaluation of the Active Metadata Management market. Paying forward what you’ve learned to the next data leader is the true spirit of the Atlan community! So they’re here to share their hard-earned perspective on an evolving market, what makes up their modern data stack, innovative use cases for metadata, and more.
In this installment of the series, we meet Abraham Tom, Director of Data at Generate Capital, who shares his 20-plus-year history in Data & Analytics, why Data Governance matters so much to his organization, and how he and his team use Atlan to improve developer productivity.
This interview has been edited for brevity and clarity.
Would you mind introducing yourself, and telling us how you came to work in Data & Analytics?
I’ve been a data professional for over 20 years, building out Netezza, Oracle and SQL server warehouses, moving on to Hadoop, and now Cloud systems. I’ve worked on all aspects of data, ranging from movement, governance, engineering, science (before it was sexy), BI analytics, graphical representation, and specializing in ground-up implementations of new data warehouse initiatives. My motivation is showing to executives why data-driven decision making is valuable, then building a roadmap to achieve that state.
Could you describe Generate Capital, and any important initiatives you and your team are working on?
Generate Capital PBC is in the business of re-building the world to save humanity. We build, own, operate, and finance affordable and reliable infrastructure solutions for clean energy, water, waste, transportation, and smart cities’ infrastructure technologies. Data Governance needs to be the cornerstone of our data warehouse program so that everyone that needs to be informed or that needs to make a decision has the right information to do so.
What does your stack look like, and how has that evolved over time?
I started as the second tech person in the organization. Generate Capital has the unique challenge where we get data from different lines of business and coalescing all that data into an aggregate view. This required quickly devising a solution that can scale, and implementing a set of tools that allowed for that. Our primary repository was Snowflake as a cost-effective scalable platform, we use Streamsets as our primary data ingestion, dbt as our transformation engine, and Tableau as our primary visualization and delivery technology. We also handle a lot of the data integration between our ERP and CRM systems, as well as being the hub for data transport to other systems.
How did you come to look for an Active Metadata Management solution? Why did Atlan stand out?
Originally, I wanted to find a simple data dictionary tool. While dbt has lineage capabilities and some documentation constructs, it was limited to what dbt is aware of, and not all the systems we use. Atlan is a great solution to not only to house our dictionary, but the lineage and glossary functions have elevated our team’s knowledge so that they can see the impact of their new development.
What have you built with Atlan, so far? What value have you been able to drive?
Right now, we’re using Atlan as more of a tech resource than a business resource, and have found good value.
Our product team has become our first non-developer users of the Atlan platform which has improved overall ticket and specification writing. Tickets, feature requests, and onboarding have all vastly improved. Our Product Managers can do research on how the data flows and can write very specific requests with better acceptance criteria, which has resulted in improved developer efficiency, less back-and-forth with questions, and improved overall delivery, all of which increased our velocity.
We have also integrated Atlan’s impact analysis with our dbt pull requests as a GitHub merge dependency check. This gives our developers awareness of the downstream impacts of the changes they plan to merge before the final commit happens. This has helped us ensure any new feature requests do not adversely affect existing production items. Usually, this requires senior resources to review code, and while we still require peer code review, that review is a lot faster, and everyone is informed of the potential upstream change and impact.
As part of our Governance initiative, we are also starting to leverage Atlan’s glossary function as a capture mechanism for the business definitions. Atlan assists us in tying that business definition to the system dictionary.
Any final thoughts?
Atlan is a great tool for developers to see data lineage and to help them figure out what needs to be done and how their work impacts downstream systems. All of my developers regularly use this to understand what they are building. It also provides other development teams transparency of the work we do and where their work ends up.
Photo by Bastian Pudill on Unsplash
