Driving Self-service Data with Atlan
The Active Metadata Pioneers series features Atlan customers who have recently completed a thorough evaluation of the Active Metadata Management market. Paying forward what you’ve learned to the next data leader is the true spirit of the Atlan community! So they’re here to share their hard-earned perspective on an evolving market, what makes up their modern data stack, innovative use cases for metadata, and more.
In this installment of the series, we meet Joel Micalizzi, Director of Data Engineering at Hinge, who shares how their data engineering function has grown, their journey toward improving the transparency of their processes and data assets, and how he and his team are lowering the barriers to entry for Hinge’s data consumers to find, understand, and apply data.
This interview has been edited for brevity and clarity.
Could you tell us a bit about yourself, your background, and what drew you to Data & Analytics?
Starting in about 2007 when I graduated from college, I joined a Software as a Service, Big Data company, 1010data. When I joined, it was a pretty small company, and I was what they called a Data Product Developer, basically a jack-of-all-trades person who would work on data coming into our system from clients. We would sell a process where we would take your data, help put it onto this platform, and then you as a user would be able to do a big data analysis. I did that for a while. I was there for about 10 years and did a whole bunch of things. I learned a lot about how to structure and analyze data, learned a lot about doing ETL processes, and learned a lot about management as I eventually came up through the ranks at that company.
But then after about 10 years, I was looking to do something a little bit different. I was seeing other kinds of competitors or different kinds of styles of how engineering and coding was done that I wanted to make sure I was getting some exposure to, because I was at that company for so long.
So I went to Squarespace. I was there for about a year and a half working on one of their data teams for their finance department. There, I got a lot of exposure to more open source technologies like Postgres, using Airflow, and a bunch of other tooling there, and managing it as well. It was turning into a place where there was a need for some data engineering practices that required a very strong engineering background, but it just wasn’t, I think, a good fit for me. So I started looking.
And that’s when I joined Hinge, and that was about four years ago, as the Director of Data Engineering. And here, we’ve done a lot. We’ve built from having one person to a full-on data engineering team. And we’ve built up a much more robust infrastructure than we have had previously so we can support a very fast-growing company.
Would you mind describing Hinge and the role your team plays?
Hinge is a dating app available on Apple and Android. Our tagline is, “Designed to be Deleted”. So we try to get users on good dates, making sure that they are safe and happy with the product.
The data engineering discipline here at Hinge supports moving data from multiple different sources to people who are interested in analytics. So that’s anywhere from our Product Managers to Data Scientists, Analytics, and BI.
Why search for an Active Metadata Management solution? What was missing?
There were a couple of reasons. One is that we hired another data engineering manager who came from another company and was utilizing a data lineage tool and was saying that they got a lot of value out of that. Generally, we’re not going to invest in something just because a new hire says, “Hey, this is good value.” But we had a number of things here that we thought a data lineage tool could potentially support.
We have a bunch of processes from a much younger time in the company that are very difficult to track and understand what they’re doing. We were growing at that time and are still considering further growth today. So there have been a lot of data products, data models, and assets being created all over the place, and we needed a better way to track what was touching what so we could do better optimizations, prioritize work when we needed to, and things of that nature.
Another big thing we started looking at also was Data Governance. At the time, we had another project to improve our data governance functionality, which was very manual. We were thinking, “If we can better track where data is going and flowing in our system, it might be easier to automate it, or at least more easily find out where the data is and in what location.” So that was another big use case.
Did anything stand out to you about Atlan during your evaluation process?
When we first did a demo with Atlan, the first thing that really stands out when you’re looking at it is the UI. I like the search bar, and I really like the way to find assets. I felt like it was pretty intuitive for very non-technical people, which is useful. And then delving into it more, something that we weren’t initially looking for was a better store of metadata, and you could see the power of putting all of this into one location.
The data lineage tool was easy to follow. We liked that it had column-level data lineage, which was a feature that we didn’t initially look at, but then realized we needed to. Then the automation; the Playbooks feature, which made it easy to just plug automation into everything and edit with a GUI. And if we needed to, there is an API to utilize, which I’ve actually already done, and it’s totally serviceable and usable for our needs.
I think there were some other cool things that we didn’t know we would like to have, like the Insights panel. I thought it was really neat that we could query the data in Atlan. Additionally, the announcement of new AI tools is something I’ve been excited about.
We did evaluate other vendors, but liked that Atlan was a little bit more mature. We saw other vendors that just did not have a similar feature set but were comparable in price.
What do you intend on creating with Atlan? Do you have an idea of what use cases you’ll build, and the value you’ll drive?
I think this is a pretty common problem that any data organization has, which is that finding the source of truth of your metrics or business logic is very difficult. So we’re hoping to centralize that using Atlan as much as possible.
Making it easy for people to use Atlan is very important to us. We want to make sure there’s very few barriers to entry, including minimal training time to get them up and running.
We’re also actively planning to utilize Atlan’s tagging features and access controls. The idea is to automate a lot more of our data governance in order to make this process much easier for our engineers and analysts.
Did we miss anything?
I’m thinking a lot of Atlan as a partner. Already, the Atlan team has been proven to listen to our feature requests and at least take them at face value. They’ll tell us if something is useful or not. I’ve also seen this open communication go two ways and the team will tell us, “This is coming out. You guys will need this feature.” That’s been really useful.
Photo by Wiktor Karkocha on Unsplash
