Understand the DataOps principles, framework and learn how to implement DataOps within your organization Let’s start with a simple definition of DataOps (aka data operations). DataOps is a data management framework that borrows from the agile methodology, lean manufacturing and DevOps to democratize data, build trust and improve team collaboration. What is DataOps? Sounds quite straightforward, right? If only it were…
Discover how Atlan, a modern data catalog, helps in ensuring data quality Does this sound familiar? The Head of Sales asks for a report of gross sales in Q2 2019. Three months later, he asks for the same report and gets completely different numbers. Why is this happening? Is the new report incorrect? Or was it the original report? Or…
First, if you’re a human of data – aka someone who works with data for a living – here’s your virtual medal. 🏅🏅🏅 Why? Because we know how frustrating it can be to manage data and all its appendages, from metadata to data lineage and beyond! Now we know we can’t wave a magic wand and wish it all away…
DataOps can help you bring together your data, team, tools and processes to become a truly data-driven organization Did you know that Gartner listed DataOps (i.e. data operations) as one of the three innovation triggers listed in data management in 2018? That’s because there’s a lot of interest around the concept. Several people are actively taking part in conversations around…
If you’re a CDO (Chief Data Officer), you already know this—it’s an exciting time to be a CDO! The role is soaring in popularity and importance—the number of CDOs in Fortune 1000 companies shot up from 12% in 2012 to 67.9% in 2018. And companies are all too aware of how CDOs are core to their growth and transformation. Today’s…
Data is most valuable when you have something to compare it to, but these comparisons aren’t helpful if the data is bad or irrelevant. Data is most valuable when you have something to compare it to. For example, it’s nice to know that your program helped 150 people this year, but that doesn’t tell you what you should do next…
In our last post on Apache Airflow, we mentioned how it has taken the data engineering ecosystem by storm. We also talked about how we’ve been using it to move data across our internal systems and explained the steps we took to create an internal workflow. The ETL workflow (e)xtracted PDFs from a website, (t)ransformed them into CSVs and (l)oaded…
Even if you haven’t worked on Kubernetes, chances are you’ve at least heard or read about it. It is already one of the most popular open source projects ever, and it’s still being developed. We need to understand where we’ve come from to appreciate where we’re today. The same rings true for Kubernetes. I was recently told, ‘Boring infrastructure is…
What is the first thing that comes to your mind upon hearing the word ‘Airflow’? Data engineering, right? For good reason, I suppose. You are likely to find Airflow mentioned in every other blog post that talks about data engineering. Apache Airflow is a workflow management platform. To oversimplify, you can think of it as cron, but on steroids! It…