Data is most valuable when you have something to compare it to, but these comparisons aren’t helpful if the data is bad or irrelevant. Data is most valuable when you have something to compare it to. For example, it’s nice to know that your program helped 150 people this year, but that doesn’t tell you what you should do next…
In our last post on Apache Airflow, we mentioned how it has taken the data engineering ecosystem by storm. We also talked about how we’ve been using it to move data across our internal systems and explained the steps we took to create an internal workflow. The ETL workflow (e)xtracted PDFs from a website, (t)ransformed them into CSVs and (l)oaded…
Even if you haven’t worked on Kubernetes, chances are you’ve at least heard or read about it. It is already one of the most popular open source projects ever, and it’s still being developed. We need to understand where we’ve come from to appreciate where we’re today. The same rings true for Kubernetes. I was recently told, ‘Boring infrastructure is…
What is the first thing that comes to your mind upon hearing the word ‘Airflow’? Data engineering, right? For good reason, I suppose. You are likely to find Airflow mentioned in every other blog post that talks about data engineering. Apache Airflow is a workflow management platform. To oversimplify, you can think of it as cron, but on steroids! It…
When discussing data collection, outliers inevitably come up. What is an outlier exactly? It’s a data point that is significantly different from other data points in a data set. While this definition might seem straightforward, determining what is or isn’t an outlier is actually pretty subjective, depending on the study and the breadth of information being collected. So what’s the…
Once your data is structured and cleaned at a basic level, you may need to carry out a series of functions across your data set. Delimiting and concatenating are two simple operations that will help you manipulate your data and make it ready for analysis. Delimiting Data Delimiting data is helpful to automate data segregation. Imagine you are analyzing responses…
Nothing is more frustrating than wrapping up a lengthy data collection exercise, aggregating all the data and looking through it, only to find missing data. At best, these missing values are a nuisance that can be fixed with a bit of work. At worst, they pose an intimidating threat to data quality and your sample size. How can you assess…
For data collected through both paper and digital surveys, you should conduct some basic data checks before carrying out thorough data cleaning. Keep reading for 4 basic data checks that you can use to check for underlying errors in almost any data set. Number of Respondents vs. Rows For any kind of survey, you should always match the number of rows…
The number of villages in India is anywhere between 600,000 and one million, according to various government databases. The number and the definition of villages vary across databases, making it challenging to plan across sectors for a village development plan. There are around 649,481 villages in India, according to Census 2011, the most authoritative source of information about administrative boundaries…