We use GitHub issues to keep track of all issues. Please do not report bugs or issues in this blog’s comments. Instead, post them on GitHub as an issue. Before submitting a comment with an issue, please use GitHub search to look for existing issues (both open and closed) that may be similar. The PDF (Portable Document Format) was born out of The…
In our last post on Apache Airflow, we mentioned how it has taken the data engineering ecosystem by storm. We also talked about how we’ve been using it to move data across our internal systems and explained the steps we took to create an internal workflow. The ETL workflow (e)xtracted PDFs from a website, (t)ransformed them into CSVs and (l)oaded…
Even if you haven’t worked on Kubernetes, chances are you’ve at least heard or read about it. It is already one of the most popular open source projects ever, and it’s still being developed. We need to understand where we’ve come from to appreciate where we’re today. The same rings true for Kubernetes. I was recently told, ‘Boring infrastructure is…
Landsat is without a doubt one of the best sources of free satellite data today. Managed by NASA and the United States Geological Survey, the Landsat satellites have been capturing multi-spectral imagery for over 40 years. The latest satellite, Landsat 8, orbits the Earth every 16 days and captures more than 700 satellite images per day across 9 spectral bands…
What is the first thing that comes to your mind upon hearing the word ‘Airflow’? Data engineering, right? For good reason, I suppose. You are likely to find Airflow mentioned in every other blog post that talks about data engineering. Apache Airflow is a workflow management platform. To oversimplify, you can think of it as cron, but on steroids! It…
“A little knowledge that acts is worth infinitely more than much knowledge that is idle.” –Kahlil Gibran Scalability, at the end of the day, has a lot to do with time management. Every little act that results in improved efficiency counts when you’re trying to scale a process or an organization. Yet organizations have come to embrace so many inefficient…
The number of villages in India is anywhere between 600,000 and one million, according to various government databases. The number and the definition of villages vary across databases, making it challenging to plan across sectors for a village development plan. There are around 649,481 villages in India, according to Census 2011, the most authoritative source of information about administrative boundaries…