Find out how your remote data team can 4x its productivity with scrum
If you’re here, I’m hoping you’ve read my previous article on the toughest challenges that remote data team leaders faced with agile. Now let’s roll up our sleeves and dive right into the solutions.
Like I mentioned in the previous post, our co-founders and I were on a quest to find the right framework for our data team.
While we knew we wanted to be agile, we had no clue which framework to adopt.
Now I always considered that no single framework could be generic enough to solve all the problems for all the varied kinds of companies. And since the agile manifesto was written mostly for IT teams, I didn’t believe that there was a framework for data teams with diverse skill sets and profiles.
But that was before I came across the story of how the FBI built a central platform for predicting crimes and terror threats (aka the Sentinel program) with the help of… wait for it (and no, it wasn’t IBM, SAIC or Lockheed Martin)… agile experts. 😱
An unlikely agile success story: The FBI’s Sentinel program
For those of you who aren’t familiar, here’s the gist of it.
A dream of going 100% paperless
The FBI, back in 2000, wanted to build a centralized digital repository for its data on crimes in the US and then, using data science models, stop crimes before they happen. (And in the process, get rid of their decades’ worth of paper files.)
Their initial budget for the platform (known as the Virtual Case File system) was USD 170 million. What began with great enthusiasm initially quickly turned into a soul-sucking government project that was going nowhere—within nine months, they’d exhausted the entire budget with little progress.
Five years later, the entire operation was shut.
Same dream, a different contractor
Fast forward a few years—new management, new contractor, triple the original budget and more time. Outcome? Another debacle. 😓
That’s when they realized that they can’t keep making the same mistakes and expect a different outcome.
More money, new management and extended deadlines isn’t a solution if the process itself is broken.
So they reevaluated the entire process and incorporated a popular agile framework with the help of agile experts.
The FBI gets agile
The FBI needed a mechanism to deliver quickly while adapting to changing requirements. And agile checked all of those boxes.
Guess what happened next? They took the entire project in-house, adopted agile, used a two-week iterative framework and built Sentinel. It took them slightly longer than the initial deadline, but they managed to deliver and were well within their budget! 🙌
The framework that transformed an expensive, complex and time-consuming government IT project into a success
What was that magical iterative framework, you ask?
Scrum! Scrum is a popular agile framework used by teams worldwide across companies such as Google, Spotify and IBM. And as you can see from our FBI story, it’s not just for tech or digital native companies but for all sectors—even government agencies.
Now before we proceed any further, let’s pause and do a quick review of scrum (for agile, not rugby).
Scrum 101: An overview
The underlying principle is simple:
Just like a machine learns and makes better forecasts after every iteration, the Scrum framework says that if the same exercise is repeated again and again, we can get better at it by working on the weaker areas.
So here’s how it would work (and don’t feel put off by the jargon right now, I’m going to explain them in a bit):
- You plan to accomplish certain things in two weeks (i.e. sprint planning)
- You and your team execute for two weeks (i.e. the sprint)
- At the end of two weeks, review everything that worked and the challenges you faced (i.e. the retrospective)
If you keep repeating the cycle of Plan → Execute → Retrospect, then over time, your team will reach its optimal productivity and do twice the work in half the time.
That’s exactly what the agile experts at FBI did to get the Sentinel program up and running.
Now let’s look at the jargon. To make it an easy read, I’m dividing the terms into three main categories:
1. The framework and its components
- Sprint: Each cycle of Plan → Execute → Retrospect; usually lasts 1-3 weeks.
- User stories: The smaller tasks that make up a sprint written in a way that the end-user understands; instead of “Build a CRON job on Airflow for the Twitter data”, say “User can access Twitter data every day without any manual intervention”.
- Scoring: Assigning scores using the Fibonacci sequence (0,1, 1, 2, 3, 5, 8, 13, 21…) to understand the complexity and time required to finish each user story.
- Scrum board: Tells the story of each sprint; generally divided into four sections:
- Backlog: User stories that the team decides to do for the entire project.
- To-do: User stories the team plans to complete during the current sprint.
- WIP (Work in Progress): User stories that the team is currently working on.
- Done: User stories that have been completed.
- Velocity: On the last day of the sprint, everyone adds up the scores of all the stories in the ‘Done’ section of the scrum board to calculate the total velocity of the team. If it’s lower than the score of the To-do section, then the velocity is negative.
So far so good, yeah? Bear with me, we’re almost done. 🙏
2. The people
- Scrum master: The person who aligns the team in the right direction and ensures that the processes are being followed.
- Product owner: The person responsible for the product and all of its features that need to be built.
3. The processes
- Daily standup: Every day, the entire team meets for 10-15 minutes to discuss the roadblocks and dependencies. The main questions to answer are:
- what they did yesterday
- what they will do today
- what is stopping them from going 10x
- Retrospective: At the end of every sprint, the team regroups to discuss what went right and what didn’t to understand the team’s performance.
- Demo: Also at the end of each sprint, demonstrate everything that got done and get feedback from the entire team.
Woohoo! You made it! 💯
Now comes the fun part—implementation.
How did Atlan implement scrum for its remote data team?
Being remote, this isn’t easy (and you already know that).
We started by briefing our team on scrum and explaining how it works. Then we had our first-ever sprint! It lasted a week and we faced some hiccups (bound to happen with something that none of us had ever done before).
Despite that, we could see the results as it had already improved our productivity.
So we went through a few iterations, experimented with the framework itself on sprint duration, formats for the rituals (i.e. standups, planning and retrospective sessions) and scoring.
Why experiment?
Because initially, our sprints lasted just a week. But that impacted our planning sessions as the scrum masters didn’t get enough time to prepare for planning and retrospectives. This lack of planning (more on this below) reflected in our sprint productivity and thus, we realized that we needed longer sprints.
Eventually, after six months, we managed to achieve double in half the time. 🎉
During this six-month journey, we had some useful insights on how to make scrum easier for your remote data team. I’ve summed them up into five main points here.
5 top learnings from scrum for remote data teams
1. Check-ins and planning sessions for better collaboration
We’ve had bad sprints and low-productivity weeks.
Guess what these had in common? Bad sprint planning.
Either we took shortcuts to plan or didn’t spend enough time prioritizing what needs to be done first. That, in turn, led to misunderstandings, miscommunication, bottlenecks and frustrations everywhere. That’s why the first learning is… 👇
1.1 Take enough time to plan
I cannot stress on this enough—ensure that the sprint leader of your remote data team (be it the scrum master, product owner or team lead) spends plenty of time:
- Understanding the complexity of the job at hand
- Meticulously planning the user stories for each sprint
Even for a team of 5, more than 8 hours go into the stand-ups and sessions every week.
As I mentioned before, we even extended our sprint durations, going from one-week sprints to two-week sprints, to accommodate for the time spent in planning.
1.2 Do daily check-ins
Another problem we faced when the data team went 100% remote was not knowing who was online at what time.
So everyone posts their available hours and focus for the day via daily asynchronous check-ins.
This helped us with not just collaboration but also organizing planning sessions when everyone was online.
2. Documentation and retrospectives to not lose sight of the big picture
When new people started joining the team and following our rituals, we faced a big challenge—they didn’t know why we did what we did.
It’s important to ensure that everyone understands:
- Why we follow scrum
- What motivated us to adopt this framework
- All the accomplishments and failures so far
We found two approaches to tackle this challenge.
2.1 Document everything
If you’re wondering why your remote data team should adopt a documentation-first approach, check out my previous article in this blog series.
When we initially switched to being fully remote, we tried G Suite, Github, Airbnb’s knowledge repository, Trello, Asana and Slack to start documenting our practices, processes and sprints. (I know, I know… that is a lot of tools! 😥)
Eventually, our data team moved on to the Atlan platform for all data-related discussions and documentation.
As far as recording our sprints, onboarding and documenting our code is concerned, we use Github. This makes things easier, especially for the newcomers. So whenever someone new joins the team, the Atlan Wiki repository on Github is their starting point and has all the information (i.e. context and knowledge) they need to get started.
2.2 Take retrospectives seriously
Retrospectives are critical to the success of your sprint and achieving 4x productivity.
Initially, we’d miss out on these sessions as it’s hard to talk about what went wrong. However, discussing the wrongs is essential to find out what’s not working and take steps to fix it.
So we started taking this seriously.
At the end of each sprint, the entire team would discuss what went wrong and what needed to be improved. Atlan values being straightforward, which is ingrained in every team member and that made retrospective sessions slightly easier for us. 💪
To make it more fun, we’d all share a GIF that showed how we felt about our productivity. This helps get the brain juices flowing on productivity and efficiency.
P.S. It’s fun when everyone starts introspecting and comes up with interesting (and funny) GIFs—instant energizer and mood-lifter!
3. Context switching for greater productivity
The 21st-century skill to do multiple tasks at once is there in most of the job profiles these days. Unfortunately, this is killing our time, brain space and productivity without us knowing. 🤯
If we work on two projects simultaneously, we spend 20% of the time in context switching and if we are on five projects, we waste as much as 75% of the time.
While we still haven’t been able to fully solve this problem, what has helped is acknowledging that it’s a problem.
Something that helps us is the practice of restricting team interactions before 5 PM; the morning hours are reserved for focused work.
As a result, everyone is more mindful of not context switching. This, in turn, helped us adopt the practice of discussing priorities and dependencies during our daily standups (more on this below).
4. Virtual scrum boards and daily standups to build trust and maintain accountability
Our initial days as a remote team were as painful as eating an ice-cold popsicle with a bad tooth. Ouch! 😰
Slack messages would go unanswered, meetings would start late and dependencies would skyrocket. Also, since we used a physical board as our scrum board, trying to find the right virtual tool that everybody loved was another problem.
So what did we do?
4.1 Virtual scrum board
Let’s start with the easier issue—the tool. We started using Github as our scrum board and it has been working for us so far.
Here’s how we use it during our planning and retrospective sessions as well as standups:
- Everyone updates their user stories before the call.
- During the call, the person who’s speaking shares their screen to walk everybody else through their list of tasks and mention dependencies, if any.
4.2 On-time standups
We use Zoom for our standups and the Google Calendar integration with Slack to remind everyone.
Also, to ensure that the meetings start on time, we share the Zoom link in advance and the Slack alert would guarantee that everybody shows up on time.
The standup format is also set. Everyone has to answer four questions:
- What did you do yesterday?
- What will you work on today?
- What dependencies do you have?
- What is stopping you from going 10x?
Another rule we have is that nobody interrupts the person sharing their update. This brought a sense of order to the otherwise chaotic standups.
Lastly, we insist that everyone came prepared to the standups as that just transforms the quality of our standups instantly.
5. “Fix it now” culture
If something takes less than 5 mins, do it now.
The motto at Atlan
If you find a problem or a bug, fix it right away and don’t postpone it.
When you face the problem, that’s the moment you have the maximum context. So why postpone fixing it?
One of the reasons why Toyota had a better and more efficient production line than BMW was because they would stop the entire line when they saw a defect and everyone would try to fix it.
Last but not least, keep experimenting with new things after every sprint. When I look back and notice how far we’ve come, I realized that none of this would have been possible had we stopped trying new things to see what works best.
Now it’s your turn. What are you doing to make agile easier for your remote data team?
