From collaboration and dependencies to productivity, find out what are the top challenges that remote data team leaders face with agile.
Raise your hand if you’ve ever planned a data project. 🤚
You, my friend, deserve a standing ovation. Because you know how difficult, complex and frustrating it can be. You have to:
- Work with your data engineers to extract data from a multitude of sources and formats
- Coordinate with your DevOps engineers to process the TBs of data pouring constantly into your systems
- Communicate with geospatial experts to understand and interpret complex satellite data
- Liaise with economists to build the business use cases and then with data scientists to perform exploratory data analysis to get the answers to those use cases
It’s like a pipeline with humans, where everything has to fall in place and work in sync. If one thing gets delayed, then the entire data project suffers.
Imagine achieving this at scale. If it sounds nightmarish, you’re not alone. I went through this with our data team at Atlan and to find some answers, I started reading more about product management methodologies. That’s how I came across agile and scrum (psst… more on this later).
Now agile isn’t the cool new kid on the block. IT and software engineering teams have been following agile practices for a while now.
But data teams are not like software engineering teams. That’s because software engineering teams generally know what they are building and are able to estimate goals and timelines. Data science, on the other hand, is exploratory. A task could take anywhere from 24 hours to a week. 🤷♂️
As if agile for data teams wasn’t complex enough, we added yet another complexity to it—going fully distributed.
Getting started with agile as a remote data team
When we started following the principles of agile, for the first few weeks, things were messy.
We would plan and follow everything that the agile manifesto said, but suddenly we couldn’t meet our timelines. Miscommunication, misunderstanding, dependencies and bottlenecks became rampant.
Instead of working together as a pipeline, everyone within the team was isolated, not unlike “human silos”. 🏝
As the head of data science, I was stumped. But then along with our co-founders, I went on a quest to tackle these challenges we faced with agile for remote teams.
And guess what?
Six months into our transition, we managed to do twice the work in half the time. And if our team managed to reach this stage of mega-productivity, so can yours.
In this two-part blog series on agile for remote data teams, I’m hoping to share our challenges and how we overcame them to create a productive data team—all in the hope of helping data teams around the world (especially in the times we are faced with today).
As the introductory blog of a two-part series, this article will focus on the biggest challenges that data team leaders face with agile when their teams are transitioning to a remote lifestyle.
1. Dealing with remote collaboration
Business people and developers must work together daily throughout the project.
Agile manifesto
When your teams are distributed, one of the biggest challenges is virtual collaboration. Coordinating across different time zones, building a rapport with teams that don’t work in close proximity or even at the same time is hard.
For example, your data science team based in the US works from 12 PM to 9 PM PST whereas your data success team based in Singapore works from 10 AM to 6 PM SGT, which is 7 PM to 3 AM PST. The only time both teams are online simultaneously is just a two-hour window.
The agile manifesto also preaches the power of face-to-face communication.
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
Agile manifesto
Sure, there’s no denying the effectiveness of face-to-face conversations that happen in person. But since we can’t really have those in-person conversations especially due to quarantine and social distancing, we have to figure out a way to get the same results while remote.
P.S. Wondering whether there are some quick fixes you can implement to improve virtual collaboration? Check out our blog on top tips for collaboration within a remote data team.
2. Keeping sight of the big picture
As a product person, I’m always curious about marketing and sales progress.
What do our customers have to say about our platform? How does the data ecosystem perceive our product?
When everyone’s co-located, I can easily walk up to marketing and find out these answers for myself. Not ideal, but doable.
All that goes for a toss when we’re remote. So how can each member of the team stay in sync and align their work towards a common team goal?
How do I, as the Head of Data Science, ensure that we are solving for the things that the customers need through our marketing and sales efforts?
Similarly, how can the business teams understand what goes behind building a great product that the customers value? How can they demystify the black box that is the data team’s work?
These are hard enough to achieve even when everyone’s at the same location, so imagine doing the same while remote.
BTW… demystifying what the data team does is a concern even for teams that are co-located. A data catalog can come in super-handy at such times.
3. Building trust
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
Agile manifesto
When everyone’s in the same room, the energy levels are higher, everyone’s pumped up and eager to help the company achieve its goals. It’s also easier for leaders to build trust and team morale.
But all that changes when your entire team’s remote. What makes it even more challenging is when your managers don’t have any remote leadership experience.
When we started going remote, it wasn’t uncommon to hear comments like this.
How do you know your remote folks aren’t slacking off?
Ouch! 🤭
Thoughts and concerns like these bring the trust factor under scrutiny, which doesn’t make things easier at all for a team to work in tandem.
4. Keeping productivity high
Going remote makes it harder for managers to keep track of productivity and accountability. I used to ask myself questions like these all the time.
How do you know if your team members are delivering their best? How can you ensure that they don’t face too many roadblocks, not just due to remote collaboration but also due to their home environments?
Take, for instance, the office environment, which is well-equipped with everything that your teams need.
However, when your team members are remote, you cannot really assure that their environment is supportive and offers them everything they require to do their best work. Especially when everyone’s living in times of uncertainty with an unprecedented crisis.
Another challenge with remote productivity is that it gets difficult to quantify work, especially in a situation that the team’s never experienced. Add to that the fact that most of the work we do hasn’t ever been done before… you get the gist. 😩
💡A quick tip: We’ve put together some of the best tips and learnings on productivity for every remote data team in our latest ebook—the ultimate guide for remote data teams. So go ahead and check it out.
5. Managing dependencies
At Atlan, we faced challenges with time management, which eventually led to a chain of delays as most of the data team relied on each other for almost everything.
Imagine this scenario.
Our data analyst estimates that preparing a report on factors that impact sales would take two hours. Halfway through the report, the analyst realizes that they need some data aggregations and for that to happen, they’ll have to loop in our data engineer, who already has a lot on his plate for the day. Since they work different hours and are at different locations, just syncing with each other takes almost half a day.
That’s why while incorporating agile as a remote data team, you should also be asking yourself:
- How can we make sure that a task estimated to get completed in 30 minutes does not end up taking half the day?
- How do we ensure that delay in one thing does not impact work for others in the team and result in a chain of further delays?
So there you go—five of the toughest challenges remote data teams face with agile. Now the question that comes to mind is this—do we have a way out?
While we don’t have perfect solutions, we have some learnings and best practices to share based on our experience. Stay tuned to find out what we did to tackle these challenges, and build a team that managed to do twice the work in half the time. 🚀
