From collaboration and dependencies to productivity, find out what are the top challenges that remote data team leaders face with agile.
Raise your hand if youâve ever planned a data project. đ€
You, my friend, deserve a standing ovation. Because you know how difficult, complex and frustrating it can be. You have to:
- Work with your data engineers to extract data from a multitude of sources and formats
- Coordinate with your DevOps engineers to process the TBs of data pouring constantly into your systems
- Communicate with geospatial experts to understand and interpret complex satellite data
- Liaise with economists to build the business use cases and then with data scientists to perform exploratory data analysis to get the answers to those use cases
Itâs like a pipeline with humans, where everything has to fall in place and work in sync. If one thing gets delayed, then the entire data project suffers.
Imagine achieving this at scale. If it sounds nightmarish, youâre not alone. I went through this with our data team at Atlan and to find some answers, I started reading more about product management methodologies. Thatâs how I came across agile and scrum (psst… more on this later).
Now agile isnât the cool new kid on the block. IT and software engineering teams have been following agile practices for a while now.
But data teams are not like software engineering teams. That’s because software engineering teams generally know what they are building and are able to estimate goals and timelines. Data science, on the other hand, is exploratory. A task could take anywhere from 24 hours to a week. đ€·ââïž
As if agile for data teams wasn’t complex enough, we added yet another complexity to itâgoing fully distributed.
Getting started with agile as a remote data team
When we started following the principles of agile, for the first few weeks, things were messy.
We would plan and follow everything that the agile manifesto said, but suddenly we couldnât meet our timelines. Miscommunication, misunderstanding, dependencies and bottlenecks became rampant.
Instead of working together as a pipeline, everyone within the team was isolated, not unlike âhuman silosâ. đ
As the head of data science, I was stumped. But then along with our co-founders, I went on a quest to tackle these challenges we faced with agile for remote teams.
And guess what?
Six months into our transition, we managed to do twice the work in half the time. And if our team managed to reach this stage of mega-productivity, so can yours.
In this two-part blog series on agile for remote data teams, Iâm hoping to share our challenges and how we overcame them to create a productive data teamâall in the hope of helping data teams around the world (especially in the times we are faced with today).
As the introductory blog of a two-part series, this article will focus on the biggest challenges that data team leaders face with agile when their teams are transitioning to a remote lifestyle.
1. Dealing with remote collaboration
Business people and developers must work together daily throughout the project.
Agile manifesto
When your teams are distributed, one of the biggest challenges is virtual collaboration. Coordinating across different time zones, building a rapport with teams that donât work in close proximity or even at the same time is hard.
For example, your data science team based in the US works from 12 PM to 9 PM PST whereas your data success team based in Singapore works from 10 AM to 6 PM SGT, which is 7 PM to 3 AM PST. The only time both teams are online simultaneously is just a two-hour window.
The agile manifesto also preaches the power of face-to-face communication.
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
Agile manifesto
Sure, thereâs no denying the effectiveness of face-to-face conversations that happen in person. But since we canât really have those in-person conversations especially due to quarantine and social distancing, we have to figure out a way to get the same results while remote.
P.S. Wondering whether there are some quick fixes you can implement to improve virtual collaboration? Check out our blog on top tips for collaboration within a remote data team.
2. Keeping sight of the big picture
As a product person, Iâm always curious about marketing and sales progress.
What do our customers have to say about our platform? How does the data ecosystem perceive our product?
When everyoneâs co-located, I can easily walk up to marketing and find out these answers for myself. Not ideal, but doable.
All that goes for a toss when weâre remote. So how can each member of the team stay in sync and align their work towards a common team goal?
How do I, as the Head of Data Science, ensure that we are solving for the things that the customers need through our marketing and sales efforts?
Similarly, how can the business teams understand what goes behind building a great product that the customers value? How can they demystify the black box that is the data teamâs work?
These are hard enough to achieve even when everyoneâs at the same location, so imagine doing the same while remote.
BTW… demystifying what the data team does is a concern even for teams that are co-located. A data catalog can come in super-handy at such times.
3. Building trust
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
Agile manifesto
When everyoneâs in the same room, the energy levels are higher, everyoneâs pumped up and eager to help the company achieve its goals. Itâs also easier for leaders to build trust and team morale.
But all that changes when your entire teamâs remote. What makes it even more challenging is when your managers donât have any remote leadership experience.
When we started going remote, it wasnât uncommon to hear comments like this.
How do you know your remote folks arenât slacking off?
Ouch! đ€
Thoughts and concerns like these bring the trust factor under scrutiny, which doesnât make things easier at all for a team to work in tandem.
4. Keeping productivity high
Going remote makes it harder for managers to keep track of productivity and accountability. I used to ask myself questions like these all the time.
How do you know if your team members are delivering their best? How can you ensure that they donât face too many roadblocks, not just due to remote collaboration but also due to their home environments?
Take, for instance, the office environment, which is well-equipped with everything that your teams need.
However, when your team members are remote, you cannot really assure that their environment is supportive and offers them everything they require to do their best work. Especially when everyoneâs living in times of uncertainty with an unprecedented crisis.
Another challenge with remote productivity is that it gets difficult to quantify work, especially in a situation that the teamâs never experienced. Add to that the fact that most of the work we do hasnât ever been done before… you get the gist. đ©
đĄA quick tip: We’ve put together some of the best tips and learnings on productivity for every remote data team in our latest ebookâthe ultimate guide for remote data teams. So go ahead and check it out.
5. Managing dependencies
At Atlan, we faced challenges with time management, which eventually led to a chain of delays as most of the data team relied on each other for almost everything.
Imagine this scenario.
Our data analyst estimates that preparing a report on factors that impact sales would take two hours. Halfway through the report, the analyst realizes that they need some data aggregations and for that to happen, theyâll have to loop in our data engineer, who already has a lot on his plate for the day. Since they work different hours and are at different locations, just syncing with each other takes almost half a day.
Thatâs why while incorporating agile as a remote data team, you should also be asking yourself:
- How can we make sure that a task estimated to get completed in 30 minutes does not end up taking half the day?
- How do we ensure that delay in one thing does not impact work for others in the team and result in a chain of further delays?
So there you goâfive of the toughest challenges remote data teams face with agile. Now the question that comes to mind is thisâdo we have a way out?
While we donât have perfect solutions, we have some learnings and best practices to share based on our experience. Stay tuned to find out what we did to tackle these challenges, and build a team that managed to do twice the work in half the time. đ
