Discover how to successfully handle collaboration, security, team morale and management for remote data teams
Going 100% remote for data teams is #$%*ing hard and chaotic.
Miscommunications can become commonplace. Productivity can be difficult to monitor.
Remote team members (i.e. your people) can easily lose track of the big picture and feel isolated, disconnected or left out.
That’s why so many data team leaders were reluctant to make their teams go remote.
But that was before 2020.
Like it or not, remote work is our new reality and everyone has to find their own way of making things work.
So if your hair’s turning grey with worry over becoming a fully remote data team … you’re not alone. We totally get it.
That’s because although we’re a remote data team ourselves now, our transition to remote was also forced. And with it came our fair share of struggles.
What forced team Atlan to go from on-premise to fully remote?
It all started with an electrical outage a few years ago, causing a fire. 🔥
We had to immediately run out of the office, leaving behind everything—our machines, personal belongings and our servers.
For our Head of Data Science, Himanshu Sikaria, it still feels like it all happened yesterday:
I remember the panic that our engineers and I were in. Back then, all of our data was on the servers on-premise. Years’ worth of work, petabytes of sensitive customer information, everything we had … could have gone up in flames in a matter of seconds. While we stood outside, helpless and powerless to control anything.
Himanshu Sikaria, Head of Data Science at Atlan
Luckily for us, there was no major damage. 🙏
Once the fire was put out, the first thing we did (after checking for structural and electrical damages, obviously) was to plan a complete migration to the cloud.
Within two months, Atlan was fully cloud-first—something we hadn’t even planned for that year. And boy was that a good move! We scaled up our delivery and deploy times and significantly reduced our operating costs.
Since then, we started experimenting with more ways of improving our productivity, guess what was next? Going remote!
We started small—letting our team work 60% in-office and 40% remote. Eventually, we managed to build a fully distributed remote data team, which was behind iconic data projects such as building an alternative data set to predict and combat malaria in India.
Having been through a “forceful move to remote” ourselves, we understand how complex and frustrating it can be. That’s why we’ve put together our learnings and best practices in this resource.
Here are four ways remote data teams can improve productivity and collaboration.
1. Better collaboration within remote data teams
When teams become distributed, the first thing that gets impacted is collaboration. How can you collaborate as you do in the office remotely?
1.1 Adopt over-communication as a habit
The first step is to understand your team:
- Working styles
- Work hours
- Skills
- Abilities and interests
At Atlan, we have daily virtual standups within our teams on Slack or Zoom.
For example, some data teams do an asynchronous Slack check-in every day while other teams have daily video calls to discuss their priorities for the day.
Also, all of our teams do quick check-ins and check-outs on Slack threads, posting their work hours and availability.
While this handles most of the collaboration challenges, it doesn’t solve all problems. So what do we do next?
1.2 Document everything
Like we said, knowing everyone’s work hours helps you figure out when to message someone, but what if you message them while they’re online, and they still don’t reply?
Here’s a true story from one of our analysts.
You are on a client call where the client asks for data manipulation for a report you’d presented. Since you’re not quite familiar with data manipulations, you ask your data engineer to “quickly” do the aggregation and send the report back to you.
He agrees to do it in the next 30 minutes but doesn’t respond for over an hour. Now you’re getting anxious—you are not sure if you should give him a call or wait for some more time. You wish you could see where he’s stuck, or maybe that you knew how to do the aggregation yourself.
A data analyst at Atlan
Sound familiar? 👂
Well, what would make the life of both the analyst and the engineer easier here?
Better documentation!
That means documenting every step of a data project—metadata, column descriptions, business READMEs. At Atlan, we use two tools:
BTW, you could also stitch together a bunch of open-source tools such as Airbnb’s knowledge repository (we’d tried it out before building Atlan for solving our data collaboration needs) and Lyft’s Amundsen to help you with your documentation and cataloging needs.
Psst… Would you like to learn how to build a documentation-first approach in your organization? Then reach out to us for a free consultation today.
1.3 Add some context to your data
You might have heard this a lot, but we felt it’s worth reiterating—context is everything. Ever had scenarios where someone from your team couldn’t:
- Understand what a variable meant?
- Figure out what a data set was used for?
- Find the information they needed for their analysis?
Because we have. And whenever we faced such scenarios, we relied on a few people who knew everything about all the data we had—our own data search engines!
Let’s hear it from Himanshu.Our lead data scientist used to be Google within the company—the data hero upon whom our team depended. So anytime anyone came across a variable name that didn’t make sense, or was looking for a data set to help build their model, they’d go to her desk and ask her.
As we grew, we realized we can’t continue like this. The COVID-19 pandemic is proof that we can’t rely on a few heroes. So we built an internal catalog for data, complete with a data dictionary, business glossary, metadata information and first-level logical checks.
Himanshu Sikaria, Head of Data Science at Atlan
1.4 Prevent the loss of human tribal knowledge
Raise your hand if something similar has happened in your team. 🤚
Someone from the data team wrote a script for churn analysis. The entire process—writing the algorithm, testing it, implementing it in practice and monitoring the results—can take a few months.
Six months after this script went live, this person leaves the team and a new member comes in. They believe that there isn’t any script in place for churn analysis and start writing it—reinventing the wheel. ☸️
Guess what would have come in handy? Keeping records of the human tribal knowledge using a catalog for data. That would make onboarding new remote data team members a breeze!
1.5 Action points: Do this now
- Over-communicate: Do daily standups through Slack threads or Zoom video calls, remote check-ins and check-outs.
- Document everything: Use tools such as Github, G Suite, Airbnb’s knowledge repository, data catalog with business glossary and metadata information.
- Context: Use a data catalog, metadata repository, data dictionary, business glossary or even plain old Excel to capture context.
2. Dependency management within remote data teams
2.1 Communicate dependencies
Let’s go back to our previous example of the analyst waiting for the engineer to get back about some data aggregation.
An update from the engineer on his “focus for the day” would have helped the analyst know how long it would take for the engineer to get back. These updates bring everyone on the same page, help set team priorities and unite everyone towards a single goal.
How do we do it at Atlan? Slack! Our data science team uses Standuply integration with Slack to share their updates.
2.2 Reduce dependencies on engineering
Set up platforms and practices that enable self-service across the analytics lifecycle to get rid of bottlenecks.
In our early days as a remote data team, we taught our analysts how to use the Airflow console to troubleshoot data pipelines.
Today, our analysts leverage the Atlan collaborative projects environment with a drag-n-drop builder to set up visual workflows and schedule data pipelines.
In addition to reducing dependencies on engineering, it also helps with troubleshooting when things go wrong. For example, when a dashboard number is wrong, a business user can just open up a pipeline and understand which step failed.
2.3 Action points: Do this now
- Over-communicate (again): Mention your dependencies with daily standups.
- Adopt self-service analytics platforms: Reduce bottlenecks on engineering and IT by using platforms like Atlan to build workflows and data pipelines.
3. Better security and governance for remote data teams
We don’t have to spell this out for you, but security is one of the main reasons why data teams struggle to go remote.
But where do you even begin? With a move to the cloud. ☁️
3.1 Migrating to the cloud
You might have heard several stories about migrating to the cloud and how, sometimes, it could take years to be fully operational.
However, that’s not always the case. There are tools available—Snowflake, BigQuery, Redshift—that let you set up your initial use cases very quickly.
For example, you could pick a single use case for your team, use an integration such as Fivetran to push data into Snowflake and use this system for further analysis.
This is easy to implement, relatively less time-consuming and in the long run, it will also save costs. If this works out, then you can test it out for other use cases.
3.2 Handling access and infrastructure
What problems did our data team face during our initial days?
When we started working from home, one of the major security concerns was restricting access, which is easier when everyone’s in a protected office environment using IT-issued desktops with massive computing capacities.
But when my team went remote, we switched to IT-issued Macbooks—lower computing power. It also exposed our data infrastructure to several risks as home computers using home internet aren’t as secure as office desktops.
Himanshu Sikaria, Head of Data Science at Atlan
Is that it? Well, Himanshu said there’s one more thing—request approvals.
During the initial days, our approval mechanism was cumbersome as we would assign individual roles on AWS. The IT admin would be flooded with requests—800 requests over 2 months!
For each request, he’d log into AWS, verify the request and edit the IAM role. Imagine going through such an elaborate process at 4 AM?
How did we solve these issues? One step at a time.
Using IT-issued Macbooks with the latest security protocols
Let’s start with the problem of restricting access.
All of our infrastructure and data is on the cloud and can be accessed only through an office-approved VPN.
Also, our authentication process uses Okta’s single sign-on (SSO) protocol and we maintain backups and logs to track usage and regulate access.
Switching to a container-based architecture
Next up: the problem of computing power.
Now that our team is cloud-first, the machine size or computing capacity doesn’t matter. What makes things faster, safer and better is our container-based architecture (Docker and Kubernetes).
Implementing an easy-to-use and agile governance framework
Lastly, let’s look at the approval mechanism.
For access, we switched from IAM roles on AWS to Atlan. Now, controlling access is a simple, two-step process on Atlan: add a user and assign a role. For new requests, all it takes is a few seconds to verify and approve.
We also track usage and adoption across our ecosystem with logs and audit trails, performance stats and alerts—extremely powerful when you have multiple users with IT-issued Macbooks accessing data from remote locations.
3.3 Action points: Do this now
- Moving to the cloud: Start by implementing a small use case and see how things go. Find what works and repeat the process with another use case.
- Securing your infrastructure: Use VPNs, SSOs and firewalls to keep local machines secure and reduce the risk of a breach.
- Backups and logs: Make sure to have backups and logs for everything.
4. The human factor for remote data teams
Humans are social beings and even remote data teams crave that bond. 👭👬
Without it, there’s no way to keep the team motivated, driven and working towards the same goal as the entire company—the remote work culture.
Culture is easier to implement in person. Think team lunches, brainstorming over coffee, celebrations (be it birthdays or festivals), Friday evening hackathons … all of which are great ways to boost morale, spark innovation and cultivate team spirit.
Any remote culture that focuses on simply the tools and processes but forgets about the humans, will end up falling short. So what can remote data teams do?
4.1 Boosting morale and motivation
To start, have daily asynchronous check-ins and video catch-ups within teams. What about team morale, you ask?
So, either before or after a team meeting, have some informal conversations focusing on the positive—talking about your kids or family, new habits to pick up indoors or even what to watch online.
Why? Because otherwise, video calls tend to be very agenda-driven, often missing the human factor.
What else? Have a Slack channel to discuss random things.
Here’s another one: organize a virtual “coffee and chill” session for your remote data teams to just get together and have casual conversations.
Or a remote workstation battle.
4.2 Encouraging innovation
Now let’s take a look at ways to encourage innovation within remote data teams. Says Himanshu:
We were working on a data science model for one of our clients and the model was stuck at 85% accuracy—not acceptable at all. So we took a coffee break and had a discussion with the engineers. I feel that when diverse people come together and discuss the problem, it fuels our creativity and ideas.
And guess what? That discussion led to a breakthrough—our Eureka moment!
Can you have those Eureka moments remotely? Of course! Here are a few ideas to get you started:
- Set up virtual hackathons over coffee.
- Keep asking open-ended questions over Slack and let your team brainstorm the answers.
- Organize “learning Tuesdays” where each team member does a short webinar on something informative and educational.
- Share relevant content to keep the exchange of knowledge and ideas going.
Again, these are just some tips to get you started.
Start with these and who knows, you might end up discovering a routine that works perfectly for your remote data team. 🙌
4.3 Reiterating the big picture
For remote data teams, it’s easy to lose track of the big picture and develop a task-centric mindset.
At Atlan, we take care of it in several ways.
For example, during product planning sessions, we bring together everyone from product and business to revisit the big picture, discuss everyone’s roles and how their work impacts our company goals.
Another thing we do is organize Hill Hacks—once a year, an entire team gets to work from the mountains together, exchanges ideas and develops a better team spirit.
4.4 Action points: Do this now
- 1-on-1s: Have as many as possible with your team to keep them engaged, motivated and happy.
- Innovate: From hackathons to AMAs to virtual brainstorming sessions, think of ways to constantly innovate.
- Big picture: During planning sessions, involve people from different teams to re-emphasize the big picture and make everyone feel like they’re part of one team.
Going remote has numerous benefits, even though the circumstances in which you’re going remote are unfortunate.
So look at this period as an opportunity to unlock some amazing possibilities for your team and build a strong remote work culture.
Do you have any interesting anecdotes, observations or learnings to share on going remote? Share it with us in the comments below.
