This post will be talking about the data process you need in managing your data team. There are so many different factors that go into building a successful team, but there are five data processes that represent the most important in my experience. This article is the first in a series of posts on Building an Analytics Platform; make sure to read up on the other four articles for a comprehensive guide to your data architecture. There are five main areas we are going to examine:
Before we dig into the five main topics, it’s worth talking about culture for a second; so much of what makes a good team is tied up in their culture. However, this is a tricky thing because you can’t force it.
Culture grows organically from the people you bring into the team, and the way the leadership manages things; it can be greatly influenced by how the company as a whole operates. While I’m far from an expert on creating business cultures, you should get your team together, take an afternoon out somewhere, and agree as a group what you want the team to be.
In my last post, I listed out the people that go into making a great team. The structure I propose here is just one of the many ways you can organise, but it’s one that works well.
It’s worth noting this is a delivery structure rather than a line management structure, so you don’t have to have people reporting to each other if it isn’t suitable for your business.
There’s nothing particularly revolutionary here. The structure of the team is kept quite flat in order to avoid a load of unnecessary middle-management and to drive a sense of responsibility across the board. Nearly everyone in the team is doing some coding or development maximising output.
The structure above represents a full complement of team members for a more mature team, rather than one just starting up. If you’re starting with a much leaner group of people, then avoid the hierarchy until it becomes necessary. As ever, you should adapt to your circumstances with this, and make sure you have something that works for everyone.
When it comes to organising around a specific project or deliverable, you can use squads. Squads take a cross-section of the team and put them together for a while to get a piece of work done. Normally 2-4 people in size, try to bring together a cross-section of skills, including design and development.
One thing I see people doing is trying to align the data engineers to a specific subject area (in retail, this might include stock, logistics, e-commerce, or finance). I’ve tried this before, and it didn’t work. People get bored being pigeon-holed into one area, and it stifles personal development for people in the team. While it is useful to get people building up a deeper understanding of a certain business area, you should balance this with avoiding over-reliance on a single person for any one thing.
I’m going to keep this section very short. There’s a lot of material out there on Agile methodology and I’m certainly not the right person to tell you about it.
What I will say is that when it comes to delivery, it goes without saying that agile is king at the moment. I certainly advocate it, but the focus should be on adapting it to the needs of the team rather than living by the book.
Having a product backlog, organising into sprints and having scrums will help you get things done, but don’t get too worried about doing it all perfectly. Much better to find a process that works for the whole team and more importantly one that people can stick to. As you grow, you’ll likely find you need more rigidity and structure about how you do things so keep reviewing your process.
Running the team using a DevOps model is a great way to take greater control of your delivery, turn around work faster and provide a better level of support to your customers.
Recently, I was trying to hire a DevOps software engineer, and I kept hearing the same thing from the recruiter. When she talked to candidates about being a DevOps software engineer, they said, “I don’t want to do the other software engineers dirty work”.
I found this strange. While a big part of DevOps is certainly around automation of environment and code deployment and enabling a more rapid release cycle, for me DevOps was about having a development and operations team that were one and the same thing. This means that anything you build, you support and if anything was broken you would prioritise the fix to that.
This is a very effective way to grow and scale the team when you have a limited number of headcount and don’t want team members solely dedicated to operations. As the team expands and you have more live code to support you will likely end up bringing in dedicated operations staff.
I call these OpsDev. The difference between the two being the DevOps focus on development first and ops second and OpsDev focus on ops task first and Development second. This way, when everything is working well, your operations team are working through the development backlog and speeding up delivery.
I’m a big advocate of having support activities handled within the team rather than handing it over to a centralised ops team. No one is going to have more expertise on the platform than the team that built it and it avoids wasting time on incomplete handovers. All in, incidents are resolved faster and become less of an issue.
I think we can all agree that most businesses have way too many meetings. When did we all start to think that’s how work gets done? Of course, some are necessary. Here’s my selection of meetings you’re going to need.
And that’s it. That’s enough planned contact points to run the team without spending your life in meetings. The team leads should be doing whatever they can to help keep the development team out of meetings so they can focus on their tasks. Coding requires unbroken concentration and having a meeting every other hour won’t aid this.
To finish off, I wanted to recommend a couple of my favourite tools that support the team. These are far from the only options, they’re just ones I have experience using and have found to be highly effective.
There is so much more we could talk about in this space, but I’ve avoided laying out rigid delivery patterns for you to follow as it’s all about finding the right fit for the team. In fact, if there’s one thing to take away from this it’s that you need to find something that works for you. The key things to remember are:
If you found this article valuable, you might be interested in our next webinar: Considerations for a Successful Data Platform. This hour long session is led by James Lupton, coaching leaders in business, data and tech on how to build a data platform.
Cynozure research calls for stronger data leadership in US financial firms to unlock data benefits
A CLEAR™ Path: How to Foster a Robust Data Culture by Leveraging Concepts from Industrial and Organizational Psychology
Outsmarting the Bots on demand: A practical application of machine learning