How to introduce data-driven culture to your dev team

TL;DR

-Tailoring the “why” for your metrics differently to different people helps w/ translation

-Starting small reduces risk and increases the likelihood of bottom-up adoption

-Adapting your metrics to company goals can increase business alignment

-Embedding your use of data into daily rituals can help with stickiness

The first time I ever thought about using data to help run my team was in 2015. CloudLock was growing fast and overnight (it felt like) I went from leading a single team of 6 devs to leading 5 teams with 50+ devs combined (as VP of Engineering).

I had no experience dealing with an organization of that size. The processes that had worked for us when we were smaller weren’t scaling for our current size and situation. I was pretty sure I needed to be doing more to help my team but I wasn’t sure what.

I’m not afraid to ask for help so I hired an agile consultant to come in and talk to our entire team – engineering, product management and UX. She made a huge difference. I’m still using a lot of what I learned from her today.

One of the most common challenges I hear from software development leaders is…

“I want my team to be more data-driven but I’m not sure where to start.”

Every time I share how we do things at LinearB I hear Kara Minotti Becker’s voice in my head. So I figured I better see what she’s doing now. It turns out she is still a software development consultant in high demand. It also turns out I still have a lot to learn from her.

The ideas and videos below come from our Zoom conversation on Friday, April 10th.

After we caught up (she has a 7-year old girl, I’m expecting my first later this month), Kara asked me…

What prompted you to reach out to me in 2015?

The short answer is, I was in over my head 😀

I asked her what she remembered about our engagement.

She mentioned “You guys were really hungry for understanding the cultural piece of agile. How it helps people work together. Not just how we set the machinery up.”

I remember that too. I’ve always cared about culture. When it comes down to it, as a dev leader, I can’t do anything without great engineers who are driven to help the team. So if I don’t create an environment where talented people want to work, where they feel safe to be themselves, where they can make friends and explore their creativity and learn and have fun… I have no chance to deliver results for the business. I also believe in the power of data. And that’s why I’m passionate about this question.

What steps can engineering leaders take to get their entire team to buy-in to a data-driven culture?

Kara and I landed on four key areas.

1. Start small and take an agile approach

“Engineering leaders need to be careful not to crush their teams under the weight of too many metrics.”
-Kara

When it comes to a new product or feature, we collect the most important requirements, ship an MVP, collect feedback and iterate over time to make it better. For some reason we forget to take that approach with our internal processes.

If we try to measure too many things from the beginning, we introduce risk. Scope creep. Delays. Loss of momentum. Just like a new feature, the first version of your metrics probably won’t be perfect regardless of how long you take to roll them out. You’re better off shipping something fast because you’ll start learning sooner.

Starting small also gives you a better chance of getting bottom-up support. When you pick 1-2 north star metrics that really represent the main goal for your team, it’s easy for everyone to get on board. Anything more than that increases the likelihood of confusion.

PRO TIP: At CloudLock, when I had a new indicator I wanted to measure, I would try it out with a single team to start. After they spent some time using it, I would incorporate their feedback. Not only did this help me work out the kinks before rolling it out broadly, but it also created champions with the team who helped me convince everyone else it was a good idea.

See Kara explain the importance of starting small and iterating.

2. Align your engineering metrics to business objectives

Obviously the next big question is…

What are the most important things I should be measuring?

If only it were that easy 🙂

Kara emphasized “There are no standard metrics that all agile teams should measure. It’s all about knowing what your company’s goals are and picking metrics that help you figure out how you’re contributing to that.”

She also believes that what you measure needs to adapt over time as your company’s goals change. “If your metrics are not aligned to company goals they are pointless.”

I went through three iterations in 5 years at CloudLock.

Value mode: Early on the company’s mission was to deliver as much new product value as possible. So for engineering, it was all about speed to value for us. We had a lot of incredible ideas for our product and our job was to implement them as quickly as possible. I was an individual contributor developer for most of this phase and I got promoted to team lead towards the end.

I remember our leadership focusing a lot on velocity during this time – the number of story points each team delivered in each sprint.

Disclaimer

I think velocity is the most dangerous, most misused metric in software development. It can be useful to inform sprint planning within an individual team. But it should never be shared outside of that team and it should definitely never be used to measure productivity.

Our leadership was trying to measure dimensions of our process like speed and efficiency. We probably would have benefited from measuring Cycle Time but back then the ideas and tools didn’t exist to get that data easily. So we settled for what we could measure – things like velocity.

More on Cycle Time later in this post.

Growth mode: At this point, we had a lot of customers but adding more was still the main goal for the company. We had a bunch of salespeople and they were making commitments to prospects. We had a lot of new marketing people and they had launches and PR announcements. We still needed to ship fast but it was even more important to hit our deadlines. Changing from the mindset of “ship new value as fast as possible” to “ship new value more predictably” was not easy. We were still under pressure to deliver more value but we couldn’t change priorities on the fly anymore.

This phase became all about predictability. By now I was the VP of Engineering. I measured iteration churn (how many items came in and out of the sprint after it started) and the percentage of story points delivered versus committed. These metrics were the best proxies I had to determine our consistency. And, to be clear, we only looked at this data for each team and never published these to the whole org or compared teams against each other using these metrics.

Customer mode: Eventually we reached the point where 500+ large enterprise customers used our product every day and really relied on it. Keeping them happy became our priority. We brought in a VP of Customer Success. We had a top-line company goal of 90% customer retention year over year. This phase was all about quality.

You might think changing our mindset from predictability to quality would be a little easier than changing from speed to predictability. It was actually harder – for three reasons. First, our org was much bigger at this point so there more people we needed to get buy-in from. Second, going fast feels good to everyone. Devs like it. And the business likes it. Nobody likes being told we need to delay a release even if they know a potential bug can have big consequences. Finally, our most important internal customer changed. In predictability, your main internal customers are still sales and marketing. With quality, your main internal customer becomes customer success and technical support. There were new people we had to get to know.

We measured the number of issues, bugs and outages per release which we called “change failure rate”, as well as mean time to restore and mean time to resolve support tickets.

PRO TIP: Looking back on it, we were using a lot of output metrics when we should have been using process metrics. When you focus on an output like how many story points you delivered or how many bugs you had, what do you do when you miss your mark? Measuring the process helps you figure out where and how you can improve. And many process metrics are leading indicators of your outputs so measuring them actually helps you predict when you have a problem coming and allows you to do something about it proactively.

For example, at LinearB right now, instead of using velocity to measure speed, we look at Cycle Time which is the time it takes us to deliver new work from first commit to release. By measuring each phase of our cycle, when this metric goes up like it did for us recently after our transition to 100% work-from-home, we know exactly where to focus to get back on track.

3. Translate the “why” behind your metrics to different audiences differently

“Members of software development teams, including engineers, are often scare of metrics… who they’re going to be exposed to.”
-Kara

I think Kara is right. And it’s not just engineers.

When shared in the wrong way, engineering metrics can be confusing to the rest of the business too.

“It is true that you have to show executive staff numbers that have meaning at their level.”

But that doesn’t mean we shouldn’t make an effort to teach our business the key aspects of software development.

As dev leaders we are somewhat caught in the middle. So what can we do?

Context is everything. If you tailor the “why” behind each of your metrics differently for different audiences, you’ll have a better chance to land your message.

Using LinearB as a case study…

We’re a 25-month-old start-up with strong initial traction and a big vision which means we’re in value mode right now. Our mission is to deliver as much value into our product as possible.

Since we care about speed to value, Cycle Time is a core metric we track. We look at it in every CEO staff meeting, sprint retro and dev all-hands. And we’re especially focused on it in Q2 because our Cycle Time went up a lot last month after our shift to 100% work-from-home.

Cycle Time has a different significance to different people in our company.

Our dev leaders care about maintaining delivery momentum. They like Cycle time because it shows bottlenecks in our process. For example, we were able to see that our coding time and pick-up time were the main contributors to our Cycle Time going up in March. So to get us back on track, our dev leaders adapted some of our processes in those areas to fit work from home.

Our business leaders care about acquiring more customers. They like Cycle Time because they see if our delivery speed is trending positively and they can correlate that to new feature releases.

In Q2 our execs are also looking closely at a phase of Cycle Time called Review Time – the amount of time it takes to merge a PR after the review process starts. Review Time is fairly in the weeds but we’re using it as a proxy to measure teamwork between developers. If Review Time goes up, it means we may need better processes or technology to help our remote devs work together better.

Our developers care about innovating, building cool stuff and trying new things. They like Cycle Time because a faster cycle is a signal that they get to take on more challenges more often. They also like it because bottlenecks in the cycle (like in Release Time, for example) can be an indication that more automation is needed. Devs hate non-creative, manual work 🙂 And of course our developers like to see the connection between their work and the success of the company. They think it’s cool that our CEO looks at an engineering process metric every week.

PRO TIP: Kara says when it comes to your dev team, “make sure they know what good looks like.” This can actually be hard when you’re starting out because you may not always have benchmarks for everything you want to measure. But you can’t get everyone to rally around a metric if they don’t know what they’re shooting for. Set yourself up for success by creating small achievable milestones to start. This will make it easy to celebrate small wins along the way which builds momentum for the bigger goal.

4. Look for opportunities to include metrics in your existing rituals

The best way for your metrics to really become part of your culture is to “operationalize” them into your existing activities.

Daily stand-up, 1:1s, scrum of scrum, retrospective, all-hands, coffee talk… many of these meetings will be better with data.

We used to broadcast our LinearB dashboard on the TVs around our office. Now we pull them up in many of our Zoom meetings.

But be careful not to overdo it. It is possible to have too much of a good thing. When you’re thinking about whether data needs to be incorporated in to a meeting or ceremony, ask yourself… is data in this case going to help us reach our desired outcomes or will it be noise?

PRO TIP: Early in my journey of using data to run my team, a mentor told me to give praise and constructive criticism in the context of the metrics we care about as a team. Not only will that reinforce data as a key figure in how the team works, it will make your feedback more concrete and actionable. For example, at LinearB this quarter since we’re focused on pick-up time, each week we acknowledge the dev who helps their teammates by picking up PRs the fastest.

Using metrics in your executive staff meeting

Kara rightly points out “At scale, which is where most executives are working, there is far too much volume of people, of numbers, of processes for them to know the individual details and context of projects. They have literally no choice but to manage by numbers at least to some extent. So the fitness-of-purpose, accuracy, and context of those numbers become ever more important to help executives manage confidently. ”