Metrics Runbook

for Dev Teams

Welcome, Dev Leader. This site has everything you need to start and run your engineering metrics program. Check it out and let us know if you have any questions. 

Hero Lady

You're busy. We get it.

Starting your metrics program

A personal note from Dan Lines, former VP of Engineering, Co-Founder and COO of LinearB

Thanks for coming! This site is full of ideas for what you can measure and how you can measure it. Before we get into that, Let’s start with the “why”. 

The first time I ever thought about using data to help run my team was in 2015. The start-up I worked for was growing fast, and overnight (it felt like) I went from leading a single team of 6 devs to leading 5 teams with 50+ devs as VP of Engineering.

I had no experience dealing with an organization of that size. The processes that worked for us when we were smaller weren’t scaling. I knew I needed to be doing more to help my team but I wasn’t sure what.

I dove head first into the problem by talking to every expert I could find. Other VPs of Engineering, agile consultants, start-up investors – anyone who could offer a clue about what scaling successfully looked like. They all said the same thing, you need data to lead a rapidly growing dev team. 

So I reassigned one of our data engineers to do nothing other than compile internal data about how our team worked. I learned a lot and made a ton of mistakes like focusing on individual performance metrics. Spoiler alert: don’t do that! And eventually I came up with a runbook that we now use at LinearB to run a data-driven development organization. Everything I learned and everything we use today is here in this web site. I hope it helps.

Thanks for coming,

dan-blue
Dan Signature White

3 metrics we really like

Cycle Time

Cycle Time measures the time from first commit to production release. At LinearB we call this our super metric because not only does it show team efficiency, it  exposes bottlenecks in the development cycle.

Frame 887 (1)

REwork ratio

Rework ratio measures the percentage of your code that is being rewritten within 21 days of being merged. It’s a predictor measurement of downstream quality issues or an indication that a requirement was missed. 

Frame 888 (5)

Investment Profile

Investment profile measures how much time your team spends on different types of work. It helps you determine if you are striking the right balance between customer-facing stories, non-functional requirements and bug-fixing. 

See your team's metrics in minutes

LinearB automates your metrics program so you can drive continuous team improvement.

Click your Git provider to setup your free-forever account

Group (10)

Metrics scorecard template for your weekly staff meeting

These are metrics we show every single week. Keeping them the same helps reinforce key messages we are trying to teach the business about how engineering works. Cycle Time is about efficiency, pickup time is about teamwork, deployment frequency is about risk and rework is about quality.

We also show investment profile every week because the business is always interested in knowing what we’re working on and what is holding us back from investing more in new features. Our dev leaders like showing Investment Profile because it helps us justify investments in non-functional projects.

We use section 3 to highlight different metrics that are relevant in any given week. Sometimes they stay the same for a few weeks in a row. Sometimes they change. For example, our PR review time went way up in March after work-from-home started so we monitored it here for a few weeks until it came back down.

The cost of doing business is always worth noting. In this section we like to highlight our server costs over time, as well as show more specific costs that are custom to every business and product.

We believe in sharing the bad and the ugly with the business. We use section 5 to flag projects that are behind schedule or have abnormally high risk so we can proactively communicate trade-offs. 

Translating engineering to execs 5 lessons learned

LESSON #1
Think of yourself as a translator

It's hard being the ranking dev leader. Your peers think of you as technical. Your team thinks of you as a suit. You're trapped in between two worlds. And that's our opportunity to provide value to our organization. I learned to embrace my role as translator. My job was to bring business context to the dev team and visibility to the exec team. Without empathy and understanding on both sides, your company will never see the full value in your team and your people will never reach their full potential.

LESSON #2
Focus on process as well as outcomes

As a dev leader, the #1 question we get from non-technical execs is "when will feature ABC be ready?" Of course delivering new value is our job. But allowing all of the conversations to just be about outcomes is a dangerous precedent. The more my CEOs understood how software is built and delivered, the less likely they were to have unrealistic expectations and the more likely they were to offer help (instead of criticism) when times got tough.

LESSON #3
Educate your business with metrics

Metrics are already the language of sales, marketing and finance departments. ARR, LTV, MQL, CAC... Your business not only understands what these metrics mean, they also understand how these terms fit in to the overall business process. There's no reason they can't understand key engineering terms and metrics too. I brought metrics to the CEO's staff meeting every single week to teach my peers about how we work and how to measure our successes and failures.

LESSON #4
Bring your execs into the weeds

Trust me. They can handle it. It's a big mistake to say "This stuff is too technical. They won't get it." What's technical about Cycle Time? It's just a process with phases that represents time and efficiency. Just like a Sales Cycle, which they completely understand. I'm not say I showed code to our CEO. But I did teach them how to talk to our developers about individual branches, PRs, and the work that gets done in the weeds. Non-technical leaders will like the additional sense of confidence it gives them.

LESSON #5
Explain the trade-offs

There's nothing the exec team hates hearing more than "we just don't have enough resources to do that right now." It leaves them with a powerless feeling. Instead educate them on the dependencies for building the new feature they want. I always like to start with "yes" and then go through a step by step process to show the team what would need to be moved around to accomplish the thing they want. I found that approach actually led to me getting approval for more hires and more non-functional investment.

Slider

Connect with Dev Leaders

Dev Interrupted is a community for dev leaders to network and discuss scaling teams, building culture, using metrics, and getting the most from LinearB. 

Dev Interrupted White

17 metrics to drive dev team improvement

Measuring the right indicators will help you accelerate delivery, maintain positive culture, and translate dev activity to business value. So what should modern dev leaders measure?

Keep reading to discover 17 metrics that matter for dev leaders and how to use them.

Delivery Efficiency

Efficient delivery pipelines lead to predictable value delivery, happy developers, happy product owners, and happy customers. Frustrations arise with inefficient pipelines. Measuring the stages of your delivery pipeline allows for bottleneck detection. This provides a high leverage point to increase your delivery performance because it impacts all teams and contributors

Cycle Time
Definition
The amount of time from work started until work finished.
Why it matters
Cycle time is the #1 indicator of your speed to value and efficiency ratio.
Image is not available
Coding Time
Definition
The amount of time from work (coding) began until PR is issued.
Why it matters
This is where the magic happens. Shorter code times indicate appropriately sized chunks of deliverable work.
Image is not available
Pickup Time
Definition
The amount of time it takes from the pull request submitted until review begins
Why it matters
Efficient teams have a low pickup time. The less time PRs spend waiting fo review, the faster they are released. This metric is important for all dev teams, but is even more critical for remote dev teams.
Image is not available
Review Time
Definition
The amount of time from when the first PR comment is posted until it is merged.
Why it matters
Long reviews delay delivery and pose quality risks.
Image is not available
Deploy Time
Definition
The amount of time from Pull Request Merged to Production Release
Why it matters
If time to release is high, it could mean you need to invest more in continuous deployment (CD) or that you have an opportunity to move to a micro-service architecture.
Image is not available
Deploy Frequency
Definition
The number of times your team deploys a release per day.
Why it matters
This is a strong indicator of how much value your team is capable of getting into the hands of customers. Even if you have an efficient pipeline, if your deployment frequency is low, you may not be
delivering enough value.
Image is not available
previous arrownext arrow
Slider

Definition

The amount of time from work started until work finished

Why it matters

Cycle time is the #1 indicator of your speed to value and efficiency ratio.

Definition

The amount of time from work (coding) began until PR is issued.

Why it matters

This is where the magic happens. Shorter code times indicate appropriately sized chunks of deliverable work.

Definition

The amount of time it takes from the pull request submitted until review begins

Why it matters

Efficient teams have a low pickup time. The less time PRs spend waiting for review, the faster they are released. This metric is important for all dev teams, but is even more critical for remote dev teams.

Definition

The amount of time from when the first PR comment is posted until it is merged.

Why it matters

Long reviews delay delivery and pose quality risks.

Definition

The amount of time from Pull Request Merged to Production Release

Why it matters

If time to release is high, it could mean you need to invest more in continuous deployment (CD) or that you have an opportunity to move to a micro-service architecture.

Definition

The number of times your team deploys a release per day.

Why it matters

This is a strong indicator of how much value your team is capable of getting into the hands of customers. Even if you have an efficient pipeline, if your deployment frequency is low, you may not be delivering enough value.

Delivery Quality

Most teams have experienced the situation where low quality leads to missed delivery dates, iteration interruptions, long hours, unhappy customers, and a frustrated engineering organization. There are many different metrics that you could measure as an engineering leader. Some of the classics range from test coverage to service uptime. While these metrics are great, we have found that there are a few other metrics that really help to measure delivery predictability.

Rework Ratio
Definition
Percentage of recently delivered code your team is already rewriting.
Why it matters
A high rework percentage is a predictor of future quality issues. It could also happen when a feature requirement was missed, indicated a communication issue with product management or the customer.
Image is not available
Refactor Ratio
Definition
Percentage of previously delivered (>21 days) code your team is rewriting.
Why it matters
The implication of a high refactor ratio is context dependent. You could be fixing a section of code that has created bugs, or you simply could be updating to accommodate a new feature.
Image is not available
High-risk Branches
Definition
Number of branches with large changes and high rework or refactored work.
Why it matters
The general rule on most dev teams is that the larger the change, the higher the risk (i.e. branches with 300 lines of code are riskier than small branches). Branches with a high percentage of rework and refactored work are also riskier. High risk work is a leading indicator of quality.
Image is not available
PRs Merged without Review
Definition
The ratio of pull requests merged without review.
Why it matters
In addition to helping understand potential quality issues, analyzing unreviewed pull requests can sometimes be an indication of a training or culture issue on a given team.
Image is not available
Review Depth
Definition
The average number of comments per pull request review.
Why it matters
This metric is an indication regarding the quality of the review and how thorough reviews are done. Reviews are an important factor for improving code quality and finding quality issues in code before it is merged and deployed.
Image is not available
previous arrownext arrow
Slider

Definition

Percentage of recently delivered code your team is already rewriting.

Why it matters

A high rework percentage is a predictor of future quality issues. It could also happen when a feature requirement was missed, indicated a communication issue with product management or the customer.

Definition

Percentage of previously delivered (>21 days) code your team is rewriting.

Why it matters

The implication of a high refactor ratio is context dependent. You could be fixing a section of code that has created bugs, or you simply could be updating to accommodate a new feature.

Definition

Number of branches with large changes and high rework or refactored work.

Why it matters

The general rule on most dev teams is that the larger the change, the higher the risk (i.e. branches with 300 lines of code are riskier than small branches). Branches with a high percentage of rework and refactored work are also riskier. High risk work is a leading indicator of quality.

Definition

The ratio of pull requests merged without review.

Why it matters

The general rule on most dev teams is that the larger the change, the higher the risk (i.e. branches with 300 lines of code are riskier than small branches). Branches with a high percentage of rework and refactored work are also riskier. High risk work is a leading indicator of quality.

Definition

The average number of comments per pull request review.

Why it matters

This metric is an indication regarding the quality of the review and how thorough reviews are done. Reviews are an important factor for improving code quality and finding quality issues in code before it is merged and deployed.

Priority Focus

Dev team spend a significant amount of time planning their work. Epics, stories, and issues that need to be completed by a certain time for the team to be successful. Priority focus metrics help the team ensure they are on track to meet objectives. 

Investment Profile
Definition
Breakdown of how the dev team spends time across issue types (bugs, stories, tasks)
Why it matters
The most valuable asset that your organization possesses is your people’s time. Your investment profile is a data-driven representation of the types of work in which your team is spending effort.
Image is not available
Context Switching
Definition
How often your team members switch from working on one issue to another because of being blocked or managerial randomization.
Why it matters
Large amounts of context switching is indicative of an inefficient team. As a manager, it shows you who on your team is blocked or if you need to help the team focus.
Image is not available
Focus work
Definition
Work spent on most important stories for the sprint.
Why it matters
Often called success criteria, in every sprint there are stories that must ship on time for the team and business to be successful.
Image is not available
Unmatched Branches
Definition
Branches that do not align to a story. Often called "Shadow Work"
Why it matters
Shadow work can derailed the successful delivery of a sprint. It can be an indication that the lead needs to better communicate priorities or that a dev has been distracted by a request for an "urgent" issue.
Image is not available
previous arrownext arrow
Slider

Definition

Breakdown of how the dev team spends time across issue types (bugs, stories, tasks)

Why it matters

The most valuable asset that your organization possesses is your people’s time. Your investment profile is a data-driven representation of the types of work in which your team is spending effort.

Definition

How often your team members switch from working on one issue to another because of being blocked or managerial randomization.

Why it matters

Large amounts of context switching is indicative of an inefficient team. As a manager, it shows you who on your team is blocked or if you need to help the team focus.

Definition

Work spent on most important stories for the sprint.

Why it matters

Often called success criteria, in every sprint there are stories that must ship on time for the team and business to be successful.