Improving dev teams through objective data on software development is here and it all starts with DORA metrics.
For a long time, it was thought that data couldn't help inform the software-development process. Thought leaders like Martin Fowler and Joel Spolsky basically said measuring developer productivity was impossible.
Well, it was impossible to do. But now, with the rise of tools like Git, Jira, and other project management platforms, it became clear that the data is there to enable us to get a closer, more metrics-driven look at what is going on inside software development projects. That data just had to be revealed.
What are DORA metrics?
The most influential development in understanding how to think about measuring software development came from the DevOps Research and Assessment (DORA) organization. The group comprised of different engineering leaders surveyed thousands of DevOps engineers over six years, coming up with a set of four key metrics that were considered critical to the success of software development projects.
The four DORA metrics are:
The first two metrics -- deployment frequency and mean lead time for changes -- measure the velocity of a team. Time to restore service and change failure rate measure the quality and stability of a project. All four metrics can be derived from leveraging tools that are common on most dev teams.
These four DORA metrics have become the standard way for CTOs and VPs of Engineering to get a high-level overview of how their organizations are performing.
By keeping an eye on the DORA metrics and organizing their work around improving them, development teams can ensure they are doing the right things to move their projects, and, more importantly, their business, forward.
Of course, understanding what the metrics actually measured and what they mean is necessary to make them useful. In addition, knowing the current state of these metrics is required for improving them as you move forward.
So let’s take a look at these four key measures.
Deployment frequency measures the number of times that code is deployed into production. It’s usually reported in deployments per day or per week.
Now, production can mean different things to different customers. For a SaaS company, it normally means actually delivering code to the production platform used by actual customers. For other companies, it might mean “made a version available for use by customers.”
Why is deployment frequency an important DORA metric?
Improving deployment frequency indicates team efficiency and confidence in its process. A team that can deploy more frequently is moving work through their pipeline faster and being more efficient about all of their work products.
How is this DORA metric calculated?
Deployment frequency is derived from the total number of deployments an organization does in a single day. As noted, the definition of “deployment” can vary between organizations. This metric can be automated if a team has a Continuous Integration/Continuous Delivery(CI/CD) tool that provides an API into its activity.
How do you improve deployment frequency?
If you want to improve your deployment frequency, you should invest in:
- Improving automated test coverage
- Integrating with CI/CD tools
- Automating the release validation phase and release process
- Reducing the error recovery time on production
Lead time for changes
Mean lead time for changes is the average time it takes from code being committed to that code being released into production.
Some organizations begin tracking the time from the first commit of the project’s code, while others measure it beginning from merging the code to the main branch.
Many organizations roll mean lead time for changes into a metric called cycle time, which is discussed below.
Why is lead time for changes an important DORA metric?
A lower mean lead time for changes means that your team is efficient in coding and deploying projects and is adding value to your product in a timely manner. Attempting to lower the average incentivizes teams to properly divide work, thoroughly review code, and have a fast deployment.
How is this DORA metric calculated?
Each project is measured from start to finish, and an average of those times is calculated.
How do you improve lead time for changes?
This metric can be improved by:
- Set goals around industry engineering benchmarks for things like pull-request size and code review time
- Create workflow automations for the code-review process
- Focus on the cycle time portion of lead time because that is where you have the most control over improvements. Read our guides on how to improve coding time, pull request pickup time, pull request review time, and deploy time.
Time to restore service
Time to restore service is the amount of time it takes the dev team to recover from a failure in the system.
“Failure” can mean anything from a bug in production to the production system going down.
Why is time to restore service an important DORA metric?
Obviously, downtime is not good, and the quicker a team can recover from it, the better.
Keeping an eye on time to restore service will encourage the building of more robust systems and increased monitoring of those systems.
Quick recovery and response times are a reflection of the team’s ability to diagnose problems and correct them. Measuring time to restore service can have the effect of making the team more careful and concerned about quality throughout the entire development process.
How is this DORA metric calculated?
Normally, this metric is tracked by measuring the average time to resolve the failure, i.e. between a production bug report being created in your system and that bug report being resolved. Alternatively, it can be calculated by measuring the time between the report being created and the fix being deployed to production.
How do you improve time to restore service?
Time to restore service can be made better by:
- Find out your mean time to restore
- Set up automatic alerting and allot resources to manage incidents
- Implement automations to prioritize recovery from failure over all other tasks
Change failure rate
Change failure rate measures how often a code change results in a failure in production. Changes that result in a rollback, in production failing, or in production having a bug all contribute to this metric.
Why is change failure rate an important DORA metric?
This metric is important because all time spent dealing with failures is time not spent delivering new features and value to customers. Obviously, lowering the number of problems in your software is desirable.
How is change failure rate calculated?
Normally, this metric is calculated by counting the number of times a deployment results in a failure and dividing by the number of total deployments to get an average number. A lower average is better.
How do you improve change failure rate?
Change failure rate is improved when you:
- Use automated testing as part of your CI/CD pipeline and ensure all new code is covered
- Schedule robust, blameless retrospectives after failures in order to improve processes
- Implement static code analysis and automations to flag potential issues before they make it to production
The benefits of tracking DORA metrics
Informed & empowered decision making
Consistently tracking DORA metrics will enable you to make better decisions about where and how to improve your development process. Doing so will reveal bottlenecks, and enable you to focus attention on those places where the process may be stalled. Trends can be identified and the quality of decisions about what was focused on can be validated.
DORA metrics can help focus both the development team and management on the things that will really drive value. They allow you to make decisions based on data rather than merely a finger in the wind or a gut feeling.
Real proof of engineering delivering value
DORA measures the value that your team is delivering. If your DORA metrics are favorable, your team is delivering value to your customers and maintaining the quality necessary not to be distracted from that focus. And that’s pretty much the bottom line for any business, delivering value to your customers.
Incentivize organic improvements
When anything gets measured, it will likely be gamed. That is, people will change behavior to optimize that which is measured. Many times this can have a negative, distorting effect on what a development team does.
DORA metrics can be gamed, but the great thing is that you want them to be gamed. You want your team working to optimize these metrics. Gaming them results in good things.
Normally, gaming a metric has a negative impact on teams, but these metrics were carefully devised to do the exact opposite, create high-performing teams. Since they highlight inefficiencies and wasted time, gaming them will increase efficiency and reduce waste.
How much do DORA Metrics Cost?
Long story short, DORA metrics used to require thousands of dollars from dev teams' monthly budget, but can now be obtained for free.
Since their inception, DORA metrics have required engineering managers to pay various SaaS companies to generate them in real-time or deliver them on a scheduled basis. Notable stories include early metrics companies actually printing out PDFs of their customers' DORA metrics and sending them over after they were calculated.
As measuring and integration technology has improved, the costs of actually gather the data required to produce DORA metrics have drastically decreased. Due to this reduction in overhead costs to produce them, DORA metrics can now be obtained for free, regardless of company size or number or size of dev teams.
How to measure and improve DORA metrics
With the right software delivery management platform, DORA metrics can be tracked easily. Popular metrics providers usually offer DORA metrics dashboards right out of the box that can be easily displayed and tracked.
A dashboard like this could be useful by giving senior members of your software development organization a higher-level view of the DORA metrics for the organization. With this simple view, leaders can see at a glance how the team is doing and what mid-course corrections might need to be made.
In addition to the actual DORA metrics themselves, LinearB can track other metrics that can help improve your organization’s performance.
Leading indicators like pull request size, pull request review depth, and pull request review time can all be monitored and when improved, will reduce mean lead time for changes and deployment frequency.
The first 3 steps in getting your DORA metrics
- Start tracking your DORA metrics with a dashboard & identify your biggest bottlenecks
- Set goals to improve upon those bottlenecks. We find that most organizations struggle with pickup and code review times, and can find the most immediate impact here
- Implement a culture of continuous improvement (LINK TO CONTINUOUS MERGE GUIDE TO DORA - COMING MONDAY)
Discover your DORA metrics in minutes, for free
DORA metrics are based on years of research into what really matters for software development teams. Focusing on them will result in more value being delivered through your development pipeline. More value means happier customers.
LinearB can help your team track them consistently and bring about a profound and lasting impact on your software delivery process and your business.