Accelerate:
The "Four Key Metrics"

The Accelerate Book

In the book, Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations, Dr. Nicole Forsgren, Jez Humble and Gene Kim studied what made separated strong performing technology organisations from their less effective counterparts.

The book summarises years of rigorous research from years of State of DevOps Reports, built upon 23,000 datasets from companies all around the world. The organisations studied included start-ups and enterprises, profit and not-for-profit organisations and companies which were born digital alongside those that had to undergo digital transformation.

The Four Key Metrics

The research identified that just Four Key Metrics distinguish the performance of various technology organisations. These "North Star" metrics serve as indicators of overall software engineering health.

These metrics aren't "The Goal" of a business, but organisations which did well against these metrics had higher rates of profitability, market share and customer satisfaction. In other words; they allowed organisations to experiment faster, ship reliably and prevent burnout.

Likewise; these goals aren't "leading indicators" or "local metrics" which tell you whether you need to increase, say, Unit Test coverage or cut build times - they measure the entire engineering health of a team. By analysing these metrics and drilling down into inefficient areas, you can ensure you're constantly optimising things that will improve the end-to-end engineering health on your team, rather than optimising in a local area that will make no improvement.

The Four Key Metrics were as follows:

• Cycle Time (Change Lead Time) - Time to implement, test, and deliver code for a feature (measured from first commit to deployment)
• Deployment Frequency - Number of deployments in a given duration of time
• Change Failure Rate (CFR) - Percentage of deployments which caused a failure in production
• Mean Time to Recovery (MTTR) - Mean time it takes to restore service after production failure

We have covered the definitions of these metrics, their averages and further information in both the video above and in our Knowledge Base in the article Measure Engineering Efficiency & Developer Experience with Haystack Metrics.

Where's The Evidence?

Rigorous appraisal of these Four Key Metrics (in research led by Dr Nicole Forsgren) has shown that higher performers are 2x more likely to meet their commercial goals (productivity, profitability, market share, number of customers) and their non-commercial goals (quantity of products or services, operating efficiency, customer satisfaction, quality of products or services and achieving organisational or mission goals).

Indeed, companies which do well under these DevOps metrics have a 50% higher market cap growth over 3 years.

The mechanics of how metrics drive organisational performance is also well understood and has been rationalised by industry leaders like Martin Fowler. For example; faster Cycle Time means you can test ideas in production quicker, getting feedback from real-world users. This means your are able to test business ideas in the real-world quicker and gain invaluable user feedback. In addition to this, shipping features faster makes your customers happier and is of lower risk technically.

Drilling Down

Each of these "North Star" metrics is formed of multiple "Leading Indicators" which allow you to drill-down and understand how to optimise a process at a more local level.

Is most the Cycle Time being spent on development time or review time? How much of the review time is being spent on First Response Time, Rework Time and Idle Completion Time?

Once we've drilled as far as we can into our problem area, we start to review the outlier data points. From here you can identify why they are outliers. For example; you may find that Pull Requests with a particularly long First Response Time are those which Haystack flags as having a Big Diff risk factor. This could tell you that your team are being deterred from reviewing particularly large Pull Requests.

This then allows you to have informed conversions and help measure changes to remedy the bottlenecks.

Free Consultation

Haystack Analytics offers the tooling you need to start to measure the Four Key Metrics for DevOps performance, additionally Haystack offers real-time alerts to allow you to spot issues before they materialise.

Unlike many of our competitors, Haystack only uses metrics that are proven to bring benefits to team performance. This means you are safeguarded from introducing snake-oil metrics which risk causing more harm than benefit.

All this means that you can prevent burnout, experiment faster and ship reliably. On average, teams using Haystack are able to ship over 70% faster.

When you apply for a free trial of Haystack, we are currently offering free consultations to help you understand where your bottlenecks are with one of our expert engineering managers.