If you’ve adopted some of the technical, cultural, and process capabilities of DevOps, you may wonder whether your hard work is paying off. Building a deployment pipeline and learning the associated DevOps tools and techniques requires time and effort. You need to know whether the changes are worthwhile. You can do this with metrics.
To measure your work system, it needs to have the properties of observability. In Observability Engineering Charity Majors, Liz Fong-Jones, and George Miranda define observability as:
…a measure of how well internal states of a system can be inferred from knowledge of its external outputs.
If a system is observable, you can gauge its health from its outputs. You’re already putting this into practice if you’re monitoring your software. You need to do the same for your process so you can see if it’s healthy and generate improvement ideas.
Designing good metrics is difficult. Your DevOps system consists of individuals in teams creating value, which Jim Benson calls The Collaboration Equation. Trying to measure a complex system with a single metric causes people to focus on one area to the detriment of many others. You need to create a balanced set of measurements that meaningfully reflects the state of the system.
Thankfully, you don’t need to start from scratch. You can measure software delivery and operational performance using the DORA metrics and the SPACE framework will help you design metrics to narrow or broaden the system under measurement.
The goal of your DevOps metrics is to measure and improve the performance of the whole system. The measurement process should help you understand and improve real outcomes for teams and organizations. You should use metrics alongside the DevOps capability model to drive your continuous improvement process.
A healthy measurement system
A perfect measurement system would quickly tell you whether a change is making a positive long-term impact on performance with no side effects. In practice, no such system exists. It takes time for changes to be reflected in meaningful measures and unintended consequences are hard to avoid. When you measure a characteristic in your organization, it sends a signal to people about what is important. This changes individual behavior and usually has side effects.
It is also difficult to establish whether one specific change is responsible for a performance improvement when so many other variables are changing at the same time. There is also the difficulty in establishing whether a performance change is positive or negative over the long term, which may differ from the impact it has over a shorter timescale.
The individuals and interactions involved in building software creates a complex and dynamic system, so it’s one of the most difficult things you can attempt to measure. Your best efforts are needed to operate metrics that are healthy, not harmful.
You need to:
- Collect and use the data for its intended purpose
- Balance the need for fast feedback with the importance of tracking real outcomes
- Constantly adjust to keep it healthy
Intended purpose
For DevOps, the metrics help teams understand their performance relative to the industry. This helps them identify areas to improve in their process and deployment pipelines.
If the metrics are used for individual objectives or performance reviews, they can become toxic. When misused, the metrics exhibit unintended and often damaging side effects. For example, if you have been given an objective of reducing lead times, you might be tempted to achieve the goal by skipping critical steps in your deployment pipeline, such as testing.
Tell me how you will measure me, and then I will tell you how I will behave. If you measure me in an illogical way, don’t complain about illogical behavior. - Eli Goldratt (in Critical Chain, 1997)
Be clear about the intended purpose of your metrics and watch out for misuse within the team or in the wider organization.
Fast feedback and real outcomes
Leading indicators help you predict future performance. You can use their early signals to predict changes in performance and take action to adjust the future outcome. Lagging indicators tell you what happened in the past, but rather than being predictive, they’re typically factual.
You can combine both. The leading indicators give you fast feedback and the lagging indicators confirm that the positive signals from the leading indicators result in real business outcomes.
In the DevOps structural equation model, software delivery performance predicts organizational outcomes as long as there is sufficient operational performance. This probably holds for your organization, so you can use software delivery and operational performance as your leading indicator.
Whatever leading indicators you select, the later but factual outcome measures are an important check that makes sure you are getting a real benefit to all the software delivery improvements.
Keeping metrics healthy
Wherever possible, let teams collect measurements for their own use. By limiting the extent of publication for the metrics, you’ll have more control over how they are used. The metrics should help you to identify improvement opportunities, not be used to compare teams or rate individual performance.
As a team makes progress, internal and external customers will notice the improvement without needing to know the numbers.
Our white paper on measuring Continuous Delivery and DevOps provides general guidance on metric design and explains types and levels of measurement. You can also learn more about the common mistakes in DevOps metrics on our blog.
In Out of the Crisis, William Edwards Deming warned of the dangers of depending entirely on numbers to manage work.
Management by use only of visible figures, with little or no consideration of figures that are unknown or unknowable.
You shouldn’t rule out metrics that are hard to get or include metrics just because your existing tools make them easy to find. Select measurements that give you a real indication of progress and work out how to report on them.
It takes more than numbers to be successful in DevOps, so as well as a healthy set of metrics you’ll need to have a strong sense of purpose for your organization and its products and services. You’ll also need to look beyond short-term outcomes in your search for success over the longer term.
Metric overload
You may be overwhelmed by all the different frameworks and measurements. Don’t be. You don’t need to measure everything all the time. Use your situational awareness to apply the right metrics at the right time.
In most cases, you should be collecting measurements at the team level, for use within the team. This keeps the set of metrics manageable and easy to change when they aren’t useful.
DORA metrics
Developed by DORA (DevOps Research and Assessment), the DORA metrics balance competing software delivery concerns by measuring throughput and stability. Operational measures help you to unlock the potential benefit of software delivery on organizational outcomes.
Learn more about DORA metrics.
The SPACE framework
The SPACE framework provides a structure that encourages good metric design. By combining levels of measurement across a range of categories, you can create a robust measurement strategy for your organization.
Learn more about the SPACE framework.
MONK metrics
If you have a platform engineering team, the MONK metrics will help you track the effectiveness of the internal developer platform.
Learn more about MONK metrics.
Developer experience metrics
One of the more challenging things to measure is developer productivity. The developer experience (DevEx) metrics provide a 3-dimensional system for measuring teams.
Learn more about DevEx metrics
Continuous Delivery statements
You can use the Continuous Delivery statements to perform a qualitative assessment of your software delivery performance. Although you can’t display the results in a graph, this is a powerful way to check your capability and explore areas to improve.
Learn more about Continuous Delivery statements
Further reading
These resources will help you find out more about DevOps metrics:
- Measuring Continuous Delivery and DevOps (white paper)
- Common mistakes in DevOps metrics
- The new DevOps performance clusters
- Out of the Crisis by W. Edwards Deming (1982)
Help us continuously improve
Please let us know if you have any feedback about this page.
