If you ask ten engineers
how they measure the success of their DevOps strategy you are likely to get ten
different answers. Success is measured differently by different
people, so it’s not unusual to observe these different
perspectives. What is the DevOps value proposition? This blog will
explore a number of things to consider when you try to answer the
question “How do YOU measure the success of DevOps?”
What is the goal of DevOps?
The most important
question you can ask yourself when it comes to measuring the
success of DevOps is what are your goals, and most importantly,
what are the goals of your company? After all, isn’t it the company
that needs to benefit the most from DevOps? Without goals, you have
no direction and therefore will never know if you achieved
anything. The goal for DevOps is simple: Economic Benefit. If the
outcome of implementing DevOps doesn’t increase cash flow, raise
share price, reduce costs, increase profitability, or improve some
other financial measurement then you need to rethink your plans.
Ultimately, you want to identify some metrics or KPIs that support
the attainment of economic
DevOps Metrics and KPIs
Gaining an economic
benefit is not a simple chore, and anyone who has started or runs a
business knows all too well that achieving profitability, or some
other measure of economic benefit, can be a daunting task. To
measure success, you need to establish some baseline metrics and
KPIs before you start implementing any changes. Establishing
baseline metrics and KPIs for your people, your processes, and your
technology is the only way you can empirically state that you
created an economic benefit for your company. Let’s look at the
metrics and the KPIs that can be established in each of these
A company is only as good as its people and only as strong as its
weakest link. Therefore, when it comes to DevOps, you need to
establish how to maximize the throughput of your staff. The
tricky part here is to develop metrics that everyone in the DevOps
team can contribute. You can’t focus on one group in the DevOps
toolchain because you will create bottlenecks that will frustrate
the rest of the team.
Something easy to measure
across all members of the DevOps team is activity. Where are all
the hours that are being logged, or not logged, going? If you ask
most people what they do all day, they really won’t be able to give
you an accurate estimate. Therefore, it makes sense to figure out
how people are spending their time. There are countless tools out
there that can track employee activity. Once you know how
everyone’s time is being spent, then that can become the baseline
for improvement. You may find out that 64% of the development’s
team time is spent fixing bugs, but the industry average is 32%.
Now you know you need to improve that number by 50%. There is a
good chance that instituting this practice can create a Hawthrone Effect. People will change their behavior
and become more productive simply because they are being
One problem to avoid is the “Big Brother” syndrome.
Micro-managing everyone is not the goal of measuring activity, and
that needs to be communicated clearly. Instead, this can be turned
into an incentive program. Since everyone is being measured, you
can institute rewards for employees that hit certain milestones or
achieve personal bests. These successes can be shared with the rest
of the team. It will also help identify areas that need attention
through additional training, mentorship, and coaching. Aside from
measuring activity, you can also monitor attendance, turnover,
response time, and employee satisfaction. These will all assist in
increasing the DevOps team’s productivity
we think about a process we envision all the steps that will be
completed for a given task. Those steps can be measured along two
scales, velocity and quality
. In the case of DevOps, velocity
is the time it takes to complete each step in the code delivery
cycle, and quality measures the rate of defects for each change.
There are many metrics you can measure using these two parameters
and here are twelve to start.
- MLT (Mean Lead Time)
How long does it take for a bit of code to get built, tested and
- DCR (Daily Change Rate)
Number of changes getting committed to mainline and tested per
- MTTE (Mean Time To Environment)
How much time it takes developers/testers to bring up a testing
environment for verifying each delivered change.
- MTTD (Mean Time to Detect)
How much time passes since the original commit of code until the
bug it introduces gets detected.
- MTTR (Mean Time To Resolve)
How much time it takes to resolve an issue after it’s detected
- MTTA (MeanTime To Approve)
How much time it takes to approve and verify a release. (Measured
from the moment all released content has been delivered and until
the release has passed all the defined test and verification
- BFR (Build Failure Rate)
Percentage of failed builds
- DFR (Deployment Failure Rate)
Percentage of failed deployments
- IRFR (Infrastructure-Related Failure
Percentage of build/deployment failures related to infrastructure
- RWR (Rework Rate)
Percentage of tickets being reopened
- ADR (Automated Detection Rate)
Percentage of defects being detected by automated testing
- UWR (Unplanned Work Rate)
Percentage of unplanned issues
By measuring these and other KPIs, you will determine how well
you are performing and can you take action on how to make
Eventually, everything that was accomplished with all your people,
and the processes they followed, end up running on your
infrastructure. If your infrastructure has not been correctly
architected or lacks the security and redundancy your business
demands, you would have wasted your time. Your DevOps value is
lost. There are countless DevOps tools that can measure the
performance of your infrastructure. Here are 11 KPIs
to consider when measuring how well
your technology is working. These KPIs are not meant to be an
exhaustive list, and there are certainly many more that can be
used, depending on primary business requirements.
- Availability (excluding planned downtime) - Percentage of
actual uptime (in hours) of equipment relative to the total numbers
of planned uptime (in hours).
- Percentage of outage due to changes (planned unavailability) -
Percentage of outage (unavailability) due to the implementation of
planned changes, relative to the service hours.
- Percentage of outage due to incidents (unplanned
unavailability) - Percentage of outage (unavailability) due to
incidents in the IT environment, relative to the service
- Percentage of unplanned outage/unavailability due to changes -
Percentage of unplanned outage (unavailability) due to the
implementation of changes into the infrastructure. Unplanned means
that the outage (or part of the outage) was not planned before
implementation of the change.
- Percentage of Service Desk Availability - Calculation of the
Service Desk Availability over the Reporting Period.
- Percentage of availability SLAs met - Percentage of
availability Service Level Agreements (SLAs) met.
- Percentage of (critical) infrastructure components with
automated availability monitoring - Percentage of (critical)
infrastructure components with automated availability
- Percentage of critical business processes not covered by a
defined service availability plan - Percentage of critical business
processes not covered by a defined service availability plan.
- Critical-time failures - Number of failures of IT services
during so-called critical times. The critical time is the time that
a service /must/ be available, for example for financial systems
during the closing of the books (at the end of the month, or end of
- Critical-time outage - Total outage from critical time failures
in IT services. The critical time is the time that a service /must/
be available, for example for financial systems during the closing
of the books (at the end of the month, or end of the quarter).
- Number of business disruptions caused by problems - Number of
business disruptions caused by (operational) problems.
Business Value of DevOps
When we began this
discussion about the business benefit of DevOps we learned that the
ultimate objective is to derive economic benefit. Since that is the
case, we ultimately want to prove that all our metrics and KPIs
enhance the economic benefit in some manner. By adhering to DevOps
best practices, you win by building it better, faster, and cheaper
than your competition. But as you strive for these benefits there
are many other benefits that you gain along the way.
- The ability to fix problems faster
- Reducing the complexity of your environment making everything
easier to manage
- Ability to deliver software on a continuous basis.
The entire team
wins due to the cultural benefits such
- Enhanced engagement between your employees
- Employees are more productive and happier because of increase
- Employees get better at their job and develop professionally
which elevates them in their career.
Overall the business wins because DevOps
- New features faster than ever
- Improved collaboration and
communication between employees and teams
- Operating environments that are more
- An intense focus on innovation because
everyone is spending less time fixing problems.
Let’s focus on the business value of DevOps a little more
deeply. When you organize your business like an “IT Factory”
have a blog post coming out about this in the future so subscribing
to our blog posts will ensure it’s delivered directly to your
mailbox) you are
essentially stating that you are going to automate everything.
Isn’t that why factories exist in the first place? In this IT
Factory, the objective is to turn out product. The faster you do
it, the faster you can obtain revenue. This concept is known as the
velocity of revenue.
When Apple releases a new phone, people line up to buy the new
phone, even though their old one is perfectly fine. Apple enjoys a
revenue spike each time they release hardware. You read about it
every year. Software is no different. On each release of new
software, you can measure the revenue spike. If you release
software once per year, you will get an annual revenue spike. By
employing DevOps automation tools, among other things, you can
release software each month, or more frequently, and generate
smaller, but more frequent revenue spikes. These smaller spikes
accumulate to more than an annual revenue spike. Why? Between
software releases your competition is trying to muscle in on your
territory. The longer the delay, the more time you give your
customers to get dissatisfied with your product and find a product
with the features they want now. Conversely, if your competitor
just released a highly sought after feature you can’t release for
another four months, you just lost four months of potential
The DevOps value formula is simple: Velocity of revenue
increases as software deployment frequency increases. Of course,
quality can never be compromised, or all is lost. If you release
software continuously like Amazon, Google, and Netflix you
essentially eliminate all these revenue spikes altogether because
the best version of your software is always available. Now you are
streaming revenue daily, and the competition will have a hard time
competing with an IT factory as efficient as yours.
In summary, the goal of DevOps is to create economic benefit. To
understand economic benefit , you need to take a baseline
measurement on every key metric that contributes to economic
benefit. Improve these metrics by making everyone accountable and
incentivizing them for success. Measure , Improve, Measure,
Improve. Now you have a finely tuned IT Factory churning our daily
software releases which maximize your velocity of revenue. If you
don’t know how to start FP Complete would be
happy to show you how. We have helped many companies on their
DevOps journey by helping them build their IT Factory, showing them
how to run it themselves, and being there when they need us. Our
DevOps assessment is a quick way
for us to gauge what you need to get started.
If you liked this post you may also like:
Do you like this blog post and need help with DevOps, Rust or functional programming? Contact us.