Software Measurement and Performance: ORKs, KPIs, and GQ(i)M
Modern approaches meet classical definitions: Combine OKR, KPI, and GQ(i)M for a complete measurement strategy.
You have probably heard about OKR, KPI, and (hopefully) GQM/GQ(i)M. These are the most popular approaches to clearly communicating your goals, the results you expect to achieve to meet the goals, your key performance indicators (that will tell you how you are doing), and what you should measure to know if you are getting there.
This post is not aiming to be a comprehensive text covering OKR, KPI, and GQM (I’m considering writing about them individually). The goal is to help you connect concepts of these approaches so that you can combine them in a unified approach toward aligning your organizational, tactical, and operational levels by clearly defining your goals, indicators, strategies, and measures.
By doing so, you will be much closer to having your organization speak the same language, knowing what matters when making decisions, and focused on achieving organizational-level results by managing at the operational level.
Approaches Introductions and Key Concepts
One thing to have in mind is that each of these approaches was conceived to be used independently. They focus on organizational needs that others have not explored. Therefore, even though they complement each other, their combination is not straightforward. This is the motivation for this post.
Objectives and Key Results (OKR)
You will see that there’s a very good overlap with GQM, but GQM provides a different framework to do it that will help align your OKR to your KPIs.
Objectives are qualitative, inspirational, and time-bound (typically in a quarter) goals. They set the direction toward where you want to go and will lead the choice of the key results that matter.
Key Results are always quantitative. They quantify the objective and break it into specific measures that will tell you that the objective has been achieved. It doesn’t have to be numeric, but it must be quantifiable, and how you measure it must be clear and unambiguously defined. You can (and should) determine more than one key result for each objective, but not too many. Two to three seem to be a good number.
I’ll refer you to this post to read about software measurement definitions: https://blog.pplupo.com/2022-06-17-Defining-Software-Measures/.
And this one on pitfalls you should avoid when choosing your measures: https://blog.pplupo.com/2022-05-13-Watch-out-for-these-unexpected-negative-outcomes-of-measurement/.
So, here goes a small example: Objective: Improve the user experience with your platform. Key result: Reduce the number of escaped defects from 0.012 per story point to 0.08 per story point.
In my observations, the most challenging part is not really defining the key result but choosing and prioritizing the objective. Because that’s the starting point, there’s an unlimited number of options, and people try to think of what they believe to be important - not necessarily what matters to the organization. This is where the organizational strategy for competitiveness/market leadership can really help limit the search space.
Key Performance Indicators
Before we define what a KPI is, let’s look at what an indicator is. According to the ISO/IEC/IEEE 15939:2017 “Systems and software engineering — Measurement process”, an indicator is “a measure that provides an estimate or evaluation of specified attributes derived from a model with respect to defined information needs.”
In other words, an indicator is a measure that gives insight into how well something is going according to some attributes. In this case, we are talking about indicators that tell us about the performance of something based on some attributes. And this performance is not irrelevant; it’s key to understanding how well our strategy will achieve a goal.
KPIs are just that. Simple indicators. In the OKR, we have defined a “key result”. That key result is definitely something that we would like to consider as a KPI. IT aligns well with improving user experience with our platform. Another good example could be the incident turnaround time (how long does it take to address an incident reported by a user?).
It’s important to notice that an indicator does not have to have a target value, like a specific number to be reached. It must, instead, have acceptable limits or thresholds. For instance, for the incident turnaround time, maybe we want it to be 48h maximum, and we are good with anything below that. Some other indicators may require a minimum threshold or both minimum and maximum. More complex setups may include “danger zones” (it’s close enough to the threshold that you will want to act before you trespass on it).
Also, KPIs should be SMART.
Goal Question (indicator) Metric (GQ(i)M)
The GQ(i)M is based on Basilli’s Goal-Question-Metric approach, adding indicators.
The goal-driven measurement process is based on three precepts:
- Measurement goals are derived from business goals.
- Evolving mental models provide context and focus.
- GQ(I)M translates informal goals into executable measurement structures.
It consists of 10 steps.
- Identify your business goals. Identify what you want to know or learn.
- Identify your subgoals.
- Identify the entities and attributes related to your subgoals.
- Formalize your measurement goals.
- Identify quantifiable questions and the related indicators that you will use to help you achieve your measurement goals.
- Identify the data elements that you will collect to construct the indicators that help answer your questions.
- Define the measures to be used, and make these definitions operational.
- Identify the actions that you will take to implement the measures.
- Prepare a plan for implementing the measures.
The GQ(i)M defines goals following a template:
(1) Purpose: To (characterize, evaluate, predict, motivate, etc.) the (process, product, model, measure, etc.) in order to (understand, assess, manage, engineer, learn, improve, etc.) (2) Perspective: Examine the (cost, effectiveness, correctness, defects, changes, etc.) from the viewpoint of the (developer, manager, user, etc.)
Let’s see an example!
Goal: Purpose: To characterize the development process cost in order to improve it. Perspective: Examine the effect of productivity on the costs from the engineering manager’s perspective.
Questions:
- Which activities are more expensive?
- How productive are people right now?
- How much do the changes in scope affect productivity?
- Are the teams properly structured?
- Can the manual testing effort be reduced?
- How much effort is spent fixing defects?
Indicators:
- List of activities and their average execution costs.
- Number of story points every release/sprint divided by the number of people in the team.
- Is there a correlation between story points per sprint and sprint backlog changes?
- How long do teams have to wait for specific people or other teams to complete their stories/tasks?
- How much time is spent on every release/sprint with manual tests that could be automated?
- How much time is spent on every release/sprint fixing defects from the previous release/sprint divided by the number of story points delivered in that release/sprint?
Metrics (measures):
- List of activities
- Time spent by each person in each activity,
- Cost of each person per hour
- Number of story points per release/sprint
- Number of people in the team.
- Backlog change percentage per sprint
- Time tickets stay on hold.
- Time spent with manual tests that can be automated in every test cycle
- Number of test cycles per sprint/release
- Number of defects inserted in each release/sprint
- Time spent fixing each defect
Combined Approach
As you can see, all three frameworks have overlaps:
Objective (OKR) = Goal (GQ(i)M).
Key results (OKR) = Indicator (GQ(i)M) = Key Performance Indicator (KPI).
Metric (GQ(i)M) = (derived from Key Performance Indicator definition) (KPI).
The main component that doesn’t correlate with another framework is GQ(i)M’s question, which is an excellent link to connect the choice of indicators to the goals.
My suggestion to combine the approaches is to use the following:
- GQ(i)M’s Goal template to define Objectives.
- GQ(i)M’s questions to guide the choice of KPIs.
- KPIs as a replacement for GQ(i)M’s Indicators.
- Based on the KPI’s definition, define measurements as specified in the ISO/IEC/IEEE 15939:2017 “Systems and software engineering — Measurement process.”
I believe this combination will help you define measures consistently, tightly, and aligned with your organization’s goals. It will help your organization to understand why these measures are being collected, how they will be used, and what they should aim to improve.
If you try it out, let me know your results in the comments!
Cheers! :-)
If you like this post, please share it (you can use the buttons in the end of this post). It will help me a lot and keep me motivated to write more. Also, subscribe to get notified of new posts when they come out.