When a KPI is used, its impact goes beyond the intention of monitoring objectives and achieving them. The measurements made influence the behavior of people and the results of other related metrics. These influences can be unexpected and undesirable, so they must be mitigated or another measure chosen.
Doing things is not the goal. The goal should be to achieve things
When designing a KPI, it is relevant to think about the possible unintended consequences of its measurement. If not, there is a risk that the KPI will do more harm than good.
Thinking about the unintended consequences of a potential measure is a key step before choosing the final mediation criteria for the objective pursued. In general, aspects in which the measure could be a real and distorting problem are taken into account:
- Could the measure be threatening, make people feel judged, and create fear and defensiveness that might cause them to avoid or ridicule the measure?
- Could the move be trivialized, giving people a tunnel vision approach with too little of the business outcome?
- Could the measure sabotage, where actions aimed at improving it inadvertently force other areas of performance to suffer?
- Could the measure confuse, so that people could implement or interpret it incorrectly without knowing it?
Bad KPI choices
Example 1.- Railway timetables
The KPI was the percentage of trains that were on schedule, and the unintended consequence was the cancellation of train services that were running “too” late.
Canceling the trains most behind schedule kept the KPI in good standing. But at a cost. Customers were much more unhappy if their trip home was canceled than if it was delayed. The irony was that execution was measured by the satisfaction score of customers who arrived on time. But it was driving the result in the wrong direction.
Example 2.- Pest control
When the British ruled India, Delhi bureaucrats became concerned about the proliferation of cobras in the city. To control the problem, the authorities offered a reward for the cobra skins. This economic incentive worked well, too well.
Soon cobras were being killed at will, and the government was satisfied with the metric that was based on the reward for the delivery of cobra skins. However, several enterprising Indians saw the opportunity. These opportunists began to breed cobras for their skins.
It wasn't long before the British had warehouses saturated with cobra skins. When officials discovered the scam, they withdrew the purchase option. But that is not the end.
With the bounty program cancelled, countless cobra breeders in Delhi were left stuck with "excess inventory". The bubble herpetological it had exploded and his former cobras had become lethal liabilities. So the breeders released their vipers. And once again, Delhi had a cobra problem, but this time worse than before.
The same thing happened during the French occupation of Vietnam. Rats plagued Hanoi. The authorities offered a reward for each rat tail. Result: the citizens began to breed rats to trade for their tails. When the bounty was withdrawn, the rats were released into the city.
Fort Benning, Georgia was having a boar problem. The Army offered hunters a $40 bounty for each pig tail turned in. People began buying pig tails from butchers and slaughterhouses at "wholesale" prices, then "reselling" them to the Army at a higher price.
Example 3.- Wells Fargo
Wells Fargo set unrealistic sales goals for bank employees and exerted enormous pressure to meet those goals. Result: Rampant opening of unauthorized accounts by employees to meet the goal and keep their jobs.
A bad KPI can generate behaviors contrary to what you want to measure. In adverse selection, the buyer in a transaction has more information than the seller and uses that knowledge to gain an unfair advantage. For example, a person whose car is out of warranty but needs repairs can purchase an extended warranty before taking the car to the shop. In these cases, it is the seller who has more knowledge than the buyer and uses it to take unfair advantage.
It's not always bad to be wrong
Not all unintended consequences are harmful, as in the examples above. Some KPIs can produce unexpected benefits. Like discovering that one measure has unexpected leverage to improve other business outcomes. Or that it has a predictive power that makes it a powerful indicator of the future for a better business result. Or that it drives more behavior than is needed or desired.
It does not necessarily mean that the KPI should die.
Checking for unintended consequences of a measure provides an opportunity to do something before it's too late. It might be decided not to measure it, but only as a last resort, if none of these mitigation methods work:
- When the measure shows a threat, can you focus people on collaborating to improve a business process, instead of worrying about being judged?
- When the measure is trivialized, can it be accompanied by one or two complementary measures that, together, give a more complete picture of the result?
- When the measure fails, can other measures be monitored alongside it and set their targets to find balance rather than conflict?
- Where the measure is confusing, can you simply clarify what it means or provide more background on its wording?
Where the risks of a measure are serious enough, it is possible that the measure may also be given a trial period. It would be wise to monitor unintended consequences to find out how bad they really are and how realistic it can be to mitigate them.
Summary
When considering using a KPI that you're not sure is driving the result you want, you can use some of the approaches in this article to look for potential unintended consequences. Where it can occur, evidence must be gathered to prove its prevalence relative to what you want to prove with the measurement.
When a measure becomes a goal, it ceases to be a good measure.
marilyn strathern, anthropologist
In the talks and documents of ronny kohavi Techniques for the correct selection, measurement and analysis of metrics can be seen. A good talk to start would be the CXL Live 2016 «A/B Testing Pitfalls: Getting Numbers You Can Trust is Hard«.
There is also more information in the TRPlane.com article on the OKR misuse.
