The previous two parts explained what a Key Performance Indicator is, and the cause of KPI Creep.
How do Data Scientists Cope?
Data Scientists are frequently confronted with datasets that contain thousands of variables. If we tried to to understand the relationship of everything against everything using the methods at our disposal, we’d fail.
Data Scientists don’t say, “we want to understand everything”. We know we can’t.
We would fail because:
- There’s too much complexity for a single human to understand.
- There’s no way to tell a coherent story.
- There’s no recommendation that would mean anything to anybody.
The Data Scientist copes by optimizing for a single variable. In every step of their work, they focus on a single optimization objective. Concretely, it’s the most valuable lesson I learned from Andrew Ng.
Effective optimization mandates a single real number to optimize against
The questions isn’t “what is the list of indicators that are thought to be predictive of performance?”
The question is “what performance are you optimizing?”
What kind of question is “what performance are you optimizing?”
- For those of you in mature industries, the answer might be “profit”.
- For those of you in startup mode the answer might be ‘users’.
- For those of you trying to move from being 5th place to being 3rd place in a given sector, the answer might be ‘market share’.
There should be only one (1) optimization objective in any given context. This forms your dependent variable.
The Basis for calling an indicator ‘key’
For an indicator to be useful it must rise to the level of being predictive of the dependent variable / optimization objective. The more predictive it is, the better of an indicator it is.
It’s not enough for a variable to be ‘interesting’, or ‘I heard that it’s important’. It has to be actually predictive of the single optimization objective.
In so doing, if you go far back enough in the model, you can identify levers and actions that are likely to have the biggest impact. This forms the basis of a system of thought that is not rooted in opinion or feeling, but rather, rooted in real marketing science.
By executing enough tests, the model can be updated. This generates an evidence-based approach to updating deliverables and communication. The format of the deliverables – what’s included and what’s excluded – evolves as digital marketing evolves.
Finally, all marketing is subject to constraints. By focusing on a single dependent variable / optimization objective, you can deduce the impact of a given constraint (say, paid spend) on the dependent variable. This is where true optimization comes into play.
It reduces the odds for KPI creep and maintains the utility of the report for a period exceeding 15 months.
The Problem With KPI’s
The problem with the existing 15 KPI-bloating-to 90 KPI method presently deployed is that dooms the analyst from ever focusing enough on a single variable to optimize against. It makes it exceedingly difficult for the analyst to generate incremental recommendations given dozens of opinion-based, and frequently hidden, ideas of cause-and-effect.
If we define what performance is, regardless of how horrible that experience may be, we can generate an evidence-based model on what constitutes a KPI worth inclusion.
The solution exists, and is proven to be quite effective in another field. Maybe you should give it a shot.
Go ahead – what’s your single optimization objective?
This concludes the three part series on Key Performance Indicators and the argument for establishing a single in-context optimization objective.
I’m Christopher Berry.
I tweet about analytics @cjpberry
I write at christopherberry.ca