The problem with Key Performance Indicators (Part 2)

Yesterday, I defined what a KPI is, and explained the existence of KPI creep.

A List of KPI’s Is NOT a System or a Model

A list of KPI’s, 10 to 15 in a young programme, is not a system or a model.

Take, for instance, the following list for a standard eCommerce shop:

• Net Revenue

• Gross Revenue

• Conversion Rate

• Number of Conversions

• Average Revenue per Checkout

• Number of Checkouts Started

• Checkout Abandonment Ratio

• Number of Carts-Started

• Number of Carts-Abandoned

• Cart Abandonment Ratio

• Average Items Viewed per Visit

• Number of Visits

That list, unto itself, does not constitute a system of thought. If I turned this on its side, and drew arrows between the factors, then yes, it would constitute a system of thought. It would be a model.

It would flow something like this:

“I theorize that if I have a large number of visits, and if I have a really great user experience, I can get them to view many of my wares and my merch. Since my merch is awesome, I can get them to add things to their cart. If I can keep them interested and shopping, eventually, they’ll start checking out. There are a whole bunch of things I can try to keep them from leaving, like free shipping or threshold goodies, that can reduce checkout abandonment and increase revenue per checkout. That’ll increase my overall site conversion rate, resulting in higher gross revenue, and, since I can control my marketing costs, a higher net revenue. To keep my conversion rate high and my net revenue really solid, I have to focus on driving qualified visits to the site.”
 

Well well well. Shouldn’t we also be looking at Bounce Rate? Shouldn’t we also be looking at Time Spent on Site? What about visits to our careers page? Or number of visits to our pinterest page? What about % of traffic originating from mobile? What about % of traffic coming from Google? What about the top ten sections visited? Or the top 5 items added to the cart? Should be looking at promo code abandonment rate as well? What about branding? What about NPS? What about our PageRank? What about the new visit / return visit ratio? What about returning customer retention? Aren’t these ALL salient variables in some way?

Marketers want these factors because they’re predictive of the key performance indicators. And, different marketers, from different backgrounds and different departments have different interpretations of what causes certain effects.

Different people have different ideas about the way marketing and the world works. All of them conflicting opinions, sometimes informed by experience, sometimes informed by evidence, sometimes informed by something that was read in eMarketer or the Official Magazine of American Airlines.

Just to make things worse for a minute – some people are capable of deducing very long chains of cause and effect. For them, a six step model is easy. Other people will deny the existence of intervening variables because it’s just too many links. The general rule (from public policy circles) is that anything but a direct cause-effect link cannot be communicated effectively through standard media approaches.

(And, if you’re using PPT, you’re using a standard media approach).

By the time you factor all of those additional things in, you’re left with a spaghetti report. Sure, there are meatballs. But good luck finding them in an ocean of sauce and noodles. Over time, you’re left with an experience that satisfies nobody because there’s just too much watery sauce.

Weak, Watery, Sauce

There’s no firm line between what constitutes a “Key” Key Performance Indicator, and, what is an intervening variable. The result of course are regular deliverables that balloon in complexity and decay in usefulness.

KPI creep is the result of some natural law that is inherent to collective decision making. Overlapping versions of simplicity rapidly become incredibly complex and unmanageable.

What is to be done? Tomorrow we’ll look at how Data Scientists handle such complexity.

***

I’m Christopher Berry.
I tweet about analytics @cjpberry
I write at christopherberry.ca