The impulse for novelty and discovery

Some people want just one number.

Some people want all the numbers.

For best results, seek balance.

One Number

It is very possible to summarize the performance of a business or an organization in a single number. There are two main ways to do so.

One is selection.

One is indexing.

In selection, you pick the most important metric, and you focus on it. It requires discipline and comes at the cost of myopia.

In indexing, you pick the most salient metrics and you combine them into a single number. It requires no discipline and comes at the cost of boiling the ocean to the point that all the rocks bleed their salts into the atmosphere.

When it comes to testing or anything that you’re trying to optimize in machine intelligence, you typically go with selection.

Some new vocabulary I learned just this summer was to use the words “<variable> under the constraint of <that-thing-they’re-worried-about>”. I road tested this phrase in August and September and it performs well. So you can say, “We should optimize for frequency under the constraint of customer acquisition cost” and you can induce somebody to focus on frequency, even though they’ll continue to worry about customer acquisition cost.

When it comes to indexing, you can toss anything you want into it, set a given time period to 100, and run it. This has proven to be effective in generating a list of factors.

Consider the rightfully mocked Klout Score. I’ve dismissed it. You’ve dismissed it. But for fast thinkers who simply want to focus on something else, it’s for them. It doesn’t make them think and it makes the feel security. There is a market for that kind of thing. One technically doesn’t even have to challenge anybody on why they find any of the variables salient.

You can avoid all the hard conversations with an index.

And in many cultures, that’s great.

All The Numbers

Sometimes people want all the numbers.

There are a number of reasons why.

At one extreme, consider the data militant. I’ll often ask for the entirety of the .sav file. Not because I don’t value the researcher, but because it’s just better for me to execute the 300 or so tests over the course of a weekend to understand the full breadth of the set. If I need an architects’ stamp on the drawings, I’ll include my code along with a note in the Monday email with an ask for a peer review. It’s not a question of trust. It’s a question of efficiency and it’s a question of competitive advantage.

At the other extreme, consider the malicious imposter. They’ll often obfuscate the real motivation for asking a question because if the researcher catches onto their bias, a whole bunch of inconvenient facts will appear. They behave that way because they’ve been trained to behave that way. They’ll ask for 120 metrics every which way because they truly just want one column of data that proves the business case they want to justify. And worse, they’ll ask for all 120 metrics to be reported constantly, forever, just to hide the real column. Just to conceal their secret sauce.

In the middle, where the majority of people are, you have the well meaning data civilian. They’ll ask for all the data because they’ve been trained to, and when all the cleaned data isn’t enough, they’ll ask for more. They crave certainty for their judgement. They want backup. They want to be sure. And if you don’t manage them, they’ll dither into paralysis. Like a hiker with hypothermia, they’ll strip naked and rub themselves with snow. They’ll die a heat death  looking for a competitive advantage.

I haven’t done a great job with median-majority. I don’t believe that people should really feel completely comfortable with a decision they’re making because if they do, it’s a sign that it’s too late and most of the advantage is gone. I don’t believe that data is a substitute for fantastic judgement, but it can create better judgement over time. Fundamentally, it’s only the feeling of security. There’s a market for security theater, and it makes this population very easy to manipulate.

Sadly, you got vendors running around spouting that a business only uses 10% or 1% or 2% or some fraction of the data they collect. It’s an effective sales line because it feeds into the median managers paranoia. It’s designed to play on that insecurity.

Be wary of it.

For Best Results, Balance

Your number one tool as a leader is your judgement.

You have a stance, which you defined for yourself. You got beliefs. You got habits. You have these rules of thumbs. You got stress responses – some way worse than others. You got goals.

You’re really about a kilogram of cholesterol riding a monkey. Most of that brain isn’t entirely under your direct command. The chemistry and network up there has much more influence than what is comfortable for us to accept.

You use your judgement to make decisions.

This is episodic. When you’re in the process of making a decision, it’s through the prism of your judgement. Data comes in. It’s transformed in that head. And a choice is made. That data can include the strategy, the culture, the literature, and all the other sources that go in.

That stream of data exists under the constraint of time.

The 40-70 rule is a great tool for episodic decision management.

If you make a decision before 40% of the data is in, it’s too early. If you wait until 70% is in, it’s too late.

That’s pretty balanced for episodic decisions.

What about improving judgement itself?

If you’re reading this now, chances are you’re big on continuous improvement. You’d have to be to choke down a post like this. Your judgement can always be better, and you can always use better judgement to make consistently better judgements.

The relationship between the data, your experience, and the narrative isn’t clean. There isn’t a neat little heuristic like the 40/70 rule there.

We draw a lot of data from the memories of our experiences.

Your experience is fuzzy (Olsen 2007). You may remember a particular project that you’re really proud of. You got some data. You put it through your judgement. You made choices. You experienced success, or what you defined as success at the time. And then you remember the whole thing as successful. Maybe you got bad data, had bad judgement, made bad choices, but circumstance created a good outcome. There are many columns in that table, so it’s hard for people to produce an accurate truth table about a circumstance.

We can misremember experiences and circumstances that make it quite a bit harder to form better judgement over time. The objective reality for the failure of a particular product, product, or algorithm may be a function of a dozen different factors. How you experienced time, what you remember, and how you remember it, will vary.

Often, the impulse for novel data, and discovery about why a given situation occurred, is spawned by the desire to form better judgements on the next iteration. This is especially important when examining the wreckage of a failure. You’ll want to examine quite a few factors, understand your role in them, the role of others, and then see it with a clearer, cleaner, lens.

Conclusion: Balance

Balance.

Demand only the data you need to justify your stance and suffer the risk.

Demand more data than you need to feel too good and suffer the lost opportunity.

Remember things too fondly and suffer delusions.

Remember things too harshly and suffer cynicism.

Focus on a single measure and miss everything.

Examine too many measures and miss everything.

For best results, balance the impulse for novelty and discovery.