What you need to know about automated statistical analysis

Here’s what you need to know about automated statistical analysis:

1. Automated statistical analysis is not a substitute for good judgement

Statistical tests are tools.

They help us understand why nature is the way that it is.

Nature resists being known about. But, she is knowable.

Statistical tests themselves are part of nature. The tests themselves were never meant to be substitutes for good judgement. That belief, that tests could replace people, has only ended up causing the accumulation of some pretty outrageous assumptions over the years.

Just because there is a significant correlation between Magnum Ice Cream sales and Piracy in the Indian Ocean doesn’t mean that it’s causal. Statements of causality require judgement.

Automated statistical analysis is not automated judgement.

2. Automated statistical analysis will not, in itself, cause those with poor judgement to have better judgement

Did the inventor of the hammer ever intend for people to hit themselves in the eye with it?

Tools can make those with great judgement even better.

Tools, in the hands of those with poor judgement, can be really dangerous.

People can learn. And, on this point. I believe that anybody, no matter how arrogant, pants-on-head stupid, can learn how to reason and construct sound arguments. People, if they learn from their mistakes, can form better judgement over time.

But automated statistical analysis, as a tool all by itself, without the accompanying heuristics or critical thinking skills, will not cause better judgement.

3. Automated statistical analysis will not force those with poor judgement to write out hypotheses in advance of exploring the summary

You’re supposed to write out your hypotheses before executing statistical analysis on a dataset.

How else are you going to keep track of what you think you know?

How else are you going to stay focused on rejecting hypotheses as you go through the list?

Automated statistical analysis makes it faster to run the tests we check the most; but it does not force the user to state how they’re thinking.

4. Automated statistical analysis can serve as a litmus test for sorting impostors

A recurrent theme in most talks with analytical leadership is that imposters, those that are terrific at faking it, scapegoat the data.

That is to say, if they demonstrate bad judgement, they can state that the bad data made them do it.

A mark of good management is imposter detection given short performance track records.

However, if given an automated statistical analysis, an individual periodically makes contradictory decisions without a sound justification, you can tell that this is a person that just doesn’t have good judgement.

There’s a lot that automated statistical analysis can do. There’s a lot more than it can’t do.