To what extent should analytics professionals make normative statements?

Do normative statements cause harm to analytics programs in the long run?

This may be a bit meta because I’m talking about the effect that an activity that analysts do every day has on what gets selected to study.

A normative statement expresses a value judgement.

Consider the following three statements:

• The strawberry campaign contributed to the acquisition of 1000 new customers out of the 10000 acquired last month.

• The strawberry campaign only contributed to the acquisition of 1000 new customers out of the 10000 acquired last month.

• The strawberry campaign failed, only contributing to 10% of new customers acquired last month.

Which is the most normative?

Consider the next three statements:

• The strawberry campaign contributed to the acquisition of 1000 new customers of the 10000 acquired last month, which is 100 more than the average campaign benchmark of 900. The forecast was for 5000 new customers.

• The strawberry campaign contributed to the acquisition of 1000 new customers of the 10000, while beating the campaign benchmark of 900, fell short of the expected 5000 target.

• The strawberry campaign contributed to the acquisition of only 1000 new customers of the 10000, while barely beating the campaign benchmark of 900, it fell well short of the expected 5000 target.

Which is the most normative?

Does introducing a benchmark cause the statement to become normative?

Is selecting a different benchmark effectively the same thing? (Or is that more nefarious?)

Optimization depends on extracting the good decisions that contributed to the success of a program and copying them into the standard operating procedure of the company.

Normative statements about evidence might have a great capacity to do harm.

The strawberry campaign beat the benchmark. It didn’t hit the forecast.

Was that the campaign’s fault? Was a high forecast really a product of a justification effort? Was it fueled by excessive comparatives?

It’s very rare that analysts dissect bad campaigns.

Failure stinks, so unless there’s a reason to wave around rotting meat, it’s promptly buried. This is an important point.

• It’s not as though researchers are proud to publish null results.

• It’s not as though people show up to eMetrics and present failures.

• It’s not as though the pages of adage are filled with failure.

Just bury it and forget it.

Successes are celebrated. They’re analyzed. The champions are hoisted upon the shoulders of giants while Joe Bean Esposito’s “You’re The Best” blares from the speakers. 

Normative statements really color perception. And analysts have quite a bit of influence through the written bullet point to hammer that perception. People are more likely to believe something if it’s repeated three or more times. Three or more times. That is to say, people are more likely to believe a statement if it’s been repeated three or more times.

Perception is a huge factor in making decisions about what to pay attention to. Could normative statements cause a sub-optimal learning paths?

To what extent should we make normative statements?

***

I’m Christopher Berry.
I write at christopherberry.ca