Why Things Don’t Mean What People Assume They Mean in Analytics

I live and work at one of the most amazing intersections. It’s also the cause of why things don’t mean what people assume that they mean.

There are technologists – developers and computer scientists – who grapple with the limitations imposed by API’s and big data.

There are marketing scientists – analysts and statisticians – who grapple with the limitations imposed by computability and understandability.

There are marketers – brand and channel – who grapple with the limitations imposed by budget and cognitive surplus.

It’s pretty amazing how a technologist, a marketing scientist, and a marketer can all be right within their own silo, their own way of thinking, but collectively be misunderstood and wrong. The confluence of all three types of people generates continuous tensions, innovations, stresses and misconceptions.

Just to share one such example, take the concept of a response rate, a conversion rate, and an engagement rate. We’ve had peace, among web analysts, since 2009 on that topic.

And yet,  technologists face the limitation that we cannot access certain measures at a unique individual level. Marketing scientists face that limitation, and have it compounded by natural activity inflation within the numerator. That is to say, new types of engagement are invented every quarter. Engagement rates drag higher not necessarily because the experience is better or the marketing more effective, but because more opportunities exist. Marketers face the limitation of having to understand, optimize, and ultimately prove that they matter. Which is also a potential cause of numerator inflation.

Systems don’t exist because measurement exists. Rather, systems evolve to maximize some other result. It will always be that way. As a result, the limitations enumerated above will always exist. It’s best to understand it and try to make use of it. This causes choices to be made. And like it or not, choices are made.

For instance, the marketing scientist will attempt something new, like some variation on the new LIV algorithm, to control for numerator inflation. In so doing, they’ll make the calculation impossible for a marketer to understand conceptually, or to reproduce at all, and even harder for the technologist to implement. They might try for something far simpler, but in so doing, introduce even greater problems into the equation.

This happens again and again and again. It happened in 1993.

The use of the word ‘people’ in web analytics caused one such problem. Technologists at Netscape invented the HTTP cookie, within the limitations imposed at the time, to understand revisitation rates to the Netscape.com website. Web analysts always meant ‘unique browser cookies’ when grappling with that innovation 2 years later in 1995. It was explained to marketers (and maybe marketers explained it to themselves) as ‘people’. The ‘visit’ and ‘unique visitor’ were massive advances over the term ‘hit’ (still in use by some very, very old people even to this day!), but didn’t get down to that unique 1 cookie : 1 human being relationship.

The superiority of the cookie, and the linkage to direct transactional and research behavior, is a huge step forward for marketers. It’s still very cute when a TV executive asks “why can’t digital people get their measurement together?”. Indeed, the TV executive is fully aware of the problems with ratings. They just gave up in 1954, because, well, TV doesn’t exist solely to measure it.

Things don’t mean what people assume they mean, at least in analytics, is because of the limitations imposed by reality are passed onto people who try to interpret them, and, the lack of time and brain memory space to remember that.

It’s only a problem if somebody makes a bad decision based on an incorrect interpretation of what something means.