I’m increasingly disturbed by the accuracy of Topic Bearing Word of Mouth (WOM) algorithms.
A previous study, published in this space, expressed dissatisfaction with standard sentiment analysis. My mind has since turned to the difficulty in expressing massive amounts of WOM into simple metrics that are actionable and decomposable.
So let’s just go beyond the realm of evidence based pre-optimization of marketing messages, and set the entire area of sentiment-bearing word polarity aside for awhile. It’s relevant and important. Just not the focus tonight.
Let’s turn to topic bearing WOM.
Imagine you could listen to the world, and assume that Burke’s reality is now…a reality.
If you haven’t seen the video from my ‘about’ section – here it is again. It literally is what I’m going on about:
How would you be able to make sense of the world? How would you, as a person, listen and understand all of that material? If the world is constantly changing and is what you say it is – just say.
Well indeed. So what are people saying? How do you aggregate all of that information into a format that’s understandable to mere mortals?
How could you possibly? To use a web analytics analogy – it’s akin to reading server log-files manually, one at a time, for want of a log-file reader. Or at least, a log-file reader that you don’t really trust.
The initial reaction is to do what marketing statisticians have been trained to prior to 2004: use sample statistics. I have got to ask: why use sample statistics when you have the whole data mine right there? Isn’t the only reason for sample statistics existing is for want of the database? (And nobody truly knows the overall sample size that they’re trying to project against. In the case of many topics, the n is extremely small. In others, it’s effectively undefined until semweb comes along.)
We have a massive database.
The idea of taking 1000 log files and reading them manually – and then saying that those 1000 log files are representative of the whole isn’t psychologically acceptable to most marketers. That +/- 3.1% sampling error is reinforcing your 15 to 20% interpretation error and you’re looking at a pretty dense ROE. ROE is generally not psychologically acceptable. Shows are canceled on the basis of statistical error for want of understanding to this day (and we’re 80 years into that methodology (consider radio, yup, it goes back that far)). And yet, even if you were to pitch that sampling approach and the ROE was acceptable, that really doesn’t gel because of the expectation of drillability and a broader expectation about the granularity of the data. That drillability expectation is also vital to solving the Integral Problem. If you’re a web analyst reading this, it’s just implicit within your paradigm – the way you’ve been brought up with the data – to expect that you’re able to drill into anything. It’s a bias that’s always been there.
If you’re a digital marketer or a UX strategist – you probably won’t even question that relative availability of incredibly granular data. It’s like a can opener. You just assume it. Take that away and the beans just won’t taste the same.
The big n, the overwhelming amount of data, demands a data mining approach. It demands a machine algorithm. It also demands a statistical methodology that is scalable. This heads into a domain that lies at the intersection of data mining and computability. It’s just awesome. There are many solutions, but very few solutions that will actually produce timely intelligence.
Topic Bearing WOM and the categorization of it should be, on the surface, a much easier nut to crack than sentiment-polarity, which is intensely subjective. But it’s not. If you ask 100 marketers to write a one paragraph summary of a 600 word blog, you’ll get a diversity of opinion about what the blog was actually about. Unanimity on what the topic was is extremely difficult to achieve. Not convinced? Consider the diversity of opinion about what the topic of S.11 of the Canadian Charter of Rights and Freedoms. In fact, this is a very deep problem that has been struggled against for the better part of the last decade. It’s no easier.
In the coming days, many pixels will be spent writing about the categorization of topic bearing word of mouth. There’s just a confluence of news and opinion. We might see a resurgence of opinion-mining and, in an experiment I’m doing on you – the word-of-mouth/social nexus.
So I’ll say this:
People will write. I welcome that.
Many will claim that it’s so simple. It’s not. This 892 word post has been a hike for you.
Awesome minds have been working this problem for at least 31 years, and have been really serious about it for the past six. 100% accuracy is not probable (in your lifetime). Statistical sampling is not a panacea. And even with a unified corpus even the best analysts are going to have a tough time with it. (Though, unified corpus’ are great).
Topic Bearing WOM poses a huge opportunity, and a huge challenge. It should be tackled with same amount of care that we take at Syncapse.
My point stands. I’m dissatisfied with the existing algorithms to summarize topic bearing WOM. And you should be too.