Marshall Sponder over at the Web Analytics Guru Blog, certainly said what many of us were thinking.

I was thinking about this glitzy presentation that is going around explaining Social Media – I usually find myself getting brain freeze when I look at this kind of stuff because it’s good, but it doesn’t really tell you how to adapt those ideas to your purposes (not that it’s meant to do that) or how to measure the effect of it (Social Media)…I vowed that when I present at SES and XChange next month – I won’t fall into that (The “Glitzy” type of presentation) – I will give information that tells someone how I measure Social Media

Marketing scientists have been struggling with how to measure the effect of word of mouth (WOM) for decades. What’s rather exciting about the Interwebs is that we can finally start aggregating huge amounts of data and start tracing its effect.

One potential experiment leverages the awesome power of JQuery. You tag comments such that they’re revealed only when clicked, and you record, for each unique visitor, which ones they’re hitting.

Assume that they read the comment that they click.

Then hire some five poor souls to ‘code’ the comments by categories – be it ‘positive’, ‘negative’, and so on, using the normal method that content analytics is done for media bias studies. Of course, a lone web analyst could equally code them too.

Wait a month.

You’ll get two sets of data – a set of web analytics data, and a set of content data. Match them up and run the analysis. The degree of impact that different sets of comments has on conversion, all things equal, will return an amount of lift. That gets you a fact based figure.

The perverse thing about the experiment is that it would be doable and valid. Many readers would scoff because I’m not controlling for brand affinity, or any number of tens of thousands possible independent causal variables, but it would derive a fact based figure.

That’s just a single experiment.

And I guess I’d much rather be running experiments that add to our understanding.