This is an initial commentary and an explanation about what it’s all about.

The Longer You Look, The More Likely Error will Find You

Take a very large dataset, say, all the customers of AT&T and their calling records 2001-2011, and divide it into to two random but equal sets. Say you didn’t have any hypothesis at all. You just wanted to see what was related to each other in that set. Say, each customer record has 5000 features, including gender, date of birth, credit score, average call durations, most frequently dialed number, and so on. (Note to statisticians: Assume a Pearson R correlation matrix, skip next paragraph).

Assume, further, that you’re going to compare each feature against one another. So, you compared all the ages against all the date of births. And then all the ages against credit scores, and so on. And, the strength of the relationship between those two features was expressed by a single number. The higher that number is, the stronger the relationship between the two. For instance, we might find that credit score and age are tightly correlated – the older one is, the more likely their credit score is to be positive.

You’re likely to find clearly incorrect relationships in such a large table, just by accident. You might find that in Dataset A, for instance, that’s there’s a statistically significant relationship between being a Virgo and having a negative credit score. There might be a relationship between average call duration and being a Capricorn. You know that such a result doesn’t make sense. Why would zodiac sign (derived from date of birth) affect those things? The way that chance works in such large tables is that the longer you look for significant features, the more likely it is that you’ll find a relationship that doesn’t in fact hold in the real world.

In fact, most of those relationships would disappear in Dataset B. However, new, clearly untrue relationships would appear in Dataset B that don’t exist in Dataset A. When you’re dealing with thousands of features, the likelyhood of such phenomenon increases. And that’s even holding everything we know about probability to be true.

In sum, a big reason why you go into a dataset with a hypothesis is to reduce the risk of coming up with something that is wrong, and very unlikely to be repeatable in other datasets.

Linear, Cubic, Exponential, Parabolic, Elipse

Not all relationships are straight lines. Indeed, especially in certain types of logistic regression, we can get very amazing, very beautiful and complex shapes separating one case from another. Diaper usage plotted against age is a parabolic relationship. Think about it. You use a lot of them when you’re young, you go through a lot of them when you’re very old. You don’t need too many of them in early to late age. Linear regression wouldn’t perform very well in detecting that pattern.

Enter Reshef et al and MIC

MIC stands for Maximal Information Coefficient. Reshef et al invented a neat way of looking at relationships between variables that doesn’t rely solely on a key statistical test (Pearson R) to indicate that it’s there. The authors demonstrated how MIC manages to detect correlations between all these complex relationship types – Cubic, Exponential, Sinusoidal – and does it really well. The went further. The created a program that can mine very large datasets and suggest relationships to examine.

What’s the Problem?

Remember that the longer you look, the more likely you’ll find something false, idea? The entire idea of hypothesis testing as the basis of quantitative analysis is an entrenched one. It’s an idea that causes resistance to advanced machine learning algorithms and pattern discovery. Reshef really did a great job in explaining the purpose of MIC. Reshef has merely stated that this is a hypothesis informing machine. You can use the program and MIC to discover relationships that were once really quite hidden. Or very, very difficult to discover without insanely expensive software. I think this is great.

The Opportunity

We’re generating huge amounts of data. The big feature big data problem is increasingly common. This is a great tool to rapidly inform hypotheses – to become smarter before getting smarter. It’s a welcome advancement, and worthy of attention.

If you hear of MIC, just know that a MIC of 0.00 means that there is no correlation between two variables, and that a MIC of 1.00 indicates a perfect correlation between two variables. Be aware that MIC does not imply linearity between the variables, but may be of a much higher order function. The second question you should ask upon hearing a MIC score is ‘at what confidence interval is it significant?’, and, ‘what kind of relationship is it?’. Then deep dive.

I’m excited.

It’s also the first get together after eMetrics New York, which was a major, and had big time Canadian attendance. These tend to be among the more interesting evenings. It has also been some three months since the last WAWTO event, so there should be quite a few fresh stories.

Trying to produce something simple out of something complex is…complex.

There are seven axioms that are guiding a lot of my thought in dealing with that complexity:

1. The purpose of analytics is to derive competitive advantage for the organization / firm / entity.

It follows that the purpose of Social Media Measurement is to drive competitive advantage. If the end result isn’t competitive advantage – then it has no value. That unto itself is a value statement.

Simplicity drives competitive advantage because simple is more actionable than complex. I’m often asked questions that have very complex comprehensive answers. I have to sort out that complexity based on relevancy and action-ability. Reality is always so much more complex. And yet, people can’t act on the complexity.

They act on simplicity. And if action is the vital link between the insight/competitive advantage gap – then this mandates a simplified approach.

2. Data alone does not yield competitive advantage.

A major brand might be mentioned 2.5 million times a week on Twitter alone. Having all of that data in a database is of no value if it doesn’t result in competitive advantage.

I’ll go ahead and make a statement: very few people on Earth have the capacity to read and understand what 2.5 million tweets mean on a monthly basis.

3. A sequence of progressive hypothesis testing is the most efficient and effective method to derive competitive advantage from data.

I still hold that the scientific method is the best one we have for learning right now. Someday, somebody will figure out a better algorithm. Until then, the scientific method has this wonderful blend of flexibility, creativity, and evidence.

Progressive hypothesis testing means acting deliberately with marketing messages. The goal might be known – like ‘drive sales’, but the opportunity to message a community becomes all the more useful when, over a sequence of messages, a specific hypothesis is testing. One really basic test might be: “will the community respond more to content about special features instead of content about where our spokesperson is going to be”.

Acting deliberately isn’t always possible, especially in a reactive world, but there’s opportunity to derive learning or insight that can drive the next wave. In social media, the tempo is that much higher. This isn’t 2-year website redesign land.

4. Predicting the future requires an understanding of cause and effect.

At the core of prediction is previous cause and effect. If I touch a hot pan, it will cause my hand to burn. Therefore, I can predict, by touching a hot pan, my hand will burn. Very predictive.

Not everything, especially in marketing, is so clean. At some of the more basic roots – If I spend 500,000 dollars on commercials and run them constantly, I will get 11 GRP. If I get 11 GRP, I’ll move 25,000 toasters.

Statisticians, or Social Science Statisticians, are so incredibly jaded by such simple linear models. Sure, you might get 11 GRP’s, but not all GRP’s are made the same. Moreover, what type of commercial are you going to run? Will it resonate with those who are already looking for a toaster? Will it cause people to suddenly desire a toaster who do not have one? Will it cause people who want to judge others to go out and buy the toaster so they can have a plank to judge? Will it cause people who already have a perfectly good toaster to want to buy, and remember, that toaster – five years down the line to buy that brand?

So frequently, especially when a cause-and-effect model doesn’t jive in our own minds, will we go out and try to discredit other models by introducing other factors that we ourselves deem salient to the situation.

In the end, it comes down to R Square. The percentage of the variation our model predicts the outcome of a variable we care about. A big reason why I rattle on about the importance of goals and KPI’s is because we can anticipate a world where everybody will care about the R Square.

This is especially true in Social Media Measurement. Many people speak of things ‘going viral’. Yet, how many people have truly explored the causes of going viral? There are multiple causes of why something goes viral.

Predicting anything comes from cause and effect.

5. Correlation is not always Causality.

Even a high R Square doesn’t guarantee truth. There might be a great correlation between affinity for John Cena and a love of peanut butter – but I’d be hard pressed to derive a clean causal link between the two. (Perhaps John Cena’s fan base is concentrated in regions where peanut butter is given to young children early?). Unlikely.

Correlation is useful, but without overarching respect for your own theory and your own mental models – it’s dangerous.

This is especially true in Social Media Measurement – where correlations abound – but causality can be fleeting.

6. Accuracy over Precision.

Would you take a thermometer that is right 95% of the time and you were fairly sure that it was always off by 5 degrees, or would you take a thermometer that is right 50% of the time and you were fairly sure that it was always off by just 0.01 degree?

In Social Media Measurement you can have it both ways!

7. It is possible for there to be two optimal, equally true, answers to a problem. (And Sometimes More!) (X^2 = 4, x=-2, 2).

If there are two equally true answers to a problem, surely there could be millions of wrong ones. I’m certain that will make certain people happy to hear.

In Social Media Measurement, it is perfectly possible for two solutions to be both equally right.

A specific instance would be the sentence:

“The boy crossed the busy road carefully.”

I’ll ask you: What was that sentence about? I can see a situation where one of you says, “The boy” and another person says “The road”.

Well, in my view – they’re equally true.

There are multiple right answers. There are multiple wrong ones too.

Simplexity.

The quest for simplicity is complex.

Simplification involves obliteration. It’s possible to take a column of 300,000,000,000 numbers, a massive amount of information, and summarize them into a single figure. In fact, there several numbers that can describe the central tendency of all that information: mean, median, mode. We have a number that describes dispersion of that data: standard deviation. We have numbers that describe the peakyness: kurtosis.

What should get obliterated in the quest for simplicity?

Going back to Axiom 1, variables that do not matter to competitive advantage should be obliterated. Going to Axiom 4, you need to identify the variables that cause a desired effect, in particular, looking for reinforcing effects, all the while knowing that Axiom 5 applies (your theory of how the world works could be wrong even if mathematically it works) and Axiom 7 – it’s perfectly possible for two models to be equally right.

It all comes down to an acknowledgment that Axiom 2 is right: data alone isn’t going to yield competitive advantage, and Axiom 3 is the best way to turn that data into insights that drive competitive advantage – a sequence of progressive hypothesis testing.

I don’t believe we’ve even begun at the beginning yet: what is salient in social media measurement?

We’ll need to get all of those on the table before we can talk about causality, reinforcing effects, and come out to a resolution. I’m pessimistic that there will be a single resolution that will suit everybody: but there is probably a solution that will satisfy 90% of the situations.

What say you?

“Let’s say my shoe is loose. So I decide to bend down and tie it really tighter, inadvertently creating a knot. Let’s say the next morning, I have a hard time getting my shoe on – for let’s say, four minutes. Then let’s say that I miss my bus by just one minute. And the bus has a frequency of thirty minutes. Well then – one seemingly unrelated decision, made 16 hours before and taking all of 2 minutes to execute, has a 30 minute tardiness impact 16 hours later. That’s pretty much like the Butterfly Effect. Writ Small. And Mundane. Without bad acting.”

The Star Trek: TNG way of saying it would be “There’s a cascade failure in the warp core”. But enough of the Laforging.

Cause and Effect dynamics are devilish. After all, my lateness could have been chalked up to not being ten minutes early as I normally am. Or it could be chalked up to the bus being on time, which is unusual. I like to think of the world as a whole bunch of cones converging on a single point. Taken from this point of view, there are as many explanations for something happening as there are people. We all have our perception and are all entitled to own opinions. Though, we’re not entitled to our own facts. (wink).

It’s just a matter of which model has the greatest predictive strength. Normally I’d head down the rabbit hole into a bias about multiple regression…but no. This isn’t going to be a statistical rant. No. I have something far funner to read. (I hope).

And of what implications for the social systems we create?

Twitter is an excellent laboratory to study for that.

And that’s where we’re going to get into a lot of trouble with each other, as social media scientists.

‘How one seemingly innocuous tweet could cause a cascade failure in the warp core?’ will be one of those great analyses someday. And it will be contested. Loudly. By very educated and sinecure analysts.

It won’t necessarily because they won’t accept that little things can make such big impacts. I’ll be referring them on back to this post at that point. And surely, every very educated analyst should be familiar, and indeed, should have experienced such dynamics in their own lives so as to be able to relate. The Butterfly is in the Sky.

Rather, the debate might be how much causality to attribute to the originating tweet, and how much causality to attribute to the reinforcing effects. And indeed, this sub-branch of analytics, of reinforcement-attribution theory, is still very young in marketing science literature. (I salute those of you who have made contributions. It’s just that I wish we had a unified language to describe it.). Someday I’d like to be able to say: “Take a look. It’s in a book.”

How do we understand cause, intervening variables, and effect – and how much we decide to respect where each other is coming from, is by and large going to paint future debates. I’m optimistic that there will exist one school of social media measurement practitioners that will rely on evidence to make assessments. And I’d like to be in that school. I’m certain that we can go twice as high.

There was a little theme running throughout the post.

That’s how little things can make big impacts. And how something little will make something big.

And it’s gone well.

I think looking for simple and remembering the end goal are two key ingredients. Backcasting happens a lot. Expecting exogenous shocks instead of being all outraged when they happen is another.

That’s all that’s really on the mind.

That and how much code I have left to write.

The Coles notes of that compact volume is:

“Simplicity is about subtracting the obvious and adding the meaningful”.

Wonderful.

Thanks Malcolm!

Sometimes there’s competitive advantage in Ease. In easyness. In simplicity.

A good example is the difference between Windows and the original command line DOS. Another is between Mac OX and Windows.

Simplicity isn’t easy though.

It’s far easier to pound out a five page brief than it is to write two paragraphs communicating the same thing. It’s hard to get right. But most of the time, you only have two paragraphs.

So you see, simplicity can be incredibly complex.

I’m kind of intoxicated by this relationship between simplicity and complexity.