The tl;dr summary:

“For Samsung’s 7500 and 8000 series TVs, all you have to do is say “Hi, TV,” when you walk into a room for the TV to turn on and know who’s there.”

“Think of it: The tech means an advertiser or TV programmer could, for the first time, know which members of a Nielsen household are watching a show or an ad. Cisco has even developed a system meant to read facial expressions and determine whether you’re entertained or bored.”

“Many people in the living room are multitasking with other devices. “We’re paying for that,” said Rex Harris, innovations supervisor at SMGX, a unit of ad agency holding company Publicis Groupe. “If you’re looking at other screens, then you’re not paying attention. We would like to know if we’re getting accurate impressions.”"

Commentary:

Alright – so – a simple innovation, the webcam, is jumping from the PC/DVR into a TV, and we get a few folks who come out and speculate what it could mean. It all ends up sounding like a 1984 telescreen idea, which, I’m 99% certain, is not what Samsung has/had in mind.

Broadcast isn’t digital.

Repeat: broadcast. isn’t. digital.

This has implications:

• There is enough inventory for targeted ads and offers in digital because the technology enables the creation of multiple ad treatments at scale. No such technology exists in the broadcast industry.

• People already effectively segment themselves by TV show preference.

• On Demand technologies like Netflix, and time shifting technologies like streaming and DVR’s, are already eroding the concentration of key market segments.

• Plot the S-curve adoption rate of the technologies driving market fragmentation against the adoption of new, Big-Brother enabled telescreens, and see which wins. (Hint: it’s time shifting and on-demand).

• You’re paying for junk impressions because we’re developing ad blindness, just like we’ve developed banner blindness.

No amount of surveillance is going to change that fact.

This is an initial commentary and an explanation about what it’s all about.

The Longer You Look, The More Likely Error will Find You

Take a very large dataset, say, all the customers of AT&T and their calling records 2001-2011, and divide it into to two random but equal sets. Say you didn’t have any hypothesis at all. You just wanted to see what was related to each other in that set. Say, each customer record has 5000 features, including gender, date of birth, credit score, average call durations, most frequently dialed number, and so on. (Note to statisticians: Assume a Pearson R correlation matrix, skip next paragraph).

Assume, further, that you’re going to compare each feature against one another. So, you compared all the ages against all the date of births. And then all the ages against credit scores, and so on. And, the strength of the relationship between those two features was expressed by a single number. The higher that number is, the stronger the relationship between the two. For instance, we might find that credit score and age are tightly correlated – the older one is, the more likely their credit score is to be positive.

You’re likely to find clearly incorrect relationships in such a large table, just by accident. You might find that in Dataset A, for instance, that’s there’s a statistically significant relationship between being a Virgo and having a negative credit score. There might be a relationship between average call duration and being a Capricorn. You know that such a result doesn’t make sense. Why would zodiac sign (derived from date of birth) affect those things? The way that chance works in such large tables is that the longer you look for significant features, the more likely it is that you’ll find a relationship that doesn’t in fact hold in the real world.

In fact, most of those relationships would disappear in Dataset B. However, new, clearly untrue relationships would appear in Dataset B that don’t exist in Dataset A. When you’re dealing with thousands of features, the likelyhood of such phenomenon increases. And that’s even holding everything we know about probability to be true.

In sum, a big reason why you go into a dataset with a hypothesis is to reduce the risk of coming up with something that is wrong, and very unlikely to be repeatable in other datasets.

Linear, Cubic, Exponential, Parabolic, Elipse

Not all relationships are straight lines. Indeed, especially in certain types of logistic regression, we can get very amazing, very beautiful and complex shapes separating one case from another. Diaper usage plotted against age is a parabolic relationship. Think about it. You use a lot of them when you’re young, you go through a lot of them when you’re very old. You don’t need too many of them in early to late age. Linear regression wouldn’t perform very well in detecting that pattern.

Enter Reshef et al and MIC

MIC stands for Maximal Information Coefficient. Reshef et al invented a neat way of looking at relationships between variables that doesn’t rely solely on a key statistical test (Pearson R) to indicate that it’s there. The authors demonstrated how MIC manages to detect correlations between all these complex relationship types – Cubic, Exponential, Sinusoidal – and does it really well. The went further. The created a program that can mine very large datasets and suggest relationships to examine.

What’s the Problem?

Remember that the longer you look, the more likely you’ll find something false, idea? The entire idea of hypothesis testing as the basis of quantitative analysis is an entrenched one. It’s an idea that causes resistance to advanced machine learning algorithms and pattern discovery. Reshef really did a great job in explaining the purpose of MIC. Reshef has merely stated that this is a hypothesis informing machine. You can use the program and MIC to discover relationships that were once really quite hidden. Or very, very difficult to discover without insanely expensive software. I think this is great.

The Opportunity

We’re generating huge amounts of data. The big feature big data problem is increasingly common. This is a great tool to rapidly inform hypotheses – to become smarter before getting smarter. It’s a welcome advancement, and worthy of attention.

If you hear of MIC, just know that a MIC of 0.00 means that there is no correlation between two variables, and that a MIC of 1.00 indicates a perfect correlation between two variables. Be aware that MIC does not imply linearity between the variables, but may be of a much higher order function. The second question you should ask upon hearing a MIC score is ‘at what confidence interval is it significant?’, and, ‘what kind of relationship is it?’. Then deep dive.

I’m excited.

(Note that I’m simplifying this quite a bit to be accessible.)

The first step in predictive analytics is to understand what you’re predicting. We’ll call this the Y variable.

In this instance, ‘how many visits from Boston can I expect on a given day’. My Y will be ‘Visits’.

I’m curious about it.

Have some discipline. I see way too many analysts change the Y variable before their investigation is through.

The second step is to identify all the variables that might be associated with a variation in Y. These might include factors like paid media, search, new visits, returning visits – and date. Then there are paid campaigns, posting new content, social campaigns, traditional media spend, promotions, and so on. Day of the week is another key variable, along with statutory holidays, and extending out to other factors like weather and creativity.

The third step is to extract, transform, and load the data you CAN actually access. You can spend months fighting to build an absolute complete model, or, you CAN start putting together a story with the facts that are available. I chose action over inertia. You should too.

That date field is usually pretty bad to extract, transform, and load. There are functions both in excel and SPSS that handle dates with some difficulty. Devils abound in the details around ‘the date where in the world’. If your installation is set to Eastern Time, and most of your traffic comes from Australia, you’ll be one day lagged. You ought to adjust the figures using the appropriate offset.

The figure below is what I could extract from Google Analytics in about an hour. (Collinearity abounds!)

The fourth step is to run the math against your model.

I use SPSS to run a regression. If you don’t have SPSS, you can try using open source programs like Octave or R. The reason for using software is because it’s annoying to do by hand. I didn’t enjoy a copy of SPSS at my first research position, so I had to code out linear regression in Excel. I learned a lot, but it is not expedient!

The figure below is the output from the software.

The way to read the table is Y = Constant + B1(X1) + B2(X2).

So, Visits = 4.888 – 1.872 (istheweekend).

If it’s the weekend, I can predict Visits = 4.888 – 1.872 (1). Which equals 3 visits.

If it’s not the weekend, I can predict Visits = 4.888 – 1.872(0). Which equals 4.888.

Not bad for Boston traffic! And I understand the impact of a single variable on visits.

My dataset is incredibly spikey. So, what’s causing some of that spikyness? I went through all the dates that I posted new content – reran the math, and got the table below.

The model above is the best. It explains 12.7% of the variance in the set.

The equation is: Visits = 4.496 -1.76(istheweekend) + 2.482(newpost).

I can tell – according to this version of reality – that if I want the maximum bump from Boston, posting during the weekday is best. And I can tell the proportional impact of each variable.

Sometimes this answer is good enough. There are more advanced methods – like curvilinear regression, machine learning, and neural networks. There are ways to introduce more variables into the equation. But typically – this method is sufficient to get a first idea about the relationships among variables and their relative importance, rooted in fact, as opposed to gut bias.

The fifth step is to make decisions based on scenarios.

If you take this equation and plot it out, you can engage in a few what-if’s. Would writing more weekend friendly material result in a lower Beta? Would increasing the frequency of new posts drastically improve the performance of the website? If so, by how much? The size of the newpost beta, as compared to the total number of Boston visits per day hints at that relative strength.

That’s the power of predictive analytics.

“Therefore, when Siri was an independent company, its plan was to map these domains deeply and seamlessly to automate transactions for its users within them. For example, “Buy that Steve Jobs biography book and send it to my dad”; “Send a dozen yellow roses to my wife”; “Book me the usual table for 2 tonight at 8 p.m. at Giovanni’s”; and “Get me 2 box seats for the Giants game on Saturday.”

Then comes the question of what solves our biggest problems. Ultimately, Siri’s value is that of automation and removing “friction” on the Internet. Siri achieves this by: (1) understanding speech input in natural language form, (2) mapping user requests against its knowledge base (i.e., ontological domains) and (3) activating software “agents” to interact with Internet service providers to fulfill user requests.”

Source: TechCrunch

Let’s just forget Google for a minutes and focus in on this combination of technologies.

• Understand.

• Map.

• Act.

That’s the general design pattern for a whole range of applications.

Certainly nothing new here.

They’ve solved a good problem. There are certain use cases for which Siri is a great solution.

He ignores the rest of the problem space. And that’s just fine. I don’t expect him to point out the subset of infinite use cases that Siri is woefully inadequate for.

Barriers, like a small keyboard, are soon to be resolved by virtual keypads and a range of next generation hand gestures that are sensed, not tactically received. I don’t see them as insurmountable.

Even Star Trek TNG made use of both voice and physical commands.

Siri is not a Google-search killer.

It is a nice complement.

The aim of data science should be:

• to make things better

• by influencing people and things to make better decisions,

• by making people and things more aware of better alternatives,

• based on better algorithms and more relevant data.

Language kept intentionally vague to set up the ‘well that could be anything’ argument when it suits me later.

If you do it right, nobody is really aware of the complexity of what just happened to them. The point is not to experience data. The point is to experience…an experience. And be better off for it!

And, the most interesting part is that it’s not really driven by humans with hidden agendas. Though, that could play a part. It’s driven by machines which generate rules that most designers don’t understand fully.

Haven’t heard of Data Science? You’re not alone. It’s only just become a ‘thing’ lately.

The usual fight for the soul of Data Science (the language, identity, ego) has begun in earnest. You can read the editorial summary here. This will go on for the better part of a decade, and frankly, nobody outside of the emerging data science community is really going to care. But it’ll be important to a few. And they’ll make it a big deal, solely because language contains bias about beliefs, and don’t question my damned beliefs, dammit.

I don’t have much of a dog in that fight. I’d much rather get to the good stuff.

Why am I excited and optimistic about the prospects for Data Science?

Never before has so much data about so much meaning so little to so many. The world is filled with waste and genuinely bad things. What if you could make sense of more of it? What would you do then? How much better off would we be?

This is a bit beyond the novelty of Freakanomics.

You may recall a line of reasoning that James Burke once put forward in his series Connections. He argued that we tend to believe that technological advancement causes the world to become better, when, in all reality, every technological advancement has made the environment worse off while making people relatively better off. There’s been a tradeoff. It seems that technological advancement is at odds with sustainability.

But does it have to be?

By becoming more aware of cause and effect as individuals, groups, communities, companies, organizations and societies – can we become better?

It is, after all, not just about tracking the world. It’s about making sense of all that data too. Thinkers like Jeff Jonas have been putting forward ideas about sensemaking for some time, and I take no credit for it. It’s not so much that the data excites me. It’s the opportunity that that data opens up.

I think there’s good reason to believe things can be better.

Picture related. Without meaning, how can you make sense of anything?

***

I’m Christopher Berry.

I bridge the gap between marketing science and data science.

I welcome connections on Twitter and LinkedIn.

A common behavior, well known to clicks-and-bricks retailers, is that consumers will research products before coming in store to buy them. This is especially true of electronics goods, but I suppose it’s conceivable they do it for home appliances, automotive purchases, and anything else that is generally of high consideration. Mobile offers the capability of researching while you’re physically in the store. And, since most stores are now ghost towns, it enables the consumer to help themselves.

Expect more of that in December 2011.

Note the desire for coupons and sale information. People want deals, dammit. It’s not exactly something I’d be pushing if I were a mobile marketer. Why cannibalize my in-store sales? Well – I might think of a way to drive urgency using the device. But I wouldn’t want to throw a “20% off” display ad just because I want proof linking the mobile channel to in-store sales. Certainly, there could be a mechanism. A reward of some type, perhaps.

Finally, there’s that 32% figure that sticks out. ‘Buying products’. It’s 2003 all over again and smartphones are to mobile commerce as broadband was to ecommerce.

The second set of statistics follows below. They used a control group and they’re reporting the differences. It’s suggesting that mobile is more effective at driving a number of brand metrics (not direct attribution metrics like a web analyst might assume). Their reporting on the relative impact of the channel on self-reported attitudinal changes, post exposure.

A summary states:

“According to Dynamic Logic, there are three important factors that drive a successful mobile campaign. They are the location of a brand name or logo within a mobile ad matters: left-side brand placement is generally most effective and has a strong impact on advertising recall; clear and persistent branding is important for brand awareness and a strong call-to-action encourages interactivity and engagement and helps drive purchase intent.” Source.

The take away is not “use mobile to drive awareness”. That is not a good takeaway. Mobile is not a mass awareness channel, no more than paid search is. It’s not the way the channel works and it’s certainly not the way consumers want the channel to be used with them. Do you really want to hit people with a SMS coupon every time they visit Deborah in accounting at the north side of the building? (It’s just within the 200m radius of a Starbucks). That’s the wrong takeaway, even if it is highly likely that awareness is higher. (It better be, there’s less on the screen to look at.)

Mobile, good mobile, forces much more discipline. It demands subtraction. It demands that choices be made. This isn’t a corporate webpage where everything can be added.

There’s more constraint because there’s more constraint.

Finally, there’s Korea. It’s the last piece of evidence I’ll put forward.

The video below explains how Korean marketers are assisting people rescue otherwise wasted time. In this instance, it’s shopping from the subway, using smartphones and codes.

This represents a fairly impressive increase in productivity. Mobile enables consumers to be more productive in their lives by converting what was previously wasted opportunity into rescued time. You’re also resurrecting outdoor display advertising and commanding direct consumer attention AND action. It’s awesome and goes well beyond ‘click this QR code to see our awesome marketing microsite’.

Recall the product adoption lifecycle. Innovators will try things simply because it’s novel. There’s a long chasm. Is that chasm ever brutal. At the other side of it there are early-adopters. Early-adopters will try things because it’s obvious that it will be useful. What we’re seeing here is some evidence that we’re through the chasm, at least when it comes to porting very common digital activities that used to happen on a laptop, over to a mobile device. The grayer area is the role of portable devices (tablets) and that role in driving changes in consumer behavior at mass.

How would you use mobile, not so much to increase awareness (it’s not a mass channel) but to complete the action-purchase portion of the conversion cycle?

It’s also the first get together after eMetrics New York, which was a major, and had big time Canadian attendance. These tend to be among the more interesting evenings. It has also been some three months since the last WAWTO event, so there should be quite a few fresh stories.

The vast majority of it isn’t in its raw form. Its coal. It has potential. But on its own, it has limited uses.

Algorithms are the modern day equivalent to machinery. Fire (combustion) is really just statistical analysis – a violent process that generates waste in the form of heat and soot.

Our modern day Watt Pump is Google. Their coal is HTML. The best coal used to be the HREF link.

The algorithm that drives Google’s primary product is PageRank. It runs on a massive amount of coal. Most people aren’t aware of the complexity that goes on – and why should they. All the mine owners really cared about was removing water from a mine. They knew, at a high level, how steam worked. But they probably couldn’t explain Boyle’s Law. Same with you today. All you really care about is finding the most relevant search result.

The algorithm that drives Facebook’s primary product is GraphRank. It runs on another massive amount of coal. Of course, it’s you. You are the coal. At least, you feed it. Hundreds of millions feed it. Every visit, like, read, connection and piece of meta-data you insert into it, you’re helping Facebook organize the world along the way you want to see it. You get a useful product in exchange. You most certainly are paying for it. But you pay in the form that most don’t value.

Things got real when coal was turned into electrical current and transported across huge distances. It enabled invention. A lot of invention. Anything that plugs into a wall in your home is a product of that. Power became standardized, and ultimately turned into a utility. These products are mostly usable. The fridge, at one end, is the most usable. (The light switch still requires somebody to flick it on. Don’t even get me started on the relative complexity of the dimmer). The modern PC and microwave are at the other spectrum. (Though, I still don’t understand why the microwave continues to be such a horrible appliance).

The language of power back then was electrical engineering. To an extent, it still is a language that has power.

A new power is emerging – and it’s not just purely engineering.

Apple has caused some insight here – on the intersection of usability, engineering, and algorithms. All three factors need to work together. It’s not enough for something to do something very well. It has to be usable – and – quite possibly, to the extent of joy of use. And so, this applies not only to virtual goods, like software, but also to an emerging class of physical goods.

What are you inventing? Are you actively considering usability?

It’ll be great to be back in New York.

There are a few people that I’m looking forward to seeing: John Lovett on social media, Melinda Driscoll on web analytics, Shari Cleary on media, Joseph Stanhope on mobile, Alex Langshur on government. And then there’s Michael Healy, Patrick Glinski, and me.

I’m presenting with Michael Healy on sentiment. Michael Healy is among the best thinkers in this space and is just great. There have been a few very recent breakthroughs in sentiment analysis over the summer (and as recently as last week), and I’m looking forward to explaining how to treat the measure. I understand a core problem with the application of the metric – the gap between what some want the metric to mean – and what the metric actually really measures.

I’m with Patrick Glinski of Idea Couture on Friday – presenting “Communicating Data to Designers”. It’s a really different topic, and something you won’t see anywhere else. It’s not on the radar yet as a differentiating competitive advantage. It’s new, it’s different, it’s fresh – and even a bit risky. So come on out. We won’t bite.

I’m looking forward to seeing you out.

It’s an excellent way to present data in a very accessible, very explorable way. I found it inspiring.

The call to action is “create your chart now”. A very good, honest, call to action.

The technology adoption S-curve can be a slow beast, and expectations of growth have persistently outstripped actual adoption, at least in North America, and especially in Canada. Adoption has a few drags on it in North America and Europe. No such drags exist in Asia.

The chart below compares all the countries smartphone penetration. (Click to embiggen)

That chart masks underlining maturity in each country. The chart below compares m-commerce ‘at least one time’ usage across Germany, China, and the United States. The big three economies. (Click to embiggen).

That’s a good snapshot. M-commerce is thought of as a pretty big risk in North America.

In many ways, the melding of couponing with check-ins was the right bridge into mobile for the times. As we all watch Groupon careen into the inevitable mess, we all ask ‘what’s next’. I ask, ‘where’s the utility’, ‘how can mobile be used to salvage previously wasted parts of my day?’

I look to m-commerce as being predictable. Certain firms, like Plastic Mobile, have extensive experience with eCommerce, and understand mobile. They’re not going to replicate the pain of the nineties. There’s an inevitability to it.

Though, painfully punctuated.

This would have happened sooner if it wasn’t for the double whammy of policy plus recession.