How do you, and those around you, deal with failure? Because if the answer is anything but “well” or “every fail has a lesson”, then progressive, iterative, experimentation and AB testing really, really isn’t for you. Go away. Stop reading. It’s not everyone. The way to build an experimentation and testing bench is entirely by setting sail for fail. How to build your Experimentation and Testing Bench Your bench will be built: 95% with culture 5% with technology The technology is well thought out and several vendors are excellent in this space. The technology isn’t a problem. Sure, there are a few pretty shockingly bad systems out there. Every industry has a phony. Building a testing bench begins with policy.[…]
Author: Christopher Berry
Two big announcements – HBO and CBS, two major media companies that create original content, will both offer OTT streaming services. Consumers won’t need a cable subscription to get either of them. Sports are excluded of the service. More on that below. As a Canadian, it’s even more interesting because the CRTC has been holding hearings on another consumer friendly initiative, Pick-And-Pay. It’s pleasing to see HBO and CBS work at offering audiences the entertainment they want, and how they want it. It’s the beginning of the flip from a content-centric to a consumer-centric paradigm. And that’s a lot deeper than just a set of buzzwords. It manifests itself in the activities at the media company. I was impressed with[…]
Here’s what I think. I don’t like going to a presentation where I’m shilled at for 40 minutes. I don’t like being told about why a product is the best. I hate the saccharine story telling from the biz dev guy even more than the unenthusiastic “I am so tired of flying” delivery. They hate giving them. I hate listening to them. I think we’re aligned on that front. I don’t like going to a presentation and getting pandered at for 40 minutes. I hate it even more than the commercials. And you know the ones. They get up there and mouth platitudes and buzzwords. Stuff they know you want to hear purely because that correlates to an easy 4.5/5[…]
Data Civilians. Monica Rigoti used that terrific term in a New York Times Big Data piece. And the term resonated. It’s common to think of Big Data in much the same ways as nuclear research. Everybody wants the bomb. Yet, data comes out of the ground in a raw ore. The ore has to be mixed different chemicals to create various salts. Then it has to be shoved into huge centrifuges. These enormous processes are used to separate the slightly heavier bits of data from the slightly lighter ones – a process that’s important if you don’t want to contaminate the earth with dirty bias. It has to be milled into a sphere or sometimes an ingot. And then surrounded[…]
Consider a list of metrics. Now pick the Y, the dependent variable, from that list. A human would use their judgement. A machine would use an algorithm. It’s clear that human judgement varies. There’s evidence that it does. Causing a machine to make accurate predictions about human judgement is an interesting problem because of the inherent variation of judgement within human populations. This is a polite way of saying that some people really diverge from the median in their application of judgement. Consider the following question: If given a table of Gross Profit, Sales, COGS, Marketing Spend, Working Dollars, Non-Working dollars, Discount Cost, Impressions, Paid Media Impressions, Paid Media CTR, UV, V, and PV, which is the dependent variable? The[…]
The dependent variable is the one that matters. You can’t explain why without Y. Last week, I broadcasted 10 questions on Twitter and Facebook what they thought of dependent variables. It’s what 18 smart managers, researchers, developers, strategists, planners, data scientists, executives, eComm marketers, direct marketers, and department heads thought. I thank them for their time. Thank you. Take the n and the instrument for what it is – a dipstick check on just how aligned we are. Here’s what was found. H1 >=80% will state that Conversion is the Dependent Variable Confirmed. Conversion won every matchup it was in. Number of conversions won head to head against impressions and engagement; engagement and unique visitors; impressions and unique impressions.[…]
Here’s what you need to know about automated statistical analysis: 1. Automated statistical analysis is not a substitute for good judgement Statistical tests are tools. They help us understand why nature is the way that it is. Nature resists being known about. But, she is knowable. Statistical tests themselves are part of nature. The tests themselves were never meant to be substitutes for good judgement. That belief, that tests could replace people, has only ended up causing the accumulation of some pretty outrageous assumptions over the years. Just because there is a significant correlation between Magnum Ice Cream sales and Piracy in the Indian Ocean doesn’t mean that it’s causal. Statements of causality require judgement. Automated statistical analysis is not[…]
Let’s take a look at what 16-bit interfaces could do. A great simulation game begins with just a handful degrees of freedom and explodes from there. Behold the grandeur that is SimCity for the Super Nintendo. If you’re familiar with SimCity (1991), skip ahead to Data Exploration, below. On a flat plane of pixels, you have the choice to: Bulldoze a feature. Build a road. Build a mass transit unit. Build a power line. Build a park. Build a residential zone. Build a commercial zone. Build an industrial zone. Build a police department. Build a fire department. Build a stadium. Build a port. Build a coal plant. Build a nuclear plant. Build an airport. Build a special reward building.[…]
Discovering truth in data always begins with you, and your judgement. Assume that you have some idea about the world. Something that you believe is true, and you want to discover if you’re right. Here’s how I draw out that out. It becomes a matter of organizing a dataset along those thoughts. I call causal variables X1, X2, X3… I call the single variable that I’m trying to explain the Y variable. There can be only one Y variable. For your own sanity, there can only be one Y variable at a time. There are a large number of tasks to figure out if X1, X2, X3 cause Y. One of them is to run any one of the many[…]
The full New York Times Innovation Report was leaked last week. It’s worth reading if only because it lets you look at a paradigm – an entire way of thinking, laden with it’s own explanations of culture, causal factors, jargon, assumptions, myths, systems, and heretics. It enumerates the preferences and aspirations of a small group of people (including their preferred org-chart re-org!) and highlights a long-standing tension between technologists and journalists. It may also serve as a wake-up call that continuous improvement and scientific management is already a reality at several disrupting media startups. Let’s begin. Summary if you didn’t read it (and won’t): The document contains 97 pages. The term “Competitor” is mentioned on 39 of those pages. Analytics[…]