Why don’t the campaign components add up?
Sometimes the components of a marketing channel will not add up to equal the total performance of the marketing channel. This is caused by any number of realities and limitations imposed in part by nature, and, in part, by you, the marketer.
Consider the following deliberately simple scenario:
March 2012 Impressions:
- Total Digital Impressions Delivered: 100,000,000
- Total Impressions with Chicken Creative: 25,000,000
- Total Impressions with Beef Creative: 50,000,000
- Total Impressions with Pork Creative: 75,000,000
Something doesn’t make sense. I’m telling you that 100,000,000 impressions were delivered in total, but each component of that figure: 25 million, 50 million, and 75 million, don’t actually add up.
That’s because creative can have multiple attributes. An ad may feature Chicken alone, Beef alone, or Pork alone. An ad may feature Beef with Pork. An ad may feature Chicken with Beef. An ad may feature Chicken with Pork. In a crazy twist, perhaps some creative features all three! (The madness!). Attributes can cause such complexity when it’s possible for a single thing to have multiple attributes.
The next scenario demonstrates complications that arise because of instrumentation:
March 2012 Impressions:
- Total Digital Impressions Delivered: 100,000,000
- Total Impressions served to Males: 60,000,000
- Total Impressions served to Females: 10,000,000
- Total Impressions likely served to 35 to 50 year olds: 1,000,000
All people have attributes, but not all people have attributes that can be measured.
It might very well be that for the XBOX Live component, Microsoft can report with greater certainty, owing to profile information, that the content was served to more males. And, because that particular app was geared towards males, there’s greater certainty on that end. It also might be the case that another component was on mommy blogger ad networks, however, the knowledge of the ad targeter was really ethical, and wasn’t uniquely tracking everybody, so, the ‘missing 40 million impressions’ aren’t missing.
The same goes for the age component. We may hypothesize because of Quantcast data that those impressions served on mommy blog networks were heavily 35 to 50 year old females, but, there’s nothing in the instrumentation itself that confirms that hypothesis.
Just because it may be measurable doesn’t guarantee that it will be measured.
Finally, consider the complexity imposed by time:
March 2012 Impressions:
- Total Digital Impressions Delivered: 100,000,000
- Total Impressions from Affiliate Program: 10,000,000
- Total Impressions from the RayRayHayHay campaign: 8,000,000
- Total Impressions from the A campaign: 1,000,000
- Total Impressions from the Eh campaign: 1,000,000
Well, CLEARLY the A campaign and the Eh campaign failed – since the affiliates didn’t use those creative treatments much at all. What we don’t know is time.
- Date the RayRayHayHay campaign creative was posted: January 5, 2012
- Date the A campaign creative was posted: March 1, 2012
- Date the Eh campaign creative was posted: March 28, 2012
That’s 1 million impressions served in 3 days for the Eh campaign. That’s 1 million impressions served in 31 days for the A campaign.
Such component analysis is made particularly tricky when we’re trying to do it using a monthly report or some other arbitrary unit of time.
In sum:
Channel performance analysis is not channel component analysis. These are two distinct types of analytics, aimed at answering two different classes questions. For the reasons listed above, attribute overlap, instrumentation limitations, and time, the sum of the components may not add up to the total. This is not a devastating realization if you understand the differences and how to think of them.
There’s a general optimistic sense that drillability, the ability to drill into any metric and see its components, is possible in all contexts. It is possible in some contexts. It is not possible in all contexts. Privacy and technical disruption impose long run constraints in ever being able to achieve that.
It’s not likely to be perfect any time soon, and, in some cases, the components won’t ever add up.
***
(Note to fellow analysts: I chose impressions to keep it really simple. On-site and post-click analysis is required. Statistical analysis exists for a reason, so, even armed with impression and CTR data, you may analyze performance across multiple attributes. Moreover,you ought to be aware of the biases that exist in your data set – is it the case that males really did respond better, or, is it the case that the instrumentation is just better at identifying males?)