Focus Group Junkies: I’m Calling You Out!

How many times have you heard this?

“We did a focus group with 12 people, and they said they liked this product, so, even though it’s not statistically significant, it’s still directional”

or

“How many more people would we need to get into a focus group before we’d get something statistically significant?”

Where does one even begin?

There’s a huge difference between qualitative research and quantitative research.

Qualitative research focuses on very small groups of people, and the interviewer goes very, very deep into what they’re thinking, attitudes, preferences, what if scenarios, and why.

It’s about deriving a very deep level of insight.

Recall during the 1993 election, Alan Greg showed a focus group a commercial that he felt accurately questioned Jean Chretien’s foreign policy credentials.

Jean’s main quant, down the road, walked into a focus group, and showed the same commercial. They got similar reactions to it. When Jean’s Quant suggested that the commercial was making fun of Chretien’s face, the focus group became livid. The rest of the story can be found at Wikipedia.

http://en.wikipedia.org/wiki/1993_Chr%C3%A9tien_attack_ad

Behold the power of suggestion, and the power of the focus group.

That was an insight that was taken from a qualitative session.

Let’s look at quantitative research now.

Quantitative is called ‘quantitative’ not necessarily because of the sample sizes involved, but because it’s about ‘quantifying the amount of uncertainty that a hypothesis holds across a large popultion’.

Let’s say I had the budget to ask every living Canadian, all at the same time, if they liked Chicken. I might get 30 million ‘yeas’ and 3.2 millin ‘nays’ on that question. I’d be 100% certain though that 90.3% of Canadians like Chicken.

Now, assume that for some crazy reason, the Chicken Council of Canada doesn’t want to pay two thirds of a billion dollars to find out how many Canadians like Chicken? Assume that they’ll only spend 50 grand to find out.

Then I could only ask a fewer number of people, right? Maybe I could get away with asking 1000 people?

Over the years, scientists figured out how to quantify that rate of uncertainty given a ratio of randomly selected people and the entire population. There’s a fairly crazy equation and all these proofs that go way back to LaPlace (maybe earlier, but that’s about as far back as I’d go!).

So, I plop those two numbers into my equation – 1000 and 33.2 million, (and a few other assumptions) and out comes a percentage – 3.1%.

If I ask 1000 Canadians if they like chicken, and I got 890 people ‘yeas’ and 110 ‘nays’, then I could be reasonably sure (95% certain, in fact), that between 859 and 921 people out of 1000 like chicken. And then, if I take that ratio, I can apply that to the broader Canadian population, I’d get a potential market size of 28.5 million to 30.5 million. Yes…there’s a range, but that’s the price for being economical. (The more you spend, the more certainty you can get).

Notice the differences in the methodologies here.

One is about quantifying the number of people who are something or who have an attitude or preference, and the other is about exploring why and what. That is not to say that one can’t derive an explanatory variable from a quantitative data set (in fact, we can find really incredible insights), but, you’ll need another survey if you wanted to ask additional, deeper questions. (And that isn’t necessarily always cost effective, though, that form of iterative quantitative research is possible).

If you’re a CMO or a marketing manager, here’s what I’d suggest:

Use one on one interviews and very occasionally, a focus group, to dig deep for insights.

Then, in an effort to monetize your idea, do not merely repeat the focus group 5 times in a bid to get a sample size of 65. That’s not the right tool for the job, and worse, since the facilitator knows the desired result, you’re just going to end up getting it anyway. (And it might end up being the wrong insight.)

Take those insights and put them into questions, and then run a Quantitative study. That way, you can be certain of the uncertainty as to whether or not that insight has legs.

The focus group has its place, but like a hammer, it’s not the right tool for everything.