An example of a copy analytics questions is: which call to action works better?

“Buy now!”

“Learn more…”

The answer, maddeningly, is: “it depends”.

That’s why we have a methodology for testing. It’s the scientific method. Just because we have Google Website Optimizer that can, in theory, answer that question very easily, that doesn’t mean that we throw away the scientific method altogether.

Just because it’s so easy – that doesn’t mean that we should be sloppy, either.

For instance, let’s assume that you have direct ability to go in and edit your HTML with the Google Website Optimizer code. Assume that you can do this yourself, and you have the know-how. Also assume that you’re disciplined enough to keep it simple, and you have a best guess at what you’d want to test.

If 100 visitors came to the site, and 3 clicked on “Learn More…” and 2 clicked on “Buy Now”, that doesn’t mean Learn More will result in a 6% conversion rate all the time, post test. (If 100 visitors came, 50 supposedly saw “Learn more” and 50 supposedly saw “Buy Now”, 3 clicked on “Learn more”, so 3/50 = 6/100 = 6%).

That’s a sample size of 50. It’s not enough.

How much is enough?

That depends on how accurate you want the result to be – and now we enter into statistics.

Thankfully, Google Website Optimizer has a lot of that math built in, so it’s pretty good. However, you need to have a very high sample size to make the best decision.

If you have 5000 visits for each element in your test, you’ll have a confidence interval of roughly plus or minus 1.4 percent, 2 SD. So, if you had “Buy Now” at 6.0%, and “Learn More” at 6.2%, you couldn’t be all that confident that Learn More really is doing all that much better. It is, in absolute terms, doing better, but it’s still really tight. The degree of uncertainty between the two is really tight. If “Learn More” was at 6.5% and “Buy Now” was at 5.0%, then under those circumstances, you could be fairly confident that by going with “Learn More”, you’d realize a 1.5% lift. Frequently we’re looking at smaller incremental lifts, meaning, sample sizes need to be much larger.

There’s a lot more to copy analytics than just call to action.

The leading headline, lead sentence, second sentence, and choices of pictures are important. As is adeherence to various tones. For instance, the difference between:

“The AM-C Pro Remote Control features a prominent mute button.”

versus

“So go on and shut up those annoying commercials with the jumbo sized mute button.”

The second version of copy might resonate better with a target audience. Or it might not.

It depends.