Conversion is an anomaly
Depending on who you believe and the context, average site eCommerce conversion rates vary between 0% and 12%. That’s not very helpful. In my own experience, defining conversion as number of completed checkouts divided by total number of site visitors, that rate varies between 0.20% to 2.00%.
That fact has important implications for analysis, bias, and making causal statements about what causes conversion.
- When doing an experiment, the lower the conversion rate, the greater the number of visitors that are required to make a truthful causal statement that something causes conversion.
- As a consequence, poorly converting sites that could benefit from experimentation the most are the most disadvantaged.
- Methods that are more common in the machine learning community may actually be more appropriate than what we’d call ‘traditional statistical analysis’.
As A Result:
- If the traffic to a given site is low, it is even more important to test big things that matter, than it is to fiddle with something likely to be trivial. Take big risks.
- It is preferable to increase the efficiency of the site by converting visitors into customers than it is to incur high incremental costs from driving more unqualified traffic.
- We may have more success if we treat conversion as an anomaly detection problem as opposed to a regression problem.