It takes a lot of traffic to get statistically significant results for A/B tests. High-traffic websites like Google and Facebook should A/B test everything, but low-traffic websites shouldn’t bother.
At least that’s how the argument goes. But I would like to offer a different perspective. I think low-traffic websites should A/B test, not only because doing so has enormous cultural benefits, but also because A/B testing does work for low-traffic sites.
You may have heard that keeping your ad consistent with your landing page is good. But is that a myth? And if it isn’t, how can you quantify the impact of this symmetry? In early 2013, we performed an experiment on our Pay Per Click landing page aimed at answering these questions.
Our goal was more to answer an interesting question than to find a conversion lift, but we were pleasantly surprised to discover that making our landing page mirror our ads led to a tremendous lift in conversions.
