Just like a suspension and arch bridges both successfully get cars across a gap, both Bayesian and Frequentist statistical methods provide to an answer to the question: which variation performed best in an A/B test?
Historically, industry solutions to A/B testing have tended to be Frequentist. However, Bayesian methods offer an intriguing method of calculating experiment results in a completely different manner than Frequentist. In the world of statistics, there are devotees of both methods—a bit like choosing a political party.
In this post, we’ll cover the benefits and shortcomings of each method, and why Optimizely has chosen to incorporate elements of both into our Stats Engine.