This is part two of a series on Stats Accelerator. In the first part, we explained the when, why, and how of Stats Accelerator. In today’s installment we will discuss a major roadblock to successful productionization of bandits in the A/B testing context and how we eventually overcame it. This is a high-level overview. For […]

As an experimenter, the pressure to make a decision after you’ve run an A/B test can lead to some pretty wacky interpretations of non-significant experiment results. After all, what does it even mean for something to be “trending in the direction of statistical significance” (a phrase actually stated confidently during a recent experiment review at […]

This post originally appeared on the BBC Data Science Blog. When speaking of optimization, most of us will think about increasing conversions and revenue in e-commerce, otherwise known as CRO (conversion rate optimization). More and more though, media and services brands are using experimentation as a means for increasing customer engagement and fostering loyalty; focusing […]

Just like a suspension and arch bridges both successfully get cars across a gap, both Bayesian and Frequentist statistical methods provide to an answer to the question: which variation performed best in an A/B test?

Historically, industry solutions to A/B testing have tended to be Frequentist. However, Bayesian methods offer an intriguing method of calculating experiment results in a completely different manner than Frequentist. In the world of statistics, there are devotees of both methods—a bit like choosing a political party.

In this post, we’ll cover the benefits and shortcomings of each method, and why Optimizely has chosen to incorporate elements of both into our Stats Engine.

Statistics are the underpinning of our experiment results—they help us make an educated decision on a test result with incomplete data. In order to run statistically sound A/B tests, it’s essential to invest in an understanding of these key concepts.

Use this index of terms as a primer for future reading on statistics, and keep this glossary handy for your next deep dive into experiment results with your team. No prior knowledge of statistics needed.

Classical statistical techniques, like the t-test, are the bedrock of the optimization industry, helping companies make data-driven decisions. As online experimentation has exploded, it’s now clear that these traditional statistical methods are not the right fit for digital data: Applying classical statistics to A/B testing can lead to error rates that are much higher than most experimenters expect. We’ve concluded that it’s time statistics, not customers, change.

Working with a team of Stanford statisticians, we developed Stats Engine, a new statistical framework for A/B testing. We’re excited to announce that starting January 21st, 2015, it powers results for all Optimizely customers.

This blog post is a long one, because we want to be fully transparent about why we’re making these changes, what the changes actually are, and what this means for A/B testing at large.

Updated May 8, 2014 Optimizely is a product that was created so that anyone—regardless of technical ability—could run A/B tests on their website. We know that getting up and running with a testing tool is only half the battle. In order to truly excel with website optimization, you also need to understand how to interpret […]