Statistics for the Internet Age: The Story Behind Optimizely’s New Stats Engine

Optimizely Stats Engine

Classical statistical techniques, like the t-test, are the bedrock of the optimization industry, helping companies make data-driven decisions. As online experimentation has exploded, it’s now clear that these traditional statistical methods are not the right fit for digital data: Applying classical statistics to A/B testing can lead to error rates that are much higher than most experimenters expect. We’ve concluded that it’s time statistics, not customers, change.

Working with a team of Stanford statisticians, we developed Stats Engine, a new statistical framework for A/B testing. We’re excited to announce that starting January 21st, 2015, it powers results for all Optimizely customers.

This blog post is a long one, because we want to be fully transparent about why we’re making these changes, what the changes actually are, and what this means for A/B testing at large.

French Girls Loves Optimization

Scotch Mornington

How many times have you seen Titanic? Enough to remember the moment Rose tells Jack to “draw me like one of your French girls”? Well, a group of iOS developers from Scranton, PA remember… and they created an app inspired by it.

The app has risen in popularity over the last year, surpassing 1 million downloads in July 2014. With A/B testing, French Girls’ lean team is turning the majority of those downloads into actively engaged, activated users. Here’s how they’re doing it, lessons they’re learning along the way, and why they named the app French Girls.

7 Tips to Improve Mobile App Onboarding

first time app user onboarding

Twelve—That’s the number of apps currently installed on my mobile phone that I haven’t used more than once. At one point, they caught my interest enough to install but now are just gathering dust and taking up screen real estate.

Chances are, you probably have at least a few apps on your phone that fit this bill too. Today, 80-90% of downloaded apps are used once and then deleted. That’s why everything that happens after someone launches your app for the first time is downright imperative. Here are some ideas product managers can test on their app onboarding flows…

Optimizing Content: How Kevy Writes More Without Writing Worse

Workspace station

Brooke Beach has a challenge common amongst many: producing a lot of content with limited resources without sacrificing quality.

Sound familiar?

Her marketing team has come up with a system that combines data from website analytics, marketing automation, and live chat to help create the right content for the right audiences. Intrigued as to how live chat contributes to this optimization equation, I talked to Brooke about how they go about it, and the impact it’s had on the business.

Lessons We Learned from Email A/B Tests in 2014

black and white keyboard mouse

This year, the Marketing Automation team at Optimizely got serious about tracking our email A/B tests. In 2015, Optimizely will be taking experience optimization to the next level, and continue to rigorously test campaigns in order to provide the best experience for our customers.

Looking back at some A/B tests we ran this year, here are some lessons learned, along with where I hope to take our email experiments in 2015.