Case Studies

A/B testing and conversion rate optimization case studies from the world’s largest brands and companies.

How Secret Escapes Increased LTV and Doubled Signup on Mobile


Secret Escapes iOS app

Secret Escapes is a flash-sale luxury travel company that offers discounted deals on four and five-star hotels around the world. The business is growing quickly with 300% YoY growth, expanding to markets in Germany and the U.S. with a base of 10 million+ members on their iOS/Android apps and website.

As a high-growth business in the extremely competitive online travel market, Secret Escapes cannot afford to make changes to any of their digital properties that might throw off this momentum. That’s why Secret Escapes runs all app and website changes—no matter how small—as Optimizely experiments first before rolling them out live.

How Testing Everything Transformed Blu Dot’s Retail Business

Bludot.com

Blu Dot tests everything. Really. I know what you’re thinking. Is testing everything possible? Is it even practical? The answer is yes on both counts. This is the very true story of a company that, within two years, evolved from never having conducted a split test to an organization that is continuously developing, prioritizing and validating hypotheses—big and small, broad and targeted.

A Method for Prioritizing A/B Test Ideas That Won’t Hurt Feelings

organizing backlog of experiments

Prioritization is the million-dollar question in optimization programs. It gets at the heart of our most frequently-asked question at Optimizely: “What should I test?” A strong prioritization process arms a growth marketer with the answer: “Here’s what we’ll test, in what order, and here’s why.”

Last week, we met Pauline Marol, the Lead Product Manager for optimization at Hotwire. Pauline’s prioritization process is a framework that keeps her optimization engine running smoothly with hundreds of ideas in the backlog and helps the very best ideas surface so they can be tested quickly.

As Pauline describes it, her prioritization “removes the emotion from A/B testing.”

Organized Ideation: How Hotwire Runs 120+ Experiments Per Year


Hotwire internal testing promotion poster

Curious what optimization looks like at one of the internet’s top travel companies? In this series of three posts, we’ll dive into the optimized testing process of Pauline Marol, the Lead Product Manager for Site Optimization at Hotwire, a leading travel booking site owned by Expedia, Inc. In this first installment, we’ll explore how Pauline structures brainstorming by ‘conversion veins’. In the future, stay tuned for a deep dive into her prioritization framework for experiment ideas and a big mobile experiment win for her team.

Creating Excellent Hypotheses with Website Analytics and User Research

How to generate A/B split test ideas

For an optimization team to be successful in reaching an aggressive revenue goal, both quantity and quality of experiments must be high. To support a focus on testing velocity, a strong backlog of experiment ideas is essential.

In this third installment of our spotlight on the optimization team at Move. Inc (the operator of realtor.com), we look at a couple of the thoroughly vetted experiment ideas that the team has tested.

To Make Optimization Stick, Focus on a Quantifiable Goal

2014-05-life-of-pix-free-stock-photos-aureliejouan-lights

As we discussed in the first installment of the Move story, Investing in Optimization – The First Steps Towards Building a Team, a key component of the success of the optimization program was their ability to communicate. For the core team, this enabled them to seamlessly do cross-functional work, remove blockers, and execute their experiments at a rapid pace.

In this post, we’ll talk about why communication was an essential for another part of the company’s testing culture: setting and communicating a primary goal to raise awareness and understanding of optimization internally.

French Girls Loves Optimization


Scotch Mornington

How many times have you seen Titanic? Enough to remember the moment Rose tells Jack to “draw me like one of your French girls”? Well, a group of iOS developers from Scranton, PA remember… and they created an app inspired by it.

The app has risen in popularity over the last year, surpassing 1 million downloads in July 2014. With A/B testing, French Girls’ lean team is turning the majority of those downloads into actively engaged, activated users. Here’s how they’re doing it, lessons they’re learning along the way, and why they named the app French Girls.

Optimizing Content: How Kevy Writes More Without Writing Worse

Workspace station

Brooke Beach has a challenge common amongst many: producing a lot of content with limited resources without sacrificing quality.

Sound familiar?

Her marketing team has come up with a system that combines data from website analytics, marketing automation, and live chat to help create the right content for the right audiences. Intrigued as to how live chat contributes to this optimization equation, I talked to Brooke about how they go about it, and the impact it’s had on the business.

How HubSpot Personalizes Every Customer Touchpoint


Customer Concept

At HubSpot, optimization is something we focus on every single day. It helps us deliver more engaging experiences to our customers and see greater returns on the content we push across our marketing channels.

We test and personalize just about everything. Here are a few examples of experiments we’ve run that have measurably improved our marketing programs.

The Impact of Symmetry in Online Marketing

symmetry

You may have heard that keeping your ad consistent with your landing page is good. But is that a myth? And if it isn’t, how can you quantify the impact of this symmetry? In early 2013, we performed an experiment on our Pay Per Click landing page aimed at answering these questions.

Our goal was more to answer an interesting question than to find a conversion lift, but we were pleasantly surprised to discover that making our landing page mirror our ads led to a tremendous lift in conversions.