3 A/B Testing Case Studies on Seeking The Global Maximum

seek the global maximum

As the following case studies reveal, there are huge wins to be had from thinking big and being open to questioning the status quo. There are also important revelations lurking in smaller tests that can point the way to a major redesign. And sometimes testing is the only way to find true north amidst the chaos and confusion of major changes.

Marc Andreessen Talks Killer Robots, Unicorns, and The Future of Tech


Few folks on the planet are as recognized for thinking about the future as Marc Andreessen, the co-founder of Andreessen Horowitz (a16z)* and one of the world’s most prolific venture capitalists by number of investments. Marc likes to discuss the future. A lot.

So it was fitting for our Co-founder and CEO, Dan Siroker to invite Marc to discuss the future of technology, privacy, and disruption at Opticon, our annual user conference.

Here are the 5 quotes from Marc’s conversation with Dan that really captivated the attention of the audience and gave them a birdseye view of how Marc sees the world evolving.

13 E-commerce Conversion Optimization Case Studies

shopping carts ecommerce conversion optimization

Ideas for A/B tests can come from the darnedest places. But there’s nothing like case studies from other similar companies to get the hypothesis juices flowing. Here are 13 e-commerce conversion optimization case studies from Optimizely customers. Each one has a clear takeaway you can use as inspiration for your own test hypotheses. Scroll forth and grow that list of experiment ideas!

Case Study: Testing Personalized Experiences at Secret Escapes

secret escapes search personalization

This is part two of an optimization case study with Secret Escapes, a flash-sale luxury travel company. In part one we covered how Secret Escapes used Optimizely to experiment on their iOS app onboarding flow, double signup rate, and generate positive ROI from paid mobile marketing.

In this post, we’ll dig into another series of experiments that are driving continual increases in new user signups and revenue. In both experiments, Secret Escapes is testing the impact of personalized experiences.

How Secret Escapes Increased LTV and Doubled Signup on Mobile

Secret Escapes iOS app

Secret Escapes is a flash-sale luxury travel company that offers discounted deals on four and five-star hotels around the world. The business is growing quickly with 300% YoY growth, expanding to markets in Germany and the U.S. with a base of 10 million+ members on their iOS/Android apps and website.

As a high-growth business in the extremely competitive online travel market, Secret Escapes cannot afford to make changes to any of their digital properties that might throw off this momentum. That’s why Secret Escapes runs all app and website changes—no matter how small—as Optimizely experiments first before rolling them out live.

What Marketers Can Learn From Slack’s CMO

Bill Macaitis

Slack is on fire. The company has record-setting growth metrics. It is adding $1 million dollars of ARR every 11 days. As of April 2015, over 750,000 people use the team messaging app daily. The entire story looks quite impressive on charts. And it looks equally impressive through the eyes of Bill Macaitis, Slack’s Chief Marketing Officer.

Marketers can learn a lot from Slack’s approach to marketing, especially their laser focus on creating the highest quality user experience possible — and measuring it. I’ll share a few lessons marketers can takeaway from Bill’s AMA.

A Guide to SXSW Interactive 2015 for Growth Marketers

Growth Marketer roadmap thumbnail

South By Southwest is upon us. Everyone from robotics engineers to heads of optimization will descend on Austin, TX from March 13-March 16 to experience the Interactive portion of the festival. This year, Interactive offers more than 800 sessions. 800 over four days!

To make the decision-making process slightly easier for those of you attending, the marketing team at Optimizely combed through those 800 and selected 12 that we believe are most relevant to marketers focused on growing their businesses through data and testing.

On Building an Optimization Agency: The Inception & The Future


Around 2007, Matty Wishnow had a revelation about decision-making: most decisions businesses make—whether it’s a marketing plan, a design brief, or a feature on a product roadmap—are “really a bunch of hypotheses masquerading as priorities.” He realized that most businesses didn’t really employ data or testing to drive their decisions.

“It was almost just like something that somebody had to check off on a checklist, all the analysis was postmortem; everything was still driven by gut,” Matty told me.

This epiphany drove him to start a digital agency called Clearhead. Today, some of Clearhead’s clients include Adidas, Fresh Direct, Blu Dot, Sephora, Keurig, and Patagonia.

I spoke with Matty and Brian at Clearhead, about how they approach client engagements, how they’ve seen the optimization landscape change over the years, and advice for agencies starting to offer optimization.

Your Email Open Rate is High? That’s Nice.

email open rate

A high open rate is nice. It’s an important signal you’re writing evocative subject lines and optimizing your send time. But if people don’t take action or convert on that email they opened, then the act of opening is much less meaningful. Getting maximum value out of A/B testing marketing email requires optimizing for key metrics that really matter to the business—and open rate is not one of them.