A Method for Prioritizing A/B Test Ideas That Won’t Hurt Feelings

organizing backlog of experiments

This is the second post in a three-part series about the optimization program at Hotwire, an Expedia, Inc. company.

Part 1: Organized Ideation: How Hotwire Runs 120+ Experiments Per Year

Part 2: A Method for Prioritizing A/B Test Ideas That Won’t Hurt Feelings

Part 3: A Mobile Website Optimization Case Study: Hotwire’s Cars Page

Prioritization is the million-dollar question in optimization programs. It gets at the heart of our most frequently asked question at Optimizely: “What should I test?” A strong prioritization process arms a growth marketer with the answer: “Here’s what we’ll test, in what order, and here’s why.”

Last week in part one of this three-part series, we met Pauline Marol, the Lead Product Manager for optimization at Hotwire. Pauline’s prioritization process (how’s that alliteration for you?) is a framework that keeps her optimization engine running smoothly with hundreds of ideas in the backlog, and helps the very best ideas surface so they can be tested quickly.

As Pauline describes it, her prioritization “removes the emotion from A/B testing.”

“I don’t want to be the one making decisions on what [experiments] I run and don’t run because I am biased,” she says.

So just how good is this process? It’s comprehensive, rigorous, logical, and too good not to share—so here it is, in full:

The prioritization system is additive. If the idea meets a requirement, it gets one point. If it doesn’t, it gets zero. The points are summed in a spreadsheet to give each idea an overall score out of 10. Ideas are ranked according to the overall score.

Related post: Steps for Building Your Own Prioritization Framework

Rule1 Point0 Points
Main MetricSupports the company’s main metric – new bookingsSupports a secondary metric, like CTR or Net Promoter Score
LocationTests a change to the results or billing pagesTests a change located on any other pages
FoldMakes a change above the foldMakes a change below the fold
TargetingTargets 100% of customersTargets a subset of customers (repeat only, new only, top 50 markets, etc.)
New InformationAdds new information or a new element or removes an element from the pageMakes a change to the existing elements (copy, color, UI, etc.)
BenchmarkingBorrows from a success on family sites Expedia.com, Hotels.com, or CarRentals.comNo benchmarking best practice
Conversion VeinsApplies to two or more conversion vein themesApplies to one or fewer conversion vein themes
Strategic TopicSupports a strategic company goalDoesn’t map to a company-level goal
MobileWould change an element of the mobile web experience, or encourage an app installNo mobile component
OpaquePotential to increase of opaque share for a line of business (travel industry-specific value metric)No influence on opaque share of business

Thanks to this process, prioritizing Hotwire’s backlog of experiment ideas becomes a joy instead of a chore for Pauline. And remember, she has a queue of more than 400 experiment ideas as of this moment—a number that will continue to grow.

Pauline Marol, Lead Product Manager, Site Optimization

Pauline Marol

Pauline and her team have done an excellent job of increasing the visibility of their testing program, a challenge that plagues many optimization programs. But visibility can be a double-edged sword: high visibility optimization means that many people at Hotwire will come to Pauline with test ideas. Having a completely transparent prioritization system helps Pauline take herself out of the process and properly set expectations around what makes a good or bad experiment idea and when those ideas will get tested.

“If you come to me with an idea and it’s not live in two weeks, it’s not because it’s a bad idea—it’s because I have better things to test.”

Pauline Marol, Lead Product Manager, Site Optimization, Hotwire

When everyone knows the rules and the optimization process is transparent, concerns around testing can fall away. Instead, Pauline gets to focus on being the FUN team at Hotwire—a hub for creativity, moving fast, driving business results, and being inspired by the opportunity to lead the travel industry.

Combine creative brainstorming, well-informed hypotheses, a robust prioritization process—and a dash of passion for testing and optimization—to follow Pauline’s lead and create a world-class optimization program of your own. Next week, we’ll take a deep dive into an experiment that produced the largest win for Pauline and the Hotwire team. Stay tuned for more!