Optimizely Blog

Grow your Optimization and A/B testing skills


Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more!

Get It Now

organizing backlog of experiments

This is the second post in a three-part series about the optimization program at Hotwire, an Expedia, Inc. company.

Part 1: Organized Ideation: How Hotwire Runs 120+ Experiments Per Year

Part 2: A Method for Prioritizing A/B Test Ideas That Won’t Hurt Feelings

Part 3: A Mobile Website Optimization Case Study: Hotwire’s Cars Page

Prioritization is the million-dollar question in optimization programs. It gets at the heart of our most frequently asked question at Optimizely: “What should I test?” A strong prioritization process arms a growth marketer with the answer: “Here’s what we’ll test, in what order, and here’s why.”

Last week in part one of this three-part series, we met Pauline Marol, the Lead Product Manager for optimization at Hotwire. Pauline’s prioritization process (how’s that alliteration for you?) is a framework that keeps her optimization engine running smoothly with hundreds of ideas in the backlog, and helps the very best ideas surface so they can be tested quickly.

As Pauline describes it, her prioritization “removes the emotion from A/B testing.”

“I don’t want to be the one making decisions on what [experiments] I run and don’t run because I am biased,” she says.

So just how good is this process? It’s comprehensive, rigorous, logical, and too good not to share—so here it is, in full:

The prioritization system is additive. If the idea meets a requirement, it gets one point. If it doesn’t, it gets zero. The points are summed in a spreadsheet to give each idea an overall score out of 10. Ideas are ranked according to the overall score.

Related post: Steps for Building Your Own Prioritization Framework

Rule 1 Point 0 Points
Main Metric Supports the company’s main metric – new bookings Supports a secondary metric, like CTR or Net Promoter Score
Location Tests a change to the results or billing pages Tests a change located on any other pages
Fold Makes a change above the fold Makes a change below the fold
Targeting Targets 100% of customers Targets a subset of customers (repeat only, new only, top 50 markets, etc.)
New Information Adds new information or a new element or removes an element from the page Makes a change to the existing elements (copy, color, UI, etc.)
Benchmarking Borrows from a success on family sites Expedia.com, Hotels.com, or CarRentals.com No benchmarking best practice
Conversion Veins Applies to two or more conversion vein themes Applies to one or fewer conversion vein themes
Strategic Topic Supports a strategic company goal Doesn’t map to a company-level goal
Mobile Would change an element of the mobile web experience, or encourage an app install No mobile component
Opaque Potential to increase of opaque share for a line of business (travel industry-specific value metric) No influence on opaque share of business

Thanks to this process, prioritizing Hotwire’s backlog of experiment ideas becomes a joy instead of a chore for Pauline. And remember, she has a queue of more than 400 experiment ideas as of this moment—a number that will continue to grow.

Pauline Marol, Lead Product Manager, Site Optimization

Pauline Marol

Pauline and her team have done an excellent job of increasing the visibility of their testing program, a challenge that plagues many optimization programs. But visibility can be a double-edged sword: high visibility optimization means that many people at Hotwire will come to Pauline with test ideas. Having a completely transparent prioritization system helps Pauline take herself out of the process and properly set expectations around what makes a good or bad experiment idea and when those ideas will get tested.

“If you come to me with an idea and it’s not live in two weeks, it’s not because it’s a bad idea—it’s because I have better things to test.”

Pauline Marol, Lead Product Manager, Site Optimization, Hotwire

When everyone knows the rules and the optimization process is transparent, concerns around testing can fall away. Instead, Pauline gets to focus on being the FUN team at Hotwire—a hub for creativity, moving fast, driving business results, and being inspired by the opportunity to lead the travel industry.

Combine creative brainstorming, well-informed hypotheses, a robust prioritization process—and a dash of passion for testing and optimization—to follow Pauline’s lead and create a world-class optimization program of your own. Next week, we’ll take a deep dive into an experiment that produced the largest win for Pauline and the Hotwire team. Stay tuned for more!

If you’re local to the Bay Area, join Pauline at our San Francisco Optimizely User Group—or check to see if there’s a User Group in your city. Better yet, join us at Opticon—Pauline will be speaking on the panel “Travel, Testing, and Personalization in the 21st Century.” See you there!

Optimizely X
comments powered by Disqus