Optimizely Blog

Tips & Tricks for Building Your Experimentation Program

android phoneThis is the last post in a three-part series about the optimization program at Hotwire, an Expedia, Inc. company.

Part 1: Organized Ideation: How Hotwire Runs 120+ Experiments Per Year

Part 2: A Method for Prioritizing A/B Test Ideas That Won’t Hurt Feelings

Mobile experiences evolve quickly—and with an increasing number of possible touch points between travel businesses and customers, it’s important to ensure that a mobile web experience is on par with a company’s desktop web and mobile app counterpart.

As we’ve discovered over the past installments in our series with Hotwire, an Expedia, Inc. company, Pauline Marol and her team have a keen sensibility for how to optimize both for business outcomes and the ideal customer experience. Pauline is the Lead Product Manager for Site Optimization at Hotwire.

Snapshot of mobile booking methods from "The 2014 Traveler’s Road to Decision" by Google.

Snapshot of mobile booking methods from “The 2014 Traveler’s
Road to Decision” by Google.

Mobile websites are a key component of the travel customer’s experience, and they certainly aren’t going anywhere. Google’s recently updated its search algorithm to promote mobile-optimized sites in results and, according to a June 2014 study, 45% of leisure travelers completed bookings via a mobile website, while 55% of business travelers did so.

At Hotwire, mobile apps and mobile web experiences are a core part of how travelers book flights, hotels, and car rentals. Travel customers’ research and booking needs are uniquely mobile, and their on-the-go experience should help them meet their goals quickly and seamlessly.

Optimizing for Mobile Web

When Pauline and her team worked with Hotwire’s Cars line of business to optimize various elements of their conversion funnels, the mobile web experience rose to the top of her prioritized queue of experiment ideas. The experience hadn’t been updated for several years, and they hypothesized that redesigning the Car product page would improve conversions.

The goal of the experiment was to update the look and feel of the car detail pages in order to improve the click through and conversion rate metrics. Visually, the designers on the team focused on making the page experience feel more like the Hotwire app—when the original page was designed, Hotwire didn’t even have mobile apps!

According to Pauline, the team’s hypothesis was: “If we refresh the design of our car mweb product then we can increase our Car mobile conversion rate because we will be able to provide a better visual and interaction experience to our mobile users.” Many of the friction points on mobile had already been accounted for in the mobile app experience, and the team thought that introducing that consistency to the mobile web experience would be a straightforward win.

The Experiment

They kept the experiment simple and tested the control against one redesigned variation. As you can see below, the experiment doesn’t focus on a single element of the experience but instead takes a big step away from the baseline. There were no major usability changes on the page, and most of the information presented to the user was the same. However, this was a big overall change, showcasing an example of optimizing for the global maximum.

Take a look at the control and variation experiences below. Which would you be more excited to engage with?



Hotwire mobile site controlHotwire mobile site variation

The Results

The variation beat the control experience. This experiment was the Hotwire website optimization team’s largest winner last year.

After this energizing win, Pauline and her team are focused on refinement once again, brainstorming ways to iterate on their new modern look and feel and to apply the learnings from the experiment to other lines of Hotwire’s business.

Discovering the optimal experience for each customer requires a holistic view of every interaction with the customer throughout their conversion journey.

To learn more about how Pauline and other travel industry thought leaders optimize their customers’ experiences, join us in San Francisco on June 17 and 18. Along with other travel industry thought leaders, Pauline will be speaking at Optimizely’s user conference on a panel called Travel, Testing, and Personalization in the 21st Century. 

Optimizely X