Embracing a Testing Culture

This post originally appeared on the Upside Travel Engineering Blog.

Upside is the first online travel service built just for business travelers, particularly travelers from small and mid-sized companies that can’t access special pricing or quality customer service. We embrace the “Move Fast and Break Things” culture found at the big tech giants like Facebook, Amazon and Google. And that’s all well and good but we are also a data-centric culture. We want to make decisions, not from a gut or a place of instinct, but from a place of information and facts.

Testing! Data!

These two cultures are a bit at odds. On the one hand, we want to ship code out quickly (and we do!) But on the other, we want to ensure what we’ve shipped doesn’t nerf our conversion rates. The Internet is a funny place, and one moved pixel or a bit of text has vast impact on conversion numbers. And conversion numbers have a serious downstream impact (good and bad!) on other business variables — CAC, LTV, etc.

We’re going to move fast and we’re going to break things but we’re using Optimizely to modulate the traffic going to the newly shipped and broken things. All new big features will hide under Optimizely until we are certain the feature is bug-free and stable before unleashing it on everyone.

It feels like we’re slowing down. But we’re actually speeding upWe can ship MVPs and eliminate poor paths — bad messages, bad landing pages, bad flows — without jeopardizing our current progress. We won’t waste time on failed ideas. Fail fast and move on. Without A/B testing we’re guessing and watching the impacts on conversion 7 days later. And maybe something good and maybe something bad. Who knows? No one knows.

Now we can see and start growth hacking the main product like crazed weasels. Not just landing pages and ads. Now we can rearrange entire flows to see what happens. Do customers like to pick hotels before flights? Do they like cars before hotels but after flights? What about all the machine learning fun — does the new model do better than the control?

My personal north star is to follow Facebook’s lead and A/B test everything that even smells like a feature. Even tiny changes to copy should undergo testing because the smallest things have the biggest impacts. Test UX bug fixes or changes. Test checkout changes (which we did and that was educational).

We do need to be smart about what we test. And we need to ensure it’s not onerous on the product — not hard to build, doesn’t slow down performance. But knowing is half the battle.

And besides, we have an awesome data science team, an awesome data engineering team, snowflake and looker. We should spend our days staring at charts and making very intelligent HMM noises and making the best possible choices with the information we have.

We have started inching toward a testing culture where we simply test every new change reflexively. In the end, this will make Upside better for our customers and we’ll only make the very best go to 100%.