Here’s one customer story that may provide inspiration for individuals and teams out there who just need a little push.

It’s hard to bring experimentation to a company that has no testing experts or existing experimentation culture. When uShip, an online marketplace for shipping services, began working with Optimizely in 2017 they faced tough challenges. These challenges ranged from lofty goals like having a test’s ROI be in the million dollar range to spaghetti testing (the throwing of things at the proverbial wall and hoping they stick).

In a recent webinar, Building a Culture Rooted in Experimentation, uShip’s Jamy Squillace (Director of Product Management) and Brooks Lyford (Product Manager) share some of their key challenges, successes, and tips for building a culture of experimentation in a company where testing had previously been nonexistent.

Don’t Overindex on ROI

uShip experienced early roadblocks that hindered process—like executive approval required on each test and needing multiple stakeholders to weigh in on ROI. Jamy noted that others starting out with experimentation shouldn’t let “perfect get in the way of good.”

By shipping tests faster, uShip could test more and learn from failures and successes—leading to increased ROI vis-à-vis experimentation. uShipers involved in the beginning were focused on perfecting tests and earning higher returns, but when the team shifted their focus to “testing to learn, not testing to earn,” they had better results.

An example of a multivariate test the team ran

“Don’t make things more complex than they need to be. Get the tests going, get people used to testing, learn and get the insight back to the team. Then, make sure tests are perfect.”

—Jamy Squillace, Director of Product Management, uShip

To ship more tests, uShip’s team revamped the ownership of their company’s experimentation. They decided the product team should oversee experiments and report on the success of the company’s experimentation culture. It only made sense, as the product team was in charge of the company’s roadmap. In Q1 of 2019, uShip’s product team promised six tests to the CEO, which was more tests than they ran the entirety of 2018.

Tip #1: Give the folks that own experimentation a goal to increase the velocity of testing.

Use feature flags

Experimenting with feature flags helped uShip drive immediate impact by testing changes to their payment plans. Feature flags allow for features to be tested to a select audience– one benefit of them being able to test different versions of features is that you can see what experience performs best.

Using Optimizely, they tested the impact of showing different plans to 10% of their user base. In the first iteration, uShip noticed some of their power users were caught in a variant bucket with an isolated payment option that was not suitable for their volume of transactions. This revealed to uShip that people were using their products in ways they didn’t realize. This insight uncovered possible user experience improvements that uShip’s product and development teams wouldn’t have noticed without having tested this feature.

Tip #2: Use feature flags to turn on or off new features or test different experiences faster.

Involve other teams in the organization

Uship’s product team owns the company’s experimentation program, but they are certain to share the results, goals, ideas, and more with the entire company. They use all-hands meetings, bi-weekly “experimentation guild” meetings, and organize company-wide experimentation workshops to cultivate their experimentation culture.

uShip empowered their marketing team with Optimizely Web early on. But, doing so came with challenges. For example, as the marketing organization became self-sufficient in running web tests quickly, it was possible for experiments to ship without quality assurance testing. This drove home the importance of controlled processes related to testing, an important step in experimentation maturity.

Brooks Lyford, uShip Product Manager, suggests teams be mindful of experiments going live before they’ve been tested for quality assurance. It is so quick and easy to create tests, they sometimes go live without QA, and teams must remember even with tests you have to build QA into the process. So, it’s important to make quality assurance testing part of the process to keep different departments from shipping tests or changes too soon.

Tip #3: Make sure teams are in sync and communicating to ensure everyone is aligned on experimentation initiative

Moving forward

uShip is looking to increase the company’s engagement in future testing and experimentation initiatives—involving departments beyond just product, development, and marketing. They’re also committed to improving their metrics and analytics as well as the visibility of their experimentation culture through a company-wide newsletter.

Creating a culture of experimentation—especially within an organization that hasn’t ever run an A/B test—will always be a steep challenge. But, by setting realistic goals, increasing test velocity, and sharing results and ideas with the entire company, teams dramatically increase the rate at which experimentation is adopted throughout the company.