How Optimizely is Building a Culture of Product Experimentation

Why Product Experimentation?

Before I joined Optimizely, I worked at Microsoft on what was once the world’s leading internet browser, Internet Explorer. For those of us who have forgotten about this great browser, here’s a screenshot:

As an engineer, I was tasked to work with a product manager on developing an algorithm for determining the best list of sites to show a user on the new tab page (pictured above).

The product manager and I settled on an “fre-cency” algorithm, a combination of what was frequently visited by a user (ex: user’s favorite homepage visited many times) with what was recently visited by a user (ex: user’s recent interest in a political campaign happening this month).

We tested the algorithm on about 40 employees’ browsing history and everyone agreed the new algorithm was an improvement. Great! So we shipped it! But should we have shipped it? Probably not.

At the time, Internet Explorer had the majority of the internet browsing market share (40-50%), which accounted for ~ 1 billion people surfing the internet.

Looking back, it’s crazy to think that 40 employees in Redmond, Washington could verify what the ideal experience was for nearly ~ 1 billion people across the globe.

If we had the tools to do experimentation, we could have gotten statistically significant data on what was the best algorithm for the majority of users.

We might have even found that the premise of “one algorithm for all users” was a flawed premise to begin with because a personalized algorithm would provide a better and more tailored experience.

As most people know, Internet Explorer’s market share steadily declined while Google Chrome’s steadily increased. Although there are many factors that led to this change, experimentation is undoubtedly one of them.

Microsoft’s success came originally from shipping Windows operating systems on physical disks that couldn’t be easily modified, so it’s not surprising that experimentation wasn’t core to the Windows organization in which Internet Explorer was built. In contrast, look at Microsoft Bing, an internet search engine. Bing adopted a culture of experimentation, which was a major factor to Bing’s eventual profitability.

And it’s not just a story of Microsoft’s. Many successful software companies like Amazon, Google, and Facebook share product experimentation as a core foundation upon which they are thriving:

So given that we know a product experimentation culture can be a key factor to a product’s success, I want to write about two principles that I’ve seen at Optimizely be instrumental in actually creating that culture of product experimentation.

How Optimizely is building a culture of product experimentation

Principle 1: Organize your culture

Culture can be a nebulous thing, so it’s useful to bring some concrete structure to make it more tangible and measurable. Below are three suggestions for organizing your culture.

1. Set a Leader. At Optimizely, our co-founder, Pete Koomen, became the leader of our initiative to own and drive the culture of experimentation at Optimizely. Having a leader ensured that there was at least one dedicated person thinking about how to improve the culture of product experimentation at Optimizely.
2. Set SMART goals. As you would with any other work, set SMART (Specific, Measurable, Achievable, Relevant, and Time-bound[1]) goals related to experimentation.

As an example, setting a goal of “I want to experiment more” is not a SMART goal because it’s not specific, time-bound, or measurable. Instead, setting a goal like “I want to run 2 experiments in Q2” is much more effective at measuring your progress towards product experimentation.

At Optimizely, we set a goal on a particular product engineering team to run 10 experiments or feature rollouts in Q2. Setting visible company goals around experimentation ensured that everyone was aligned to the value of investing time in building a culture of product experimentation.

3. Provide venues for questions, feedback, and iteration. Even if you have the time, space, and priority to experiment, running a proper experiment can be difficult even for the most experienced.

At Optimizely, we created a weekly Experiment Review meeting where people could bring their experiment ideas to a committee, which would help answer questions like “Is this experiment going to verify my hypothesis?”, “Are these the ideal metrics to track?”, “What will we do if a particular variation wins, loses, or ties against the rest?”

The weekly review also meant each experiment we ran was thoughtful, impactful, and a worthwhile investment so that we weren’t just running any experiment to hit our quarterly goals.

The above tips are just some examples of adding organization to what would otherwise be a nebulous culture. Since every company is different, experiment with your own ways of adding organization.

However, adding only organization is not enough. To really make the traction of a product experimentation culture stick, you have to make it easy to experiment.

Principle 2: Make it easy to experiment

Making experimentation easy will increase the likelihood that teams will have the time to experiment. At Optimizely, we did this by ensuring that experimentation easily fit into each step of our product development lifecycle. Below are three phases of development where you could think about making it easier to experiment.

1. Design phase. A first example of making it easier to experiment comes from the design phase of a project. We started with our technical design document. Our technical design document process starts with a google doc template that engineers make a copy of as a checklist of things to think about when building a feature in Optimizely.

By raising questions about experimentation in the “Launch” section of this document, we increased the likelihood that engineers think about how to validate their feature or mitigate the risk of their feature by rolling it out.

At Optimizely, one result of this was experimenting with the beta invite UX of the Mutlivariate Testing feature before it was finished developing to, not only, get beta users, but also validate the need for the feature.

Although the UX of the variations of the beta-invite did not make a meaningful difference, the demand for Multivariate Testing we measured was much higher than many other features, validating the priority and focus of our team and showing data on the difference between features.

2. Development Phase. The next example of making it easier to experiment comes from the development phase. One way we made it easier to develop an experiment was to give each team an SDK in their language of choice that provided APIs which abstract the complexity of common experimentation actions like targeting, bucketing, and sending events.

For the frontend team, we provided a JavaScript SDK and for our application backend team, we provided a Python SDK.

However, both the frontend and backend teams realized that experimentation would be even easier by building a wrapper around the language-specific SDK to provide APIs that are specific to the environment in which the SDK is used.

For our frontend team, this meant building a wrapper that made it easier to experiment with tools like React. For our application backend team, this meant providing convenient methods around gathering user state stored in the database.

Having these SDK wrappers really accelerated the ability for new engineers to ramp up on experimentation and made it much faster for our culture of product experimentation to take traction and stick.

3. Testing Phase. A final example of making it easier to experiment was in the testing phase. Recently, Optimizely went through a full company rebrand: changing our logo, refreshing our color pallette, and updating our application design.

Unfortunately, a full rebrand is not as easy as a find and replace. It’s more like a stream of whack-a-mole problems. Since we strived to make this rebrand as seamless as flipping a light-switch for our customers, we needed the ability to test changes and quickly rollback if something went wrong.

In addition to providing the ability for an experiment to iteratively move across our development environments from local to staging to production, we also used feature rollouts to slowly roll out the rebrand from 0% to 100%.

However, even then, it was best if we provided an internal light-switch to our product development team in order to verify that the experiment and rollout were operating effectively. So that’s what we did.

We configured our application so that by adding a specific query parameter to our URL, an admin at Optimizely could force a particular variation by specifying the experiment key and variation key. This allowed our internal team to be able to easily flip the experiment on and off, which made it easier to test our experiences and increased confidence that we were rebranding successfully.

This last example was an idea from our frontend team, which we shared with our backend team in order to make possible. This sharing happened in a cross-functional meeting called an Experimentation Community of Practice where we invite all experiment practitioners to bring their learnings and pitfalls on experimentation to an open forum discussion.

This meeting brings us back to the first principle that organizing your culture is important to enabling product experimentation to take traction and stick within your company.

Conclusion

If there are two things to take from this blog post, it’s that organizing your culture and making it easy to experiment are two principles that will help you build a product experimentation culture, which will give your product and your company the foundation upon which to thrive and avoid the fate of the once great browser, Internet Explorer.