Powerful PMs steal with pride and test everywhere

Watch the talks for how to use experimentation, remote configuration, and beta testing to make every feature successful and be a more powerful Product Manager.

Let’s dig into a few of the lessons shared by the product leaders.

How to build a 5-star product

The secret to building a 5-star product is treating every new feature as an experiment. Each release is an opportunity to test, learn, and iterate. When Erik Metelka, Head of Growth Product at the video sharing platform, Cameo, introduced a major feature for sharing videos, he rolled it out as a test to 20% of mobile app users. Shortly thereafter, a bug surfaced in the multi-step workflow.

Since the feature was flagged for a subset of users and could be flipped off without an app store release. Thus, the impact was minimal. The team easily fixed the problem, rolled out the improved feature out to 100% of users, and watched the rave reviews pour in.

At Reckitt Benckiser, Amy Vetter proudly admits to stealing product ideas from Instagram and Amazon. But she never assumes that what’s worked for these companies will work for her eCommerce health brands such as Dr. Scholl’s and Durex.

“I steal with pride, I steal from other websites, stuff that I think our consumers might like. But I can’t guarantee that they’re gonna like it. So you need to test first.”

 – Amy Vetter, Consumer Experience Manager, Reckitt Benckiser

 When Vetter wanted to help her consumers understand their shipping options to drive urgency, she looked to Amazon’s UX for inspiration. She decided to test a countdown feature, showing when the item will arrive if the user purchases within a certain number of hours. Rather than automatically rolling it out to all consumers and expecting greatness, she tested first. She tried adding the feature in different parts of the layout. She experimented with messaging to ensure the relevance across their brands and customer scenarios. The insights gleaned from the results proved how much value the countdown feature provides for the business.

AMC Networks brings binge-worthy original content to your living room. We can thank them for shows like Breaking Bad, Killing Eve, and The Walking Dead. In addition to great content, they provide their customers with digital products for video streaming. Like Vetter, their VP of Product, Jon Keilson, gets ideas from other industry leaders and then he experiments to see how the features land with his users. In one experiment, Keilson worked on a feature inspired by Netflix’s auto-advance experience. By introducing it as an A/B test he determined the new “watch next” experience yielded a 2% increase in video views and rolled it out across applications with confidence.

But they can’t all be winners. This is a particularly tough pill to swallow when a heavy engineering lift goes into a new feature that doesn’t drive your goals. Another of Keilson’s tests was on the homepage content. It surfaced selections that were thought to be sure-fire winners. Building out the full feature required frontend and backend development. This was a heavy investment in engineering time. Imagine how deflated the team felt when they learned that this new experience led new users to view 20% fewer videos. This experience reinforced the importance of testing everything. If Keilson hadn’t treated this feature as an experiment it would have not only cost development resources, it would have cost the business growth.

How to be resourceful and conserve developer time

Keilson learned another lesson from the failed homepage experiment. He learned to test smaller changes, faster. One way he does this is through painted door tests. A painted door experiment employs minimal product changes to measure user’s interest and engagement with a feature before committing to building it out completely. For example, with the AMC Networks homepage experiment, rather than building out the backend components before testing out the concept, they could have run a painted door test with minimal code changes to measure user reactions. Based on the signals you get from the painted door test, you can make an informed decision about whether or not to build out the feature fully and scalably for all of your users. 

At Cameo, the product team found additional ways to be autonomous and save engineering time by owning feature delivery and iteration in addition to experiments. With Optimizely’s Full Stack SDK’s they are able to manage features, tests, and promotions without waiting for app store updates. They can make time-sensitive updates and manage seasonal experiences in the product without being restricted by deployment cycles.

How to experiment across brands, markets, and channels 

Hopefully, you are convinced that experimentation is going to make your product stronger, your work more effective, and your stakeholders happier. So, you’re all bought in and want to experiment everywhere. Now you’re ready to optimize your ‘test and learn’ workflows for efficiency at scale.

Vetter talks about how tests can have entirely different impacts depending on the audience. Across brands like Scholl’s and Durex, she looks at product use cases, user behavior, as well as demographics like age, and geographic markets. Customers of these brands can have completely different reactions to features that worked well with the other groups.

The challenge of testing everywhere stretches beyond audiences and demographics into platform technology. At AMC networks, Keilson’s engineering counterpart, Yoshitaka Ito, refers to this as a permutation problem. At Opticon20, the duo shared the secret for efficiently experimenting across 5 brands and 8 platforms. They do it by templatizing their applications.

Using shared templates across apps allows them to manage the UI as components that can be updated and tested more consistently and easily. Now they can run 80% of their tests cross-platform by powering them on the server-side.

This means they can experiment on every application from native mobile apps to TV streaming devices while saving tons of engineering time.

Powerful product teams assume nothing, steal with pride, and test everything. They are resourceful and conserve developer time even when experimenting across all of your brands, markets, and channels. You, too, can employ these strategies to build better products, manage your stakeholders, and deliver value for your business.

Looking for more data-driven frameworks for Product Optimization? Watch all of the Opticon20 sessions.