Download our FREE ebook of 43 A/B testing case studies from the world's leading companies for test ideas and inspiration. Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program

X

Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more!

Get It Now



This year’s Opticon20 is in the books! Over 2,600 people joined us virtually across North America and Europe. Opticon20 was focused on the science of getting it right and highlighted key lessons from 37 digital leaders about how they are using data-driven experimentation and progressive delivery to build better experiences for their customers. 

It featured an inspiring fireside chat with Bubba Wallace, Nascar Cup Series Driver and Activist, a customer keynote by Sara Tresch, SVP of Digital Services at Charles Schwab, virtual cooking class from San Francisco non-profit 18 reasons, and a number of exciting product announcements including Operations Experimentation (OpX) and a completely redesigned Full Stack experience. Opticon20 also added a brand new event, Opticode, showcasing customer stories from product and engineering teams, technical demos, and workshops. 

Below are the five big takeaways from Opticon20 or you can now watch all of Opticon20 on demand.

1. Adapting to Uncertainty

One major theme of Opticon was how companies are adapting to uncertainty as the pandemic has disrupted industries and driven more business online. Mahesh Chandrappa, VP of Digital at State Farm spoke about their digital transformation which has been accelerated by COVID-19. To adapt, State Farm built virtual offices for their agents that included shipping thousands of wireless hotspots to agents in poor connectivity areas, building virtual claim capabilities, and a streamlined mobile app for agents. As they’ve been creating new digital experiences to respond to changing consumer behavior, they’ve scaled their use of Optimizely experimentation to optimize those experiences and mitigate risk. They are now continuing on to experimenting on their agent’s digital experiences. In the future, customers like State Farm can use Optimizely’s OpX for Salesforce which enables them to experiment within their Salesforce instance to test different call scripts and process approaches to help agents service customers better and help improve customer satisfaction, loyalty, and contact center operations. 

Quote from State Farm

2. Putting Customers at the Center

Another key theme that emerged was the importance of putting customers at the center of your digital experience. Sara Tresch, SVP of Digital Services at Schwab talked about Schwab’s core value of making decisions through “client eyes”. She highlighted how Schwab uses agile product development and design thinking to validate product ideas faster and ensure that what they are building makes things better for their customers. Every change that Schwab makes, they not only measure the impact of that change on key business metrics, they also measure the impact on their “client easy score” which is a measure of how difficult it is for someone to complete a task on their website. 

Amy Vetter, Consumer Experience Manager at Reckitt Benckiser presented about how Dr. Scholl uses experimentation to challenge their own assumptions. She highlighted experiments that won that seemed counterintuitive like putting more popular products lower on the page actually helped increase sales.

Quote by Amy Vetter

3. Being data-driven

The theme of being data-driven began in the Optimizely keynote, where Whelan Boyd, Group Product Manager, Data Platform at Optimizely showed off a number of enhancements to our data platform including Stats Engine as a Service and integrations with Snowflake and Fivetran that provide teams the ability to combine experimentation data with their own data to deliver deeper insights. Tony Wilson, Sr. Technology Director of Consumer Experience Apps, CNN discussed how they had integrated experimentation into their mobile app release cycle. He said “We improved our monitoring process, and rolled out a new feature in our mobile app more effectively as a canary release. This allowed us to use science to evaluate how our real users received the actual release. And so we were able to confidently launch to a greater percentage of that audience, safely.” Hazjier Pourkhalkhali, AVP Strategy and Value at Optimizely, showcased research done with Harvard, Stanford, and Duke’s business schools that showed that “The average company we surveyed said that their revenue went up by 9% as a result of digital experimentation”. As customers experimented more, they saw even more revenue growth.

Quote by Hazjier

4. Embedding Experimentation Everywhere

In the keynote, Claire Vo, Chief Product Officer at Optimizely, highlighted how Optimizely has evolved to enable progressive delivery and experimentation across website, apps, mobile, and OTT experiences.  Later in the keynote, Byron Jones, VP of Product and Partnerships at Optimizely, showcased three exciting and new partnerships that bring Optimizely experimentation into new areas including the ability to experiment on different Amazon Personalize machine learning algorithms, Microsoft Dynamics 365 powered e-commerce sites, and deep into the customer support experience through an integration with Salesforce called Operations Experimentation (OpX).  The importance of experimenting everywhere was also on display in a talk from AMC who uses Optimizely Full Stack to experiment on the features and user experience across 7 platforms and 5 brands.

5. Standing up for Social Justice

We closed out Day 1 of Opticon with a fireside chat with Bubba Wallace, Nascar Cup Series Driver and Activist. Bubba sat down with Optimizely CMO, Carl Tsukahara to discuss the science of race car driving and the drive for racial equality.  Bubba is the first African-American driver in 50 years to win one of NASCAR’s national series. He shared how his team is constantly optimizing and tweaking their cars to maximize performance. Small adjustments on a race car, even 3/1000 of an inch can result in a night and day difference in term’s the car’s performance. 

Bubba’s impact on the sport of Nascar, however, goes beyond just his driving performance, in the wake of the unjust killing of George Floyd, Breanna Taylor, and Ahmaud Arbery, Bubba felt like he could no longer just focus on sports. So , he spoke out in support of Black Lives Matters and went on CNN to ask Nascar to get rid of the confederate flag from the stands.  Two days later, he was speaking to the president of Nascar on the phone and Nascar committed to removing the confederate flag from the stands. In his words, “My biggest career accomplishment: Being vocal about what’s going on…and being more of a human being instead of just an athlete. I wanted to stand up and make my voice heard.”

Quote by Bubba Wallace

If you missed it, you can now watch Opticon20 on demand.  

Optimizely X