Download our FREE ebook of 43 A/B testing case studies from the world's leading companies for test ideas and inspiration. Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program

X

Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more!

Get It Now




Opticon19 may be over but the lessons learned from it will stay with us over the year to come. It was our most well-attended and comprehensive customer conference to date. From the packed to capacity Happy Hour in the Sky Room of the Marriott Marquis to the Developer Night held the same evening, the conversations around experimentation were flowing, as was the growing belief that the world of experimentation and optimization is gaining global traction across multiple industries. Opticon19 brought together 25 customer speakers, inspirational conversations with actor, entrepreneur, and philanthropist Ashton Kutcher and astronaut Dr. Mae Jemison, as well as a full expo hall. It was two and a half days witnessing the maturation of an industry that is not only here to stay but set on changing the world one experiment at a time. 

Ashton Kutcher and Carl Tsukahara Opticon19

Fireside Chat with Ashton Kutcher

There’s no better place to start than with CEO Jay Larson’s opening remarks outlining what it means to outperform, “Outperform is about outperforming your customers’ expectations and outperforming your competitors in the process.  It’s about using experimentation to build better products and run your company in a fundamentally better and more profitable way.” 

Opticon19 content will be available in the coming weeks so stay tuned, but in the interim here are five takeaways from Opticon19.

ROI ROI ROI

Not only did we introduce the ROI Model to help teams quantify the value of their experimentation but during the opening keynote, CEO Jay Larson also shared numbers as to the value that Optimizely has created on behalf of its clients, and that number was $870 million. The ROI model is designed to help teams share widely the value of their experiments, and in so doing, demonstrate their value to the bottom line that helps their company to outperform.

DIVERSITY OF TEAMS AND TITLES ARE EMERGING IN EXPERIMENTATION

Alongside our announcements of Performance Edge and Data Lab, which speak to meeting the needs of developers and data scientists there was considerable conversation in panels and in networking with a diverse cohort of experimentation professionals. Whether it be Nina Bayatti of ClassPass sharing how experimentation had begun within marketing but now was implemented company-wide across acquisition, growth and within the app or Ian Tucker of the Wall Street Journal offering that the uplift from his team is now getting baked into forecasts. Experimentation is not simply gaining greater adoption, but it’s spreading internally across enterprises as well. Perhaps the most succinct assessment of this development came from Anne Reints, Senior Manager, Growth Analytics & Experimentation at Malichimp, “We’ve gone from a tiny mighty team of 3 within product and marketing to a cross-functional team of 30.”                  

Diversity Panel at Opticon19

Diversity is Key to Experimentation

EXPERIMENTATION IS CENTRAL TO DIGITAL TRANSFORMATION

As our co-founder Dan Siroker said in his closing remarks, “Customer expectations are going to keep ratcheting up,” and the only way to meet those expectations is to stay digitally competitive through experimentation. Mark Yolton, VP Digital & Interactive at Salesforce, shared that even though his team has decades of experience, they are still consistently surprised by how often their own guesses at what will work are wrong, and thankfully by testing they are able to meet customer expectations. 

Nowhere was this more evident however than in the Northwestern Mutual Breakout Session led by Roman Geyzer, Senior Director of Product. After outlining a culture filled with downloading whitepapers, salespeople calling at inopportune times and release cycles that had to be sequential, Geyzer shared how by structuring the organization based on the client journey they were able to radically transform. In summing up this massive shift with the 150+ year company he said, “We moved from a ‘we should build’ solution mindset to ‘should we build this and why are we building this’. One sample experiment, simply changing the CTA from ‘Let’s Talk’ to ‘Get Started’ improved the click-through rate by 190%.

THE ECOSYSTEM IS GROWING

At this year’s Opticon19 we were proud to have our partner Accenture Interactive as the Diamond Sponsor. Global Lead of Digital Marketplace Services and North America Experience Transformation Lead Jeriad Zoghby shared his compelling vision of experimentation within the context of a new world of digital commerce. Zoghy, who as his title suggests travels the world sees tremendous growth in what he terms ‘partner ecosystems’.  “There is a massive market for Digital Experimentation across the spectrum from Direct to Consumer to B2BC.” B2BC being the online retailers from Amazon to Nordstrom who are in essence the new digital superstores where brands need to find a way to demonstrate their value.

We also were thrilled to have both Salesforce and AWS in attendance sharing their visions of how experimentation drives the digital experience economy and helps them to better serve their customers. Alongside these partners, we could not be happier with Contentful’s announcement of their Optimizely application, allowing for experimentation within Contentful across branding, messaging and user experience. Contentful’s development follows on our pre-Opticon announcements of both Jira and Slack integrations, not to mention our work with Cloudflare which we shared earlier in the year.         

EVERYTHING THAT CAN BE OPTIMIZED WILL BE OPTIMIZED

Dan Siroker Nina Achadjian

Prior to the closing remarks from Dr. Mae Jemison, Optimizely co-founder Dan Siroker sat down with Nina Achadjian from Index Ventures to talk about the future of the industry. A wide ranging conversation ensued covering everything from A/B testing Dan’s shoe selection to the optimal experience at Starbucks with this central and cogent point, “Everything that can be optimized will be optimized.” And this holds for the entirety of the customer experience, leading to a growth in the use of experimentation, a growth in the number of experimentation professionals and as outlined at the top, a growth in ROI for those that follow the path.

Mae Jemison

Dr. Mae Jemison Talks Interstellar Travel with a Packed House

Lastly, the closing talk from Dr. Mae Jemison was a wonderful end-note to the programming. Beyond detailing her remarkable story and how she became the first African-American woman in space and was a character on Star Trek, her exhortation to have us all contemplate the possibility of interstellar travel serves as a powerful rejoinder that when it comes to the human spirit, nothing is truly impossible.

 

Optimizely DXE Report