Download our FREE ebook of 43 A/B testing case studies from the world's leading companies for test ideas and inspiration. Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program

X

Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more!

Get It Now




Here at Optimizely, a core part of our offering rests with the intellectual property we’ve built up over the years by running thousands upon thousands of experiments for hundreds of companies. Often on the front lines of those programs lay our Strategy Consultants who are true fonts of knowledge when it comes to how to structure, launch and grow your experimentation programs and establish an experimentation culture. In the first of a series we sat down with Alek Toumert, Lead Strategy Consultant, to discuss how he came to experimentation and share some insights he’s discovered along the way. 

How did you get started with Experimentation? 

I was working at the American Medical Association and on my first day they said, “You have a call with Optimizely. We’re buying this A/B Testing software and your team is going to figure out how to use it and lead the program.” It quickly became the favorite part of my role. It was an invaluable experience as I got the building blocks of what would help me later in my experimentation career. 

In my role on the digital analytics team I had to understand internal stakeholders’ business, create proper tracking for their initiatives and understand what behaviors were impacting their most important constituents. As the AMA was an organization slower to adopt data at the time, this was a great way to teach people how to interpret data across their products.

Alek Toumert

Alek T. Leading a Strategy Session

What is one of the earliest experiments from which you saw results?

At the time the way the AMA was structured making changes to their properties was slow, and about three months in my manager and I heard that they were going to completely change the homepage of the website without testing. We got the CEO to attend one of our optimization meetings and was able to demonstrate the value of running a test when you make such a large change. We ended up being able to advocate loudly enough to run a redirect test between the old and new homepage, which spurred a lot of subsequent testing efforts. 

How did you come to Optimizely?

I was asked to speak at Opticon in 2015 on taking a program from nothing to something. A few years later I got offered a role as a strategy consultant and have been doing it ever since. 

Tell us a bit about your current role?

After a customer signs on to use Optimizely we are part of the post-sale engagement. In the beginning my main role was to be a thought leader in experimentation. Essentially, walking someone through from hypothesis creation to analysis and all the trials and tribulations you might experience between. A lot of my work is helping organizations grow and develop change management muscles and flexibility with experimentation, and how that plugs into workflow. 

What are some of the learnings that have come from your three years in the role?

It seems obvious now but Executive Sponsorship is key. You need more than a practitioner to truly build a successful experimentation program. You also need to see if you can find the right people to spearhead the program, often they exist but might be scattered throughout the company which is why you need a good RACI to achieve success. 

What’s a common pitfall early experimenters have?

Not spending enough time in experiment design and supporting analysis practices with enough resources. You need to understand your problem intimately, define experiment metrics correctly, and have some guardrails on how you make decisions. This should help increase your velocity. Increased velocity = more learnings. In the webinar I recently did on checkout flow, I really hit on two of these points.

Understand your page level metrics deeply and how they interact, correlate, and cannibalize each other. Is getting a user to scroll to the product reviews more valuable than viewing multiple product images on your PDP page?

In your hypothesis creation spend the most time understanding the problem statement. If it’s not clearly defined you could be brainstorming irrelevant solutions. Ensure your problem statement is backed up by multiple data points. 

User Metrics and KPis

Understanding How User Actions Drive KPIs

What’s a goal for you as you look out over the next year or more?

I really want to get even better at using Optimizely’s own data. As partners we have access to all the experiment data and knowing how something worked in the past, really can help plan for the future. I want us to continually improve how we help our customers and leaning on the historical experiment data is one way to do that. 

For more product and implementation knowledge specifically on ‘How to Optimize Your Checkout Flow and Engage Your Customers’ watch Alek’s webinar by following this link.

 

Optimizely DXE Report