Download our FREE ebook of 43 A/B testing case studies from the world's leading companies for test ideas and inspiration. Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program

In this episode of Test & Tell, our CEO, Claire Vo, covers the necessary knowledge, resources, and process you need to start a testing program for your ecommerce business. She covers the qualifications your site needs to meet, tools and team members, a 9-step testing process, and how to continue its success within an organization.

Listen to the Podcast

For ecommerce companies, the goal, of course, is to increase your bottom line and drive revenue. On your site, this means increasing revenue per visitor or purchase rate. But are you implementing a strategy to grow these metrics? That’s where A/B testing and optimization comes in.

A/B testing your ecommerce store pushes you to improve the site experience so you maximize the value of your existing users.  More and more ecommerce companies are investing in optimization by establishing a program that helps you A/B test consistently and achieve wins that result in compounded lifts.

What Qualifies You to A/B Test

Before you begin thinking about building a robust program, let’s make sure your site is a good candidate for A/B testing. Testing is only effective when your ecommerce store meets baseline requirements. This way, your experiments are able to reach statistical significance and give you confidence in the resulting data.

Enough Traffic – You’re going to need enough of this. We typically recommend that sites get at least 5K to 10K visitors per week in order to run a test through a full week duration.

Sample size & conversion rate – Ideally, you have a conversion rate of 3% with the goal of achieving at least a 10% lift. Why? Because the higher your conversion rate you have to work with, the less traffic required. This means a smaller sample size is needed for each experiment, allowing you to reach statistical significance faster.

There are many calculators to help you determine what you need. Each uses differing methodologies to calculate statistical significance:

If you don’t have a lot of traffic or a good baseline conversion rate, you will need to run your tests for a longer period of time.

Resources You’ll Need to A/B Test

Web Analytics

If you’re an ecommerce store, you probably have web analytics already properly set up. Whether it’s Google Analytics, KISSmetrics, or another tool, this helps greatly in getting to know your current customers and how they interact with your site. More on that later.

An A/B Testing Tool

So you’re going to need a platform to actually run your A/B tests. There are many great options out there today that are ready to use “out of the box”. Tools like Optimizely, VWO, A/B Tasty, and Kameleoon have visual editors that facilitate users who are less technically proficient. Experiment Engine is an Optimizely Technology Partner because of a strong integration between both platforms that enhance the entire testing process. Some ecommerce companies choose to develop an internal testing platform if they have the resources and time to do so.

A Cross-Functional Team

You’ll later find out in this article that testing requires a multi-step process that requires a diverse set of expertise. A team of resources that has the ability to execute each part of that process will make your testing much more efficient to cycle through. This is what a team looks like:

Analytics Expert – This person leverages that web analytics tool to learn more about your customers on your site, which leads to identifying testing opportunities.

Marketer – This person has knowledge about the incoming traffic channels, such as PPC campaigns or other events.

Designers & Developers – They build and develop your experiments so you’re able to run them.

9 Steps of the Ecommerce Testing Process

So you’ve made it this far – you’re bought in on A/B testing, your site is qualified, and you’ve acquired the necessary resources. It’s time to go through the testing process!

These 9 steps outline each part of your testing cycle:

Step 1. Identify Opportunities

Your store has multiple pages and a lot of products – where do you even begin? You’ll want to look at your web analytics and check out your site flow. Find areas (or touchpoints) where users are falling out. These pages are ideal starting points to A/B test.

Website Funnel

Anonymous example from Search Engine Watch of a website’s flow in Google Analytics.

Visitor Flow Fallout

Looking at the flow, the home page and info page have a lot of traffic. But they also have major dropoffs – making them candidates to A/B test.

Step 2. Generate a Hypothesis

After you’ve identified an opportunity, you need to come up with a hypothesis targeted at that touchpoint. This hypothesis should be something that could potentially improve the experience of the site.

For example, you’ve identified traffic coming in from a social media channel that is dropping off when they get to your product details page, so they rarely add an item to their cart. A hypothesis would be that the product details pages would be more effective if a social proof section was prominent so that these users become more confident in adding an item for purchase.

Step 3. Design the Test

At this point, if you’re using one of the testing tools we talked about earlier, they have visual editors or wysiwyg editors that allow you to easily create variations. Other ecommerce businesses may use the help of designers to create comps of variations.

Step 4. Code it Up

Your developers come in here and code up your variation so that it’s technically able to run on your site.

Step 5. QA Your Test

Make sure your experiment is set up properly by staging it and testing across browsers and devices. This helps ensure that your results are not affected by technical issues.

Step 6. Run the Experiment!

It’s time to push the experiment live! Hit the “run” button on your testing platform and wait until it hits statistical significance. (This is where those testing qualifications need to be calculated)

Step 7. Analyze the Results

Each testing tool provides experiment data, such as goal results, tracking, RPV, overall conversion rate, and more. Determine how your test affected your customers and gain learnings, whether or not it’s a win.

Optimizely Test Results

Optimizely’s goal module showing test results.

Step 8. Share those Learnings

Learnings from your tests can benefit the entire company across departments. Be transparent about the optimization process and improve every part of the business using customer insight (even from losing tests).

Step 9. Implement Winning Variations & Repeat Cycle

If your variation resulted in a conversion lift, implement it on your site to start getting gains! You developers can help you execute these changes while you continue to optimize by immediately repeating this testing process. By constantly tweaking for improvements and maximizing testing capacity, your business will achieve compounded gains.

How to Promote Optimization Within Your Organization

Show proof to leadership – After a few wins, prove the effectiveness of the program by sharing the data with leadership. It’s best to report these to executives by showing how testing directly drives ROI.

Build a comprehensive process with resources – The resources and 9-step process we’ve listed is the start to establishing a robust ecommerce testing program. To continuously optimize, you’ll need to overcome challenges as you repeatedly cycle through experiments. Help can come from hiring resources to work in-house, bringing on an optimization agency, or using new platforms & services that provide an all-in-one solution, like Experiment Engine.

Evaluate the success of the program – First, you should be collecting and documenting all of site data, such as overall conversion rates, funnel fallout, order values, and more. But metrics for your program is just as important. Dedicate detailed documentation of test data, learnings, and operations. Ask yourself if you’re happy with testing velocity and if you’re scaling with capacity as your business or website grows. Measure your testing output on a weekly, monthly, and quarterly basis. Record win rate and calculate how your bottom line is affected. Experiment Engine does this for customers by projecting incremental value added to revenue.


  • A/B testing benefits your entire business, from revenue growth to acquisition costs and product lines.
  • Start optimizing by first qualifying your ecommerce site, then acquiring resources needed to execute testing.
  • Testing is a process with 9 basic phases; that process should be continuously cycled to establish a program in order to maximize the value of testing.
  • Only when you scale 1 or 2 tests per month into a program are you able to incrementally grow your ecommerce business.
  • Always evaluate on a test by test basis first; then, evaluate the success of your entire program.

To get the resources you need to start or scale A/B testing, request a demo with Experiment Engine.

Optimizely X