9 Tips To Help You Run A/B Testing

Use quantitative and qualitative data to guide tests

Analytics data will show you how your visitors are behaving, but qualitative data will tell you why they are acting so. A website survey can give deeper insight into the reasoning behind their exit. For example, with quantitative data, you may see visitors leaving before signing up with their email, but qualitative data will tell you it was due to a long submission form, not the color of the “Submit” button as you might have assumed.

Find out what visitors are searching for within your site

Google Analytics also provides useful data to guide your test ideas with its site search data. If a significant number of users are looking for your pricing page, this may mean that a page could benefit from making that content available or more prominent. This is an easy and direct way to satisfy your visitors’ behavioral needs while guiding them towards conversion.

Run an A/A test before your A/B test

To ensure that you will be able to run your A/B test without a hitch, it’s important to test your control against your control. If you see a significant difference in your results, it means your test is not set up properly and you will have to check the functionality of your software. Therefore, an A/B test run the same way would be rendered inaccurate. On Experiment Engine, an A/A test is always used to protect the validity of your split testing.

Don’t forget existence testing

You may be excited to add an element to your page to test, such as a button, headline, or even content. However, your visitors may not be converting because there are already too many elements on the page. An existence test will test your original page against versions that have removed prominent existing elements. This way, you are able to establish the essential parts before testing different styles or new elements.

Collect enough samples for statistical significance

The goal of A/B testing is to collect actionable data to help you optimize through incremental changes. Therefore, the usefulness of that data is dependent upon the sample size. So if your variation (test idea) yields winning results over your control (original), you will not be confident that this result will occur when changed permanently if the sample size (number of users submitted to the A/B test) was too small. Make sure enough visitors went through your test will help avoid a false positive or false negative that would incorrectly drive your actions.

Run a test for its entire needed duration

Running your test for its needed duration ensures statistical significance and confidence. Calculate how much time your test needs and make sure to complete it accordingly. Ending a test early or interpreting premature data could lead to misled insight. Even if you reach your sample size early, it is possible to miss relevant data that is dependent on time. For example, the number and type of users (samples) who visit your site may differ on certain days of the week.  Of course, you will not want to go too long, either, and delay implementation of improvements (or reversion back to the control if your test did not win.)

Filter out certain outside influences

If you are aware of any influences that may skew your data, you will need to avoid putting it into your test. This includes any special occurrences that would affect your traffic. So if you are running an e-mail campaign or paid ad that will spike up the number of visitors, segment that traffic from your test so your end results are a better representation of your average user.

Use the proper tool and platform

There are a few great testing platforms that can help move you through the A/B testing process. Previously, heavy design and coding resources may have been needed, but tools like Optimizely and our own Experiment Engine technology can alleviate those needs. These tools will also guide you by making recommendations to achieve statistical significance and collect the relevant data. To simplify it further, our platform uses expert optimizers to provide informed, strategic test ideas to push you to continuously and intelligently A/B test.

Continuously run tests to truly optimize

So you have completed your first test or have done a number of split tests already…now what? Keep going. The point of optimization is the continuous improvement of your website. It is great news if you have already been implementing changes based on winning test results, but if you constantly challenge yourself, your business, and your website, you will keep improving that conversion rate. We realize that generating a steady flow of test ideas may be difficult, which is why we started a marketplace of optimization experts to help. We hope the data-driven side of you is inspired to keep testing.