Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more! Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program

X

Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more!

Get It Now




When A/B testing, companies are faced with multiple approaches to testing; a business may choose to test major changes that significantly affect their visitors’ experience or make smaller, incremental changes that continuously evolve the web page. Each method has its advantages and drawbacks and is used depending on the company’s optimization strategy, goals and customers.

Radical Testing

The philosophy behind large-scale changes to a website is direct: implement a major change and receive major positive effects in return. This approach, which some may call “radical” testing, uses large-scale changes to garner significant results every testing cycle. But while a company may see large conversion gains with each big step, there is also a high probability of failure, too. Therefore, it is a high risk, high reward situation. But if an existing website has a user experience that cannot be salvaged, a radical approach may be applicable.

Radical testing implements a major change less frequently for the potential of significant gains, but also the high probability of failure.

Radical testing implements a major change less frequently for the potential of significant gains, but also the high probability of failure.

 

Benefits of Radical Changes

  • Major Results – If a company implements changes that affect more than half of the page’s elements or user experience, there is potential of significant conversion rate results.
  • New Local Maximum – If a radical test is successful, the major changes could make big strides, in terms of results, and push for a higher ceiling.

When a Radical Approach is Needed

  • Content is Off Brand – If the company’s website is not in line with business goals, then it will need to be scrapped completely or radical tests can help revamp entire sections.
  • Architecture Prevents Conversions – A company will need a major change if their website’s experience does not guide users through the conversion funnel.
  • Major Functionality Issues – When there are multiple problems with the functionality of a company’s website, it is best to fix them in a large-scale change.

Risks of Radical Changes

  • Inefficient Time Investment – Because radical tests take more time to prepare, it will take longer to define requirements, develop, and implement. Therefore, it will take longer to find winning variations.
  • Difficult to Pinpoint Cause-Effect – If a radical change increases conversion rate, it may be unclear what specifically caused the result, providing less actionable insight for future tests.
  • Lower Than Possible Conversion Rate – After a year or prolonged period of testing, overall conversion rate could be lower than the overall results of an incremental approach.
Although there are times when radical changes produce better conversion rates, incremental changes are more likely to give significant results over time. This simulation of both approaches shows the high reward of a radical change, but also the high probability of test failures in a 12 month period.*

Although there are times when radical changes produce better conversion rates, incremental changes are more likely to give significant results over time. This simulation of both approaches shows the high reward of a radical change, but also the high probability of test failures in a 12 month period.*

 

Incremental Testing

Incremental A/B  testing takes advantage of making continuous, small-scale changes. It allows organizations to learn more details about visitors, their behavior, and which specific elements convert. By analyzing an existing website for its strengths and weaknesses, a company can begin A/B testing specific elements to determine their effect and implement small-scale changes. For example, if a landing page is underperforming, one can gather its data and assess which elements may be hindering conversions. If you hypothesize that the written copy is too long and indirect, try running an A/B test to split visitors between the existing landing page (control) and a version with shorter, concise copy (variation). If the variation wins with a higher conversion rate, implement the change and move on to testing the next variation idea, which, hypothetically, could involve the formatting of that written copy to further optimize its use. Doing this over time on many elements will help evolve an entire website incrementally, while measuring each small step to constantly optimize for conversions. Think of this approach as a cycle of continuous improvement!

Incremental testing implements continuous, small-scale changes for compound growth of conversion rate.

Incremental testing implements continuous, small-scale changes for compound growth of conversion rate.

 

Benefits of Incremental Changes

  • Quick Testing Cycle – With an incremental approach, generating test ideas and implementing changes can be completed quickly because cycles will be shorter. Although this means the change in variations could seem small, proper test setup and impact-focused test ideas can bring significant results.
  • Higher Win Rates – Because there is actionable data, incrementally testing ideas that aim to increase conversion rate can result in a higher win rate.
  • Target Effective Element Changes – Since specific elements are being tested, it is typically easier to recognize which changes are actually helping create winning tests.
  • Understand Customer Behavior – Because the insight is more detailed, testing these changes will allow you to better understand the behavior of customers.
  • Maximize Conversion Rate Growth – Taking less time to implement tests lets you run more tests per month, increasing the odds of a win. More frequent wins that build on top of each other create a compounding effect, where the increase in conversion rate is maximized.
A look at the expected trend of conversion rate when looking at both approaches over a 12 month period, assuming a 3% starting rate, 500,000 in monthly traffic, and tests running on 20% of traffic at one time*

The expected trend of conversion rate when looking at both approaches over a 12 month period, assuming a 3% starting rate, 500,000 in monthly traffic, and tests running on 20% of traffic at one time*

 

Projected ROI of Incremental Testing

So how does incremental testing affect revenue? The expected results of an incremental approach over time can directly affect performance, while providing actionable insights. Due to a higher testing win rate and compounded growth, increased conversion rate will yield a greater ROI, producing more additional revenue and customer engagement than a radical approach. For example, a company that has a current conversion revenue of $2,250,000 could project these ROI trends (based on a monthly traffic of 500,000 with testing run on 20% of visitors):

The projected ROI is calculated using on a company spending $200,000 in annual salary for a testing team. The starting conversion revenue is $2,250,000 and they are testing on 20% of their monthly traffic of 500,000.

The projected ROI is calculated using a company spending $200,000 in annual salary for a testing team. The starting conversion revenue is $2,250,000 and they are testing on 20% of their monthly traffic of 500,000.

 

As graphed, the significant results of incremental testing over time can positively affect the company’s revenue when growth is compounded month by month. Each incremental test optimizes the last winning iteration and continues that trend, resulting in a geometric return. So although there are circumstances when radical testing is needed, an incremental approach helps achieve a higher ROI in the long run. For companies with higher traffic and more visitors to test, the potential of compounded revenue growth could be even greater.

How FreightPros.com Used Incremental Testing to Increase Overall Conversion Rates

FreightPros faced the challenge of increasing conversion rates for quote submissions and demo requests. To solve this, they engaged Experiment Engine’s platform and marketplace to generate and test incremental variation ideas.

By continuously implementing multiple incremental tests, they were able to A/B test numerous small-scale changes to find winning variations quickly. In October, FreightPros tested the positioning of the company’s value propositions, which increased their conversion rate 20%. Next, FreightPros experimented with improved typography and form UX to achieve another 10% increase. The big impact of these small-scale variation ideas were tested one after another, week to week, and led to these implemented changes:

 

FreightPros saw increases of conversion rates, month to month, and were able to identify which incremental changes directly caused the results.

FreightPros saw increases of conversion rates, month to month, and were able to identify which incremental changes directly caused the results.

 

The result? An overall increase of 32% in conversion rate in 2 months due to incremental testing. This approach also provided a number of valuable customer insights that were specific enough to apply to other landing pages, which helped optimize conversions even more.

The Choice: Radical vs. Incremental Testing

Radical testing is important and necessary when full-scale changes are in the cards. This approach has the potential of returning significant results in a short amount of time. However, the risks can outweigh the reward, especially when considering the overall gains of an incremental approach. In contrast, testing smaller changes is the better approach for well-functioning websites looking to optimize. Each implementation is a measured evolution that gives a higher probability of long-term success. By testing incrementally, companies better understand customers, continuously optimize, and learn from specific changes along the way. Is your testing program and its approach effectively meeting those goals? (Help yourself with these testing tips)

 


*Graphs use a starting conversion rate of 3% on monthly traffic of 500,000. Calculations incorporate incremental tests that are implemented weekly, have an average  win rate of 25%, and produce an average lift of 25% on conversion rate. Radical tests are implemented monthly, have an average win rate of 15%, and produce a average lift of 40% on conversion rate. Both testing approaches are applied to 20% of the monthly traffic.

Optimizely X
comments powered by Disqus