4 Steps for Creating a Problem-Centric Split Test Hypothesis

Hypotheses are the statements that clarify our thinking leading up to a test. They make a clear prediction about how changes in customer experiences will impact a given goal. They provide a rationale for why customers might react positively or negatively to that change.

In other words, hypotheses are absolutely essential to getting value from every experiment you run. If you’re just getting started with A/B and multivariate testing and are looking for a guide to writing your first test hypothesis, start with this post.

Why problem-centric hypotheses are important in testing

But even with standard hypothesis best practices, there are opportunities to elevate your testing, align your hypothesis with challenges that are core to your business, and increase the odds of producing a positive result. It starts with identifying problems.

Why focus on learning to create a problem-centric hypothesis? In working with customers, we’ve found that a lot of companies dive straight into conducting tests, without first having an understanding of what they want to achieve, or which problem(s) they are actually looking to solve.

The following steps will help you craft your hypotheses carefully and set you on the road to testing success.

1. Solve real problems

The most impactful hypotheses are those that are aligned to your company metrics. They help connect the process of learning about customers’ behaviors with higher-level goals to help grow the business. What are your KPIs and company-wide goals? By designing tests and experiences that impact these metrics, you focus your efforts on questions that matter.

A great hypothesis should begin with the problem, not the solution. Examine and try to understand the difficulties your visitors are experiencing, which buttons and pages are being ignored, and then think about how it can be solved. That is the substance of a winning experiment, not testing colors or placement of CTAs on a whim.

For example, are your visitors being bombarded with too many options and too little content? If the conversion rate is low, it could be because there’s too much information to be useful to the visitor on a given page. The solution could then be to remove non-critical content in order to minimize distractions and make the path to checkout or product discovery much clearer.

2. Understand revenue leaders

The next step, and arguably one of the most important, is to understand what you are looking to accomplish with your testing strategy. Is the goal to increase subscriptions, revenue, or leads? Where does your main source of revenue come from? You can’t determine strategies or tactics for testing until a goal has been decided upon.

For example, if your goal is to increase revenue for an e-commerce website, you might want to start by examining your cart checkout rate. Strategies to achieve this goal could include emphasizing the primary call-to-action (CTA), minimizing distractions in each checkout step, or building a sense of urgency for customers to complete the checkout process.

Testing tactics could then be changing messaging, removing non-critical content, and promoting temporary offers. The results of these testing tactics would all contribute to generating a better understanding of what changes can be made to increase the checkout cart rate and therefore increase revenues.

3. Strive to test all changes

Testing requires freedom, and freedom will only come once a testing culture has been established within a company. Many of our customers have different obstacles to overcome on their journey to developing a testing culture. For instance, a financial services company we work with must have every customer experience change approved by senior staff. On the opposite end of the spectrum, there are companies that have so much testing freedom that senior teams would consider changing the company’s branding if it increased conversion.

It’s important to examine all possible areas for change, leaving no stone unturned. It is only once all possible tests have been conducted that it can be decided where the changes on the website should be made.

4. Test multiple variables or variations

Most optimization programs are started with experiments that only include two variations on a single variable: red or blue, button or no button, an image with a model or without one.

However, as organizations expand the scope of their experiments and test more variables, they see dramatic improvements test performance—we recently found that testing five variations can nearly double your split testing win rate! Don’t be afraid to explore several ideas within one test to see which can generate the biggest improvement.

For more resources on crafting hypotheses, check out the following articles:

• Design a hypothesis that drives your business goals
• Use analytics data to inform your hypothesis
• Download the hypothesis worksheet in the Testing Toolkit

The insights above were presented at an Optimizely User Group in London. User groups are held periodically in major cities in the United States and Europe, and are open to optimizers regardless of which testing or personalization platform they use.