Download our FREE ebook of 43 A/B testing case studies from the world's leading companies for test ideas and inspiration. Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program


When you’re starting your experimentation journey it can be tough to figure out where to start. I often hear from my customers that they’ve hit a rut and are short for hypotheses that can keep program momentum going. These are not rare places to be! The lifeblood of a program is having hypotheses to test against and we’ve seen quite a few over the years at Optimizely. And even the ones that don’t take a lot of development work can still provide great returns!

On Optimizely’s strategy team we have seen a set of simpler to set-up experiments that work quite well, as part of a strong hypothesis, across many customers who are looking to solve a similar problem. I can vouch for these being simpler in set-up since they are solutions that even I can accomplish within Optimizely Web (trust me, my development skills are pretty much nonexistent). Keep in mind that it is always a first best practice to run a hypothesis workshop to ensure the solutions are to YOUR customer problems. 

To help you keep your program velocity high, I’ve collected these easy to implement experiment ideas that have the potential for high impact and categorized them by vertical. For each, I provide the solution, goal, and rationale in our set of simple hypotheses. You’ll need to fill in the customer problem based on what you’ve identified from your data. Maybe these all apply, maybe not! Be sure to check based on what information your customers have provided you before prioritizing the hypothesis. Let’s dive in!

Retail

1. Overall Hypothesis: If we highlight the PDP (product detail page) attributes (e.g. size and color), then we will increase add-to-cart rate, because it will draw attention to the initial actions a visitor needs to take before adding-to-cart

  • Goal: Increase Add-To-Cart
  • Solution: Highlight the PDP (product detail page) attributes (e.g. size and color)
  • Rationale: It will draw attention to the initial actions a visitor needs to take before adding-to-cart

2. Overall Hypothesis: If we reorder the “hooks” on PDPs (e.g. highlighting reviews versus product specs), then we will increase add-to-cart rate, because we will put the best hook in front of more visitors upon scroll

  • Goal: Increase Add-To-Cart
  • Solution: Reorder the “hooks” on PDPs (e.g. highlighting reviews versus product specs)
  • Rationale: The best hook is in front of more visitors upon scroll

3. Overall Hypothesis: If we highlight the abandoned cart in the header for a return visitors, then we will increase checkout starts, because it will draw attention for a visitor to pick up where they left off easier

  • Goal: Increase checkout starts
  • Solution: Highlight the abandoned cart in the header for a return visitors
  • Rationale: Draws attention for a visitor to pick up where they left off easier

 

4. Overall Hypothesis: If we add “tags” (e.g. New!) to certain products on PLPs (product listing pages), then we will increase PDP views, because it will create additional curiosity for the visitor

  • Goal: Increase PDP views
  • Solution: Add “tags” (e.g. New!) to certain products on PLPs (product listing pages)
  • Rationale: Creates additional curiosity for the visitor

5. Overall Hypothesis: If we highlight account benefits further up funnel (e.g. banner on the homepage for free shipping), then we will increase PDP views, because it will make our value props in view for more visitors

  • Goal: Increase PDP views
  • Solution: Highlight account benefits further up funnel (e.g. banner on the homepage for free shipping)
  • Rationale: Makes our value props in view for more visitors

 

Travel

6. Overall Hypothesis: If we reorder fare classes in search results to show best value first, then we will increase route selection and AOV (average order value), because more visitors are looking for the cheapest fare

  • Goal: Increase route selection and AOV (average order value)
  • Solution: Reorder fare classes in search results to show best value first
  • Rationale: More visitors are looking for the cheapest fare

7. Overall Hypothesis: If we highlight additional purchases along the booking path, then we will increase AOV, because it will remind a visitor we can provide more to them than just a flight

  • Goal: Increase AOV (average order value)
  • Solution: Highlight additional purchases along the booking path
  • Rationale: Remind a visitor we can provide more to them than just a flight

8. Overall Hypothesis: If we localize the homepage imagery to direct destinations based on time of the year or weather (e.g. show Chicago visitors imagery of South Beach), then we will increase route search rate, because it will entice visitors more upon entry

  • Goal: Increase route search rate
  • Solution: Localize the homepage imagery to direct destinations based on time of the year or weather (e.g. show Chicago visitors imagery of South Beach)
  • Rationale: Entices visitors more upon entry

9. Overall Hypothesis: If we skip steps (e.g. skip a review screen after selecting the route), then we will increase bookings, because it will reduce clicks we require of visitor before they can pay us

  • Goal: Increase bookings
  • Solution: Skip steps (e.g. skip a review screen after selecting the route
  • Rationale: Reduce clicks we require of visitor before they can pay us

 

10. Overall Hypothesis: If we promote our packages via a banner, then we will increase route search rate and AOV, because it will remind a visitor we can provide more to them than just a flight 

  • Goal: Increase route search rate and AOV
  • Solution: Promote our packages via a banner
  • Rationale: Reminds a visitor we can provide more to them than just a flight 

B2B / Technology / Financial Services

11. Overall Hypothesis: If we use imagery of people using the product as opposed to just product or conceptual product imagery, then we will increase product page views, because it will feel more real for visitors

  • Goal: Increase product page views
  • Solution: Use imagery of people using the product as opposed to just product or conceptual product imagery
  • Rationale: Feels more real for visitors

 

12. Overall Hypothesis: If we reorder our pricing / plans table based on prior browsing behavior (e.g. viewed this tier or pricing, show this first in next session), then we will increase lead form views, because it will be in a more relevant order for most visitors

  • Goal: Increase lead form views
  • Solution: Reorder our pricing / plans table based on prior browsing behavior (e.g. viewed this tier or pricing, show this first in next session)
  • Rationale: More relevant order for most visitors

 

13. Overall Hypothesis: If we add testimonials to our product pages, then we will increase lead form views, because it will provide social proof to back up our claims (shameless plug!)

  • Goal: Increase lead form views
  • Solution: Add testimonials to our product pages
  • Rationale: Provides social proof to back up our claims (shameless plug!)

14. Overall Hypothesis: If we make our accreditation more prominent in our contact or payment steps, then we will increase purchase conversion or lead conversion, because it will ease concerns about giving us information

  • Goal: Increase purchase conversion or lead conversion
  • Solution: Make our accreditation more prominent in our contact or payment steps
  • Rationale: Eases concerns about giving us information

 

15. Overall Hypothesis: If we make our navigation naming more relevant (e.g. audience focused vs action-oriented vs product verbiage), then we will increase pageviews per visit, because it will be less how we talk and more how our visitors talk

  • Goal: Increase pageviews per visit
  • Solution: Make our navigation naming more relevant (e.g. audience focused vs action-oriented vs product verbiage)
  • Rationale: Less how we talk and more how our visitors talk

Media

16. Overall Hypothesis: If we change / promote content categories on homepage from prior browsing behavior, then we will increase pageviews per visit, because it will be more relevant for a visitor

  • Goal: Increase pageviews per visit
  • Solution: Change / promote content categories on homepage from prior browsing behavior
  • Rationale: More relevant for a visitor

17. Overall Hypothesis: If we duplicate or move subscription Call-to-Action in-line on content pages, then we will increase subscription page views, because it will be a better reminder as they consume the value we provide

  • Goal: Increase subscription page view
  • Solution: Duplicate or move subscription Call-to-Action in-line on content pages
  • Rationale: Better reminder as they consume the value we provide

18. Overall Hypothesis: If we highlight trending or important stories on the homepage, then we will increase pageviews per visit, because it will create curiosity with the visitor

  • Goal: Increase pageviews per visit
  • Solution: Highlight trending or important stories on the homepage
  • Rationale: Creates curiosity with the visitor

19. Overall Hypothesis: If we edit the placement and design of recommended content, then we will increase pageviews per visit, because it will encourage the visitor to explore more

  • Goal: Increase pageviews per visit
  • Solution: Edit the placement and design of recommended content
  • Rationale: Encourages the visitor to explore more

 

20. Overall Hypothesis: If we provide an indication of how long content would take to consume, then we will increase pageviews per visit, because it will create an expectation upfront

  • Goal: Increase pageviews per visit
  • Solution: Provide an indication of how long content would take to consume
  • Rationale: Creates an expectation upfront

All Industries

21. Overall Hypothesis: If we test a landing page with symmetrical messaging (ie mirroring language from an ad or email) versus the typical page where you direct paid, email, or other campaign traffic, then we will decrease cost-per-acquisition, because it will be more focused for a visitor

  • Goal: Decrease cost-per-acquisition
  • Solution: Symmetrical messaging (ie mirroring language from an ad or email) versus the typical page where you direct paid, email, or other campaign traffic
  • Rationale: More focused for a visitor

 

22. Overall Hypothesis: If we provide geo-specific messaging on the landing page based on the interests of that region, then we will decrease bounce rate because it will be more relevant upon entry

  • Goal: Decrease bounce rate
  • Solution: Provide geo-specific messaging on the landing page based on the interests of that region
  • Rationale: More relevant upon entry

 

23. Overall Hypothesis: If we add sample text to form fields to provide guiders for what information is needed, then we will increase form completion rate, because it will make it clear to a visitor what they need to provide

  • Goal: Increase form completion rate
  • Solution: Add sample text to form fields to reinforce information needed
  • Rationale: Make it clear to a visitor what they need to provide

24. Overall Hypothesis: If we reorder homepage (or other heavy content pages) secondary content blocks, then we will reduce bounce rate, because we will put the best hook in front of more visitors upon scroll

  • Goal: Reduce bounce rate
  • Solution: Reorder homepage (or other heavy content pages) secondary content blocks
  • Rationale: Putting the best hook in front of more visitors upon scroll

———————————

Are these questions and solutions you have thought about for your site experience? If you have tried them, what have you learned and what iterations did you leverage on your wins? 

A lot of these are even simpler to put together using Optimizely extensions. You can use what Optimizely has already provided in its library or work with a development resource on your team for a one-time build you can continually reuse. 

*Please note that depending on your website architecture, that some of these experiment ideas may need some developer love*

Optimizely X