Download our FREE ebook of 43 A/B testing case studies from the world's leading companies for test ideas and inspiration. Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program


When the holidays rolled around in 2020, the shopping industry looked very different, as there was less bustle in the stores and more bustle on the keyboards. Companies had to adapt to the increase in traffic volume and demand for online shopping. During the holiday period, Optimizely customers leveraged all their experimentation capabilities to create a seamless shopping experience and expand shipping needs during the ongoing pandemic.

The increase in holiday shopping done online was 32% in 2020 compared to 2019, nearly double any year-over-year increase seen in the five years prior (it only increased by 16% from 2018 to 2019). Accenture noted from a survey that more than half the country had opted out of in-store shopping altogether. These types of data points are clear indicators that companies must get even further ahead of their holiday planning to capture more of this market.

When our Strategy team looked back at the tactics our customers leveraged the most during the 2020 holiday period, there were a handful that are still applicable for 2021 as we get closer to holiday planning. But there’s always room for improvement on these.

Align your marketing spend

  • Ad impressions increase by nearly 50% during a typical holiday period. In the most competitive time of the year, companies need to spend money to acquire and retain customers. Countless hours are spent to differentiate the advertisement spending with unique criteria and preferences of consumers. That differentiation should be applied to your shopping experience as well.
  • Leveraging Optimizely’s personalization capabilities, companies were able to align the imagery, messaging and value prop seen in external advertisements to the users’ on-site experience. This is an easily actionable strategy because that collateral that is used externally is all you need to execute this on-site.
  • This strategy can influence three key metrics: bounce rate, purchase conversion and cost-per-conversion. If you already have type of marketing implemented, try it before the holiday season so you can plan ahead.

Set expectations

  • In 2020, almost everyone was on the receiving end of a late package, as the exponential increase in the volume of gifts sent online congested post offices across the country. But communication goes a long way. Despite delays beyond your business’ control, ensure your customers have the ability to track their packages so they know that they arrive at their respective destinations safely as you increase the likelihood of them becoming a repeat customer.
  • Free shipping is a great value proposition, but ensure you are optimizing how often and where you communicate with your users on what that entails for delivery timeline. Allowing opt-in/out of free shipping later on in the checkout process is a great way to increase conversion rate and potentially average-order-value. Free shipping as a persistent banner early on in the journey can always increase performance on top-of-the-funnel metrics but isn’t what necessarily drives home the final conversion.

Reinforce, reinforce, reinforce

  • The opposite end of the Set Expectations coin is that those value props like free shipping really matter. A lot! What may have enticed a user to your site or to take an initial action on the site can easily be forgotten among an array of products or multiple colors on a product detail page (PDP).
  • When you have those value propositions that are accessible to everyone, make sure they are always in view across the primary shopping journey. Some of our favorite places on which customers apply value propositions like free shipping are:
    • Near attribute selection (e.g., color and size) on PDPs
    • In persistent banners across the site
    • In image hover descriptions on product listing pages

Create more “saves”

  • As window shopping becomes more literal to the online shopping experience, we must accept we can’t capitalize every user in their first visit. In fact, we would recommend looking to see what your average sessions-to-conversion rates are across the site during the weeks leading up to the holidays. And how that may differ by channel or entry point of the site.
  • Once you figure out those main combinations of window shopping, look to create unique experiences to allow a user to save those moments where they start to look at another window (site).
  1. Save for later and email myself functionality.
  2. An easier way to bookmark a page.
  3. An opt-in to reenter the site on the same page they were looking when they return.
  4. Play around with these save options.

Leverage your loyalty

If you have created a delightful enough shopping experience that a user converts, be sure to fortify these types of moments and promotions in December.

What is great about using experimentation for the holiday period is that the holidays only last a few weeks. This leaves most of the year to examine some of these strategies first before going all in during the holidays. Since you may have limited resources as well to build experiments, testing some of these in advance can help you prioritize where to leverage those resources once November and December roll around.

What strategies are you and your business carrying over from 2020 to ensure 2021 is a successful holiday season? How much did you use experimentation in 2020 versus personalization?

Optimizely X