Holiday Testing: Challenging the Negatives

“I won’t be able to generalize my results from testing during the holidays.”

“Holiday testing? Our dev team would never let that happen!”

These are just a few of the many excuses I’ve heard from experimenters when talking about holiday testing. I take a deep sigh and listen, trying my best not to interrupt. The fact is, testing during the holiday season has far more benefits than it does drawbacks, and even the drawbacks aren’t necessarily drawbacks at all.

Let’s take a deeper look.

“Too much traffic will cause bugs in my experiments.”

Actually, if you do it right you won’t have to worry about that. For starters, begin working with your counterpart teams early- the earlier the better. Once you have your teams in place, it comes down to building in a strict process and QA governance/practices:

• At least one month before the holiday, all counterpart teams should be aware of the things you want to test. Holiday promotions are often shorter in length, so be sure to align with your marketing, merchandising, and creative teams to understand the holiday promotion calendar.
• Prepare test plans for each promotion. Your test plans should include your hypothesis and the expected outcome or result. These test plans should also include contingency plans, like what we to do if a test is losing, or what is an acceptable amount of time/loss before we make a decision? IT might be also be worth using a decision policy during the holiday (contact your Customer Success Manager for examples).
• Work alongside your developer/QA resources and adapt to their process. This will help build up the trust necessary to run tests over the holiday.
• At least one week before each test launches, send an email to all stakeholders with links to all variations/experiences so they can see what a customer sees. Capture their feedback and follow up on any open items.
• Send launch email communications out to all your stakeholders letting them know an experiment is live.
• Due to the sensitive nature of holiday experiments, analyze the results of your test more frequently. Have a daily check-in and provide update emails to major stakeholders daily.
• If you are an Optimizely user, consider building auto stop/starts using Optimizely’s rest API. For example, If the Primary Metric of success for any variation falls beyond a certain percentage lower than Control, programmatically pause your experiment with the rest API.

More traffic = more results = More learnings about your customers

Now that the worry of bugs is out of the way, you can focus on the huge benefit of having a higher number visitors than usual on your site. Those few months in the fall and winter are the busiest time of year for online shopping, and the more visitors you have, the more you can learn! Check out our 2015 benchmark study which shows that the leading metrics that teams use to measure success are conversion improvements from experiments. As stated in the study, the faster the pace of experimentation, the more opportunities there are for learning, conversion and revenue gains within an organization. With higher traffic, you’ll not only see higher ROI returns, but any unsuccessful changes will emerge quickly, allowing you to roll back those changes saving potential lost revenue.

“I won’t be able to generalize my results from testing during the holidays.”

Perhaps that’s true, but if you think about it, you can’t really generalize results from testing during every other season, either. In fact, I’ll take it one step further: if you are only testing during non-holiday timeframes, you don’t really have the full picture. Consumers are smart, and they have expectations about their buying experience that are based on the transactions they have made in the past. You’ll want to use their data to maximize findability, increasing the odds that they’ll discover something they can’t live without. It’s impossible to argue against the fact that sales and conversions are somewhat based on impression rate. You know that saying, “First impressions are everything?” The better your impressions, the better your conversion rate.

Holiday testing? Our dev team would never let this happen!

It’s true- you might come face to face with a very resistant development team when approaching them with the idea of testing during the holidays. However, having an open communication plan will help get the whole team on board. It’s so important during this stressful time of year that everything is planned out and discussed, from your goals to developing your strategy as mentioned earlier. Communication can happen in more ways than one, including email, Q&A sessions, town hall meetings, shared documents, and more. As long as everyone is on the same page throughout the whole process, you’re more likely to have buy-in from everyone, including your development team.

Learnings from this year’s holiday tests become the strategies of tomorrow.

Testing during the holidays will inform next year’s plan of attack when it comes to messaging, headlines, and promotions. This point is huge, as everyone views the holiday season as the most important time of the year financially. So with that said, I say, why not test everything?. If you don’t, how are you going to know what worked? How are you going to improve and make your promotions more successful next year? You can bet that your executive will want to see year-over-year growth the next holiday… but how do you plan to understand and improve if you are afraid to test and learn?

The hesitation of holiday testing is understandable, but gathering actionable data and providing a great visitor experience should be more than enough to reassure you that testing during the holidays is worth it. If that’s not enough to convince you, keep in mind that what you learn from holiday testing will set you up to succeed in the future, greatly increasing your ROI both in revenue and in resources. If you have an action plan, open communication, and well thought out experiments, you’ll be looking at a great holiday season with happy visitors, an excellent ROI, and data you can use to plan for the next holiday season.