Multivariate testing is a testing method whereby all possible combinations of a change are tested simultaneously. This method of testing can lead to the discovery of a winning variation sooner and uncover interactions between different site changes. The downside of MVT is that it requires much more traffic than simple A/B testing, which may make it impractical for websites that do not receive as much traffic.
“Interaction effects”. This phrase can strike fear into the heart of anyone who has contemplated running multiple concurrent A/B tests. At best, experimenters view interaction effects as an inconvenient reality which forces them to consider making experiments mutually exclusive. At worst, interaction effects can make it difficult or impossible to interpret the results of concurrent […]
In an increasingly challenging and fast-moving retail environment, organizations are looking for every opportunity to reduce cost and risk while improving the end-to-end customer experience.
Given the disruption created by online pure players, how can traditional retailers maximize the value of the multi-channel model of customer engagement and also gain new levels of agility and innovation, from introducing new products through to marketing?
Your customer base is dynamic and complex. If you’re trying to optimize across your entire population of users, traditional A/B experiments may fall short. In this case, you should consider a technique we call experimental design because it allows for massive increase in the amount of variance in your digital marketing campaigns.
In this post, we’ll discuss how to make large-scale experimental design campaigns actionable.
Your optimization strategy is an exercise in rigorous prioritization. When your most valuable resource is time, it is essential that your experiments run on high-priority pages and generate a measurable effect. Learn how to estimate test impact, leverage your website analytics, and develop a robust testing strategy with data that will let you you make the most of your limited time.
I’ve had the good fortune to attend several optimization talks & conferences with our CEO, Dan Siroker. One thing I admire about good speakers is their ability to distil complex ideas into simple truths.
Simple is good.
In this forum, we try to keep things short & sweet because we know you are frantically busy. After all, this is a blog—not API documentation.
With that in mind, here are Dan’s 6 “Best Practices” from 251,391 A/B and Multivariate tests
On October 21, 2010 President Obama visited Palo Alto to raise money the good ol’ fashioned way: a $30,400-per-plate fundraising dinner. This tried and true fundraising technique is a great way to raise money if you can get the President of the United States to show up to dinner. But how can you raise money if no one has heard of you and all you have is a website? Back in 2007, when Obama was running for the nomination and trailing by double digits in the polls, that’s pretty much all we had.
As Director of Analytics for the Obama 2008 campaign, my job was to use data to help the campaign make better decisions. We started with just one simple experiment back in December of 2007.
