Testing Theory: The Importance of Default Selections

Welcome to the first post in Testing Theory: Academic Findings You Can Actually Use. This series is a stepping stone to creating better tests by providing practical testing ideas based on the study of how people make decisions (formally known as behavioral economics).

Optimizely has made it incredibly easy to A/B test any change you want, but experience has shown us that not all tests are equally good. Some changes just don’t affect your site’s visitors in any significant way. Because it’s not practical to test every possible change that comes to mind, it’s important to have a strong hypothesis of why a change will be effective in helping you maximize your site’s goals.

Continue reading

Watch: AB Testing Ideas for E-Commerce Sites

In a webinar last week, we shared ideas for how e-commerce sites can use AB testing to improve their conversion rates this holiday season. As site traffic peaks for many businesses between November and February, so does the potential for fast results from AB tests. Watch how simple AB tests can go a long way for your business during the holiday season and beyond.

3 takeaways from this webinar:

  • The benefits of testing during the holiday season
  • A list of versatile and easy-to-execute testing ideas
  • How to tailor AB tests for mobile and tablet devices

Are you an e-commerce site? What types of tests are you running this holiday season?

Posted in Ecommerce | Leave a reply

ZAGG Doubles Down on Simple Image Test for 40% Revenue Increase

Background

ZAGG is one of the largest e-commerce sites for mobile electronic accessories. After success selling the invisibleSHIELD in 2006, ZAGG has grown with the industry into a leading online marketplace for mobile device accessories of all types.

Challenge and Goals

The product pages on ZAGG.com presently utilize three types of imagery for product display: static images, video and 360º product images. The original design of ZAGG’s product pages placed the static product image as the default for visitors. ZAGG’s conversion and usability analyst, Kollin Kilian, ran an A/B test to determine whether presenting the video or static image as the default on the product page led to more clicks on the “add to cart” button. He found that showing the video as the default yielded a 27% increase in revenue per visitor. Not satisfied with this testing victory, ZAGG looked for further optimization opportunities.

“The video default was selling, but you always want to be testing,” Killian says.

Equipped with Optimizely and their existing video and photo assets, ZAGG set out to design a layout with an even higher conversion rate than its predecessor.

Hypothesis

Killian constantly looks to other e-commerce sites for ideas on product page design. He found 360º imagery to be a popular format and decided to test out how ZAGGs product pages would perform with that view as the default. The team’s instinct favored video as it has been a core part of their successful product strategy – however, Killian and the team thought 360º imagery could potentially be the perfect balance between static images and video.

Original

Original page with product video

Variation

Variation page with 360º image

Solution

On the first experiment with the product page, ZAGG was quickly able to learn that the video outperformed the static image. Having already built the simple test in Optimizely, all it took was a few minutes to duplicate the experiment and add a variation that set the 360º product image as the default.

“Optimizely is too simple not to use,” Killian says. “With a couple clicks you can measure engagement, revenue, and set up multiple event tags throughout the page to see how people are navigating through it”.

Results

In the first test, using the video as the default increased revenue per visitor by 27%. The second test which placed the 360º image as the default increased ZAGGs revenue per visitor an additional 12%.

The results show that using the 360º image increased ZAGG’s revenue per visitor by 12%.

Conclusion

Killian was delighted to uncover an additional revenue gain that required virtually no additional resources to test and implement. The simple change to the 360º image as the default resulted in a near 40% increase in revenue.

ZAGG’s experience demonstrates the value of A/B tests that don’t require new resources. Gains can emerge from tests that utilize repurposed content. While your development team is exploring new product ideas to build and test, you may uncover big conversion wins by applying the resources you have available in fresh ways. ZAGG demonstrates that until new features can be developed, there’s still a valuable opportunity to refine and improve conversions on your site using the resources you have today.

Posted in Case Studies, Ecommerce | Leave a reply

Dear E-Commerce Sites, A Traffic Spike is Approaching. You Better Be Testing.

Join us for a webinar tomorrow and learn ideas for simple, effective testing through the holidays and beyond. 

The holiday season is around the corner, and most e-commerce websites will start to see the annual spike in site traffic.  Last year saw continued growth in online holiday shopping, especially for shoppers making purchases from mobile devices, and this season seems all too likely to surpass last year’s in holiday dollars spent online.  Are you ready to make the most of it?  We are here to help.

An Opportunity Not to Miss

Times of high traffic are optimal for website optimization efforts with A/B testing and multivariate testing. The potential speed of collecting results is unparalleled and the chance to act on winning results within days and start increasing revenue is not to be missed. You can run a test on Monday and Tuesday and easily use Optimizely to allocate 100% of the traffic to the the winning experience on Wednesday. For example, making the special price offer text green instead of red across product

pages may represent only half a percentage improvement in revenue per visitor. Over the course of the holiday season, this small increase may represent tens or hundreds of thousands of dollars in additional revenueWith a testing tool nimble enough to make subtle and drastic changes across the entire site within minutes, huge increases in conversions and revenue are attainable.

More New Visitors, More Incentive to Test

The increase in site traffic will certainly be a mix of new and returning visitors.  The prospect of new visitors is an incentive to start A/B testing because every new visitor is a potential return visitor. The holiday season is a prime opportunity to convert lots of new visitors and make them want to return again. Testing when you have more of these new visitors can quickly give you the most accurate picture of a better site experience and give you insight to apply to further tests after the holiday rush.

Code Freeze? No Problem.

Code freezes are no longer a deterrent to testing during the holiday season because Optimizely’s testing platform does not alter your existing site code. Every variation is contained in the Optimizely snippet, a single line of code placed in the head tag of your global header. Even if your site is locked down, you can still target and test all of your site visitors and increase the likelihood of turning them into customers.

Join us for a webinar tomorrow and learn ideas for simple, effective testing through the holidays and beyond. 

Attend our upcoming webinar: Optimize for the E-Commerce Holiday Rush – Ideas for the A/B Tests that Keep on Giving

Tuesday, October 16th 11 am-12 pm PST

Reserve your Spot Today!

Posted in Ecommerce | Leave a reply

First they ignore you, then they laugh at you, then they fight you, then you win.

On the heels of our announcement on Tuesday announcing our 100,000th experiment and TechCrunch breaking the news that we’ve surpassed Omniture Test&Target, we were thrilled this week to see a direct response from Adobe, Omniture’s parent company. Anecdotally, we’ve heard from customers that Adobe has had us in their sights for some time, but not until now have they officially made a statement. As the famous Gandhi quote goes, “First they ignore you, then they laugh at you, then they fight you, then you win.”  Today, I think we’ve officially moved into fight mode.

Here is a great quote from their blog post:

Test&Target serves over 1.2 bil­lion test page opti­miza­tion requests each day. Wow. Com­pare this to one startup who has bragged about serv­ing up 1.3 bil­lion over the course of a few years.

As our blog post and the TechCrunch article make clear, we announced 1.3 billion visitors, not requests. Clearly, we serve many more requests than visitors. So not only is this statement inaccurate, it’s also interesting for two other reasons. First, they are directly attacking us without mentioning us by name. Why are they so afraid to mention this “one startup”?

The second reason why this quote is interesting is that they choose to focus on requests and not visitors, as this highlights one of the biggest differences between our two products: the implementation. Back when I was using Test&Target as part of the Obama campaign, we had to install an mbox code snippet for every part of the page we wanted to test; and every time you wanted to test something new you needed to add an mbox. What’s worse is that Omniture charges you for each mbox request. Some of the customers who have switched from Test&Target to Optimizely have referred to this as “mbox hell.” This frustration was one of the many reasons we created Optimizely. We have built the product I wish we had in 2008 to make it dramatically easier for anyone to A/B test their website. Optimizely’s implementation is a single snippet that is unique to your account. With this single line of code  you can run as many experiments on as many parts of your site as you like.

As for ease-of-use according to their blog post:

Even basic test­ing is very easy to learn and to deploy — I’ve wit­nessed opti­miza­tion man­agers learn to run basic tests within the tool within an hour’s time.

Having used Test&Target personally and having listened to the stories of customers who have made the switch from Test&Target to Optimizely, this statement comes to me as quite a surprise. But, as a data-driven marketer, I’d gladly be convinced after seeing some data that shows Test&Target is in fact easy to use. In the spirit of real world A/B testing I’d like to offer Omniture the Optimizely Challenge (akin to the Pepsi Challenge): put your product directly up against ours and let’s see which one marketers think is easier to use. Anyone reading this post is welcome to go directly to http://www.optimizely.com/ and enter any URL you’d like to test (feel free to even try creating an experiment on adobe.com). We put our product where our mouth is. Omniture, are you willing to let marketers try your product and see how it compares?

Another great quote:

So, I encour­age you to dig deeper than the rhetoric you’re hear­ing this elec­tion year and demand the facts when look­ing at the future of your test­ing pro­gram.

Finally a point we agree on. I also encourage you to dig deeper than the rhetoric your’re hearing. Try both products and see which you prefer.

Posted in News & Events | 18 Replies