The Optimizely Blog

“Testing is a way of life for us. We’re increasing our testing resources, and building testing into our long-term, strategic plans.”

- Shayne Tilley, Commercial Manager, Avalanche Technology Group

Introduction

Here at Optimizely, we are always excited to hear about testing success stories of all shapes and sizes.  Much to our delight, for many of our clients, testing is fast becoming an even more vital piece of long term optimization strategy, which helps keep the success story flow plentiful.  Today, we wanted to share a short, yet powerful a/b testing story that highlights how even very simple testing can make significant impacts on …. drumroll please… REVENUE!

Background

Avalanche Technology Group is the Australian distributor of popular anti-virus software AVG, and testing has long been a core part of their marketing culture. When they recently examined their shopping cart conversion data they suspected there was room for improvement. Shayne Tilley and his team knew the checkout process would be an incredibly sensitive funnel to test, and challenged themselves to devise a testing approach to confirm their suspicions without getting too drastic.

The Experiment

Avalanche decided to run a simple experiment, which they designed to minimally impact the checkout flow and require minimal resources to implement. The test removed the site's header navigation links from the shopping cart pages, which they knew would reduce visual noise, and keep traffic more focused on actually checking out.

Original page with Header Navigation Links:  

Test page removing Header Navigation Links:

Results

By removing the irrelevant navigation links, Avalanche increased their conversion rate by 10%, which in turn increased revenue by 16%:

Conversion results: 

Revenue results: 

Conclusions

When considering tests on potentially sensitive areas of the funnel (in this case, the checkout process), don't be afraid to start small.  Getting everyone to buy in on a massive overhaul is always going to be tougher than getting agreement on a (seemingly) minor change.  After all, just because a change may be visually minor doesn't mean the positive impact won't be significant.  While the 'Less is More' theory is no more a sure-fire bet than any other change you may be pondering at a given moment, its at least one more testing idea to toss into the quiver.

Ready to see what you can do next? Take a shot at setting up your simple test on Optimizely.

They say a picture is worth a thousand words. But what is the potential value of a single word?

Today, we are thrilled to share a guest post from Optimizely user Vitals, whose recent testing success story may shed a little light on the matter. Every day, a wide variety of A/B Testing experiments are run through our platform, ranging in complexity. We are delighted to introduce this post as the first in an on-going series intended to bring you insights and ideas from real world A/B testers of all types.  Without further ado, we turn it over to Irving Weiss, Vice President, Web Marketing at Vitals.

Background

Vitals is a leading website committed to helping patients find and connect with quality doctors, hospitals and other health specialists. Every doctor on our site has a summary page that we traditionally refer to as the "Snapshot" page, and a more in-depth page that we refer to as the "Full report." The purpose of this dual page design is to protect our in-depth data, and to quickly show users all of our data capabilities. It’s very important to us that the user proceeds from the "Snapshot" page to the "Full report." 

Experiment

One of our employees suggested we run an Optimizely test to rename the “Full report” tab to “Full profile.” The theory behind this test was that people are more apt to associate the word "profile" with a person and the word "report" with an object, making it less obvious that the "Full report" is about a doctor. 

Most people on our team found it hard to believe that a single word could really have a meaningful impact on user behavior, so this experiment was definitely met with resistance.  Prior to Optimizely, testing usually meant more people needed to be involved, which made for a longer process, and didn't typically allow for a quick and easy test.

Once the experiment got the approval, it didn’t take more than a day to discover that the naysayers were totally wrong; the new treatment brought about a 65% increase to the CTR!

The original treatment: "Full report" tab gets a 5.5% CTR

The new treatment: "Full profile" tab gets a 9.1% CTR

This Optimizely result screenshot shows the whole story:

Lessons Learned

• Even small modifications can make a whopping difference. In this particular case, not only did we get a much better CTR on the tab, but this change has lowered our bounce rate and allowed us to present our full offering to many more people. 
• Listen to the ideas of all your employees. Sometimes the old timers are so stuck in their ways that they can’t think of doing things differently.
• In high traffic areas of the website, you can get a great test result in as little as a day. We did let the test run for a week just to be sure it wasn’t an anomaly.

About the Author: 

Irving Weiss joined Vitals from its launch in January 2008 and was single-handedly responsible for building its visitor base. Currently, he oversees a traffic team that helps draw over 200,000 daily visitors and millions of visitors to Vitals monthly.

How did they do it?

Watch the quick video below to see how easily you can set up a text change experiment using Optimizely:  

Comments, questions, or want to share an A/B testing story of your own? Let us know.

Well, its been a while since a healthy update - pardoning the delay, we have much to report. We have been hard at work at Optimizely and are excited to announce a couple of new features and a handful of other exciting tidbits.

Saved Goals
Today, Optimizely users can begin taking advantage of our new Saved Goals feature. Saved Goals will offer a faster, easier, and more powerful way to manage conversion goals for your experiments. 

Saved Goals highlights:

• No need to recreate goals for each experiment, we’ll save your goals for you from now on.
• Set up basic or advanced “Click” goals on any page, anytime
• Track revenue, pageviews, and engagement with ease

Watch the video about Optimizely’s new Saved Goals and read more at our knowledge base.

 Geotargeting 

In the past year we’ve released a number of enhancements to our powerful targeting capabilities and today we’re excited to announce our latest addition to the fold: geotargeting! Optimizely Platinum clients can now target their experiments using a customize-able range of geographic parameters.  
Geotargeting Highlights:

• Target experiments by city, state, province, country, or continent 
• No additional latency (Optimizely still requires only a single static request directly to Akamai)
• No snippet change
• Include and/or exclude any geographic area in your experiment
• Geotargeting is only open to platinum clients right now

Read more about Geotargeting and watch the video.

Optimizely is hiring... and moving

Our rapidly growing San Francisco based team has reached 15, but we are hungry for more fantastic talent. We are currently looking to expand our team, and have positions open in Sales, Engineering, and Customer Support. With an outstanding team of talent, a fast-growing company, and one of the coolest working environments ever, not to mention a very cool product with lots of happy customers, we are excited to add more firepower to the team (this could be you, or someone you know). To learn more about our current open positions, please check out our jobs page.   

With all the growth, it looks as though we have just about outgrown our current office space, which means, time for a new office. Same great cast of characters and culture - bigger and better facilities. Don't worry, we're not going too far - just hopping our way on over a bit closer to downtown. The move is slated for May, and you can be sure we will be looking forward to welcoming visitors with open arms, and much more space. We'll keep you posted.

And Finally, a Word From Our Newest Team Member

It is with great excitement that I steal a few sentences on this post to introduce myself - My name is Jodie Ellis, and I have (with much delight) just joined the team here at Optimizely to lead the marketing efforts and ensure that wherever people are talking about the best way to get A/B website testing done, Optimizely is at the center of the discussion. In both my personal and professional life, I have always had a real soft spot for high quality and great value - as such, I have steered my way into the great fortune of working with best in class people and products in a wide variety of industries and roles. From my very first conversations and interactions with the Optimizely team and product, I can tell you that I have never been more charged up to get in to work. I truly look forward to building a first class communication bridge between Optimizely and the rest of the world, and will welcome input from all sources, be it customer, casual observer, industry insider, media mogul or anyone with a voice to be heard.

 
As always, we'd love to hear from you - please let us know if you have comments, questions, ideas or just want to say hi.

Background

New Haven, Connecticut-based startup SeeClickFix is a social technology platform for non-emergency community issues. See a pothole on your daily commute that doesn’t seem to be getting fixed, or know of a traffic light timing glitch that should be addressed? SeeClickFix partners with local governments to have issues like these monitored and fixed.

The Experiment

CEO Ben Berkowitz wanted to use data to inform internal debates about website copy and design, and chose their most important page — their homepage — as the place to start. SeeClickFix had spent time designing and developing a beautiful Google Maps integration on their homepage, which showed the issues their users were filing in real-time:

Ben suspected however, that while the map was a great showcase of their product's usage, it might actually be reducing page engagement. Here is his original hypothesis:

I want to do something that would previously have created a debilitating debate with the team. I want to drop the map that we worked so hard to make beautiful and functional. My goal is to focus eyeballs on the search box. This is where the culture shift comes in. Optimizely is so easy and time-light that the only reasonable outcome to suggesting the dropping of the map is, ‘lets just test it.’

Here's what they tested instead:

Results

The new design increased engagement by 8.3% — Optimizely measures engagement as the percentage of visitors who click on any part of the experiment page. It is a handy way to measure the holistic impact of a large change. You can think of engagement as the opposite of bounce rate so the higher engagement, the better.

Here is a screenshot of the Optimizely results page with some of the numbers redacted:

Conclusion

This initial test accomplished two things for SeeClickFix: it increased their product's usage, and it introduced data to their company culture as a change agent. From Ben’s perspective even a 5% shift in engagement or conversion, positive or negative, means an experiment was worth running, because the data resolves debates, and improves the product and user experience.

If you are interested in learning more about this experiment check out Ben's blog post on the topic.

I did a fun interview with Andrew Warner of Mixergy and shared a few tips about how to get started with A/B testing:

Dan: Yeah. Well, the first simplest thing I’ll say focus on the one thing you want somebody to do on that page. Once you’ve identified that one thing, remove almost everything else. Try a variation of your page, which is focused entirely on that one thing you want them to do, on our home page it would be enter your URL, click get started. On your page, it might be entering, clicking the banner at the top of the page and getting people to sign up. Whatever that one thing that you really want them to do, focus in on that and remove almost everything else, remove especially copy, remove images, remove text and start from that. Start with a very dramatically sparse page, and that’s a great starting point because what that’ll show you is the impact of all the stuff you think helps, all of these great screen shots or links to show that you were on the New York Times and all that stuff, maybe that’s actually distracting people from what they should really be focusing on, which is the one call to action you have on your page. I would say that’s my best recommendation for a good place to get started. And once you’ve done that, you’ve traversed this local maximum problem, you’re now actually at a point where that’s a great part to say, “Now that we have the sparse with the core beginning essentials of what are needed on our landing page, now let’s start adding things to it and see what actually will improve the conversion rate.”