Monthly Archives: August 2012

Optimizely Meetup Recap: 3X the Fun

By on August 30, 2012

When 50 people come together to talk A/B testing and hear from Optimizely testers with experiences and tips to share, sparks fly and minds are blown.

Without a doubt, sparks flew at SPUR Urban Center in downtown San Francisco Wednesday night when we hosted our first ever Meetup event, A/B Testing: Who, What, Where, When, Why and How? The crowd heard from three seasoned optimization advocates, David Malpass from Prezi, Lizzie Allen from IGN, and Melinda Byerley from Poll Everywhere.

You, too can be part of the future optimization excitement by joining our Meetup group.

With a development team far across the Atlantic in Budapest, David Malpass has limited technical resources to work with in Prezi’s San Francisco office dedicated to improving the website. However, that definitely hasn’t hindered him from establishing a deep testing culture at Prezi, a presentation SaaS company, where they test everything possible. At the Meetup, he drove home the idea that engineering is not the only department that benefits from developers; marketing teams can harness their super-powers too, for advanced A/B testing and site improvements. He emphasized the importance of testing the pricing page. When he ran a test comparing the performance of monthly or yearly payment plans, monthly won by a long shot and drastically improved Prezi’s bottom line.

Screen_shot_2012-08-30_at_3

David’s motto: Always be testing.

Lizzie Allen spoke of her travels down a long, arduous road filled with weeks of crashing meetings she wasn’t invited to and overall hustle to cultivate a testing culture at IGN, a prolific gaming information website. As a data analyst, Lizzie said she would always come prepared to meetings with data evidence to support her points and question faith-based marketing. “Why are we doing that?” she frequently asked decision makers about the site in order to promote data driven decision-making. The story of challenges, successes and occasional failures ends on a positive note for IGN as testing with Optimizely has now permeated through the company culture. Lizzie overhears people saying, “let’s test that” in the hallways.

Screen_shot_2012-08-30_at_3

Lizzie’s motto: Question assumptions and instinct.

When jumping right into A/B testing, don’t start with the homepage, Melinda Byerley said, as it’s the hardest place to agree on changes. The HIPPO (Highest-paid-person’s-opinion) is hard to win over on the homepage. Melinda has left a trail of testing at every company she’s worked at in the past eight years, including PayPal, eBay and most recently Poll Everywhere. To garner the most insights from A/B testing, Melinda advised using Optimizely’s traffic allocation feature, just in case a certain HIPPO is nervous about running 100% of traffic through any variation.

Melinda’s motto: Be bold. Start big.

Wednesday’s meetup was a bonafide success and we look forward to hosting more in the near future. Stay tuned, and join the meetup group for more optimization gatherings.

 

About Cara Harshman:

As a Content Marketer at Optimizely, Cara is constantly looking to tell innovative stories about how businesses are using A/B testing to become more successful. Have a data-driven story that deserves to be told? Email her at cara [at] optimizely [dot] com. Before joining the team in 2012 she was reporting the news in Madison, WI.

Posted in Inside Optimizely | Leave a reply

A/B Testing All-Star: Boris from Bigstock

By on August 24, 2012

Name: Boris Masis
Company: Bigstock 
Role: Product Owner (Product Manager)
Bigstock’s most frequently searched term: Background

Boris says, “It’s easy to make A/B testing a part of your development flow and process. After you’ve done the first few tests, its natural for you to want to continue. It gives very tangible results and helps you see what impact you’re making.”

Optimizely: How long have you been A/B testing?

Boris: We’ve been testing for about a year. While searching for a testing platform, we looked at VWO and Optimizely. We got started with a demo from Optimizely and fell in love. Since then, testing has played a significant role in our roadmap. It’s part of our product rollout strategy. For example, we just introduced two fairly large features that we tested first: new checkout flows, and a brand new search algorithm. Testing ensures that the work is adding value and quantifies that value. Of course, we also do testing when we are unsure of which path to take.

What’s your testing process like?

We have a white board where projects go from “planning,” to “in development,” to “A/B tested,” to “fully delivered.” We try to run every large project through A/B testing before we call it done to make sure it’s actually helping.

How have you built a testing culture throughout Bigstock? 

From a developer point of view, it’s been really well perceived and embraced. People are excited to monitor results. We have a stats TV in the office that rotates through the metrics and there are usually one or two Optimizely results in there. It’s great to have measurable outcomes we can talk about. A few months ago, we got the whole Bigstock group together to watch the Best Practices & Lessons Learned Webinar recording. People who aren’t even working on the site can get a sense of how the process works, and give their ideas. It’s a little bit of a challenge to get everyone engaged, but it’s pretty informal. When we’re not sure of something, testing is the obvious answer.

Case study: With Fuzzy Search Algorithm Search Results 10% More Accurate

Challenge:  As a stock image website Bigstock and our parent company, Shutterstock are always trying to suggest the most accurate search results. There’s definitely some amount of serendipity involved in searches and people use many methods to arrive at what they’re looking for. We wanted to test the performance of a new search algorithm that would lead people to exactly what they were looking for. A developer designed a “fuzzy auto suggest” algorithm that guessed the intended search term if the word was spelled wrong (for example, “calendar” is commonly misspelled on our site). We tested it against the “exact auto suggest” algorithm that populated results based off the exact search term. Like any successful experiment, the test started with a goal.

Goal: Our goal was to determine whether a fuzzy suggest algorithm would present better results to users and improve the overall user experience. So if the algorithm presents the word you actually meant, are you more likely to select it? We served as a proxy for a lightweight test that would also apply to Shutterstock.

Hypothesis: I didn’t know whether it was going to make a difference or not. Usually when we do a new design, the winner is fairly obvious. For this test, I didn’t have an intuitive sense.

Measurement: We measured the success of the test based on the total number of results selected.

Original (Exact Auto Suggest):

Variation (Fuzzy Auto Suggest):

Results:

People selected results from the the fuzzy auto suggest 9.6 % more often. We saw significant improvements throughout, including a 6.52% increase in the number of images added to the cart, and a 3.2% increase in downloaded images.

We ran the test for five days. Normally we run things longer, but this was a very high traffic experiment with clearly separated conversion rates. We also looked further downstream with Google Analytics at how it affected the rest of the image selection and purchase process. There we track many additional details about user behavior.

4 Key Testing Takeaways: 

  • You need to test because sometimes your intuition is wrong.
  • Interpreting results is difficult and the more supporting data you have the better, in other words, integrate your analytics platform.
  • It’s easy to make A/B testing a part of your development flow and process. After you’ve done the first few tests, it’s natural for you to want to continue. It gives very tangible results and helps you see what impact you’re making.
  • Focus on the big pieces first and do exploratory work before optimization work. For example, try out entirely new algorithms or page layouts first, and then do smaller tests like tweaking buttons.

 

About Cara Harshman:

As a Content Marketer at Optimizely, Cara is constantly looking to tell innovative stories about how businesses are using A/B testing to become more successful. Have a data-driven story that deserves to be told? Email her at cara [at] optimizely [dot] com. Before joining the team in 2012 she was reporting the news in Madison, WI.

Optimization Intro Essentials: 5 Tips for Testing Triumphs

By on August 23, 2012

Are you new to website optimization? Are you an old-timer with hundreds of tests under your belt? Either way, take these tips as a foundation for any testing endeavors you may (and should) embark on.

1. Getting started: Your site is better than you think!

Bigstock-are-you-sure-its-gonna-make-me-616813

Website optimization starts with exploration. First, it’s crucial to understand your existing site visitors, who they are and what their usage patterns are. Then define a goal for them. At the end of the day, it’s all about influencing users (both new and returning) towards the usage patterns that make sense to your business. If you can’t do that, it doesn’t matter what your site looks like.

When speaking with people about their site, it’s amazing how many make excuses before opening it up in the browser. “Well, it’s not good at all, we’re actually planning a redesign soon,” they say.

Did it ever occur to people that some users don’t care about design if the offering is clear? Fun fact: IE6 users still exist. This is not to say you should create a site with no sense of design. Redesigns can be great, but their value can be lost unless you have a pre-defined end-goal(s) to analyze the investment against.

Whatever your current site looks like today, you have a great start. You have something that you can use to learn from.

To get started optimizing a website, ask yourself these questions and always keep a pulse on them.

  • Where is the bulk of traffic coming from and navigating to on the homepage or landing page?
  • What is the current bounce rate?
  • What do I want people to do on my homepage or certain landing pages?
  • If more people do that, what effect will that have on my bottom line?

Keep track of these stats/concepts on a regular basis via your analytics platform. Data is your friend! If your business is reliant on web leads of any kind, this IS as important as revenue tracking. This is a call to B2B companies where we see a huge opportunity for improvement considering the value of a single lead.

2. Your best foot forward: The call to action

Bigstock-shoes-10570346

At Optimizely, we preach testing your “call to action.” A call to action is a button or link on a page that directs visitors to click in order to take a specific action. It should answer the question: what do you want people to accomplish on your homepage or landing page? Sign up, Watch now, Free trial, Add to cart, are all calls to action. They are prominent, drive attention and are amazing places to run simple tests. If you don’t have a call to aciton, create one. Test turbocharged phrases. Apply a color scheme to your button, keeping in mind which colors are resonate with your visitors. Consider the placement on the page relative to the natural eye-path.

***Test Tip*** Increased clicks on the CTA does not always indicate a net positive. Always look further downstream to see how increased clicks affect the your ultimate goals. Looking further downstream may also yield new test ideas. ***/Test Tip***

Tap your design team, or even use a site like 99designs.com, oDesk.com, e-lance.com, etc. to whip up a few different buttons or visuals that you can easily place on your site.

3. Administering the test: Strategy is key

Bigstock-golden-key-416568

When setting up a test it’s crucial to have a process. Every test should include some form of the following (high-school science class flashback!):

Examine Data & Generate Test Ideas – What insights do you gather about your site? 

 Start with the data and answers to the questions you asked about your visitors. Maybe other internal sources like customer feedback or suggestions from colleagues point to a few testing opportunities. However, we cannot stress enough, the use of existing visitor data for test idea generation. Opinions are important, but not always in your site’s best interest. Your site visitors are unpredictable. Rely on real data, not hunches.

Define a Goal – What do you want to achieve?

Once you’ve nailed down a few ideas, prioritize them and keep the bigger picture improvements at the top. Maybe you want longer page views, more page view, more clicks, form fills, or items added to carts and checkouts completed. The sky is the limit, but it’s up to you to define the end goal.

Make a Hypothesis - What do you think will happen from this test?

Laying out the thought process and all possible outcomes will lead to a more purposeful and enlightening test. This step is especially important as you start running tests more frequently. Example: you see a lot of people going to your “about us” page which is not leading to any conversions. You believe that if you remove the link from the top navigation and only leave it in the footer, you would see more clicks on other more meaningful parts of your page.

Set the Parameters - How long will the experiment run for? Which traffic segments will be subjected to this test? What does succes mean?

Set a timeframe for the start and end date of your experiment. This ensures that you do not fall for any early results and quit the test prematurely. Also work with your team to determine what success would look like for this test.

***Test Tip*** Traffic on Tuesday and Sunday are different. Let tests run their course. ***/Test Tip***

Analyze - What do the results indicate?

Once your test has come to an end, examine the results, but not just the winning variation. Dig deeper into the data to see how this winning variation manifested throughout the entire site. Are two variations too close to call? If so, consider testing again, with more distinct changes.

Implementation -  Act on the results you’ve discovered.

Maybe the original version is best, maybe the new variation blew it out of the water. Either way, this is where you take advantage of the results and move forward with implementing the change to your site, not to mention learning more from the test. With Optimizely’s testing tool you can allocate 100% of your site trafficto the winning variation while your dev team makes changes to your backend.

4. Other testing ideas: What we’ve seen from our clients

Bigstock-choices-concept--34051715

Once you have some basic results back and a better grasp on your visitor behavior, keep testing other areas of your site. Here are a few common places where we see a lot of tests in specific industries/verticals:

e-Commerce:

  • Product pages –> Will targeted product placement based on visitor location increase sales?
  • Payment forms –> Can you reduce the number of clicks to buy?
  • Suggested products –> Which format of suggested products garners the most clicks?
  • Main navigation order –> Does alphabetical or most searched/popular arrangements lead to more engagements?

Lead Generation:

  • Forms, forms, forms –> Can you make them shorter and more concise?
  • Landing pages –> What do you need to include on each one?
  • Colors —> Do colors impact your visitor behavior at all?
  • Geo-targeting your pages –> Should visitors from China see a different landing page than those from Canada?
  • Free trial offers –> Which offers will lead to more sign-ups?

Publishers:

  • Popular article/videos –> Should you gate this content?
  • Headlines –> What are the most effective words?
  • Navigation bars–> Which category breakdowns receive the most traffic?
  • Suggested content features –> Which position will make more people navigate to this content?

SaaS/B2B:

  • Request demo placement –> Are they easy to find?
  • Phone numbers for different variations –> Can you offer multiple to determine which one caused more inbound calls?
  • Support questions –> How do visitors report questions or feedback?
  • Product demo video –> Where will it receive the most views?

***Test Tip***Source your test ideas from every industry. For example, e-commerce sites will benefit from testing ideas for publishers. The more you diversify your testing, the more you will learn.***/Test Tip***

5. The more you test, the more success

Bigstock-greedy-business-partners-4045219

Website A/B testing is a tool to amplify your optimization strategy, guaranteed. Search engine optimization, online marketing, social media, conversion rate optimization all work together to attract more eyeballs on the content you spent hours creating. Through testing, you will discover which user experience fits best for your site. Test your site to validate your content, your new headlines, buttons, landing pages, video placements, and the list goes on.

Examining results from how different variations of your site perform is a sure-fire way to achieve your goals whether they’re based on making more money, generating more leads or increasing your content’s visibility. It’s also a pragmatic way to tailor content to your array of visitor sources and decide which sources are worth investing in.

The most salient fact to take away from this post is that all websites have room for improvement. Commit to continual testing because one test will impress you, but many tests will stun you and have you rolling in dough.

Only one question remains: What are you waiting for? Start testing immediately.

About Cara Harshman:

As a Content Marketer at Optimizely, Cara is constantly looking to tell innovative stories about how businesses are using A/B testing to become more successful. Have a data-driven story that deserves to be told? Email her at cara [at] optimizely [dot] com. Before joining the team in 2012 she was reporting the news in Madison, WI.

Calling all Bay Area Optimizers

By on August 21, 2012

If you live in San Francisco (or the Bay Area) and are reading this blog post, then mark your calendar for Wednesday, August 29th between 6:30 and 8:30 pm. We are inviting you to join us at our first ever Meetup to learn from real-life testers who are optimizing their business’s websites with A/B testing. If you find that every conversation you attempt to start about website optimization is falling flat in your friendship circle, then come to SPUR near the corner of Mission and New Montgomery, and make new friends who perk up when they hear anything about conversions, calls to action or enagement. 

With refreshments and appetizers in hand, you’ll hear from: 

  • David Malpass of Prezi about how running A/B tests on the homepage increased conversions
  • Melinda Byerley of PollEverywhere about best practices in testing
  • Elizabeth Allen of IGN about their coming of age testing story and how data ultimately triumphed
  • The Optimizely team about any optimization inquiry or fun fact you might desire 

Space is limited – so open a new window now and RSVP “yes” to be a part of the event. You’ll be so glad you did. 

Looking forward to seeing you all there. 

-The Optimizely Team

About Cara Harshman:

As a Content Marketer at Optimizely, Cara is constantly looking to tell innovative stories about how businesses are using A/B testing to become more successful. Have a data-driven story that deserves to be told? Email her at cara [at] optimizely [dot] com. Before joining the team in 2012 she was reporting the news in Madison, WI.

5 Questions A/B Testing Will Answer

By on August 17, 2012

Do you ever find yourself puzzled at which site design would serve your business best? Good. You’ve come to the right place. 

The possible combinations of headlines, colors, buttons, images, forms, placement, etc., are dizzying. Often times, the highest paid person’s opinion dictates the site layout based on personal hunches and preferences. Hunches are risky. Why not let actual data drive the decision? Act on your questions. Demolish your assumptions. Start experimenting with different variations of your site and let real data collected in real time inform your site design. A/B testing, split testing, or bucket testing – go with whichever name is least intimidating to you – has become essential for optimizing both the user experience and the traffic flow through your site. You can learn surprising facts about your audience, just by serving up alternative versions of your pages. Find answers to these five questions when you start testing. 

1. What do my users like best? 

Present them with multiple options and let them tell you. With testing, you can quickly understand what grabs your users’ attention best. Big words? One word? Images of smiley women? Or images of casual men? The team at 37signals tested the homepage for their CRM, Highrise with a few radically different ideas and sure enough, the specific human smiling/staring/frowning back at users mattered. They found some surprising results. A long-form sales letter about Highrise performed 37.5% better than the original illustration heavy, app-centric information page. Quite a significant increase, but they didn’t stop there. They tested a homepage featuring an oversized portrait of a user. It performed 102.5% better than the baseline design. That’s a turbocharged difference. The test got even more nuanced when they started testing different portraits of users. Beware, testing can be addictive.

2. Why do people abandon their shopping cart? 

Video Fun Fact: Before the internet, people were still abandoning carts

The possibilities are many. But one thing’s for certain: you can influence (and decrease) the rate at which people walk away from a full cart. Articles about best shopping cart design practices are bountiful online. Implement those insights with A/B testing and you will have yourself some answers and more revenue. Shoppers walk away from about 50-70 percent of online carts for various reasons. E-commerce sites have huge testing potential. Companies like Fab.com, Amazon and Groupon aggressively test shopping cart elements to drive revenue. Even the slightest improvement in the shopping cart funnel can lower your abandonment rate and generate more money for you. Who doesn’t want more money? Test it and figure out what works. 

3. What will it take for people to complete a form? 

Forms can have a huge bounce rate. People see a page of blank boxes and freak out. Designing a form is an art, and a valuable testing opportunity. Don’t underestimate how important each element of the form is. Ask yourself these questions while creating your form: What are the most essential pieces of information you need? Do you really need a phone number? Apart from the questions themselves, what parts of the page might be distracting a user away from completing the form? Will moving the form to a different part of the page do anything? Probably. A/B testing on forms is a thing of genius and a very effective way to achieve many of your company’s goals fast.  

4. Which words prompt action?

Sharp-edges

Call to action words, are the focus here. Running experiments on your call to action buttons are great tests to run early that yield surprising results. Your site exists to fulfill some purpose, and that purpose requires visitors to act. Make the certain action obvious. Play around with different combinations of words – don’t be afraid to try something out-of-the-box – and test them live concurrently to see which receives the most clicks. You will be glad you did.  

5. How do returning users navigate my site versus new visitors?

Your site may not be a one-size-fits-all solution for every visitor. Returning visitors want to use it one way, first-time visitors another. How do you figure out who wants what? With A/B testing, you can specify which type of visitors see which variations. In other words, you can target different populi with different content, buttons, layouts, etc. You can even show visitors who land on your site via Facebook, Pinterest, Twitter, etc, with specific pages. What an incredible way to learn about all types of visitors.

A/B, split or bucket testing is an explosive tool. Do it to learn about your users, how they engage with your site and what they respond to best. Look at the data. Implement those winning changes and start achieving goals you never even knew you had.

 

About Cara Harshman:

As a Content Marketer at Optimizely, Cara is constantly looking to tell innovative stories about how businesses are using A/B testing to become more successful. Have a data-driven story that deserves to be told? Email her at cara [at] optimizely [dot] com. Before joining the team in 2012 she was reporting the news in Madison, WI.