Download our FREE ebook of 37 A/B testing case studies from the world's leading companies for test ideas and inspiration. Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program


Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more!

Get It Now

When it comes to A/B testing, the who can be just as important as the what. Even with a stellar product and an arsenal of testing ideas (dare I say 71?), it’s up to the end user—the testing hero—to make optimization an integral part of any business.

Sometimes testing heroes are marketers. Sometimes they’re product managers. And sometimes, they’re a fictional blonde girl with a hankering for porridge. These folks may come from different backgrounds, professions, or realities, but are all united by their love of data.

  1. The Bachelor


  1. Product managers – “We try to dial in things like navigation, pricing, shipping options, and promos before the holiday season and then use Optimizely to launch season specific tests, like ‘Order in the next X hours for guaranteed delivery by Christmas’ or to display things like gift certificates more prominently.” – Nick Schlegel, Product Manager, Backcountry
  2. Designers – Qualitative + Quantitative = <3

  3. High school students – Named Sarah Click. I kid you not!

  4. Developers“We wanted to try making dramatic changes to our pricing page to increase conversions. Optimizely made building the experiment quick and painless. Within two weeks we’d figured out how to boost trial sign-ups by 24% and purchases 11%.” – Paul Godfrey, Senior Developer, Zendesk

  5. Email marketers – Optimizing for opens, clicks, and engagement.

  6. The guy who invented Internet Explorer

  7. Goldilocks (and Goldilocks Cat)goldilocks cat

  8. SEO & SEM wizards – These crafty marketers have already invested dollars into paid marketing and know optimizing experiences after that initial ad click is the best way to get the biggest bang for their buck.

  9. Fast Company – It’s an A/B/CEO test.

  10. Software engineers – Optimizing for speed.

  11. Data analysts“One of the biggest mistakes a company can make is to just assume that the functionality on the big screen will work on the little one. You have to be smart, and pick and chose. That’s why we test mobile.” – Scott Ehly, Optimization and Usability Analyst, RentPath

  12. Oregon Trail pioneersoregon-trail-aoregon-trail-b

  13. The leader of the free world Obama_New-Media

  14. Aziz Ansari

  15. A small start-up in Montana

  16. A global corporation in Seattlestarbucks

  17. Rupert Grint (swoon) – Wondering which pick-up line might work best (I’d chose A!!!!)

    rupert-arupert b

  1. Online dating sites Rupert might frequent

  2. And this brilliant man who tested his way through the online dating system

  3. MTV


  1. Here’s a puppy enjoying bath time – He doesn’t seem to be testing, but we’ll let it slide.

  1. SaaS Companies“Optimizely was a superb tool for the task, saving many hours of “engineering time” and making the test results easy to view and understand for team members with less technical skills (but other priceless qualities).” – Shai Alon, Front End Engineer, Totango

  2. Content marketers – …but I won’t give up my tricks that easily!

  3. These jubilant soda enthusiasts


  1. Sock sellers – SmartWool. Smart website.smartwool

  1. Non-profitsDonation maximization.

  2. Financial advisors“We are constantly looking for ways to increase conversions. Using Optimizely, we added ‘100% privacy, you will never be spammed’ text to the page and altered our sign-up button. These small changes resulted in conversions going from 12.5% to 32.5%!” – Peter Olson, Partner / Marketing Director, OnCall Advisors

  3. Networkers – A/Business card testing! optimizely-business-cards

  4. NetflixTest all the things!
  5. ApartmentGuide – Optimized mobile experience = optimized revenue.

  6. Airbnb – Adding a little heart increased engagement by 30%.

  7. This Hall & Oates impersonator hall-oates-impersonator

  8. Really, really, ridiculously good looking people.zoolander

  9. Directors of e-Commerce – At companies that makes really, really ridiculously cool gear.

  10. Global enterprises“What it really came down to for us was pricing and support. For an enterprise company like us, where downtime equals dollars, having that guaranteed access to Optimizely’s customer support gives us the peace of mind that we need.” – Mike Burk, Electronic Arts

  11. Small businesses“We’re three people and man-hours has always been a bottleneck for us. Optimizely has helped greatly in this regard by allowing us to A/B test without bringing in users, which at this stage is not feasible. We’ve been able to identify the major friction points in our funnel and test alternatives. You can’t get better if you don’t test!” – Sam Gimbel, CEO, BrewCrate
  12. This brave musician/amateur fox expert – In pursuit of answers to the age old question: What does the fox say? what-does-fox-say-awhat-does-fox-say-bwhat-does-fox-say-c

  13. The world’s largest media companies

  14. Also, US Weeklykanye-kim

  1. Growth hackers – #buzzword #sorrynotsorry

  2. Moustache lovers“There’s a lot of passion involved in nonprofits. You’re driven, not just to do job, but to do a good job for a good cause. We all have strong opinions and ideas about how things should be done. Optimizely tells us, out of this mix, which is the best way forward.” – Andrew Ng, Movember

  3. Founders – “We’re maybe as good as a coin-flip at guessing what’s going to work best for our users. We rely on testing to just make better decisions. People are really fascinating and interesting… and weird! It’s really hard to guess their behaviors accurately.” – Peter Koechley, Co-founder, Upworthy

  4. Warby Parker wearers warby parker

  1. Burrito eaters

  2. Online retailers – Checkout funnels and product pages and promotional offers. Oh my!

  3. Ops teams“How can we be a legit A/B testing company if we don’t A/B test in-house? It was my first week on the job and I was tasked with finding out what chair we should use in the new office. We had 4 chairs and some old school paper ballots. I named each chair (Vesta, Bill, Ted, and The-Chair-Who-Must-Not-Be-Named). It was simple. Once you tried out every chair you voted on what you liked most. At the end of the day, there was a run-off between Vesta and The-Chair-Who-Must-Not-Be-Named. Vesta eventually won and now we all sit comfortably in our A/B tested chair democracy.” – Jen Kilty, Operations Wizard, Optimizely

  4. Electronic Arts – Wondering how SimCity 5 sold 1.1 million copies in 2 weeks? Wonder no more.

  5. Digital agenciesWhen it comes to CRO, these testing pros know what’s up.

  6. B2B companies“Ensuring that sales teams have solid leads to follow is fundamental in the B2B marketplace. With Optimizely, we were able to improve form validation to deliver better leads to sales without burdening our busy internal IT team.” – Nimesh Parmar, Online Manager, Iron Mountain

  7. B2C companies“Optimizely was extremely important in our website redesign process. Instead of launching the new site and analyzing the results later, we could try different variations and see what worked during the design process.” – Do Kil, Creative Director, Spreadshirt

  8. 1-800-Dentist – Improved sign-up flow increases leads by 23%!

  9. Here’s a puppy who’s just as excited about these testing heroes as we are – It’s all happening! Read on.

  1. VPs of Marketing — “Testing with Optimizely pays for itself. With all the gains in conversions, in addition to how fast I can get a test turned around – literally the man hours I’m saving by not having to leverage the tech team as much – those are huge impacts to our business. It’s not just value creation, it’s also resource saving.” – David Yeom, VP of Marketing, The Honest Company

  2. Madden gamers – Testing various settings of a video game to manipulate outcomes.

  3. Brewmasters – Cheers for A/Beers!

  4. Experimental marketers

  1. Google – 41 shades of blue?

  2. And also, Google – They test so much, they deserve two spots.

  3. Harvard Business Review

  4. People who write books on A/B testing

  5. Hillary Rodham Clinton hillarysuittest

  1. Morpheus morpheus

  2. Dennis Rodman dennis-rodman

  3. Hamlet – Variation A: To be? Variation B: Not to be?

  4. This baby birdare-you-my-mother

  5. These jocks – It’s an AB TEST y’all! sit-ups-v-planks

  1. Caffeine addicts – Starbucks vs Dunkin Donuts…

  2. Scheming coyoteswileawileb

  1. Marketing Automation Ninjas – “It’s really a no brainer. Marketing automation is meant to scale marketing efforts to many, many customers at once. Because you automatically have so many people running through, it’s the perfect environment for testing. Plus it’s a non-trivial expense. If you plan to spend hundreds or thousands on a system each month, don’t you want to know you’re getting your money’s worth?” – Amelia Lin, Marketing Automation Specialist, Optimizely

  2. Optimizely – Duh.

  3. …and YOU! – Type in any website URL below to see how easy it is to run an A/B test today.

Optimizely X