6 Ways to Generate A/B Test Hypotheses

TL;DR

• In 1848, an accidental explosion caused a 3 ½-foot iron rod to pass through the brain of Phineas Gage—somehow he survived.
• The severe brain trauma left Phineas fitful, impatient, obstinate, capricious and vacillating.
• At times, your website and app users can behave like Phineas Gage.
• You can help them—and your bottom line—by embracing psychological heuristics to inform a hypothesis-driven optimization strategy.

Autumn in Vermont is simply stunning: brilliant leaves of yellow, orange, red and green explode across a sylvan tapestry that could stop you in your tracks. But in mid-September of 1848, it wasn’t the fall foliage that stunned a 25-year old railroad worker named Phineas Gage.

Rendering of Phineas Gage’s skull as a result of the accident.

The story of Phineas, one you may have heard in a college neuroscience class, goes something like this… While setting a powder charge in a rock outcropping, Phineas’ tamping iron unexpectedly sparked. The explosion propelled the 3 ½-foot iron bar into Phineas’ head—in through his left cheek, through his frontal lobe behind his left eye, then out through the top of his skull. Somehow, he survived. This remarkable story has become the subject of books and innumerable medical studies because of the fascinating lessons about human behavior we can glean from it.

The Connection Between Phineas and Optimization

He is fitful, irreverent, indulging at times in the grossest profanity (which was not previously his custom), manifesting but little deference for his fellows, impatient of restraint or advice when it conflicts with his desires, at times pertinaciously obstinate, yet capricious and vacillating, devising many plans of future operations, which are no sooner arranged than they are abandoned in turn for others appearing more feasible.

Source: Recovery from the Passage of an Iron Bar through the Head

Does this sound familiar? You betcha.

Due to shortening attention spans and increasing broadband performance globally, today’s digital visitors behave much like Phineas Gage post-torpedo brain. Why? Every day, 822,000+ new websites are launched, and 300+ new apps go live. Your website and app users are quickly becoming less patient and more distracted with a near-limitless degree of choice.

So what is the marketer or web designer to do when hypothesizing experiments to improve their websites and apps?

Psychologists have identified practical “rules of thumb”—called heuristics—to help us approach how humans (aka Phineas and your site visitors) might think and act when they land on your site.

Below are six common heuristics that serve as a great starting point for experiment hypotheses. When hypothesizing A/B tests, try thinking about what would engage visitors with a Phineas persona. How would Phineas act?

6 Heuristics to Generate A/B Test Hypotheses

1. Scarcity

The perceived rarity of a product or service is positively correlated with inferred value. Remember, Phineas is “capricious” and “vacillating”—entice him with scarcity!

• For example, try adding timeliness or urgency to your copywriting. Here are real two examples of this heuristic in use.

Example of scarcity heuristic in action.

2. Recognition

The comfort of the familiar. Phineas will likely pay you a premium if you can show him content he recognizes.

• For example, try personalizing aspects of your site to large segments of your visitor base.

3. Naive diversification

Simultaneous choice vs. sequential choice. Don’t cause Phineas to confuse himself with making too many choices at the same time!

• For example, if the goal is to drive visitors to sign up remove all distractions and just focus the messaging and calls to action on that action.

4. Anchoring & Framing

Thanks to Daniel Kahneman and Amos Tversky, we know that Phineas will tend to 1) rely too heavily on what you show him first (anchoring), and 2) judge things—like price—in relative terms, not absolute (framing).

Anchoring: in short, don’t sandbag yourself. The first price you show (or tell) a customer is extremely important because it sets the bar for Phineas’ subsequent value calculations in his mind. For example, if you show Phineas a pricing option of $399/mo on your website, your sales team is going to have a heck of a hard time convincing him to pay $4,000/mo because he’s been anchored so low.

Framing: Dan Ariely’s Predictably Irrational showcases a great example of when The Economist used a price framing strategy (image below) to increase the price of the print-only subscription (red box) which, in turn, increases the perceived value of print + web ($125 + $59 = $184) which makes the print & web subscription seem like steal for $125.

• After conducting a study with 100 of his MIT students, Ariely observed that the presence of the $125 print-only subscription increased sign-ups for the combo package by 165% and reduced the selection of the cheaper option ($59) by 76%.
• Extrapolated to 10,000 Economist buyers, the inclusion of the “print-only” price frame would add $343,000 in additional revenue.

Source: Predictably Irrational by Dan Ariely

• Another strategy would be to sidestep the anchoring & framing bias altogether by forcing all visitors to contact sales like Workday does.

5. Fluency

The more skillfully or elegantly an idea or product is communicated, the higher the inferred value. Phineas likes a silver-tongue: fluent, fast, smooth. Like the chef from South Park, childr’n.

• Conversion focused copywriting is a powerful tool in your optimization toolkit. Try experimenting with the types of unique selling propositions you use to market your product or service.

6. Social proof

Touting popularity and ubiquity is a very effective form of persuasion. People are influenced by (or act upon) the actions of others, e.g. if Microsoft says its good, it must be good.

Tips to Turn This Into Action

Start with some basics:

• Empathy, empathy, empathy: Did you know that less than 1 in 10 executives has actually gone through their visitor sign-up flow or used the product they sell? Yes, you read that correctly. Most shot-callers are completely disconnected from the experience their product or service provides. Don’t let “HiPPO” run your company.
  • Action: Get REAL user feedback. Check out UserTesting.com for video screencasts and live audio of your Phineas Gage-esque users experiencing your website. Feel their pain, and generate a pain-mitigating hypothesis you can test.
• Less is more: Reduce choices by being a modern-day minimalist. Don’t make it hard for Phineas—he is very capricious!
  • Action: Compare the two forms below. What form would Phineas rather fill out? Exactly. Where can you streamline your web or app flow?

Source: kylerush.net

• Faster is better: We’ve all heard “patience is a virtue” but our dear friend Phineas couldn’t be less patient. Like ferrets on methamphetamines, 40% of visitors today abandon a website that takes more than 3 seconds to load—80% of which will NEVER return. Goodbye Phineas! [EXIT stage left on a Chilean stagecoach]. For mobile, 74% of visitors will bounce if the app takes longer than five seconds to load! Holy mackerel that’s a lot of visitors.
  • Action: If you’re an Optimizely customer, keep your snippet streamlined by 1) archiving old experiments, and 2) removing jQuery if you already have it on the page above the Optimizely snippet.

Take a look at your digital properties and ask how would a Phineas-like persona behave here? Look for opportunities to apply these heuristics as you develop hypotheses to test out across your digital channels.

How can heuristics apply to your business, your visitors, and your optimization strategy? From the Optimizely staff, we’d love to read your thoughts and comments below—please share!