Download our FREE ebook of 43 A/B testing case studies from the world's leading companies for test ideas and inspiration. Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program

X

Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more!

Get It Now



As Optimizely’s Experimentation Program Manager, I was very intrigued by the launch of our new and improved Multi-Armed Bandit. I was excited to start using this powerful, machine learning optimization tool that was designed specifically to increase conversions. This was especially exciting for my Marketing team, who wanted to get more conversions on key landing pages and webinars, so I wanted to share some of the best use cases for this powerful feature!

First, a quick definition of what a Multi-Armed Bandit is:

  • Scientific: The term “multi-armed bandit” comes from a hypothetical experiment where a person must choose between multiple actions (i.e. slot machines, the “one-armed bandits”), each with an unknown payout. The goal is to determine the best or most profitable outcome through a series of choices. At the beginning of the experiment, when odds and payouts are unknown, the gambler must determine which machine to pull, in which order and how many times. This is the “multi-armed bandit problem.”
  • Practical: In other words, the better a variation performs, the more traffic a multi-armed bandit will send its way. So, if all you care about is conversions over a specific time period, like a weekend promotion or a webinar, and don’t need statistical significance – this is a great tool! As a reminder, A/B tests don’t do this. Instead, they keep traffic allocation constant for the experiment’s entire lifetime, no matter how each variation performs.

Here is a visual representation of how a Multi-Armed Bandit will re-distribute traffic automatically across variations.

mab_gif.gif

You can also see this visually from an example Optimizely results page from a Multi-Armed Bandit I ran recently. You’ll see that the number of visitors in each variation is different depending on which variation is converting best. The Multi-Armed Bandit will automatically send more traffic to the higher converting variations.

Metrics MAB

Traffic Distribution from Real-World Multi-Armed Bandit

When does a Multi-Armed Bandit make sense for you?

  • You do not need statistically significant results: Multi-Armed Bandits are designed to drive conversions over a given period of time, however, they do not provide statistically significant results. If you need to experiment on an experience you plan to roll out permanently, this would not be a good use case for leveraging Multi-Armed Bandit.
  • You have a time-bound or conversion oriented experience to optimize: Multi-Armed Bandits are conversion driven, so they are best used for things like a week-only promotion or single webinar happening on a specific date OR highly specific, conversion-oriented landing pages where you could run a Multi-Armed Bandit indefinitely.
  • Conversions are happening close to the change being made: Multi-Armed Bandit used the Primary Metric to determine traffic allocation, so the primary metric should not be set to something that is a delayed conversion or far down the funnel from where you are running your Multi-Armed Bandit. For example, I could run a Multi-Armed Bandit on my webinar pages to increase conversion to that webinar, but optimizing for that lead then becoming a Sales Qualified Lead (SQL) would be too far down funnel.

So, with that context in place, here are 5 great use cases for Multi-Armed Bandits!

  • Webinar: In advance of a webinar happening, creating a Multi-Armed Bandit with many variations specifically to drive more people to register can help drive registrations for the webinar.

Optimizing CTA Copy for Webinar Attendance

  • Holiday Weekend Promotion: Optimizing different promotions for a given holiday weekend to see what drives the most conversions such as add to cart or making a purchase. See an example below:
  • Conversion-oriented Landing Page or Modals: Many companies have landing pages or modals that are specifically designed to drive downloads of content, capture emails, or generate leads. Creating a Multi-Armed Bandit to optimize for the right messaging or call to action to specifically drive conversions can be impactful.
  • Headline Testing: During the first few days of a specific news article being released when you are especially looking to drive clicks and views on that content, leveraging a Multi-Armed Bandit to optimize for the right headline to drive clicks and views can be impactful. Below you could imagine optimizing the headlines for click-through to achieve maximum clicks.
  • Travel Promotions: Driving conversions for a specific flight or hotel can be helpful to ensure that you have a fully booked flight or property in advance of a specific date. Leveraging a Multi-Armed bandit to see what promotions will drive the most conversions in advance of a specific date can be impactful.

I hope this helped you better understand the exciting possibilities for Multi-Armed Bandit. If you need any help with set-up, please reach out to your account teams for help.

What optimizations are you planning to run? Comment below or tweet me @bexcitement.

Optimizely X