Download our FREE ebook of 43 A/B testing case studies from the world's leading companies for test ideas and inspiration. Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program


Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more!

Get It Now

When building out your experimentation program, there is one metric that you should be sure to look at: the number of experiments run.  The most successful companies in the world run the most experiments in order to stop guessing and start knowing.

Jeff Bezos of Amazon states “Our success is a function of how many experiments we do per year, per month, per week, per day.”

However, most companies still think of experimentation last, after the web page has been developed, the content created, and the site has been launched to the public. This often leads to optimization being reprioritized and can make scaling experimentation challenging. Mature experimentation companies build experimentation into their content creation and development process ahead of launch.

So why then, given the obvious success of industry leaders such as Amazon, do companies still fail to integrate experimentation into their DNA? Could it be the perception that integration with the existing stack might be complex?

At Optimizely, one question we hear often is “Do you integrate with our CMS?”. While there is not an out of the box, one-size-fits-all integration, you can create integrations and processes that are use case-specific.

In 2014, as part of a hack week project, we launched a WordPress Plugin that allows users to launch headline experiments from within the CMS, view the results, and then launch the winners. This solution was one of the first that brought experimentation into the content creation process and provided the ability to launch headline tests. Since its launch, the WordPress Plugin has had over 3000+ active installations and has resulted in over 10,000 experiments, averaging 3 variations per experiment.  

While WordPress is still considered one of the most popular content management systems out there, a new breed of enterprise-ready solutions are enabling developers to configure their CMS solutions for the modern web. Headless CMS solutions are a new trend that enable developers to quickly launch new content, across multiple devices, using the language they prefer. Even the most popular CMS solutions like WordPress and Drupal are developing headless solutions.

The CMS that we are using at Optimizely is called Contentful.  Contentful, provides content infrastructure for digital teams that is highly configurable. It’s ability to customize the end user experience by developing UI Extensions was one of the main reasons we chose Contentful. This ability to customize has allowed us to begin building experimentation into our content creation workflow.

Implementing a Modular Content System

Before we move on to how our marketing site is using Contentful and Optimizely together, it is important to understand how most of the pages on our site are created. Our homepage, products pages, and most other pages were built on a module system. This allows for content creators to use a preset group of modules to launch new landing pages with minimal help from our web team. On the left is a screenshot of our homepage, and on the right is what it looks like in Contentful.

Experimenting with simple changes like text, images, and rearranging elements are easy using our visual editor. However, sometimes you’ll want to experiment with completely different layouts. With Optimizely X Web you have two options: create two separate pages with two separate URLs and launch a redirect experiment or make all of the changes using only the visual editor.

Enter Optimizely Full Stack. With Full Stack we are able to run a split page experiment using the exact same URL, and since the content is rendered on the server side, there is no chance for flashing or reduced client side performance.

Integrating Optimizely with the Contentful CMS

Let’s walk through how we have integrated Optimizely into Contentful to enable our content creators to launch split page experiments from directly inside of the CMS. While the examples provided below are focused on Contentful, these concepts can also be applied to any CMS that allows a developer to customize the UI.

We had a hypothesis that if we added a carousel to the homepage in the hero section, it would result in more homepage engagement. Since this change is above the fold, and is completely configurable in Contentful, we figured this would be a great Full Stack test.

First we created two different versions of homepage, one with the carousel module, one with the regular static module.

Now we can create a new piece of content that we call ‘OPTLY – Split Page’. Simply give it a name, the location you want the experiment to run, and the two different versions of the homepage.

Once we’ve set up the basics of our experiment, we’ll need to track events and consider our audience. This is where UI Extensions come in. We have built three different UI Extensions that plug directly into the Contentful interface that allows users to choose events, audiences, and even view the results of the experiment directly in Contentful. This way, a content creator can alway see how their experiment is doing.

Now that we have the content ready to go, all we have to do is publish this item in Contentful. When the content is published we are using Contentful’s Webhooks feature to then send that data to a worker process. That worker process will accept the POST from Contentful with all of the content information. We then use that information with the Optimizely REST API to create a new Full Stack experiment.

If we go into the Full Stack experiment, you’ll notice that we set the name of the experiment to include the Contentful ID of the OPTLY – Split Page test. This will allow us later to query Contentful, if the user is bucketed into the experiment.

You will notice that the variation names or keys are simply just the Content ID’s of the different versions of the homepage we used to be a part of the experiment. Once again this is so that when we activate the full stack experiment, it will tell us which variation key to use and we can use that ID to show the correct homepage to the user.

When the experiment is started, anyone going to the path specified by the user that meets the audience conditions will be bucketed into one of the variations and shown the corresponding index page.

Just to recap, here is the process that will need to be set up to allow a user to launch experiments from their CMS:

  1. User creates content in their CMS
  2. A worker process is set up to listen for new content
  3. The worker process uses the Optimizely REST API to create a new experiment

Useful API Methods

Here is a list of Optimizely REST API methods we used to create our UI Extensions and automate the experiment creation process.

  • Create an Experiment: Used to create an experiment when a new piece of content has been created in your CMS.
    • TIP: One event is required when creating an experiment.
    • TIP: Make sure that the allocation for each variation adds up to 100 or it will throw an error.
    • TIP: All audiences and events are arrays, even if you only have one value.
  • Events: Used to get back all of the events that have already been set up for a particular project used in the events UI Extension.
    • TIP: In order to create an experiment, it must have at least one event associated to it, so this should be required in your UI.
    • TIP: Do some initial work setting up all of the events you care most about. This will enable your team to add events when setting up an experiment without needing developer help.
    • TIP: For Full Stack experiments, use the Javascript SDK to send back client side events.
  • Audiences: Used to get back all of the audiences that have already been set up for a specified project.
    • TIP: If you do not pass an audience ID when creating an experiment, it will default to everyone. Your UI should reflect that to avoid confusion.
  • Results: Used to display the results of the experiment in Contentful using a UI Extension.
    • TIP: Once the experiment is created, store this experiment ID in a field associated with the content in the CMS.
    • TIP: Depending on the number of events an experiment has, your UI should be able to accommodate experiments with many events.

Other Use Cases

Split page testing is just one of the many integrations we plan to roll out to Contentful. Also, even though this use case uses Optimizely Full Stack, there are other integrations that we have on our roadmap to bake personalization and experimentation into our content creation process.

Module Personalization

We currently do a lot of personalization on a few key modules: Hero and Promo Cards. We already have Optimizely Extensions that we use when personalizing the content. You could setup a Content model in your CMS that maps directly to the extension fields. Then, instead of process that creates a variation using Full Stack, it would use the Optimizely REST API to create a variation using the prebuilt Optimizely Extension.

Copy Testing

Similar to the Optimizely WordPress Plugin, we could set up a content model that allows a user to enter multiple variations of text to be used not just for headlines, but for everything from CTA buttons to Hero messaging.

Image Testing

We recommend you test everything on your site: products, articles, or any other content such as imagery. Similar to copy testing, you can allow a user to upload multiple images instead of just one.

Once you have taken the time to build experimentation and personalization into your content workflow, it will ultimately result in more winning experiments. Not only will it allow your web team to focus on the more unique and higher impact experiments, but it will also transform your content creation process and make experimentation an innate part of the process, rather than an afterthought.

What integrations have you built into your CMS that have helped increase your experiment velocity? What other ideas do you have to help build experimentation into the content creation process?


Optimizely X