Quantify the ROI of Your Experimentation Program

The Optimizely ROI Model was created to empower experimenters like you to better understand and demonstrate the value experimentation is driving within your business. Most of you are so busy driving significant business gains through experimentation that there isn’t time to do the data analysis to develop a credible model. Yet this is what you need to do to show impact and gain executive level support for investing in experimentation.  

The good news is that we’ve created this model for you – fully customizable and based on your data. Across all of our customers in the last year we’ve driven $1B in incremental revenue through experimentation. With our ROI model, we help you translate your Statistically Significant Lift into Revenue. 

At Optimizely, as we set about to build the product we outlined the following ROI Model criteria:

1. Provide an approach rooted in logic and math.  
2. Build something logical and transparent.  
3. Flexible to our customers’ organizational needs.
4. Supported by our customers unique experimentation data

To ensure we met the above criteria, we have begun with an Excel based model approach, where we custom-deliver a model with your historical experimentation data.  This data can be refreshed anytime via the Optimizely Success team or the Optimizely Application. The model is fully unlocked with a detailed assumption and instruction tab to walk your through best practices for use.  

The Optimizely ROI Model centers around three key concepts; Metric Alignment, Time-Horizon Discount, and an Intangible Discount. 

First, Metric Alignment. Not all actions you can influence in your digital touchpoints are of  equal value. For most organizations there are 1-3 core metrics that are revenue driving and are the ones that  you are looking at when measuring success. Our Model takes a metric first approach that aligns your calculations around those core metrics. Each custom model is populated with historical experiment data.  Over time, you will find the more consistently metrics reused, the easier it will be to calculate quickly and compare your programs evolving success.

The Time-Horizon Discount is a concept that captures a few different ideas like The Novelty Effect or Diminishing Returns which all center around the idea that the impact you see on day one will not be at the same level in perpetuity.  To capture the notion that impact will reduce over time, we looked to our historical experimentation data set to help us. We leveraged the data from the million plus experiments run with Optimizely to do an Isotonic Regression to see how they actually decayed month over month.  We were able to calculate this by the type of Metric for more accuracy. We have built this flexible discount into the ROI calculations and will continue to analyze over time to increase accuracy.

Finally, we incorporate an Intangible Discount.  The Intangible discount is meant to capture the immeasurables likes interaction effect.  Although, interaction effect can be controlled using Multivariate Testing, for most experimenters this isn’t practical because of the additional time and complexity it takes to run them.  Accordingly, we have given you a range and discount framework to help you incorporate this discount in program results.

Optimizely is excited to share a custom version of our ROI Attribution model with all customers moving forward.  We are excited to partner with you, our customers, to help you make this model your own. At Optimizely, weve seen first hand how experimentation drives not only innovation but revenue – we are excited for you to as well!

Reach out to your Customer Success Manager or CustomerSuccess@optimizely.com to get your custom model and training session today!