5 Ways to Gain Greater Visibility for Your Experimentation Program

I’m a strategy consultant at Optimizely, and I’m often asked by our clients how to push for greater visibility and impact within their organization. Given how important alignment and visibility by executive leadership is to any team in the field of experimentation, I wanted to share a few best practices that can easily be adopted into your program regardless of size or scale.

Show the success of the program

It’s vital to establish and measure critical success metrics of your program. These come in the form of both ROI impact as well as more operational metrics such as velocity of tests you’re executing, rate at which you are hitting statistical significance, quality of tests executed, or efficiency of executing tests. These are great measures to check the overall health of your experimentation program. If you are having trouble hitting topline objectives, it may be helpful to assess your program based on these metrics to see where you can improve and what levers you can effect to drive more impactful ideas

A little bit more on ROI…

Almost every experiment practitioner has heard the question: “What is the ROI of experimentation?”. Experimentation has so many inputs and dependencies there isn’t always a linear line to ROI. We are also often testing at the same time that various campaigns are launching There  are seasonality factors, “newness effects” or other larger macroeconomic circumstances that affect the business. Given all these complexities, many programs go unmeasured. In fact, a Marketing Sherpa survey asked how many people have measured the ROI impact of optimization, and 53% said they did not or could not measure ROI.

It is important to set the expectation that ROI should not be the lone metric to measure the success of your optimization program. There is a constellation of metrics that are critical for showcasing the value of experimentation. Common metrics we recommend are test velocity, win rates, stat sig rates, operational efficiency, number of teams testing. For most of us, however, that isn’t going to always be received well by executive stakeholders.  To help address this question, a few ways I’ve seen to approach having a ballpark for ROI:

• A general estimated approach through doing a linear analysis and caveating that this is a ballpark, optimistic number.
• Deploy a test into production, but keep a small hold-out group which maintains the old experience to gauge long-term impact. (A word of caution: this can be time-consuming and also means you’re not optimizing to that percentage in the hold out group.

The best I’ve seen (and admittedly I’m a biased) is a calculator Optimizely launched at our user conference Opticon. The ROI calculator has clear and transparent assumptions, is flexible to your organizational needs, and uses specific data to calculate it. Going forward, using this ROI calculator, you’ll be able to answer the ROI question clearly with data and logic to back it up.

To learn more about the calculator and get access, you can check out the blog post.

Repetition is the key to understanding.

Don’t be afraid to constantly shout your wins from the rooftop, using clear communication via emails & newsletters, Optimizely program management, as well as a robust public testing calendar with clarity on what is being prioritized to launch and why. The “public relations” component of a program in which you are using multiple forums to talk about how you prioritize, what criteria there is for testing and why the rigor is important, is often overlooked.However it can be extremely effective in ensuring the right quality of ideas coming through the pipeline. Quality in equals quality out.

Further, do you have an executive sponsor for experimentation? You should. Having a sponsor can be exponentially helpful in communicating the importance of the program, rigor and value of experimentation. A sponsor is instrumental in driving  the momentum of the program, clearing roadblocks and investing more resources to drive growth.

Velocity or Quality? It’s a balance

Generally, organizations newer to testing focus on velocity as a primary metric for program success. Velocity is a highly impactful metric to hold yourself to.  It ensures adoption of experimentation, increases the likelihood of hitting statistical significance and helps you arrive at insights more quickly. But you can hit a tipping point where the approach of a high volume of small, iterative tests begins to slow your ability to drive incremental gains.The key is to find the right balance between continuing to increase velocity, but making those ideas more sophisticated and going deeper into areas that you haven’t tested before. Some thought-starters:

1. Constantly educate your teams on the correct  rigor and methodology of ideation. Optimizely uses the “Problem /Solution /Result” approach and robust prioritization frameworks to come up with clear and solid hypotheses that can highlight the most impactful ideas.
2. Move from testing the average to testing the  segments. Yes, it’s all about personalization. Once you’ve gained insights from your past testing on what works for everyone, it’s time to dig deeper. Look at your data and investigate nuances like  “new” vs. “returning” customers, geo, affinity, loyalty, previous purchasers vs. 1st purchasers, etc. and begin testing experiences on those segments to extract more value.
3. Find new sources of ideas. Try looking at different reports and data. If you’ve been using primarily Google Analytics funnel data to identify opportunities, try look at heatmaps, customer insights or survey data. If you are getting test ideas submitted from other team members, look for teams/functions that haven’t yet been involved and see if they have new ideas to bring to the table.
4. Think end to end, the entire ecosystem of your product from ideation to development to marketing. Most companies who are beginning experimentation, generally start with changing superficial elements of a page, such as buttons, layouts, colors, copy, images, or order of content. To move experimentation further, it’s time to start exploring concepts of features, functionality, algorithms, page speed, and moving onto new platforms like testing on your apps. Using Optimizely Full Stack unlocks a whole new set of quality test ideas. You can begin to think about ideas such as sort algorithms, recommendations logic, introducing new functionality like chatbots, all while also reducing page latency and flickers.

Plan for growth

Every leader generally sees their success exemplified through increased ownership and the growth of their team. Get your execs excited by approaching them with a point of view on how the experimentation program can grow organizationally. Identify what organizational model you are at the moment, and think through how you may need to evolve to continue to grow.

Most early maturity programs begin as an individual team, testing for a single function or business unit. Another option is a centralized model, owning testing and execution for multiple teams. For more on team structure you can read my colleague Alek Toumert’s blog post on this topic. With increasing maturity, most orgs move to a council or center of excellence model to decentralize ownership to multiple execution teams. This is a great model for broadening adoption and getting more teams testing.

In addition, articulate  who on your team is currently dedicated to experimentation, as well as gaps in expertise that may be holding your program back. From there begin to craft what headcount you need on the team in order to achieve more.

Think outside the box.

How can you shine a light to execs and the broader org on all the great work and insightful updates you have through testing? There are so many ways to generate excitement and involvement:

• • Host an Experimentation Summit. It’s a great method to share your achievements and gain agreement across the organization on what experimentation does, how it helps add value to the company and what key initiatives your team plans to go after.
  • Newsletters. You can include fun content of “did you know?” facts about experimentation, quizzes showing control and test variations asking readers to guess the winner, and it can include snapshots of your roadmap to inform everyone what tests have launched or are planned to launch.
  • Brown Bags/Company meetings. This is a fun, informal way to educate people who may not know there is an experimentation program, or have a very limited understanding of how it works and why it’s important.
  • Start a Learning & Development class. The goal here is education on important concepts, methodology and approach to testing that your team is undertaking.
  • Run a contest. Incentivize your teammates to get involved. It fosters interest, healthy competition and who doesn’t love a prize?
  • Office hours. Create transparency and ease of access to you and your team.

There are so many other ways to drive your experimentation forward beyond these 5 ideas. In truth whatever feels most authentic to your company and process is what will be most successful. What ways have you driven experimentation in your organization? Leave a comment!