Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more! Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program

X

Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more!

Get It Now




This content originally appeared on the WiderFunnel blog.


The optimization industry has transformed over the past decade. Today, leading companies understand that experimentation is much bigger than A/B split testing on a landing page.

The C-Suite at organizations like Amazon, Netflix, Google, Walmart, and Microsoft are leveraging experimentation as a foundational growth strategy, testing across the customer journey. And other companies are rapidly following suit to avoid becoming obsolete.

But what actually characterizes the organizations that are more successful at experimentation? What are they doing differently? What are they doing better? What are they most focused on?

WiderFunnel set out to attach data to assumptions about “experimentation maturity” with a comprehensive research report. With support from Optimizely, we surveyed marketers, product managers, and growth strategists at some of North America’s leading brands like Nike, United Airlines, Showtime, American Express, Hotwire.com, MailChimp and many more.

We wanted to know:

  • What are the elements of a mature experimentation program?
  • What is the ideal approach for experimentation success?
  • What is the experimentation status quo among more mature brands?

And the findings are eye-opening.

In this post, you’ll find an overview of 6 key findings from the “State of Experimentation Maturity 2018” research report. And an analysis of the impact of these key insights for organizations that are looking to scale their experimentation programs.

Get the full research report for more in-depth benchmarking data to help you build a business case for scaling experimentation from a departmental initiative to an organizational strategy.

1. Most respondents are still only scratching the surface when it comes to experimentation.

Experimentation at most organizations is still largely in its infancy, with zero percent of respondents achieving the highest level of maturity in this report.

50% of respondents are in the first two levels of experimentation maturity.

Half (50%) of respondents are still in the first two stages of maturity, and 33% are in the third stage. None of the respondents in this report have reached the highest level of maturity: “Driving”.

2. Most organizations plan to pick up experimentation speed (literally) in the next 12 months, indicating the strategy is gaining momentum.

While there is room to mature, experimentation is clearly gaining traction for many organizations as a worthwhile strategy. To maintain momentum, most optimizers represented in this report are striving to increase their testing velocity.

“A large percentage of experiments don’t win. Improving throughput (or number of tests per property per month / week) will ultimately generate more success due to the law of large numbers.”

Hudson Arnold, Senior Strategy Consultant at Optimizely
In fact, the majority of both Small and Medium Enterprises (52%) and Large Enterprises (64%) plan to increase experiment velocity in the next 12 months. Not a single respondent to this survey is planning to decrease experiment velocity in the next year.

The majority of respondents plan to increase experiment velocity in the next 12 months.

There are several factors that influence an organization’s ability to experiment at a high velocity, such as:

  • Sample size – the number of people in a given test will affect how long that test takes to complete (at statistical confidence)
  • Resources – how efficient are your tools and how proficient is your optimization team?
  • Workflow – how easy is it to proceed from one step to the next when designing and launching an experiment?

“Determining when to call an experiment complete is not just about sample size. You also need to consider your existing conversion rate and the elasticity of visitor behavior on your primary conversion goal in relation to sample size.”

– James Flory, Optimization Strategist of WiderFunnel

3. At 63% of more mature organizations, Executive teams are involved in metric-setting for experimentation, pointing to Senior-level buy-in.

Success metrics and key performance indicators ensure optimizers are held accountable for contributing to business goals and driving actual growth, ensuring the longevity of the strategy.

And when the Executive team is involved in the creation of these metrics, experimentation is linked to the goals they care about, and the experimentation program has visibility at the highest level of an organization.

100% of organizations at the “Scaling” level have success metrics for experimentation. At 63%, Executives have been involved in setting these metrics.

In fact, Executive teams at 63% of more mature organizations are involved in setting success metrics for experimentation, showing an investment in the outcomes and, even more so, in the strategy’s long-term importance.

But when we analyzed the entire respondent pool, we discovered that less than 50% of respondents have metrics set at the top level.

For the majority of organizations, the Executive team is not involved in setting success metrics for experimentation.

Without buy-in and alignment at the top-level, optimization champions may experience roadblocks in sharing results and learnings, obtaining resources, and scaling experimentation efforts across the organization.

Involve your Executive team in the creation of success metrics for your experimentation program. This will allow you to:

  1. Ensure experimentation is linked to the goals your Executives care about, and
  2. Ensure your program has visibility at the highest level of your organization.

“If your Executive team doesn’t have any input on your experimentation program’s key metrics, you might be running experiments on goals that they really don’t care about – that aren’t really important to the business.

– Nick So, Director of Strategy at WiderFunnel

4. Structuring your experimentation program in a combination organizational model may pave a path to experimentation maturity.

Experimentation programs often fall into one of the following organizational structures:

  • Decentralized (All-owned testing): Optimizers exist in different teams across the organization. Each strategizes and executes experiments according to their own KPIs, as they are positioned to reach larger organizational goals. This model does not have the oversight of a central team.
  • Centralized (Center of Excellence): Often in companies where individual teams have their own sites or domains. A central body that owns experimentation across the business, encouraging the growth of skills and the expansion of the program.
  • A Combination Model: When there is shared ownership of digital initiatives, a central body works with all relevant stakeholders to prioritize and plan for experimentation. But individual teams have the ownership to execute tests.

There are pros and cons to each experimentation program structure, and optimization champions will need to figure out what will work best for their organization and their unique culture.

While a decentralized program may enable speed of testing, without proper standards and central oversight, there is a risk that teams will clash, cannibalize each other’s goals, and pull in different directions rather than pushing the organization in a unified direction.

A central body will most likely have the holistic customer journey in mind, and can prioritize and deliver experimentation campaigns that focus on the most important parts of the business. But it’s unwise to assume a central body will have the necessary depth of expertise to experiment on certain parts of the business.

58% of the most mature organizations in this report use a combination organizational model for experimentation.

A combination model attempts to combine the best of both worlds and is favored by the most mature organizations, with 58% using a combination model.

…and it clearly works. We found a correlation between higher experiment velocity and a combination organizational structure.

In the combination model, there is shared ownership of digital initiatives – a central body works with all relevant stakeholders to prioritize and plan for experimentation. But individual teams are able to own and execute tests.

52% of organizations that are running experiments at a high velocity are using a combination model, showing the agility of the organizational structure.

“While a centralized experimentation team like a Center of Excellence may be the ‘hallmark’ of a mature experimentation program, it is not surprising that a combination of both team structures is most efficient.

It truly speaks to the collaboration required in achieving organizational experimentation maturity. No matter how mature the central team is, if the organization is lacking the buy-in from outside teams and business units, that experimentation team cannot reach it’s full potential.

While the centralized team should act as the core of the experimentation, part of that is to disseminate the mindset across the organization for alignment. It should not be the sole isolated contributor to an entire organization’s experimentation activity,” explains Nick So.

5. More mature organizations overwhelmingly tend to seriously consider any experiment idea that is backed by evidence.

95% of organizations at the highest maturity level in this report – Scaling – “Agree” or “Strongly Agree” that all experiment ideas are seriously considered as long as there is evidence to support them.

This means that more people can submit potential test ideas for consideration as long as they can provide solid data-backed reasoning.

It also means that more people are accessing data, including experimentation results, in order to form solid hypotheses to fuel their experimentation programs.

…but ⅓ (32%) of overall respondents are neutral or disagree with that statement.

In contrast, ⅓ (32%) of overall respondents are neutral or disagree.

These organizations may be missing out on winning experiment ideas. Because no one person or single team at an organization will have in-depth expertise at every touchpoint within the customer journey.

Mature experimentation organizations understand that every member of a company may have a brilliant idea to test to improve the customer experience.

“Every department should be encouraged to submit ideas for experimentation. But this should only be done when a company is also confident it can complete the feedback loop and provide explanation as to the acceptance or rejection of every single idea.

An incomplete feedback loop – where people’s ideas get lost in a black hole – is one of the most detrimental things that can happen to an experimentation culture. Until a feedback loop can be established, it is better for a more isolated testing team to prove the value of the program, without the stressors caused by other parts of the organization getting involved.”

– Mike St Laurent, Senior Optimization Strategist at WiderFunnel

The challenge for these organizations as they attempt to scale experimentation is to create a system that educates everyone about the program, that requires evidence with each idea submission, and that provides a closed feedback loop for the consideration of ideas.

6. Across the board, organizations are focused on resourcing for Web Development and QA skills.

When building a team for optimization, organizations at every maturity level are highly focused on Web Development and QA. These skill sets are important to ensure experiment variations can be coded and tested for quality pre-launch, and that winning variations can be hard-coded quickly.

Experts in Web Development and QA are in high demand across the board.

The other skill sets – Design, Experimentation Strategy, Data Science & Analytics, and Project Management – are a much lower priority for all survey respondents. Resourcing for Design and Experimentation Strategy is only slightly preferred to resourcing for Data Science and Project Management.

Mature organizations, however, are also highly focused on hiring Experimentation Strategists.

While Web Development and QA skills are a priority for every organization surveyed, organizations at the “Scaling” maturity level are equally focused on Experimentation Strategy.

Respondents in this category reported double the average number of team members involved in “Experimentation Strategy” relative to any other maturity level.

This focus on strategy indicates that mature organizations are hiring experts who have a combination of strong product management basics – the ability to define requirements, collaborate with teams and stakeholders – as well as experimentation and analytical rigor – the ability to interpret test results properly, estimate test durations, and own the science of experimentation.

“Getting the right team in place to do optimization is essential. I think a lot of it is about the ability to evangelize experimentation across the organization. To do it in a way that isn’t in your face, and is very smart. To do it in an emotionally intelligent way.

When I was building my team, emotional intelligence was an extremely important consideration – it wasn’t just about technical capabilities and skills. And that has contributed a lot to the success of where we are today.”

– Lauren Schuman, Director of Growth at MailChimp

Leading brands see experimentation as a strategy to enable business transformation. However, the research clearly shows that most companies are still in the early stages of maturity.

The most mature organizations are making huge gains in terms of getting organizational buy-in and creating a culture of experimentation. These organizations have adopted a test and learn business mindset.

Instead of operating at the gut-level, they are building data-driven strategies with their experimentation at the heart of every new product or feature; their Executive teams are embracing the you-should-test-that philosophy and applying it across the entire customer journey. And this is leading to more refined, more delightful customer experiences.

But with the competition this stiff, the real question to ask is: How does your organization compare?

Get access to the full research report, and find out!

Optimizely X