Download our FREE ebook of 43 A/B testing case studies from the world's leading companies for test ideas and inspiration. Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program

Almost 90% of companies see optimization as crucial to their digital strategy, with 72% of those companies finding success. But most companies are still not satisfied with their testing and believe they could be doing better:

Econsultancy CRO Satisfaction

Improve your optimization and increase revenue by solving these five major program challenges:

Irregular and Unprioritized Test Execution

According to CRO expert, Bryan Eisenberg, approximately 61% of companies carry out less than five tests per month. Econsultancy reports that companies with significant increases in sales are completing an average of 6.45 tests per month. Most organizations just aren’t able to execute enough. If a company isn’t maximizing testing capacity, it’s losing revenue.

Those companies want to execute more, but they’re not equipped to scale due to an inefficient workflow process. To avoid this for your business, create a testing pipeline to plan upcoming experiments.

You’re able to add to the queue and prioritize each test for a consistent execution schedule. This pipeline ensures that you keep your program healthy and are able to scale testing velocity when needed.

Asana Roadmap

Example of a testing pipeline in Asana’s management software showing touchpoints as sections and experiments as tasks with variation details within them.

Asana Testing

Example of an experiment tasked in Asana with variation details described and phases of the test as subtasks.

To create a pipeline, use organizational management software such Asana or Confluence. You are able to manually track and create tasks or projects in these tools to facilitate your testing process. You could even use Google Docs to prioritize your tests with this spreadsheet process shared by Moz‘s Director of Growth Marketing, Andy Nelson.

However, these tools aren’t integrated with testing platforms and require additional resources just to manage a pipeline, which slows down your testing.

EE Testing Pipeline

Testing pipeline viewed within Experiment Engine’s complete testing solution.

On the Experiment Engine platform, we’ve included a testing pipeline within the solution so that users are able to collaboratively plan projects, manage test priority, and go right into execution – seamlessly in one place. The process becomes quicker and easier and your program continuously optimizes without a hitch.

  • Manage your entire testing roadmap from one place
  • Plan areas of focus to test (site touchpoints) & create project briefs
  • Queue new variations to be executed
  • Prioritize upcoming tests and find bigger lifts

Lack of Resources for Variations

To fill that testing pipeline, you’ll need variations. But many organizations find difficulty creating variations when resources are necessary to design and develop them.

Even with today’s testing tools that relieve technical needs, optimization resources are still required to generate variations with high-impact. Without a way to repeatedly source variations, companies are unable to optimize effectively.

EE Variation Preview

A preview of a variation on Experiment Engine. A description of the test version from the optimizer and a view of the original version is also available.

Variations need to be guided by insight to increase win rate and achieve higher lifts. To overcome resource challenges, we’ve established a network of experienced optimizers who can design variations based on your creative brief. No development or design resources needed. Variations are reviewed and approved before being added into the testing pipeline, ensuring that your optimization cycle never stops.

  • Access a network of optimizers for expert insight
  • Source fully developed, fully designed variations
  • Preview designed variations across desktop, mobile, and tablet platforms
  • Fill your testing pipeline & find new opportunities to optimize

Shortage of Actionable Insight & Learnings

Each test and its results must be saved for future takeaways. Without that insight, your program is not improving; it’s not increasing win rate nor driving more revenue. Companies come across this problem when they’re not learning from previous tests. Whether it’s a winning or losing test, the results identify areas of focus. They provide data about users that drives future variations.

For example, a test involving pricing display on a category page may result in a significant change that drives further testing on pricing across the entire site. But many companies miss this type of insight because it requires detailed documentation over time, which is needed to collect true learnings.

Spreadsheet and data management software can be used to manually log in results of each experiment. But this isn’t integrated with testing platforms, either, and requires resources that take away focus from actual testing.

EE Completed Experiments

Completed experiments logged in Experiment Engine by projects.

In Experiment Engine, each executed test is automatically documented in the platform’s “Completed Experiments” area, where data collects over time. This library of results provides users and the EE customer success team with actionable insight to drive future tests. No need to manually log data or worry about missed takeaways.

  • Keep track of winning or losing experiments to drive future testing success
  • Manage the evolution of each project and learn from each incremental change
  • Gain insight about customers to facilitate product managers & marketers
  • Segment, cross-report, and deep-dive on all testing results for relevant learnings

Uncertainty About Program Health Status

Reporting on a program’s health requires a constant survey of: the number of variations in the pipeline, the number of completed experiments, the win rate, and a comparison through time. Tracking test results are challenging enough, but if a company wants to improve its optimization, it also needs to track its program’s health.

Carefully measure your program and evaluate it against roadmap goals and industry standards. The detailed management required here is yet another obstacle, though.

EE Program Health

Experiment Engine dashboard monitors displaying program health.

On Experiment Engine, users track testing regularly with dashboard monitors that display variations in the pipeline, completed experiments, testing velocity and win rate. It compares data to program health from the past and measures against industry averages. Keep your program productive and hit milestones as planned.

  • Monitor an overview of your program health from the dashboard
  • Track testing velocity month after month to push for faster optimization

Difficulty Translating Program Results to Business Metrics

MarketingSherpa found that 44% of CMOs consider using their website as an integral part of customer research for overall strategy. An IBM report also states that 63% of CMOs believe program ROI is the most important measure of success in 2015. But translating testing results into meaningful insight for a business’s growth is difficult.

For the optimization department, they need to set realistic expectations and goals when they provide executive reporting. It takes an understanding of how optimization metrics align with business metrics.

EE QBR and ROI Projections

Quarterly business review and ROI projections reports provided by Experiment Engine.

Report the progress of your optimization program with the company using actionable metrics and projections. Experiment Engine provides high-level reports covering testing velocity, win rate, and more to show what it means for potential revenue. This way, you’re able to easily access and deliver reports that drive decisions across the company.

  • Report important program metrics to the company
  • Compare your program against industry standards
  • Share ROI with incremental revenue projections


Any optimization program will come across hurdles. But preparing for those challenges and putting solutions in place early will prevent bottlenecks and ensure a greater return. If a business wants to increase its conversions, it has to improve its process.

Empower your team with a testing platform and service that facilitates each phase of optimization so that workflow management comes easy, giving more attention to actual testing and implementation.

To see how customers have used Experiment Engine, read our case studies.

Optimizely X