Best Practices and Templates for Making Experimentation Impact Visible Consistently Across your Company

I’m Becca Bruggman, Optimizely’s Experimentation Program Manager. My job is to make sure we are “drinking our own champagne” and run a best-in-class experimentation program.

This is the fifth installment of a six-part series designed to help you run a best-in-class experimentation program. In the series we are covering everything you need to build your program, develop it and have it running efficiently as well as create a well oiled machine and make it visible.

If you read my recent blog post, I shared a variety of frameworks and tactics for deciding how to prioritize all your experiment ideas and running a weekly experiment meeting! Now, let’s discuss how to make all the impact from your experimentation visible!

When it comes to program visibility, I’m a believer in using multiple methods for sharing information. Additionally, doing as much as possible to “meet people where they already are” by using email lists and meetings that already exist to piggyback on those pre-existing audiences and processes.

For Optimizely on Optimizely, I’ve used a variety of tactics for making the program visible over the last few years, bulleted here and each covered in more detail below:

1. Physical Walls showing experiments planned and in progress
2. Leverage already existing meetings or processes to share experimentation updates
3. Sending company-wide emails when an experiment is launched or completed
4. Regularly presenting interesting learnings from experimentation during team meetings
5. Sharing learnings to specific groups on Slack who would be most interested
6. Blogging to share new tools, tactics or learnings about the program externally

So, what do I mean when I say physical walls? Check out some of the ones I’ve used over the last few years:

Including experimentation into product delivery agile processes

Summarizing all current experiments running and planned against a specific quarterly goal

The first image shows all experiments individually by status within agile product delivery and the second shows a summary of all experiments currently running or planned for the quarter. I found creating physical spaces to show experimentation program status especially helpful when we were first getting the program off the ground. It helped with bringing experimentation out of the digital world and into the physical world, where people could see tangibly where they already were. 

In the first picture, these cards exist on our wall of work that represents all engineering and product work from dreaming to doing. Having the experimentation work also represented on the wall of work makes sense as product experimentation is being done by the same product squads that are represented on the wall of work and prioritized similar to other product work, so it should be included within this existing process. Additionally, we already had a weekly meeting in place where we shared updates from the wall of work, so I was able to get time to share experimentation updates or, even better, have the product managers themselves share updates by adding cards to the already existing wall.

Similarly, we have an email alias called “Visible Changes” that already existed for sharing changes to the product with the whole company. Since experimentation also made changes to the look and behavior of the product, I piggybacked on that email alias and started sending experiment launch emails with the title “[visible-changes][experiment]”. 

Informing your company every time an experiment is launched is standard best practice. This also heightens and maintains the visibility of the program. The extended team knows experiments are being launched consistently and that experimentation is an important, active program. Additionally, making sure customer-facing teams are aware of customer-facing UI changes cuts down on surprises when demo-ing or at customer on-site visits. 

I also aim to have those who had the idea for an experiment send the visible changes email, for several reasons. If I send the email every time, people are more likely to tune it out. Additionally, seeing teammates contribute to experimentation and getting company-wide visibility can encourage other team members to contribute via the “Culture of FOMO” and wanting to get that sort of visibility as well. Finally, if people after reading the email have questions they know who to go to with questions based on who sent the email.

These emails are, by design, lightweight. Below is an example below of one I’ve sent at Optimizely and the Experimentation Program Toolkit includes a template that you can use for your own program. I recommend sending these in advance of the experiment launching and to as large an email alias as possible, I personally send them company-wide.

Optimizely Example of Experiment Start Email

Just like the experiment start emails, these emails can be lightweight. I’ve included one that Perla, a member of our Demand Generation team, sent at Optimizely and the Experimentation Program Toolkit includes a template that you can use for your own program to share results. Just like with the experiment launch emails, I aim to have the experimenter respond to the previous experiment start email, including details on the results and next steps.

I recommend sending these when you are ready to pause the experiment based on statistically significant results and/or learnings that you are ready to take action upon and share with the team. Again, we send these company-wide.

Optimizely Example of Experiment Results Email

I also regularly attend team meetings to share interesting experiment learnings, especially when it’s an area that the team is especially invested in themselves.

Sharing Experimentation Updates at Show and Tell at Optimizely

I’ll also share experiment updates asynchronously over Slack. I aim to share in pre-established team channels and with teams that would be especially interested in the results. This can be especially fun if you’ve shared the experiment at launch as well for the team to emoji vote, as I shared a few blog posts back. Marcos is happily surprised by the results!

Finally, as you can see from my blog posts – I love sharing learnings and making our program visible externally! Some organizations are more comfortable with this than others, but sharing best practices to level up the community is great to show off the amazing work you are doing, demonstrate your thought leadership and recruit more great people to your company.

You can find all templates noted above [here]. 

What are your tactics for making experimentation visible within your organization? How do you celebrate learnings? Comment below or tweet me @bexcitement.

See you in the next and last post on showing the ROI of your program!