Stefan Thomke: 3 questions organizations need to answer for good experiments

The book focuses on how business experiments have become an essential part of continuous innovation, which is in turn vital for driving profitable growth and customer engagement. We’ve previously written about this powerful book and this is the second in a series pulling out insights from it. 

To give you an idea of what to expect, let’s take a look at Professor Thomke’s thoughts on what makes a good business experiment.

“Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day.”—Jeff Bezos, CEO, Amazon

It’s a famous quote that gave shareholders—and everyone else for that matter—a rare insight into Amazon’s innovation engine. Amazon is a great example of an experimentation program at scale, but many of our customers ask us for tips on how to start out. Professor Thomke addresses this. Here are three tips from his book Experimentation Works on how to develop good business experiments.

Question 1: Have stakeholders made a commitment to abide by the results?

Before you begin any product experiment, your stakeholders should agree a way forward once the results are clear. That will include a commitment to consider all the findings, rather than cherry-picking data that supports a predetermined point of view. Even more importantly, the book continues, you must be open to walking away from any proposals if they aren’t supported by the data.

Question 2: How can we ensure reliable results?

The truth is that companies typically have to make trade-offs between reliability, cost, time, and other considerations. In those cases, randomized field trials, blind tests, and data analysis can increase the reliability of your results. In general, you should also consider measures to account for systemic bias, whether conscious or unconscious, and make sure the characteristics of the control group match those of the test group. 

Question 3: Have we gotten the most value out of the experiment? 

Many companies invest in experimentation, but then fail to take full advantage of the potential. on offer. To avoid that mistake, Professor Thomke explains that you need meet to focus on the areas likely to show the highest potential returns. Instead of merely concentrating on “What works?” you should be asking “What works – and where?”