Opticon19 Europe: How the BBC Gets it Done. With Scale & Sophistication.

And a vast array of content broadcast across radio and TV, websites and apps, live and on-demand. Experimentation certainly has its work cut out at the BBC.

Recently at Opticon19, BBC senior analyst Heather Miller and experimentation lead Mamshad were on hand to answer two of the day’s most fundamental questions:

1: How does such a vast, diverse organization like the BBC continue outperform the competition? 

2: Did they plan the matching red tops that they had on stage? (The answer was no, by the way.)

Back to question one, Mamshad and Heather summed up their experimentation strategy in two words: scale and sophistication. 

Talking through Experimentation at the BBC

Scale: test everything, everywhere. 

In practice, scale entails measuring the impact of new features at every step of the user journey and across every discipline including product, UX, and editorial. Mamshad and Heather explained how the BBC’s center of excellence plays a key role in making that happen, supporting up to 30-40 digital teams all the way from getting started to analyzing results. This centralization also allows learnings to be quickly shared across the entire business – be that in a monthly experimentation newsletter or via a central results hub.

Sophistication: maximizing the quality of experimentation 

For the BBC, sophistication means making the best use of development resources. Take cross-platform experiments, for example, which provide a complete view of the way a single service – such as iPlayer – is used across devices. People may browse for content on a phone, but then save it to watch later on a TV.

Integrating Optimizely with an existing wealth of analytics data allows the team to drill down into experimentation results. And by adopting an audience segmented approach to targeting and reporting, the team is increasing its capability to optimize for their varied user base to have a more personalized experience. 

And then there’s the R.U.B.

As a not-for-profit organization, how does the BBC measure the impact of experimentation in the absence of commercial targets? It represents the four key metrics that safeguard the ultimate goal of high-quality public service content: Reach, Usage, Breadth and Sign-ins. 

In pursuing their strategy of scale and sophistication, the BBC is able to quickly quantify the impact of new features. The BBC is able to use experimentation to power innovation in a fiercely competitive market, as well as to mitigate risk by avoiding financial and reputation mistakes.