Why Experiment Server-Side?

Experimentation was a core tenet of our development process: every change to Google search results was A/B tested on a small fraction of users before launching to everyone. During my time there we made thousands of improvements to Google’s search results, each one scientifically tested, with measured impact on our key performance metrics.

What is the impact of switching the ranking of the first two search results from Google? On clickthrough rate? On ad revenue?

Experimentation provided a way for us to answer questions like these quickly and easily. Not all our experiments involved ranking algorithms – simply changing things like the spacing between search results could have a significant impact.

My first project at Google was helping to build our internal platform that was used to run live experiments on our search results page. Having a centralized platform was essential to support hundreds of experiments running at one time across a large team. The platform also provided a common set of key performance metrics that were an input to every launch decision. Each week, at the Search Quality meeting, we’d look at experiment results from live traffic before making a decision on what would launch to users. You can see a video of one of our meetings here.

One of the core tenets of our experimentation practice was that experiments were run server-side. We experimented by changing code on Google servers rather than modifying content inside the end user’s browser or device. Running experiments server-side was essential in our case as we were experimenting on algorithms that ran on our servers. But more importantly, running experiments server-side meant that they were deeply ingrained into our product development process.

In this post, I’ll talk about the benefits of experimenting server-side, and some best practices we’ve learned from talking to product development teams.

Should you run experiments client-side or server-side?

The diagram below illustrates the basic difference between a client-side and server-side experimentation architecture. Client-side experimentation is done by experimenting directly in the end user’s web browser, mobile app, or other client device, i.e. the treatment is determined as content is being rendered to the user. Server-side experimentation is done by experimenting in code on company servers, i.e. treatments are determined before content is returned to the client.

Optimizely was founded in 2010 on the premise of making experimentation on a website as easy as possible: with just one line of JavaScript, you could empower any team to set up experiments in a visual editor and deploy them to their website instantly. Today, experimentation using client-side JavaScript has become an established practice across almost every industry, with hundreds of vendors including Adobe and Google.

But while the client-side approach continues to gain rapid adoption, it’s also become increasingly clear that it is not, and never can be, a universal solution for experimentation.

Product development teams building new features, such as the algorithms that power Google’s search results or the backend logic of a retail or travel site, are unable to experiment on those features client-side. More and more websites are using single page app frameworks with server-side rendering (i.e. isomorphic applications) making client-side implementations more difficult. As digital consumer channels proliferate, many companies are consolidating their technology stack with a unified data layer for multiple channels. For these reasons and more, server-side experimentation has been in increased demand and has been one of the top requested capabilities from Optimizely customers.

Client-side and server-side experimentation both offer distinct advantages depending on the needs of your organization. Below is a summary of some of the key differences we’ve learned talking to customers:

Put simply:

• If you’re a marketing or growth team and want to enable anyone on your team to create experiments with a visual editor, without the need for code releases, you should consider running experiments client-side. 
• If you’re a product development team and want to experiment deeply in your product, with minimal performance impact, and across multiple channels, you should run experiments server-side.

Of course, many companies have a need for both. Most of the more sophisticated companies we talk to are doing a combination of client-side and server-side experimentation, involving both marketing or growth teams as well as product development teams.

We’ve also noticed a big difference in process for teams running experiments in the client vs. those running experiments on the server. Consider the lifecycle of an experiment, from initial hypothesis to launch:

The client-side approach has a clear benefit that experiments don’t need to be built on the server up front. If the experiment is a loser, as the majority of experiments are, then no server code release is required. As such, experimentation can be performed as a standalone practice that doesn’t require a development team. Many companies we talk to running a client-side program at scale have a dedicated team for experimentation with a roadmap of experiments to run.

The server-side approach calls for a different organizational model. Each experiment may require a code release and therefore requires the involvement of the product development team every step of the way. Rather than having a centralized experimentation roadmap, each product development team should simply consider rolling out every feature on their roadmap as a A/B test. In the more experienced organizations we talk to, every member of the development team incorporates experimentation into their rollout process, rather than running experiments in isolation. This is the practice we had established on the Google Search Quality Team and was essential to ensure that every change to our search results was having a positive impact on users.

If you’re on a product development team asking yourself “what should we experiment on?” then the answer is simple: your product roadmap is your experimentation roadmap.

Optimizely X Full Stack

Last year we launched Optimizely X Full Stack to enable customers to run experiments server-side or in any part of their technology stack. We’ve also launched upgraded Mobile SDKs as well as new OTT SDKs for experimentation in connected TV applications.

Our vision is to enable product development teams to experiment on everything they build, using one unified experimentation platform. We’ve started with SDKs in Java, Ruby, Python, PHP, Node, JavaScript, iOS, tvOS, Android. The code below shows an example usage of our SDK in Python:

from optimizely import optimizely

# Initialize an Optimizely client
optimizely_client = optimizely.Optimizely(datafile)

# Activate user in an experiment
variation = optimizely_client.activate('my_experiment', user_id)

if variation == 'control':
  # Execute code for variation A
elif variation == 'treatment':
  # Execute code for variation B
else:
  # Execute default code

# Track conversion event
optimizely_client.track('my_conversion', user_id)
  

Over the past few years, we’ve heard a huge amount of interest from product development teams who are looking for a better way to experiment and roll out features in their applications. Doing experimentation at scale is hard — our ambition is to make it easy. And we’re just getting started.

Over the coming months, we’ll be introducing a C# SDK for experimentation in .NET applications, and more options to run experiments in any server language and compatible with any CDN provider. We’ll be introducing new workflows to make it easy to toggle or roll out new features. We’ll also be releasing a number of features designed to scale to large enterprise organizations, with more controls around permissions, audit logging, program management, and more.

These are just a few of the areas we believe that will make a comprehensive experimentation platform suitable for any product development team. We’re excited to keep investing in this area and helping our customers innovate with experimentation.

Start experimenting everywhere in your tech stack!

You can start running experiments in your applications using Optimizely X Full Stack today. Create a Free Trial or ask your Customer Success Manager to enable a trial in your account.