The Most Misleading Measure of Response Time: Average

One of the most common questions we hear from customers is, “How will Optimizely affect my page’s load time?” They have good reason. Page load time has never been more important. Pages with faster response times reduce bounce rate and they can even improve your ranking in Google searches.

Any third party code snippet you add to a web page will impact its overall load time. This is because, in most cases, the third party code needs to finish loading before the rest of the page can begin to load. If the vendor providing the code is doing a good job, that impact should be so small that your visitors don’t notice it at all.

Some vendors attempt to mitigate impact on load times by providing an asynchronous code snippet—in other words, a snippet that loads simultaneously with the rest of the page. This is primarily used for background services, like Google Analytics, where the code isn’t making a visible change to the page. In most cases, it isn’t appropriate to load the code snippet for an A/B testing platform like Optimizely asynchronously. This is because parts of the page may load before the snippet, causing the user to see the original page flash on the page and then change to the appropriate variation. That’s why we recommend including the Optimizely snippet at the beginning of the head tag. Some vendors provide timeout thresholds to prevent this from happening, but doing so limits your sample size to visitors within a certain response threshold—also not ideal.

Regardless of the implementation, a vendor snippet needs to load before the rest of the page to be useful, so minimizing snippet response time is still extremely important. The first step toward managing response times is knowing which metrics to use to measure it, and unfortunately, this is where a lot of businesses make mistakes. Frequently, businesses will turn to “convenience metrics” average (or mean) response times to gauge overall performance. However while the average might be easy to understand it’s also extremely misleading. Why? Because looking at your average response time is like measuring the average temperature of a hospital. What you really care about is a patient’s temperature, and in particular, the patients who need the most help. The best way to measure this is to track the 99th percentile response time: the worst 1%.

Similarly every visitor to a website experiences a different response time, and the slowest 1% of response times represent the visitors having the worst experience. Let’s look at a real world example. This chart shows the overall distribution of response times for the Optimizely snippet worldwide, with the average response called out explicitly.

Source: Catchpoint.com, data from Oct. 15, 2013 to Nov. 25, 2013 for 30KB Optimizely snippet.

Now, let’s add in the data for the 99th percentile:

Source: Catchpoint.com, data from Oct. 15, 2013 to Nov. 25, 2013 for 30KB Optimizely snippet.

As expected, the 99th percentile is much higher than the mean, as it represents the slowest 1% of response times—the worst 1% of experiences. We also compared the 99th percentile response times by geography:

Source: Catchpoint.com, data from Oct. 15, 2013 to Nov. 25, 2013 for 30KB Optimizely snippet.

The 99th percentile for North America is the worst across the geographies we measured. With the majority of web traffic coming from North America, we needed to find a way to improve response times at the 99th percentile. There are many ways a website owner can cut response times, but using a CDN is one of the most effective.

At Optimizely, we had already been using Akamai, one of the world’s fastest and most reliable CDNs, so we decided to try adding another. We tested a balanced CDN architecture, combining Akamai with EdgeCast, another leader in the CDN space. The result was a major reduction in 99th percentile response times across the board:

Source: Catchpoint.com, data from Oct. 15, 2013 to Nov. 25, 2013 for 30KB Optimizely snippet.

We found that using this new, balanced CDN architecture improved average response times by 64%, and more importantly it improved the 99th percentile of response times by 42%. As you can tell from the chart above, this is a substantial improvement for the slowest 1% of response times.

We also looked at how this improvement in the 99th percentile broke out across geography:

Source: Catchpoint.com, data from Oct. 15, 2013 to Nov. 25, 2013 for 30KB Optimizely snippet.

With a balanced CDN architecture, the 99th percentile became 161% faster in North America and 42% faster worldwide. For customers, this means that even in the worst case scenario, the Optimizely snippet will load faster making it even less noticeable to visitors.

This is why we’re excited to announce that we have made this new, balanced CDN architecture available to all customers regardless of plan level. If you’re an existing customer, there’s no further action you need to take. Your snippet code is already available on both of our CDNs and it’s being served by both as appropriate. If you’re not an Optimizely customer, this is a great time to test it out on your website.

For more information on CDN Balancing and how to measure website response times, download our white paper on the topic, here: The Most Misleading Measure of Response Time.

Maintaining low response times has never been more important than it is now. Web apps continue to grow more complex, while consumers’ tolerance for slower load times continues to wane. One second of unnecessary page load time can cost a company millions of dollars. At Optimizely, we’re committed to providing a platform for testing and personalization that integrates seamlessly with our customers’ websites and scales with their businesses—even in the worst case scenario.