Whenever possible, product managers at The Guardian make decisions about functionality and design on news pages and other web properties with A/B tests. Through a combination of A/B and user testing, PM Kerstin Exner dramatically improved paid registration conversions for The Guardian’s online dating site, Soulmates.
“At the Guardian, we are very much driven by data and not individual opinions,” Kerstin says. “This is one of the reasons website testing has become an important discipline for us, in an effort to make the website better.”
Launched in 2004, Soulmates monetizes via paid monthly subscriptions. When Kerstin started hypothesizing ways to increase registrations to the site, she saw that part of the puzzle was already solved—traffic. The ads on the theguardian.com/uk that generate the majority of traffic to the site were performing well in click-throughs, but the landing page the ad linked to was way under performing on conversions.
Kerstin’s task was to optimize the landing page to turn more visits into registrations.
Kerstin hypothesized that the landing page was confusing and too limited for first time visitors because it showed them just one profile (the profile of the person featured in the ad) with a call to action to “Join now for free” to see the full profile. It didn’t have any context on what Soulmates is about or options to search for other singles if they didn’t like the profile they saw.
She corroborated her hypotheses with user research. In feedback sessions, people said that want to easily see a broad array of profiles when browsing the site. Using Optimizely, Kerstin decided to A/B test this feedback.
She designed a radically different test landing page didn’t include a call to action to sign up directly. Instead, it tried to engage users by giving them many more options to find singles in addition to the one from the ad. She added a search bar, headshots of other singles you might be interested in, a headline explaining what Soulmates is, and quotes from other couples who met on Soulmates. Where the original landing page was highly focused on the user profile from the ad, the new page enabled more discovery and search for other people.
She used Optimizely’s AB testing tool to track the number of successful registrations for both landing pages.
The variation landing page improved conversions by 46%. She used Optimizely’s traffic allocation feature to push 100% of site traffic to the better converting landing page while the engineering team built the changes into the site’s code.
This test was a big win for Kerstin and Soulmates, but not all tests have yielded such a high conversion increase.
“Of the 14 tests we have run so far on Soulmates, seven have been successful, three have been negative, and four have been inconclusive,” Kerstin says.
But knowing what doesn’t work can been just as informative as knowing what does work. See what else The Guardian tested and get the full story of what they learned by downloading the customer story.