How To A/B Test Price When You Have A Sales Team

From order of pages in a signup flow to the exact headline that incites people to click, conversion rate optimization is a core component of marketing and growth in the business-to-business world. Companies can see massive performance increases through constant A/B experimentation if done correctly.

One key element to test is price. Steven Sinofsky, a board member at Andreessen Horowitz recently wrote that, “Nothing is more critical to a software-as-a-service (SaaS) business than pricing strategy.” I totally agree and experimenting is a key piece of pricing strategy. Testing things like how price is displayed, amount to charge, and how often is all part of the puzzle.

By running an experiment on price at Bizible, we were able to not only look at differences in lead volume, but also the change in lead to opportunity conversion and the big one: change in opportunity value (spoiler: it increased 25% for us).

For companies with a sales team, A/B experiments on price brings up three key questions:

1. How will sales know which price a lead saw?

2. How can we know what price they are going to see if they haven’t visited the pricing page yet?

3. How will we make sure everyone at the company sees the same price?

This post will walk you through our process for answering each of these questions using an Optimizely experiment with the Optimizely Salesforce integration built by Bizible. We hope you can use our experience to run A/B experiments on your price too.

1. How will sales know which price a contact saw?

In order for sales to effectively close deals, it’s important for them to maintain flow and confidence. Before each demo, our sales team visits Salesforce to brush up on the particular potential customer. To maintain this process, we worked out a way for them to easily check which variation a contact saw directly in Salesforce, so they can quote the right price on the fly, without hesitation.

Setup in Optimizely:

To do this, you’ll need to actually create two experiments. One that holds the original variation and one with the test variation.

1. Create the first experiment with the title of “Pricing” and delete the automatically generated “Variation #1” variation.
2. Since the remaining variation is titled “Original” and this is the original price, there are no other changes to make. Save the experiment.

3. Create the second experiment also with the title of “Pricing” and again delete the automatically generated “Variation #1” variation.
4. Rename the “Original” variation to “New”.
5. Make the desired pricing changes. Save the experiment.

Bizible’s Optimizely + Salesforce integration adds both the experiment and variation to the lead, contact, case, and opportunity, so all sales needs to do is check Salesforce before engaging with the contact.

2. How can we know what price they are going to see if they haven’t visited the pricing page yet?

It’s just as important for the sales team to know the price a contact is going to see as the one they saw. If a visitor comes directly to our homepage and then calls, sales NEEDS to know which price they will see when they eventually visit the pricing page in order to accurately quote a price. But how do you predict the future on which page a visitor will see? You use a multi-page test.

Setup in Optimizely:

In each experiment, Select Options -> Experiment Type and choose Multi-page test.

Multi-page tests allow you to drop visitors into an experiment without them having to visit the pricing page.

Figuring out how to place visitors into a variation without them visiting the pricing page was one of the harder questions to solve. By using Optimizely’s multi-page test feature, all a visitor has to do is visit any page of the site, not necessarily the pricing page, to ensure they are dropped into a variation.

3. How will we make sure everyone at the company sees the same price?

As a software product that sometimes has multiple decision makers, it’s important that everyone in the company sees the same price. Oftentimes we have multiple people attending a demo, so if they were to all visit the pricing page and see different pricing, we’d lose trust and would need to default to lower price. Not only would this muddle the experiment, but also cause frustration and loss of trust among prospects, likely hurting sales. To make sure everyone in the company sees the same price, we’ll use IP Targeting in Optimizely.

Setup in Optimizely:

In one experiment, go to Options -> Targeting. From here, type your root domain and select “IP address” then type:

(?:[0-9]{1,3}\.){3}[0-9]?[0-9]?[5-9]$

and select “Regular expression.” Follow the same steps for the other experiment, but be sure to use the second IP address range which is:

(?:[0-9]{1,3}\.){3}[0-9]?[0-9]?[0-4]$

Targeting an experiment to specific IP address means that if two are people on the same IP, they will see the same experiment. By splitting up the experiments in IP ranges, if two people at same office look at the pricing page, they will both see the same price. (There are some edge cases such as if they viewed from another location such as their home computer or mobile phone off wifi, but we did not run into this issue.)

In our experiment, we ensured that Bizible’s company IP address was in the higher tier, so that if we did a screenshare demo while at the office and showed the pricing page, the higher price would always show.

Launch and test the experiment

Save all your experiments and then test the setup by visiting from different, non-wifi connected devices (such as mobile phones) on the two different IP ranges. Be sure to visit the pages in an incognito browser. Once this is all said and done, you should have a setup that looks like this:

Finally, launch your experiment and retest the setup just to confirm it’s working as planned.

Results

After running the experiment for 30 days, here’s what we learned testing a higher price along with some of the results:

• Sales process was smoother than we expected. There were no major hiccups or awkward conversations because sales was easily able to know which price they saw.

• Large volume is needed in order to reach significance at lower stages of the sales pipeline. As a startup with a fairly skinny sales pipeline, we were not able to reach significance in 30 days.
• Since we use IP address targeting to bucket visitors into variations, the amount of traffic to each version is uneven. The new test variation had 25% less pageviews, so the rest of the results will be shared in percentages.
• Normalizing for pageviews, leads increased 10%, but opportunities decrease 56% (we expected a decrease in leads and opportunities, so were a little surprised to see leads volume actually increased).

However, the biggest outcome was the change in value per opportunity for both settled and projected revenue. The value per opportunity increased 25%, which confirmed our assumption: The new (higher) price will lead to fewer opportunities, however the value of those opportunities will be higher.

Conclusion

While the revenue uplift from the experiment is inconclusive, we’ve learned a lot about the process of experimentation and how big changes can have big impact. We’re continuing to run similar experiments to optimize not just the number of leads created, but also how well they convert through the sales pipeline and ultimately revenue.