A/B testing all-star: meet Jarred from Rocket Lawyer

Name: Jarred Colli
Company:Rocket Lawyer
Role: Product Marketing Manager
Favorite legal term: Ceteris Paribus

Jared says, “Just because a concept you tested didn’t work the first time, doesn’t mean it’s not the right thing to do.”

Optimizely: How long have you been using a/b testing as a web optimization strategy?
Jarred: I’ve been doing this for less than a year. I’ve got a background in consulting and analytics though, so problem solving for a long time, but testing for a short time.

Tell us about an a/b testing experience where the results have surprised you.
I’m constantly surprised by the results. Usually the first iteration or idea we lead with in an experiment tends to fail against the control until it can be examined more closely to tweak the individual elements. Sometimes a page that looks much better at first glance has missing elements that causes it to perform worse than the control. I’m more surprised now by experiments that succeed on the first attempt than I am by initial failures. But ultimately, it’s not about being right the first time, it’s about crossing off ideas as quickly as possible until you’ve arrived at the right one.

Are you connecting testing to ROI in any measurable way? If so, how?
We’re not concerned with click through as much as the ultimate conversion of a user, so we gauge all of our experiments by the effect they have on customer acquisition. Ultimately increased conversion rates decrease our cost per acquisition, which makes every marketing dollar more profitable.

What is the coolest, most mind-boggling test you have ever run?
We’ve tried all kinds of things, probably some of the most complex stuff a person can do with Optimizely (if the amount of time our technical account manager, Mike has to spend on the phone with me is any indicator). But ultimately it’s the simple ideas that yield the big results, that one little thing you didn’t think of that is missing in an experience, or piece of information that is missing. User experience mistakes can be obvious, but usually they’re quite subtle.

If someone is new to testing and is looking for a good way to find a couple of testing ideas, where would you suggest they start?
Start with what your goal is, and then work backwards from there to where your users come from. Every page is a step where people are provided options to either move towards or away from that goal. Things that don’t help someone move towards that goal need to be examined as to why they are where they are, because they are at best not very helpful, and at worst a big distraction.

Is there anything else you have to share about your testing experiences?
Just because a concept you tested didn’t work the first time, doesn’t mean it’s not the right thing to do. Keep looking at your experiments to make sure your execution is exact, paying close attention to what is driving user actions.

Come up with a question to ask our next all star!
I’d like to know what the simplest change someone has made that yielded the biggest results (that wasn’t a button placement or button copy issue).

About this series:
“A/B Testing All Stars” is a new blog series that features Optimizely customers who have implemented a testing culture on their sites. Interested in being featured as one of our All Stars? Contact cara@optimizely.com.