Download our FREE ebook of 43 A/B testing case studies from the world's leading companies for test ideas and inspiration. Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program

Peep Laja generously agreed to answer questions from the CRO experts in our Slack community Test & Tell. Peep studied IT in college (first job out of college was a developer), Peep worked as a professional fundraiser for 2.5 yrs for an international nonprofit, ran an SEO & PPC company, a SaaS company, did direct response marketing for years and started ConversionXL in 2011.

The top 3 questions were offered some sweet CRO prizes.

[Skip To The Winners]

Here is a recap of the AMA with Peep Laja

Keep in mind this is a raw version of what happened in our conversion rate optimization Slack Community. It’s unedited but I don’t think this is a case where grammar mistakes are costing us money. To get the full experience, you would have had to be there because the recap doesn’t do the AMA justice. In addition, to missing out on the rad emoji combinations, you might miss some context as well.

peep laja ama test and tell

Andra Baragan Asks:

How do you calculate the needed sample size on experiments that should impact the average order value/overall revenue and not conversion rates necessarily?

Andra Baragan

Peep Laja Answers:

You need a different formula to calculate revenue sample sizes, no need to know it by heart, either locate an online calculator, or find an Excel spreadsheet that does that for you.

I use either or For revenue sample calculator use:

Gab Goldberg Asks:

How can one predict, or at least make educated guesses/forecasts, about which opportunities to increase conversion will yield the greatest gains?

Gab Goldenberg

Peep Laja Answers:

You need to conduct full conversion research first – that will tell you what and where the problems are. Here’s the guide to ResearchXL framework to do that. Then you can look at drop-offs in GA, so let’s say only 30% of people on your eCommerce Cart page proceed to checkout, and right now your revenue is X, if you know what the top 5 problems are, you can tackle them as opposed to testing random stuff. Industry average for eCommerce cart page drop off is 50%, so there’s 20% opportunity there if you know which problem to attach with the treatment so in a nutshell, look at dropoffs to assess possible impact, and use conversion research to identify what the actual problem is.

Taavi Kalvi Asks:

Is there a time to de-optimize a process? A time to add friction, make something harder For example, how do you feel about unsubscribe or cancellation processes that are difficult to complete. Should marketers make them intentionally so?

Taavi Kalvi

Peep Laja Answers:

Yes there is time to add more friction – when you have humans part of the sales process, you dont have to have a high-converting lead gen form generating leads that dont have the money to afford you, we had this at our agency, lots of leads coming in, and my salespeople and I spent hours on the phone with people with tiny budgets, after 1 hr sales pitch one guy said I have $800 for this, inside I was thinking “It cost the business this much to have this call!
People’s time is valuable, so we added drop-downs with expected budgets (with a minimum threshold) and 90% of the crappy leads disappeared overnight.

If you have a no-touch model – where people can signup/pay themselves, there’s no need to do that.

As for making unsubscribe / cancel even more difficult – no, they will hate you even more. The only thing you can do is create some FOMO (try canceling your facebook account – it will show you all your friends who will miss you) or offer a deal, and neither of these options might be doable esp for email unsubscribe. You don’t want people on your list who don’t read your emails that hurts your deliverability so make it as easy as you can.

If it’s canceling a paid subscription, offering a deal etc might save you, ever tried canceling your home cable / internet? 75% instant discount! Or upgrade to the biggest plan with no extra cost.

EJ Lawless Asks:

When starting an engagement with a company, do you find its better, in general, to be inclusive and bring in more stakeholders from the company or have the program run through one person/organization?

EJ Lawless

Peep Laja Answers

I use the following (that has worked great):

  1. There’s one single contact person on the client side – the buck stops with him/her, every meeting is with that person etc
  2. I start every engagement of mine with a 30-45 long conversion research project. after it’s complete, I request to present the findings to ALL stakeholders, CIO, CEO, IT, legal, branding, cleaning etc. The presentation is always a 100% success – they will be very impressed. Now that single-point-of-contact person will get buy-in from all stakeholders and will get the resources/support needed.

Brian DeLoach Asks:

When launching a new website/product what type of user research do you conduct before launch? What tools do you use to provide structure and gather feedback?

Brian DeLoach

Peep Laja Answers:

Well your standard lean startup + customer development stuff applies here, validate that there’s a need, and talk to your target audience to learn about the problem you’re gonna solve, before building anything, understand how your solution fits in. Secondly, I’d build something fast (MVP) that you can now get actual feedback on when starting out, you won’t have massive amount of people so you can call up your clients and get straight qualitative feedback user testing is def a super cheap and quick way to spot any usability issues. But – I’m not in the business of consulting new ventures, so ;).

Mike Fiorillo Asks:

Balancing your work on ConversionXL (blog and conference) with running a CRO agency must be really tough. How do you manage your time, and how much of your time is spent on CRO client work vs. all the rest.

Mike Fiorillo

Peep Laja Answers

It was all me in the beginning but as you grow, you can delegate more and more to your employees. I’ve gotten really good at delegating over the years. I used to write every single blog post for years, work on every single client project now – I write a few / year, work with like only 1 agency client at a time and I mainly work ON the business, not IN. I do feel that it’s critical to do hands-on work too, to sharpen the saw also, half-life of CRO knowhow is like 2.5 yrs most everything you know will be obsolete if you dont keep practicing.

We have famous “marketing gurus” on sites that sound like fast and sprouts – offering SEO advice that worked in 2011, because it’s the last time the founder did any hands-on work, so I fully recognize the need to keep doing client work, it’s fun too, natural evolution I think, you have to change if you want your company to grow + good leaders create more leaders, I’m super happy to make way for new optimization stars.

Brad Rozran Asks:

How did you convince your first ever client to let you test their site?

Brad Rozran

Jimmy Pham Asks

Tell us about a difficult time you tried convincing someone to start optimizing. They weren’t having any of it. How did you end up persuading them? How would you do it in the future?

Jimmy Pham


Peep Laja Answers

Use the classic “many-yeses” tactic.

Do you want to make more money? Yes/No

If I could improve our conversion rate by 20%, would you like that? Y/N

What if we’d be losing money every day, but there would be a way to fix that , would you want to do that? Y/N

Make them see the obvious. How can you say no?

Instead of telling them “let’s change things”, tell them “let’s run an experiment” The worst that can happen is – well – nothing at all!

It’s important to understand and communicate that optimization consists of 2 parts

  1. Figuring out what the problems are, where they are, how big they are, this is what all of you guys can do (using ResearchXL or other frameworks)
  2. Figuring out best solutions to the problems, nobody has a crystal ball – that’s why we need to run tests in Step 2 if we’d know what would work, we’d all be billionaires we CAN know what the problem is, but we can’t know what will be the best treatment without testing.

Karolis DzejaAsks

According to the people who tested Wikipedia’s donation banner, throughout 214 tests, if they had stopped at the first sign of 70% confidence (with a minimum of 15 conversions), they would’ve picked the winner 90% of the time and the loser 10% of the time. They calculated that their tests we’re on average 72% too long. They could’ve done 3.6 times more testing.(Source video @ 40:40)

Knowing this, if you’re testing and hunting for home runs rather than incremental gains, does it make sense to optimize for the number of swings rather than more certainty at the cost of time? Is 95% confidence a dogma that should be questioned? What’s the pragmatic business decision here?

Karolis Dzeja


Peep Laja Answers

A minimum of 15 conversions? I haven’t seen the video, but I would like to call BS on this. So I can issue a formal comment on the video later, after I watch it if they consider 15 conversions a minimum stopping rule, I suspect all of their tests are bullshit, you should totally ignore the statistical significance levels until 2 other conditions have been met: Minimum 1 business cycle test duration and pre-calculated sample size achieved, significance is a meaningless number before that.

Now – when it comes to low traffic sites, and you can NOT detect a 5% win or 10% and you choose to swing big – lots of big changes in your treatment – you would want to see a big difference right away, testing 1 single element at a time is a luxury for high volume sites, 1% win for Marriott (8 billion online revenue) is a shit ton of money, 1% relative increase for you might be nothing, also, for low traffic sites I recommend you go for Bayesian Statistics, like the one VWO employs, will offer you richer data to take calculated risks on tests potential loss vs potential lift.

Update Post Watching the Video: 

Took a look at the video and well these guys are quite clueless and they think p-value is a stopping rule. I didn’t watch the whole thing, but they’re only talking about confidence levels. This is a good (but not easy) read for anyone about what p-values are.

Also read, Stopping A/B Tests: How Many Conversions Do I Need?

Alex Birkett Asks:

As VP Pre Party at ConversionXL, I’d like to ask you what you are most excited about for this year’s ConversionXL Live? Any unique activities this year or things that haven’t been announced publicly?

Alex Birkett

Peep Laja Answers

There might or might not be a blind wine tasting – figuring out if ppl prefer $5, $15 or $25 wine (same variety)
Quite looking forward to our copywriting panel – 3 copywriters will each re-write a piece of copy, and revealing the AB test results there.

I’ve heard rumors about a new amazing product launch happening which the attendees might get free access to and a great pre-party, 2 live bands, 2 parties and one with free drinks by BounceExchange.

Ej Lawless Asks:

What types of people will benefit the most from attending ConversionXL Live? Follow up question, what should people do to get the most out of ConversionXL Live?

EJ Lawless

Peep Laja Answers

Optimization requires traffic, so the people who will benefit the most are the ones that has established businesses. Can’t run ab tests when you have 50 sales per month. So it’s not for companies just starting up, but for existing business that want to grow faster.

My best tip for getting the most out of it: Network! be very conscious of building new relationships and learning from your peers.

JC Cavanaugh Asks:

When determining which clients to work with what would be the top 3 things you look for.

  1. Is it tons of traffic to get faster results?
  2. A Google Analytics check that reveals alot of opportunity?
  3. Clients expectations being realistic?

Leading into, who is your ideal client?

JC Cavanaugh

Peep Laja Answers

  1. I go for a minimum of 1000 transactions per month. Less than that and your testing velocity is really slow, and if your test results in “no difference” you have a resultless month. I sell “conversion research” to these people instead.
  2. The good old funnel – measure your funnel drop-offs, also steps that come before the actual sales funnel, e.g. traffic from Category pages to Product pages, Product page to cart adds.
  3. I say a typical growth rate is 5 to 15% increase in conversion rate per month, depending on how optimized the site is, the crappier the site, the more early wins, life has shown that the bigger the company you work with, the longer the engagement bigger in terms of transaction volume, aim for companies that do at least 1M / mo in revenue.

Luiz Centenaro Asks:

Eye Tracking and Biometric User Research: Why did you start this service and what do you hope to achieve with it?

Luiz Centenaro

Peep Laja Answers

In a nutshell: to be better able to measure UX.

What do people actually look at? what do they ignore? which visual cues actually work to drive attention?

You can’t answer these questions with mouse tracking, so eye tracking gives you a new layer of data to operate with to inform your A/B tests and such. It gets even cooler with Biometrics – when they read/see/watch your sales copy/video etc – does it have an emotional impact?

People buy with emotions, even in B2B setting, so we are now able to measure emotional valence, how much emotion do people feel, and whether it’s positive or negative, and get quite specific about it too, great way to test your ads too before spending tons of money on campaigns.

Mike Fiorillo Asks:

What’s the #1 mistake that clients trying to run an optimization program themselves tend to make?

Mike Fiorillo

Peep Laja Answers

They test random crap, to validate their own opinions, or “what would happen if…” that results in a totally random process. The opposite of what a CRO process should be like – highly-structured, learning-driven, if the site is crap, it’s quite easy to target the low-hanging fruits, but you’re gonna be done with those pretty soon then what? That’s why you need to do conversion research.

Taavi Kalvi Asks:

If you were giving a lecture to a group of college students, what books or life philosophies would you share with them? & Which one is better – Estonian beer or American beer? Which one do you offer at ConversionXL Live to attendees?

Taavi Kalvi

Peep Laja Answers:

1. My top 2 recommendations for students:

A) Be proactive. Always take responsibility, ask yourself what could YOU do to achieve X, make something better, don’t wait for permission or direction

B) Everything in life is about relationships. Nepotism is still the way the world works. Whom you know is what opens doors all the best deals, partnerships, jobs etc are given to people they know. In college, prioritize partying!

2. Beer: United States is way bigger, so also 1000x more selection – anyone can find 20+ beers they really like so the sheer market size gives the advantage to the US great beers in both places, altho Estonia doesn’t have that many dark beers (porters, stouts), the kind I like at ConversionXL Live we’ll serve Texas beer.

And Now Announcing The Winners. Drumroll Please..

drumroll please

1st Place: Gab Goldenberg winning 45% OFF ConversionXL Live.

2nd Place: Taavi Kalvi winning FREE access to Peep Laja’s Conversion Rate Optimization Course

3rd Place: Mike Fiorillo winning a FREE Copy Of How to Build Websites that Sell: The Scientific Approach to Websites Kindle Edition

If you haven’t already make sure you register for CXL Live and use the promo code “EE” for 25% OFF the best conversion rate optimization conference.

Optimizely X