Investing in Optimization – The First Steps Towards Building a Team

step one in building an optimization team

Suma Warrier is an optimization veteran. Over the past 10 years, her digital marketing career has focused on building and scaling optimization programs at the likes of Symantec, Sephora, and most recently Move, Inc. (operator of, a News Corp company.

Her latest challenge? Building an optimization program from the ground up, and enabling her organization to unlock the power of rapid testing while working towards an aggressive revenue goal. In this series of three posts, we’ll cover how she approached team structure and process, how the team set goals, and examples of experiments her team has run.

In 2014, Suma and her team ramped up ideation, prioritization, execution and analysis of’s tests. In this post, we’ll cover how the team is structured, the importance of their cross-functional stakeholders, and the agile processes they adopted to ensure a smoothly running optimization program.

Opticon 2015 — in 1 Minute

one minute mondays

This week, a #OneMinuteMonday on Optimizely’s annual conference: Opticon.

There are more than 225 million people set to attend conferences this year. There’s probably more than 1.5 million conferences that are going to happen in 2015 alone. Yes, there’s a lot of gatherings for professionals to better themselves… But there is none like the one I’m about to talk about: Opticon 2015.

For Killer Response Rates in Large Campaigns, Try Experimental Design

mass variance in dishes

Your customer base is dynamic and complex. If you’re trying to optimize across your entire population of users, traditional A/B experiments may fall short. In this case, you should consider a technique we call experimental design because it allows for massive increase in the amount of variance in your digital marketing campaigns.

In this post, we’ll discuss how to make large-scale experimental design campaigns actionable.

Opportunity Cost and Testing — In 1 Minute

one minute mondays

Welcome to One Minute Mondays, because on Monday, you only have one minute. Each Monday, we’ll post an education-packed tip presented by someone from the Optimizely team. Learn something new and connect with the Optimizely team — all in 60 seconds!

This week, Opportunity Cost and Testing from Khattaab Khan, a Strategic Consultant at Optimizely.

Actionable Tips for App Store Optimization [Infographic]

rainbow of apps

Search and app store browsing is the most used method for discovering and downloading new apps – accounting for up to 63% of all app discovery. This means the ease in which new users can locate your app in the Apple App Store and Google Play Store is imperative. That’s where App Store Optimization (ASO) comes in, making it as easy as possible for potential users to find and install your app.

In the following infographic and post, I’ll walk through key areas of your app that can be optimized to improve app store rankings, as well as best practices to serve as a starting point for testing and ASO.

A Guide to SXSW Interactive 2015 for Growth Marketers

Growth Marketer roadmap thumbnail

South By Southwest is upon us. Everyone from robotics engineers to heads of optimization will descend on Austin, TX from March 13-March 16 to experience the Interactive portion of the festival. This year, Interactive offers more than 800 sessions. 800 over four days!

To make the decision-making process slightly easier for those of you attending, the marketing team at Optimizely combed through those 800 and selected 12 that we believe are most relevant to marketers focused on growing their businesses through data and testing.

42 Mobile A/B Testing Terms You Need to Know

two iOS phones

As mobile continues to eat the world, the number of A/B tests running on mobile applications continues to grow. While A/B testing isn’t new, the vocabulary to talk about A/B testing mobile apps is new for many people. Succeeding in mobile A/B testing starts with the right vocabulary. Following in the footsteps of the 14 CRO Terms You Need to Know article, we have created one for a mobile lexicon. This alphabetized glossary of 42 terms and examples will help you communicate about mobile optimization.

Why 2 Opposing Statistical Schools of Thought are Actually Both Essential


Just like a suspension and arch bridges both successfully get cars across a gap, both Bayesian and Frequentist statistical methods provide to an answer to the question: which variation performed best in an A/B test?

Historically, industry solutions to A/B testing have tended to be Frequentist. However, Bayesian methods offer an intriguing method of calculating experiment results in a completely different manner than Frequentist. In the world of statistics, there are devotees of both methods—a bit like choosing a political party.

In this post, we’ll cover the benefits and shortcomings of each method, and why Optimizely has chosen to incorporate elements of both into our Stats Engine.

On Building an Optimization Agency: The Inception & The Future


Around 2007, Matty Wishnow had a revelation about decision-making: most decisions businesses make—whether it’s a marketing plan, a design brief, or a feature on a product roadmap—are “really a bunch of hypotheses masquerading as priorities.” He realized that most businesses didn’t really employ data or testing to drive their decisions.

“It was almost just like something that somebody had to check off on a checklist, all the analysis was postmortem; everything was still driven by gut,” Matty told me.

This epiphany drove him to start a digital agency called Clearhead. Today, some of Clearhead’s clients include Adidas, Fresh Direct, Blu Dot, Sephora, Keurig, and Patagonia.

I spoke with Matty and Brian at Clearhead, about how they approach client engagements, how they’ve seen the optimization landscape change over the years, and advice for agencies starting to offer optimization.

Your Email Open Rate is High? That’s Nice.

email open rate

A high open rate is nice. It’s an important signal you’re writing evocative subject lines and optimizing your send time. But if people don’t take action or convert on that email they opened, then the act of opening is much less meaningful. Getting maximum value out of A/B testing marketing email requires optimizing for key metrics that really matter to the business—and open rate is not one of them.