Download our FREE ebook of 43 A/B testing case studies from the world's leading companies for test ideas and inspiration. Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program



In November, we hosted our first one-hour Digital Labs live session for a truly global audience. Speakers from groundbreaking brands Segment, ClassPass, Zoopla and Creative CX discussed their insights into building a smarter tech stack – and why it’s come to such prominence in recent times.

Missed it? Here’s what we covered.

First, meet the Panelists:

Veronica Hulea – Zoopla
Nina Bayatti – ClassPass
Sudhee Chilappagari – Segment
Kenneth Kutyn – Optimizely

1. Why a smarter tech stack?

We kicked things off by asking the obvious question: why do we need a smarter tech stack in the first place?

The answer is simple. One size doesn’t fit all. The days of mass communication and generic product offering are gone.

For Zoopla that means recognizing customers with very specific needs: buyers and sellers, renters and landlords.

“We are trying to build dedicated journeys that are mapped to various customer types. We are doing that by building a comprehensive view of the customer, understanding their behavior across the web and native app and which marketing channels they prefer,” Veronica Hulea explained.

A smarter tech stack is the key driver to building a comprehensive view of the customer and opening up targeted and relevant experiences.

Customer Journey Mapping

2. Making it personal

True personalization is built on a deep understanding of users. Until now, data collection has usually relied on website analytics from companies like Google & Adobe.

Web Analytics Tech Stack

But as users increasingly expect deeper and more seamless personalization, you need to keep up with new and more powerful functionality – and that relies on a robust tech stack.

We heard how Segment takes data collection and activation to a new level. Profile Unification allows teams to understand user behaviors across platforms, while Audience Builder delivers tailored messages via email or Facebook all the while testing and optimizing data to understand what works best for customers.

Customer Profile Unification

“The analytic stack is an integral part of this effort to deliver personalized journeys going forward,” Veronica told us. “The combination of Segment and Optimizely is powerful because we can send all these audiences to Optimizely, find out how we can grow our business, and also how we can work on the personalization of a journey.”

3. Adapt to the virtual world

Obviously COVID-19 has had a huge impact on how a lot of businesses operate, none more so than those involved in fitness and wellness. With lockdown restrictions shutting down fitness classes, gym visits, beauty and wellness appointments, Nina Bayatti of ClassPass talked about the shift from in-person classes and appointments to virtual experiences.

“We introduced livestream classes as part of our product offering. This helped partners earn much needed revenue during the shutdowns, as well as giving our members a way to continue their routines.”

Zoopla has also moved physical interactions online, with virtual viewings allowing users to look around new homes from the comfort of their existing one.

Approaches may have changed by necessity but they are often for the better – shaped by the combination of experimentation and smarter data insights.

Both Zoopla and ClassPass have leveraged tools from within their tech stack, including Optimizely and Segment, to adapt their product offering to better serve their customers. For them, experimentation has provided an effective way of testing these new features and product offerings.

4. Look at your data. Listen to your users.

One question that kept coming up from viewers concerned the kind of user research behind some of the different optimizations.

For ClassPass, this means not just looking at the data but also hearing from users and partners. “Many of our recent customer surveys were undertaken simply to understand what would give our users the confidence to book?”

Panel discussions with partners also allowed teams to understand what was going on in the gyms and how they were adapting their facilities to the crisis. This approach has been warmly received by both sides.

“I think part of that comes from the fact that our customers are really interested in trying to get back to some sort of like routine and so they’re also eager to give that kind of feedback. And it’s the same with our partners who are really looking for ways to adapt their businesses.”

5. Two major tech stack trends

At this point Sudhee Chilappagari from Segment came back in to comment on two major trends he’s seen in the ways teams have been experimenting with tools in their tech stack during the pandemic. “So the first trend was B2C companies doubling down and investing in digital advertising tools, adding Google Ads, Facebook Pixel, Twitter Ads and LinkedIn Insight Tag to their advertising tech stack.”

And the second trend? “This concerned B2B software companies who rely on a subscription-based model,” said Sudhee. “As churn became more heightened during the pandemic, they started doubling down on customer success. In fact, when we ran an analysis to find out fastest-growing software tools on our Catalog, 3 out of top 10 tools happened to be in the customer success category.” These will include the likes of Salesforce, Zendesk, Kustomer and Vitality.

6. Experiment your way out of crisis

Finally, Kenneth Kutyn of Optimizely talked to viewers about the sheer number of experiments now containing the word COVID or coronavirus. A smarter tech stack gives teams the capability to lean on experiments and quickly get new messaging out there.

Digital Experimentation Impact on COVID-19

“Some best practices we’ve observed were small things like just notifying your customers that it’s business as usual. At the same time, companies are also experimenting on all kinds of new business models – say curbside pickups or, as we’ve seen, virtual classes and viewings.” These experiments will often bring previously hidden features to the surface to create new value. “They’ve already done the hard engineering work on these features. And now they’re just using very lightweight experiments to alert people to their existence.”

Thanks as ever to all of you who attended this first session in our new Digital Labs Series, dedicated to helping you build your own smart tech stack.

More sessions are on the way featuring panelists from some seriously great brands. They’ll be discussing various aspects of the smart tech stack, all backed up by real-world insights based on their own experiences.

If you haven’t signed up for the series, you’re welcome to join us: Sign Up for Digital Labs

Optimizely X