Download our FREE ebook of 43 A/B testing case studies from the world's leading companies for test ideas and inspiration. Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program

X

Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more!

Get It Now



Today’s most valuable companies (aka FAANG) are admired for the return on investment they provide for customers, and are an aspiration for all of us in the digital landscape to measure our work against. Recently, I had the opportunity to have a chat with James Governor, Co-founder of RedMonk, the developer-focused industry analyst firm, and Lawrence Bruhmuller, Optimizely’s very own CTO, to discuss the future of building good products with both Progressive Delivery and Experimentation; one of the core, shared features of the FAANG companies. 

What Makes a Good Product Team?

What makes these FANNG companies so successful when it comes to delivering great products and services? One of the commonalities between these companies is that they all have strong product development teams and product delivery processes. These teams often have tight-knit, collaborative groups of engineers, product managers, designers and analysts that are all in lock-step when building products and services. There is deliberate focus on how teams are set up such that each has complete ownership and are able to define and manage software delivery in a data driven manner with the right controls in place. They have been practicing progressive delivery and experimentation for quite some time. In fact, digital-first companies have been doing this since they started, they adopted early, a mindset of “we need to continuously learn and can make some mistakes along the way as long as we have the controls built into the process.” In many cases and in some industries, this approach has helped them become a competitive disrupter. Platforms like Optimizely have made it easier to accomplish continuous learning with progressive delivery (feature flagging, gradual rollouts, one-click roll backs) and experimentation, in a unified platform.

Software Delivery Meets Growth Hacking with Progressive Delivery & Experimentation

The concept of growth hacking and content marketing has been around for quite some time, in fact, Optimizely was founded out of the practice of experimenting on content to quickly learn what worked and what didn’t, but more importantly, creating a solution anyone could use. Our Co-Founder Dan Siroker had an idea while volunteering for the 2008 campaign, “What if we change the image and call to action on the donation page, how much more donations could we get?” Well, the answer is 60 million dollars more. Following that campaign and experience, Dan Siroker and Pete Koomen founded Optimizely because they wanted to make it easy to run experiments and learn as quickly as possible. 

Growth Hacking and Software Delivery

But we don’t stop the effort there. Optimizely believes that any change you put in front of real users means real insights. In James Governor’s research, he is seeing a similar desired approach to moving fast and getting it right, where he has coined “Progressive Delivery” as a way to improve software delivery pipelines and encourage them to move fast with confidence via experimentation. He asks the question during our chat, “How do we bring this growth hacking, move fast approach to software delivery to enable better product management and a culture of experimentation?” The answer is Progressive delivery and experimentation.

Why Should Engineers Care about Progressive Delivery and Experimentation?

In the Dora’s DevOps 2019 assessment, elite performers were making more changes and not breaking things, why is this? They had better quality tests built into the CI/CD process, which allowed them to ship code with higher quality. It defeated the long-term stigma of “if we move fast, we’ll break things.” They also had controls in place to progressively deliver software with feature flags, gradual rollouts, creating efficiencies in deploying code, and giving product owners and business stakeholders the ability to control when a product is released to users. But product development doesn’t stop there, elite performers are also adopting a culture of experimentation to continuously push the envelope on business performance, and having a unified platform that does both creates synergies between development and product owners. 

Why Should Product Managers Value Progressive Delivery? 

Because of improvements in the CI/CD process, the notion of deployment and release are being decoupled from each other. What this means for the business and Product Managers is they now have more control over what they release, when and to whom, so they can better coordinate with marketing and PR. Product Managers can remotely update features without deploys and quantify impact using A/B tests. AWS is using feature flags and rollout controls for their cloud computing features, Datadog using rollouts of their kubernetes clusters, and Stripe tests their APIs to make sure they are usable and readable by anyone.

Product Development Teams Need Progressive Delivery & Experimentation in a Single Platform

Pulling together controls for developers and product managers in one platform is bringing Progreessive Delivery and Experimentation to humans. Having the ability to easily create features with multiple programming languages, different types of experiments, configure metrics, measure experiments with confidence, in a single platform, while being supported by a global services and success partner, is what the modern product management teams need in order to continue building great products and services.

To hear more practical examples and recommendations on how to begin this journey of Progressive Delivery and Experimentation, watch our recent webinar on the topic. 

If you are ready to get started on this journey, we recommend you try out our free feature flagging with experimentation solution.

Optimizely X