Product experimentation: what’s in it for backend developers? (part 2)

Ever wonder how to leverage experimentation as a backend developer? Especially as it applies to features that don’t have a direct impact on your product’s UI? This post is part of a series for engineers who work on algorithms, APIs, architecture frameworks, migrations, and more. We recently published Part 1, an interview with Optimizely Tech Lead, Frank Hu on the value of experimentation for backend developers and it’s a great starting point for this conversation. Since then, I talked with Staff Software Engineer Ali Rizvi, to get his take on the topic.

Perri: Ali, you’re an Optimizely veteran! Can you share a little bit about your engineering background and role at the company?

Ali: I’ve been working as an engineer for 9 years and have been at Optimizely for over 5. I started out on our partner integrations team before working on the original team that built our server-side experimentation solution. That has since evolved to be Optimizely Full Stack. Today I am on our Developer Experience team, working on our SDKs.

Perri: Before coming to work at Optimizely, what was your experience and perspective on A/B testing and experimentation?

Ali: Back in 2012, I spent a summer interning at Facebook, and that was the first time I was exposed to A/B testing and feature rollouts. I used a version of PlanOut to build out the feature definition and used an internal tool then known as Deltoid to analyze the results. From then on, treating feature development as an experiment has been my process.

I start by rolling out my work to an audience of 1, that is I only enable the feature for myself in production. Gradually, I make it available to a small internal group, then beta customers and once it’s validated we make it Generally Available (GA). This process wasn’t always as easy as it is for me today. 

Before Optimizely had Full Stack, we managed everything from feature flagging to user whitelisting directly in the code. This meant that any changes we wanted to make in production required a deployment. Since releases can now be decoupled from deployments with Full Stack, we now can define exactly what we want to be rolled out in production, who sees it and we can “pull the plug” right away if something goes wrong. This has improved our internal processes dramatically. We have an entire Slack channel devoted to providing visibility across the org into every feature flag or test created.

Perri: Tell me about one of the early and impactful experiments you ran. What were the results?

Ali: One of the first experiments that comes to mind is one that I ran on a transactional email that is triggered to prompt new users to set up their password. We were seeing low engagement on the email which meant that many new users were not updating their password. We felt this could pose a security risk that we wanted to help them prevent.

By simply testing a new email subject line, the winning variation reflected a 37% higher password change rate. It turned out to be a powerful test that came at a time when we had not yet built any of our Full Stack SDKs, and this was the first time I saw that we had this huge opportunity to provide server-side testing as a product for our users.

Perri: Does experimentation serve a different purpose for backend processes than UI features?

Ali: There are different use cases with the backend for sure. Rolling out new versions of services as an experiment can help optimize the nature of a backend service. For example, we have a service QA automation that uses a cross-browser testing tool. When we decided to invest in a new 3rd party tool we used an experiment for a ‘Vendor Face-off’. The variations of the test featured 2 different cross-browser testing vendors to see which vendor was most performant. 

Recently, I used an A/B test to validate a minimum viable product (MVP) I was working on during one of our competitive hackathons. It’s a very powerful way to show how a new approach compares. In my case, the goal was to reduce the time for something to compile and having tangible metrics to illustrate the impact of the proposed solution helped me get buy-in for the idea.

Perri: Do you have any advice for backend engineers who want to get started with experimentation in their workflows?

Ali: It’s totally doable to adopt experimentation practices into your workflows slowly. I’ve learned that if you are going to invest in experimentation, it’s really important to think about the long term. We’ve seen so many of our customers go from 0 to hundreds of feature flags and product experiments in a matter of months, so you really want to think about how to scale from the start.

You’re going to want to invest in services to do the heavy lifting for you rather than multiple implementations across your technology stack. One way to do so is to have a centralized service that only does experimentation and feature flag evaluation. Your entire organization can use that service for all interactions involving evaluating a feature/experiment.

Every time a user interacts with your platform, every touchpoint is part of a test flow. It’s critical to take the opportunity to understand the results of that test and to optimize them whenever possible. Think about how you want to evolve your organization and the practices you want to set up to get the full benefit of adopting experimentation in the development lifecycle.

For Ali, taking an experimentation-driven approach to software development is just as much about feature management and controlled roll outs as it is about A/B testing solutions against one another. And feature flagging must be scaled thoughtfully, with the right technology solution. Optimizely Rollouts offers feature flagging for free and is built using the same Full Stack SDKs and platform.