3 ways engineering teams are running more product experiments

In 2018, the big takeaways for developers in experimentation were managing experiments within your own repo, experimenting in serverless applications, and integrating with 3rd party analytics tools. In 2019, the big shift for engineering in experimentation was around improving performance by examining your development process, driving experimentation through flexible content creation, and getting more out of your data. 

Here are the 3 big themes we learned from developers as they look to increase their experimentation velocity to ultimately drive a culture of experimentation. 

Scrutinize your development workflows and processes

Engineering teams are driven to stabilize and control product launches to reduce vulnerabilities and rollbacks. Enter feature flags. Feature flags are a software development technique that turns certain functionality on or off without deploying new code. They also serve as underlying levers to experimentation because they empower you to control, measure, and experiment with the features you’re launching.

If you’re using Optimizely Web to run A/B tests and multi-page experiments, there are multiple ways to install the code snippet that runs Optimizely but now there’s a more performant way to do this that maximizes its effectiveness. Working with your CDN (content delivery network) provider is paramount to speeding up and securing your Optimizely Web implementation. 

HP, the 80-year old technology icon, reported a $21 million incremental revenue impact which they attribute to the almost 500 campaigns they have run through Optimizely. All of these experiments take place across several teams at HP so that’s a lot of information to return to customers. The more information you have in the Optimizely Web snippet, the larger the snippet gets. HP invested time in simplifying the development process and sharing code between team members to increase productivity and drive experiment velocity. They were able to reduce experiment code size by an average of 50% and increase the number of experiments by 42% YoY.

Moe Alabandi, former Lead Experimentation Engineer at HP

Drive experimentation through flexible content creation

How many times has your engineering team received ad-hoc requests to change visible elements on the fly? Engineers are now challenging the current process to make these UI changes and questioning what, or who, are the bottlenecks. Tools like Extensions in Optimizely Web give you the power to add, customize, and control visual components, without depending on code deploys. Creating new elements or adding third party elements to your site was classically dependent on the work of engineers. Extensions in Optimizely X Web will empower your marketing team to reuse and change templated elements that you create in a one-time setup.

Your CMS (content management system) should expand your experimentation capabilities, not gate them. Contentful announced an integration with Optimizely that showed us that you can run experiments on top of structured content for the first time, across all digital platforms – web, mobile, TV, wearables or even voice interfaces. The best part — all of this is done without code deployments. 

Ask for more from your data 

Do you ever get impatient waiting for your data to reach statistical significance and are tempted to look at your experimentation results earlier than you should? This is a statistics concept known as peeking. If you look at the data earlier and it looks like an experiment winner is determined but it is in fact false, you’ll be acting on false information. The main takeaway from Alan Malek, one of Optimizely’s statisticians, was don’t peek at your results! It’s hard to be patient, but it’s even harder to make decisions with false positives.

If you’re a data-driven company attempting to provide personalized user journeys, keeping track of who’s who in your giant dataset can be challenging. Customers have the ability to access your business through many channels using many different devices. Bill Hinderman, Web and Experimentation Engineering Manager at VividSeats, was able to recognize users in a cross-device ecosystem and connect those data points across all the devices he serves experiments on to create a powerful customer profile and ultimately, user experience. 

Given the plethora of analytics solutions offered today, many companies often have “several sources of truth when it comes to their data. By combining experimentation-focused data and more generic web traffic data using Optimizely’s custom Google Analytics integration, HEMA (a Dutch store-chain) was able to reduce data discrepancies down to 1.6% by verifying their experimentation data with Google Analytics.

Jessica Chong, Senior Software Engineer at Optimizely

One thing made very clear at Opticon19 — if you’re currently not experimenting, it’s not too late to start. Innovative companies will continue to push their teams to experiment in order to deliver the most seamless user experiences to ultimately increase revenue. Without experimentation, companies are just simply guessing what products to build and what experiences to serve. If you’re interested in learning more about Optimizely’s products for developers or attending more events like Developer Night, check out our developer hub and join our meetup group!