Download our FREE ebook of 43 A/B testing case studies from the world's leading companies for test ideas and inspiration. Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program

X

Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more!

Get It Now



Data is Evolving Flywheel


Data is evolving

Across all industries, product development teams are collecting and processing more information than ever before. Investments in big data have given rise to the data scientist, and aided product teams in identifying, rather than simply capturing, key performance indicators and revenue-driving metrics. To compete in today’s digital economy, organizations must consider producing, consuming, and analyzing data a primary component of their product and business strategy. 

As we hold conversations with development teams and data teams about experimentation, we’ve learned that our customers have moved away from this idea of the monolithic data pipeline, where a single tool is used to collect, transform, and analyze data. These teams are building much more modular features into their systems, that tap into a variety of data sources for generating key insights about their experiments and their business. Ensuring data quality throughout these event pipelines is paramount. We’re excited to be launching Enriched Events Export, Optimizely’s most flexible, dependable data export yet, in support of this need.

How is Optimizely meeting the needs of data teams?

Optimizely has always relied on a well-defined set of events to calculate experiment metrics and results, in order to properly measure statistical movement in your audiences’ behaviors. One of the things we’ve heard you find most valuable is our export product, which gives you direct access to these events, for querying by any SQL or data analysis tool. Optimizely’s exports are often joined with third-party data in data warehouses, pulled into notebook environments as inputs to predictive models, and imported into visualization tools like Tableau and Looker for experiment performance and impact monitoring. 

Exporting Data from Optimizely

Up until now, we’ve offered two forms of export. Our Raw Events Export gives you access to your events exactly as they’re received by Optimizely, which is great for things like verifying receipt, validating schemas, and debugging the event pipeline. Our Results Export gives you access to session-aggregated events, and is often used to reproduce the metrics and statistics on your Results pages. Because the Raw Events Export does not provide Optimizely-backed attribution, and the Results Export does not provide event-level views of your data, neither addresses core use cases completely.

With Enriched Events Export, we are introducing a new event-level dataset, enriched with event tags, user attributes, and source-of-truth experiment and variation assignments. You can now export all experiment data as sent and represented on the Optimizely Results page, from a single source. 

All use cases of the Raw and Results Exports can be accommodated with this dataset, and a whole lot more can be unlocked! Prior to this, a complete view of your Optimizely data required stitching together the Raw and Results events, to compute advanced metrics on specific segments like “all visitors who saw variation A and clicked through from the homepage”. We are thrilled to be releasing a dataset where this type of segmentation is possible using one export.

Enriched Events Export takes the best features of our existing export products, and improves upon them by:

  • Preserving all event-level records you send to us. This event format means that event tags you set on individual records are preserved throughout the processing task, enabling you to perform in-depth data analysis on exported data.
  • Enriching events using Optimizely’s internal attribution models. This enrichment process gives individual events formal assignments to experiments, variations, and sessions, so you can have confidence that the events you’re exporting are the same ones attributed to your experiments on the Results page.
  • Improving data storage and compression with Apache Parquet. Through higher data compression ratios and a more intuitive partitioning scheme, your events are easier and faster to pull into external systems, and more straightforward to query once they’re there.
Optimizely Data Enrichment Process

Enriched Events Export gives you access to your data after internal enrichments are applied, and before Results page aggregation has occurred. It is the most detailed and flexible version of our export datasets.

3 use cases for Enriched Events Export

Enriched Events Export unlocks a number of use cases for you across Full Stack and Web. Here are some examples of how it’s being used today.

Tracking experiment and external business KPIs alongside one another

Using event tags set by you within Optimizely, coupled with automated experiment attribution from the Optimizely server, you can utilize additional information like order ID and product type in your experiment analyses. This was not possible with the Results Export, which gives you a session-aggregated list of your events that strips away event-level metadata. Enriched Events can be exported into the systems where your business metrics live, mapped into KPIs like lifetime value and product page retention rate, and viewed side-by-side with your experiment results.

Focusing data analysis on only the events that matter most

We found that customers who were sending us large volumes of data sometimes struggled to parse through an exhaustive set of user events, when seeking the most important metrics for evaluating an experiment. With the Enriched Events conversion dataset, partitioned by event name rather than experiment ID, you can ingest experiment events more selectively, cutting down on data transfer file size, upload time, and analysis time. Our customers are using this to pull in feature-level events only, rather than all possible user behaviors that might be correlated with a given experiment.

Generating views into your experimentation program as a whole

When experiments are analyzed in piecemeal fashion, and at irregular cadences, organizational insights around experimentation are often lost. If you are interested in scaling out your experimentation program, the Enriched Events dataset will be fundamental, helping you create sweeping views of testing processes, develop testing best practices, and generate cross-experiment insights. Enriched Events Export exposes all your decisions and conversions for every experiment you’ve run over the last year, formatted in a logical and consistent manner, and verified as Optimizely’s source-of-truth dataset. This means we’ve double-checked your events against our internal attribution models, and are using this data further downstream to build your results. Slice and dice this enriched dataset any way you need (like viewing lead conversions on the marketing site across all experiments) in order to produce the big picture around your experimentation program.

Getting started with Enriched Events Export

To get started with Enriched Events Export, reach out to your account team or drop us a line. Your data can be accessed securely via Amazon’s API, SDK or Command-Line Interface (CLI) using credentials provisioned by Optimizely.

You can find more technical information in our developer docs.

What we’re building next

Over the next few months, Optimizely is exploring a user-friendly API for querying Enriched Events, which would make this data available for ingestion within minutes of processing. This would enable real-time data capabilities for you, like on-demand analyses of particular subsets of experiment data, without you having to write and orchestrate an ETL to pull those events into your data warehouse. Let your Customer Success Manager know if you are interested in getting early access.

And if you are looking for a platform to get started, reach out to us today. 

Optimizely X