Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more! Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program

X

Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more!

Get It Now




Ever find yourself lost in aisle and bin numbers at an Ikea? Would it be easier to Venmo a co-worker via Slack? Want to send a selfie to the cloud?

I’ve tried to solve some of these ‘problems’ while learning new technologies and educating developers. It’s always fun solving pain points by using an API in the ecosystem and then publishing for others to learn on GitHub. It’s been exciting to see other developers using and extending my projects to solve their problems. My favorite project, thus far, has been going serverless with AWS and Twilio SMS to push selfies to the cloud, which has brought attention to many on Twitter!

Send selfies to the cloud via SMS

These projects have allowed me to constantly improve my coding skills, as well as interact with the most passionate and thoughtful developers in their fields. Developers are always pushing the limits and, although it can be challenging, meeting with developers and advocating for them internally has been very rewarding.

Joining Optimizely

Last year, I became Optimizely’s first Developer Evangelist, to build out our developer programs and community. We’ve now revamped our developer experience and launched a suite of developer SDKs for experimentation in software applications, called Optimizely X Full Stack. I’ve spent the majority of my time building developer resources and interacting with the developer community first hand. Internally, at Optimizely, I’m viewed as the voice for developers — I constantly provide feedback from the field to our product teams, which heavily influences our roadmap.  

Developer Products

Many know Optimizely as a platform that enables marketers to experiment on the web — while that is true, there’s several new products and tools for developers, and we’re just getting started! Everything is documented on our developer site with resources like getting started guides and code samples. I’ll touch on each product below.

Optimizely X Full Stack, Mobile, and OTT

We recently launched developer SDKs that enable experimentation in any codebase. This marks a huge transition from moving to not only supporting marketers, but also product teams, including product managers and developers. Full Stack enables developers to experiment with feature flags and rollouts, search algorithms and results, backend performance updates, outbound marketing campaigns, pricing strategy, and much more in any of their projects. We currently have SDKs in Java, Ruby, Python, Node, PHP, Swift, Objective-C, as well as wrappers for Android, tvOS and Android TV.  These are all available open source on our GitHub and are documented on our developer site. I built a demo application using Python and Flask that illustrates A/B testing two different sorting algorithms.

Web Experimentation

Our Web product enables any stakeholder to experiment by using our visual editor. This allows for website A/B testing with messaging, user experience and product design, to name a few. In addition, our Javascript SDK allows developers to run more fine grain experiments on the web by modifying the default behavior of UI elements whose behavior can be defined by any of the many Javascript practices.

Developers can also build reusable templates for the web, which we call Extensions. This enables anyone in the organization to easily add UI elements to their web page. Extensions can be, for example, a customizable countdown timer, banner advertising a promotion, or a pop-up to collect email addresses. Marketers can then experiment on the content in the extension, without additional help from developers. I’ve put together a public extension library and would love to see developers in the community contribute! Feel free to reach out or submit a PR directly on the GitHub page.

REST API

Optimizely offers a REST API to further extend and build on top of our platform. It allows you to programmatically manage your experiments and projects, among other data models. Developers can build custom workflows or complement internal analytics pipelines with our results endpoint. The API allows any project that can make HTTP calls to fully integrate Optimizely into their stack regardless of what language they’re writing their project in. I’ve spoken with several developers building interesting applications on top of our REST API. For example, General Assembly used our REST API to track experiment results on a horse betting track where each horse represented a variation. Check out the neat photos below!

Mock horse race to track variations

Slack channel for placing bets

Leveraging Experimentation

As a developer, the ability to experiment is very valuable — when I’m coding, I tend to distance myself from the end user, which then makes it difficult to design an optimal experience for others. Leveraging experimentation allows me to rely on data to test a few different options when making design decisions. I’ve run many experiments to improve user experience, verify new product features and measure algorithm performance.  

I recently built a web app to display tech conferences and I wasn’t sure how to handle the date format. I questioned using the simplified month/day/year format or would a better UX lend itself to month name followed by the date and year. After running an experiment, it turned out users were more engaged with the shorter format. Additionally, I struggled with the correct balance of number of conferences per page. Check out my experiment below.

Confir.io Screenshot

Experimenting with number of conferences displayed per page

Building a habit of experimentation in your own developer workflow will allow you to more confidently roll-out new features and verify technical design decisions using data.

Stay in touch

I’d love to hear from our community about their experiences or pain points with any of the aforementioned tools or those curious about developer evangelism or experimentation in general.

Optimizely X
comments powered by Disqus