Ship faster with less risk. Get Optimizely Rollouts, free feature flags for developers. Create Free Account

Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more!

Get It Now

Let’s talk about chatbots. My first interaction with a chatbot, like millions of others, was SmarterChild on AOL Instant Messenger, in the early 2000s. SmarterChild offered an early glimpse into the potential that an automated bot could offer to real world interactions. However, personally, it felt like a novelty in the midst of an era of existential crisis for internet technologies.

Fast forward to today, and we can see that the proliferation of smartphones, messaging platforms with open APIs (Twilio, Messenger, Slack, WhatsApp etc), and advancements in artificial intelligence have graduated chatbots from internet novelty to business necessity.

With the potential to reach more customers in more places, chatbots offer the potential to greatly improve how customers interact with businesses. Research from 2016 found that users spent 85% of their time in as little as 5 mobile apps each month. Recently Facebook announced that its Messenger platform has surpassed 1.3 billion monthly users,  displaying steady user growth and showing its huge reach. Despite the opportunity, there are many hurdles to creating a successful chatbot, most notably insight into the optimal chatbot experience.

As companies such as KLM, eBay, and TD Ameritrade bring their digital experiences to platforms like Messenger, it’s crucial to understand what works, what doesn’t and how users will engage with your bot early on in your development cycle.

Enter Optimizely Full Stack. Optimizely Full Stack SDKs allow product development teams to experiment and roll out features across all channels, including chatbots. To get a better understanding of how companies can truly experiment everywhere, my colleague Martijn Beijk and I collaborated to build our very own Facebook Messenger bot with Optimizely.

Building Our Messenger Bot

For our bot we emulated a recommendation feature, similar to eBay’s Messenger bot. For our experiment, we wanted to know whether it’s better to recommend one product or multiple products at a time.

We built the bot in NodeJS and Express, in part because Node’s concurrency works well for applications handling large amounts of requests, and ease of setup. Alternatively, we could’ve easily accomplished the same functionality using another language/framework such as Python + Flask, or Ruby + Sinatra.

First we created an Optimizely Full Stack project, and implemented the Optimizely Node SDK into our Node Express app.


Next we configured the Optimizely SDK in our app to use a helper function to grab the Optimizely datafile and initialize the Optimizely client:

For a production use case, you could also grab the datafile via Optimizely’s REST API and leverage Optimizely Webhooks to keep the SDK in sync with changes you make in the Optimizely interface.

Instantiating Optimizely in our Chatbot App

With the helper function to initialize Optimizely setup, we could then easily instantiate the Optimizely Full Stack client when our server starts so that we can begin using the SDK to run experiments.

Once that was completed we built out the endpoint to accept incoming events from Facebook – for example when a user sends our bot a message, or takes an action from within Messenger.

Facebook Helper Functions

We also created some helper functions to handle sending a message back to Facebook and the user.

Creating the experiment in Optimizely

Now that we’ve finished setting up our bot, we can build our experiment. For this simple use case we only need two variations for testing the single product vs. multiple product recommendation.  We’ll also want to define our metrics for this experiment. In this case we simply tracked if a user clicked to purchase an item, and the associated revenue generated. However, since users can also click through the bot links to our website.  

If we want to, we  can also track events that occur outside of Messenger such as new account creations, products viewed, or purchases on your website or in your mobile app. We can easily do this using the Full Stack JavaScript SDK to track these events. The key thing that we will want to do is share the same datafile between our NodeJS server code and JavaScript client code so that we can be in sync.

Implementing the Experiment in Node

Finally, we built out the new functionality that we wanted to test. Here we added a condition to check if the user was asking for a recommendation as well as the logic for how to recommend products using hard coded product data.

We set up a rule to parse incoming requests from Facebook to detect the word “recommend”. If anyone sent our bot a message looking for recommendations, we’d then activate them into our experiment, and either return one product or three different products.

Messenger bot in action!


Just like that we created two different variations for how our chatbot should respond with recommendations, and were able to track the effectiveness of each variation in terms of revenue and conversions.

This is just one simplified example of how Optimizely Full Stack can be implemented and applied to emerging channels like chatbots when it comes to experimentation. Product teams building chatbots can also use Full Stack to improve the customer experience with more advanced conversation algorithms, better timing of message sends, and even when to allow live agents to take control from the bot.  Experimenting in chatbots with Full Stack opens up a world of possibilities for teams to build better experiences and drive more customer loyalty.

Want to see our demo in action? Go to and ask for a recommendation via Messenger!

Additional Links:


Optimizely X