Tutorial: Get alerted when feature flags change, via AWS Lambda and Webhooks

In the last few months, we’ve found this has been especially helpful for our end-to-end testing. Being able to toggle features on and off for each language we test makes running these features much faster.

Our SDK team is using a Full Stack projecr to configure our E2E tests. We do this by setting up our changes as feature flags in Optimizely, and creating an audience for each SDK we manage (e.g. when we developed the workflow for feature flags, we set this up as feature=”feature_management”, audience=”node_sdk”). This allows us to turn off features for certain SDKs or all SDKs during testing. Below is a snapshot of the feature layout.

We’ve seen good results from using feature flags to manage these tests. But recently, we hit a problem. With many different engineers working on E2E tests all at once, what happens when someone else turns off a feature that you may also be working on?

In order to track this better, I wanted to get a slack notification to our developer channel telling us what changed in our datafile (the configuration file capturing the state of all our experiments and feature flags).

I created an AWS Lambda with API gateway to accomplish this. The API gateway endpoint is used to register a webhook with Optimizely. When the datafile is changed, the webhook is fired. The lambda function reads the datafile from the CDN and then from the DynamoDB instance if it exists (if not it creates the DB table and stores the current version of the datafile before publishing a “no difference” message to your slack channel). The function compares these two JSON datafiles and tries to send a human readable diff. Below is an example diff using the feature above that someone toggled from on to off.

There’s room for readability improvements. But, from the above I can tell that the rollout for feature_management has changed and that the feature has been disabled since featureEnabled was set to false and status went from Running to Paused.

Getting started was easy. It was trivial to setup the Lambda and the webhooks. The hardest part was figuring out the permissions for the lambda.

*Since the datafile is not changed often and the lambda only published to a slack channel, this implementation is a low cost solution. You could use this type of setup to notify your developer channel of project file changes or even QA for staging and debugging before release.

For example, the developers may have gated features and initially had them set to false. When it’s time to flip on the features that need to be tested, your QA team could be notified in a channel and begin testing. This could actually be dogfooded company wide not just for QA.

*Using the webhook, you could also post notifications to your servers via some service such as AWS Simple Notification Service. This would work nicely within AWS. Your Elastic Beanstalk instance spins up and registers for a SNS. When a notification comes in, your server instance would just read the file from DynamoDB latest instance or directly from the SNS message. But, that’s a blog for another time.

Implementation

This document assumes you already have an AWS account and know how to create an Optimizely project. Once you log into the AWS account console you can easily find the lambda section. The three areas we will touch with the AWS console are the lambda function, the DynamoDB, IAM console, and CloudWatch for logs.

Above is a basic breakdown of what the webhook lambda will look like. This is the Design view of the lambda. When you click on the individual components, the views below it change for the appropriate module (such as the lambda showing the code). In the example below we can talk to the API Gateway, access logs, and connect with DynamoDB.

The first thing we need to do is create the webhook forwarder. To do this, you can use a lambda template. Go to the AWS Lambda Management Console and create a function Using blueprint microservice-http-endpoint. The microservice-http-endpoint blueprint gives you a setup for gateway API and DynamoDB using node.JS.

Name your lambda and create a new role with “Simple Microservice permissions.” Also, create and name your new API endpoint. Don’t worry about the code right now. We’ll just create the lambda to start.

Edit your DynamoDB permissions by accessing the IAM console and adding “Create table” permission.

*It may be a good idea to create the DB beforehand and just use that DB instead of allowing the lambda to create the DB. I decided to have the lambda create the DynamoDB file.

Ok, now we can look at the code. The template code provided by AWS shows you how to update, delete, and insert into a DynamoDB instance using various HTTP methods (POST, GET, PUT). We only care about making POST calls so lets start there.

First change the example code so that you are only switching on the POST or of course default. Then, let’s just print out the webhook project id and url from the Optimizely webhook payload.

let body = JSON.parse(event.body);let projectId = body['project_id'];let url = body['data']['cdn_url'];console.log(‘project url:’, url);break;

Next, create a test using the API Gateway test template (upper right hand corner next to the Test button) and replace the following as your body property of the payload:

"body": "{\"project_id\":1234,\"timestamp\":1468447113,\"event\":\"project.datafile_updated\",\"data\":{\"revision\":1,\"origin_url\":\"https://optimizely.s3.amazonaws.com/json/1234.json\",\"cdn_url\":\"https://cdn.optimizely.com/json/1234.json\"}}",

Validate that your endpoint is working correctly. Now, let’s register our webhook. Rather than run through the whole process, this help article here covers how to register your webhook with Optimizely. Your webhook URL is your API Gateway Invoke URL that is available by clicking in the API Gateway Button in the Design view of your lambda.

Now, you can test this by actually updating your project and looking at your logs. You should see your URL there. Next, you need to add your secret key and test for that in the payload. So, create a environment variable and add your secret key there. You will see environment variables below your coding view. Below is a snippet of code from our lambda showing the testing for the secret key:

const crypto = require('crypto');
 const hmac = crypto.createHmac('sha1', process.env.SECRET_KEY);
 let hubHeader = event.headers['X-Hub-Signature'];
 hmac.update(event.body);
 let secretCompare = 'sha1=' + hmac.digest('hex');
 console.log(secretCompare + ' == ' + hubHeader, secretCompare == hubHeader);

If the secret key and the header don’t match, don’t honor the request. Keep in mind that if you are servicing multiple projects then you need a way to either have multiple secret keys or not use secret keys. One way for multiple keys would be to store the key in projectId:secret key value pairs in multiple environment variable or a single environment variable.

This document includes the gist of the lambda so you can look through all the code at one time.

So, we have our url, we can tell if the event coming in is legitimate, now, let’s process the request.

When a request comes in we will:

1. Load the new datafile from the webhook payload.
2. Look to see if the datafile exists in DynamoDB.
3. If it does exist, use it to find the latest differences, otherwise, store the current copy in DynamoDB and say nothing has changed yet.
4. Finally, we do a diff, update the datafile in DynamoDB, and publish the difference to the Slack channel.

First, the way I implemented the diff of the datafile might not be best. It uses recursion since we know the datafile is relatively small. It tries to print messages that tell what actually changed. You may want to replace that with a npm json diff package or tweak the existing to your needs.

I’m not going to go through each function. But, I would like to discuss how we setup the webhook publish to Slack. If you are logged into Slack it’s really easy. You can just open up a browser and go to https://api.slack.com/incoming-webhooks. Or, you can go directly to https://my.slack.com/services/new/incoming-webhook/. Enter your channel and create a Slack webhook. In the sendWebhook call in our lambda, you will add that webhook to the appropriate area. Notice that the send portion is just the last part of the url.

function sendWebhook(data, done) {
 var url = 'https://hooks.slack.com/services/LONG_UNIQUE_KEY';
 const options = {
 hostname: 'hooks.slack.com',
 protocol: 'https:',
 //port: 80,
 path: '/services/LONG_UNIQUE_KEY',
 method: 'POST',
 headers: {
 'Content-Type': 'application/json'
 }
 };
 const req = http.request(options, (res) => {
 res.setEncoding('utf8');
 res.on('data', (chunk) => {
 console.log(`BODY: ${chunk}`);
 });
 res.on('end', () => {
 console.log('No more data in response.');
 });
 });
 req.on('error', (e) => {
 console.error(`problem with request: ${e.message}`);
 });
 req.write(data);
 req.end();
 done(null, {'result' :'200'})
}

Conclusion

Setting up a webhook to notify the team of project configuration changes helped me to know when we were potentially stepping on each others’ toes. The lambda webhook can also be used to notify your servers, as well as send a Slack message. Using lambda functions for webhooks is a powerful tool in project management.

I hope that this document makes it easy to understand and create a webhook lambda to digest through Slack. Finally, you can find all of my index.js code on GitHub. You can simply replace your code with the provided index.js and add your YOUR_KEY_HERE from Slack and you’re ready to go. Don’t forget to add your secret key as a environment variable. Happy Coding!