Keep Your Development Trains Running Through the Holiday Code Freeze

With Black Friday and Cyber Monday just about one month away, your company may already have started the season’s annual deployment freeze. The deployment freeze is a common strategy that’s used to mitigate risk during a window of time. It’s typically used during the holiday season to ensure that sites and services have 100% uptime and help prevent critical incidents from occurring. And while there are benefits to the freeze, as with all things code, there are downsides and in this blog we seek to address just how you can mitigate some of those downsides with what we call a ‘soft’ deployment freeze. 

A deployment freeze is designed to ensure that no newly introduced features cause disruption when:

• Increased network traffic suddenly creates stress on your systems. (Last year, Optimizely saw a 3x increase in event data and visitor streams from retail sites.)
• A bug gets introduced along with the new feature, causing breakages that disrupt the business. (Example: When Amazon had a one hour outage on Prime Day in 2018, it was estimated they lost $100 million in sales.) 
• Critical staff are unavailable. Engineers go on holiday like everyone else, and fewer staff are available to respond to emergencies.

So, maybe you’re thinking: the deployment freeze period must be a QA engineer’s favorite time of year. Right?

Not necessarily. As many engineers know all  too well, creating an arbitrary code deploy deadline introduces inherent risk — and can potentially make a system super unstable once the code freeze lifts and engineers start shipping code again.

It may be easy to stop the deployment train, but getting it started up again  creates its own challenges.

Here are some of the challenges and risks that developers face after a deployment freeze:

• Increased system brittleness: Software becomes more frail when a backlog of changes pile up without being merged. Once the freeze lifts, merging all those changes as a massive deploy can cause major issues and potential collisions.
• Delayed product delivery: Code that isn’t shipped before the freeze deadline will cause feature delays.
• Broken processes: Even when a deployment freeze is officially called, it’s rarely a total code freeze in actuality. Teams often need to deploy for emergency bug fixes, last second feature changes that need to be delivered, critical infrastructure maintenance, etc. Each
  of these incidents jumps the normal deployment process and causes a higher level of friction. 
• More bugs: An arbitrary cutoff date forces developers to rush to get features to production, which increases the chance of introducing bugs.

So, how can companies balance both the needs of management and developers during a Deployment Freeze? 

One approach is to use a feature flag system to implement a soft deployment freeze: what I like to call the “deployment slushie.” Deploying code behind feature flags allows engineering teams to keep checking in code and shipping to production while — at the same time — mitigating risk by deploying that new code to custom audiences that are segmented off from the entire install base.

The process works like this: 

1. The developer ships code as usual, except deployments are gated behind feature flags that are targeted to the individual team.
2. Engineering teams can test in production behind the feature flag to validate functionality. 
3. In the event that something goes wrong, anyone on the team can turn the feature flag off to remove the code that introduced it.
4. When the deployment freeze ends, the team can remove the test audience from the flag and roll the feature out to all.

This approach helps avoid the pain of changing your deployment process, creating arbitrary deadlines, and risking system brittleness. Engineers don’t have to stop deploying to production for the duration of the freeze. With feature flags, customers also won’t notice that code is being continuously shipped, and that new features are being built and shipped behind flags.

The “deployment slushie” does have some limitations. Functionality that causes downtime, such as a database change, is still off limits. Features that can’t be rolled back (Example: Migration script that would change the state of a customers experience) also still can’t be shipped. But overall, using feature flags to manage your holiday freeze — or any code freeze — allows you to confidently ship code as usual without having to worry about many of the after effects in the thaw.

In this upcoming holiday season, give your customers a seamless experience without pausing your deployments. Use feature flags and keep your development trains running all winter long.