Tl;dr

🚩 How do we use Circuit Breakers at Optimizely? Managing third party integrations and as kill switches.
🔒 Access Level? All Engineers, some restrictions depending on the circuit breaker.
😬 Risk Level? Low
👩‍💻 Tests? Manual + Regression Automation Tests
⏰ Lifetime? Usually Permanent

A circuit breaker feature flag is a manual switch that can disable a feature without deploying code. At Optimizely, we use circuit breakers in many ways, including managing third-party integrations (Segment, castle.io, New Relic) and as feature kill switches. For example, we maintain kill switches for non-essential services that are computationally expensive — in case we need them off during a traffic spike.

Circuit breakers gate features that are permanent. They’re left in place in case certain conditions arise, so those features can be disabled easily and safely. 

How do we use circuit breaker feature flags?

Circuit breaker feature flags are used by almost every engineering team at Optimizely: from feature development teams to infrastructure teams.

Generally, teams use runbooks to describe certain workflows, including when certain features should be turned off. For example, imagine that we use a third-party vendor to process certain events. When we reach a limit or cap and are about to incur overage charges, we can use the breaker to pause the event stream or selectively pause certain types of events. The runbook for the infrastructure team can spell out the conditions for executing this flag, taking the guesswork and coordination out of this process.

An example code sample of how we implement a Circuit Breaker Feature Flag

Who should have access to a circuit breaker flag?

At Optimizely, the decision to allow all developers to access and use the flag is made team by team. We tend to tighten access controls for circuit breakers that gate highly sensitive functionality. For example, a circuit breaker that disables logging to a third-party vendor may be restricted to team members to ensure that engineers from another team don’t disable valuable logs accidentally.

What is the risk level for using a circuit breaker flags?

Circuit breakers should be designed and implemented so that they can be disabled and re-enabled without issue. On the spectrum of risk by flag types, circuit breakers should be low-risk flags to turn off.

How do you test your deployment of circuit breaker feature flags?

We test circuit breakers that gate third-party integrations manually to ensure the breaker disables the integration without breaking product functionality. 

We also gate some resource-intensive functionality behind circuit breakers as a precaution for high-traffic situations, like Black Friday. For these flags, we design regression automation test suites that cover the following situations:

• Feature On – Does this feature work correctly
• Feature Off – Does our product still work when this feature is disabled

The automation test runner knows where the circuit breaker flag is gating the feature– and uses feature targeting (audience conditions) to test both paths.

When should you remove circuit breakers?

Circuit breakers usually control permanent functionality. They’re in it for the long haul. But if that functionality changes, the flag should be removed. For example, if we change a third-party integration, the flag should also be removed from the code base. 

Good hygiene is the key to success in the world of feature flags!

This is the part of a series that will dive more into the different feature flag types we have here at Optimizely, as well as the engineering teams that implement them. See you next time as we dive in deeper on the Experiment feature flag type.

If you’re looking to get started with or scale feature flags, check out our free solution: Optimizely Rollouts! 

Are you using circuit breaker feature flags at your company? I’d love to hear about your experience! Twitter, LinkedIn