5 Ways Phased Rollouts Will Protect Your iOS App

Have you ever deleted an iOS app you paid for? I have, albeit very begrudgingly. The app was Rise, an alarm clock that made waking up beautiful. I used it all the time, until they released an update that made it crash every time.

It wasn’t just me. Reviews took a major hit. People were unhappy with version 4.0.2.

The 2nd screen shows the “current ratings” for Rise’s buggy 4.0.2 release.

Since this buggy update in January 2014, Rise has done a solid job bouncing back. By most App Store standards, they are doing very well. Their most recent release (4.1.6) has 4 out of 5 stars, and they’ve increased ratings for All Versions to 3.5 stars. They were even featured as the Free App of the Week in August. 2014 with a rave review.

The point here is not to point fingers at Rise. The buggy-release-causes-backlash-of-reviews-leads-to-better-release-to-fix-bugs scenario is pretty common (last week, MacRumors reported people having major troubles with Nest’s iOS app after a new release). The point here is that these kinds of releases don’t have to be so common.

Thanks to a strategy called phased rollouts, app developers and product managers can release a new feature or design to a portion of users to get early feedback/QA before releasing it to 100% of the user base.

Phased rollouts save mobile teams from releasing buggy updates or bad user experiences to their entire user base. This strategy also allows teams to release updates or hotfixes without waiting for App Store review, a process we know takes time and can cost you users. During the phased rollout, PMs can measure their key performance indicators (KPIs) in real-time and take action based on those measurements.

Before walking through a scenario that shows how phased rollouts can help protect your app, you might be wondering…

Technically, how do phased rollouts work? 

The ability to release a new experience to a portion of your user base is possible through features like Code Blocks.

With Code Blocks, a new feature or experience is encapsulated in code before submission for App Store approval. This code then allows you to control what percentage of your install base sees this new feature or experience. This is most commonly used to do three things:

1. Measure the impact of a new release, feature, or experience.
2. Roll back a new feature or experience of it introduced bugs or wasn’t received favorably.
3. Multiple versions of a new feature or experience in an A/B/n experiment.

To start using Code Blocks and doing phased rollouts, you just need to integrate an SDK that allows you to test new experiences (you can learn about Optimizely’s SDK here) on your app.

A release scenario with phased rollouts

Let’s say you are a product manager and have spent countless hours with your team building a new design of your app. In user groups, 20 people gave positive feedback. You’re fired up to get it into the world, but not 100% of the world. You know that rolling this feature out in phases is wise because your userbase is delicate, and new releases can shake things up.

So, you’ve installed an SDK on your iOS app that allows you to have more control over the release. You release the new design to 20% of your users, leaving 80% of users with the old design for comparison.

Let’s walk through this scenario with five different outcomes:

1. A successful release

Five days into the test you have enough data to declare your new design an outstanding success with statistical significance. In a few clicks you’ve pushed the new design live for 100% of your users.

Voilà! You have now fully released your new design and are taking full advantage of the observed lift to your KPIs. Not only that, but you were able to sleep at night for the five days the test was ongoing because you knew you had a kill-switch in case things turned out differently. Everyone loves to get sleep.

2. Inconclusive, it’s a tie

Five days into the test you realize that your new design is underperforming your old version. However, with all the work your team put into this new experience—which tested excellently with focus groups—you aren’t ready to abandon ship and revert to your old design yet. Instead, you decide to reallocate users to a 50/50 split between the two designs to investigate further.

Taking this action is valuable is a number of ways.

First, it enables you to identify the main issues with your new design. Is it buggy or is the user experience just confusing? Should you fully scrap all the hard work you put into this new design or, can you dig deeper into the data and salvage some of it? You can begin to do such data analysis while still allocating 100% of users to the old design if need be.

Second, you can let the test run for more time and understand the cause of the initial underperformance. Your KPIs may improve over time as users familiarize themselves with the new design.

This is what being data-driven is all about.

3. A buggy release

Next to bad UI, bugs are the most cited reason for negative app reviews. People have very little patience for apps that crash or are buggy. You can catch bugs in the pre-release QA, but something may always slip through. In the case that your release causes the app to crash, you’d want to minimize the number of users who experience it. Hence, you do a phased rollout.

Releasing the new feature to more than just your team will help you find bugs before they cause people to delete your app.  So even if you do discover a bug, you can roll it back and tinker with the code some more to squash that bug.

4. Engagement tanks

A few days into the test you are seeing scary results: your new design is tanking your KPIs. You decide to wait three more days to see if users’ attitudes towards the new design change. Unfortunately, the results do not improve.

Although your new design is not performing well, the silver lining is you don’t need to wait for a code review—you can adjust in real time instead of waiting weeks to deploy an update.. The SDK installed on your app gives you full control over the allocation of users so you can easily throttle 100% of users back to the original design. Only 20% of users saw the new design.

5. Segment the results

If the new design performs poorly with one user segment but well with another, you can ship 100% traffic for a particular segment until you get to the bottom of why the release is performing poorly with the other audience.

Developing with your users

The ability to control the percentage of users who see the design is valuable to you and your users regardless of how your new design performs. In summary, phased rollouts can:

1. Give you confidence to release a new update to 100% of users.
2. Discover bugs you didn’t know earlier without frustrating 100% of your users.
3. Allow you to roll back buggy updates immediately without waiting for code review.
4. Ease your users into a new experience in phases.
5. Help you understand how different segments of users respond to the new release.

Regardless of how it goes, rolling out new designs or features in phases, rather than all at once is an example of a user-first development process. Develop with users, not only for them. A quote from a 2012 Facebook blog post explains this concept:

“This cycle of iteration is the engine of progress and the people who use Facebook are not just the beneficiaries but are also intimately a part of the process. We don’t just develop this product for them, we develop it with them.”

Testing through a release will allow all apps—no matter which category they’re in—to have this conversation with their users and develop the best possible the product with them.

Learn more about how you can start doing phased rollouts on your app with Optimizely’s mobile A/B testing.

Read Next: Safely Roll Out Your Next New iOS Feature