Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more! Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program

X

Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more!

Get It Now




More than 95% of the world’s smartphones run either Android or iOS, and in 2014, Android made up more than 80% of worldwide smartphone shipments. That’s why, today, we’re excited to announce the general release of the Optimizely Android SDK. Now you can use Optimizely to optimize your app on both of the world’s leading smartphone platforms. (See a full list of features and details about the Android SDK here.)

Optimizely Android A/B Testing SDK

In the weeks running up to this release, we ran a beta program with some of the top Android apps on the market. Through many conversations with teams of all shapes and sizes, we came to understand that not all mobile developers are created equally. Both iOS and Android developers are up against big challenges unique to their respective platforms that can have a huge impact on the way they structure their testing programs. However they also share commonalities in how they think about development.

We’ve distilled these similarities and differences between iOS and Android development down to a list of five traits. As you think about how to structure your mobile testing program, we highly encourage you to consider some of these key differences and similarities.

Android and iOS app developers…

1. …have different needs in terms of flexibility.

Perhaps the most apparent difference for developers between iOS and Android development is the app review cycle.

The length of iOS app review cycles are a huge unknown for mobile developers and can often take days or even weeks. This happens because Apple requires human beings to review all code for each app submission. Google recently announced that they will also have people review app updates, but the process still happens faster—hours, not days.

That said, Optimizely has made it our business to help customers get updates published quickly on iOS when they use our technology. For example, companies often use our Live Variables feature to make sections of their app’s code editable on-the-fly without having to go through the App Store review process.

Of course, we think this is a great practice for mobile developers on all platforms, not just iOS. See #4 below for more on this.

2. They test for different circumstances.

Permissions in iOS are managed on an opt-in basis, while permissions on Android are opt-out. This has big implications for what mobile developers prioritize testing.

Push notifications can be critical to an app’s strategy for retaining users. iOS requires apps ask their users to opt into receiving push notifications. Android will allow any app to send its users push notifications unless the user opts out.

We’ve seen more of our iOS SDK customers experiment with the way they request permissions for push notifications (or other opt-in permission settings) than we have on Android. This makes sense, of course—it isn’t a situation that applies on Android. This really highlights the fact that, while apps themselves may be the same across both platforms, there are platform-specific circumstances like this that will influence your testing strategy.

If you’re an iOS app developer interested in experimenting with requesting permissions, we highly recommend reading this article on the topic.

3.They both experiment heavily with user onboarding & activation.

Both Android and iOS apps prioritize testing their onboarding and activation flows early-on in the process.

Studies have shown that users delete 80-90% of apps after only one use. That’s why Android and iOS developers alike understand just how important it is to activate users early so they understand what the app can do and keep them wanting more.

Activation and retention are inherently connected. If you can’t activate and excite your users, you don’t have anyone to retain. That’s why increasing activation should be a top priority for any mobile app team. French Girls, a popular social media app we wrote about a few weeks ago, increased their activation rate on iOS by 67.5% by testing the order of their onboarding steps and the copy in their in-product tour.

Related Post: 10 Essential Tests to Drive Mobile App Downloads, Engagement & Retention

4. They both want to control feature rollouts.

App developers want more flexibility and they’re using platforms like Optimizely to build that flexibility into their dev cycles.

As mentioned above, the review cycles for submitting to the App Store can be long and unpredictable. While this is less of a problem on Android, we still find Android teams have a strong appetite for performing controlled rollouts of new app features without having to submit updates to Google Play.

Our Android SDK supports the same mobile developer tools as our iOS SDK. We’ve seen a lot of customers in our beta using features like Live Variables and Code Blocks to give them more flexibility and control over the experience their users have in app.

5. They are both usually surprised by test results.

No matter how well teams know their users, they still find themselves surprised by the results of experiments they run.

Ever since we launched Optimizely in 2010, companies have told us about how surprised they are by their experiment results. This is one thing that hasn’t changed as we’ve expanded to support mobile apps.

Lots of companies use Optimizely to deliver experiences on both their website and their mobile app. As a result they often expect their mobile experiments to yield similar results as their web experiments. However while the humans using your mobile app might be the same humans who your desktop website, their intent, behavior, and context are all quite different on mobile than on the web.

These differences manifest in different ways depending on your type of business. For this reason, it’s critical that you continue to challenge your assumptions and experiment with anything you had taken for truth based on tests you’ve run in the past on the web.

Mobile is going to continue to skyrocket. That’s why it’s never been more important to have an optimization strategy that encompasses your customers’ unique needs on both mobile and the web.

If you’d like to learn more about how to start a mobile app optimization program from scratch, check out the free eBook we’ve written on the topic called Mobile App Optimization: Create an App Users Love.

You can also start A/B testing experiences in your iOS and Android apps today. Just sign into your Optimizely account and create a new project or visit the mobile A/B testing page to test it out.

Optimizely X
comments powered by Disqus