Monthly Archives: September 2012

Ustream Increases Broadcast Sessions 12% With Icon + Text

By on September 18, 2012

Background

Ustream is a leading live interactive broadcast platform that aims to bring people together around shared interests for live interactive experiences.

The company attracts millions of users from around the globe ranging from artists to celebrities to music enthusiasts to corporate executives.

Challenge and Goals

As a live and interactive video streaming site Ustream is always looking for ways to improve their user experience.   Part of their on-going improvement process entails the constant development of new user interface design elements, each with specific goals in mind.

Recently they launched an opt-in beta site for early adopters which had multiple proposed layouts. In order to determine the optimal layout for their desired visitor engagement, they needed to gather intelligence about how each layout would drive visitor interactions.

A button in the header of the new layout provides the ability for Ustream’s broadcasters, or soon-to-be broadcasters to launch the web-based Broadcast Console. Ustream wanted to optimize the new layout to maximize the number of broadcast sessions initiated from this button. Two different proposed button designs surfaced as a viable solution to the layout challenge.

Hypothesis

The team was divided on how to garner the most broadcast sessions with a button change. They assumed that as a multilingual site, a lone camera icon would sufficiently drive new sessions. The team thought that adding the word “Broadcast” next to the icon might also be necessary; however, they wanted to save as much space as possible, since the header is a valuable part of their screen estate. The team was unsure of which experience would achieve the optimal results.

Solution

Using Optimizely, Ustream set up an A/B test pitting each of the proposed button designs against each other.  The Ustream team set up a click goal on both buttons in order to measure the number of broadcast sessions that each button design yielded.

After exposing the test to just over 12,000 visitors, a statistically significant winner emerged, and the uncertainty could be resolved with data.

“Trust your intuition, but don’t forget to measure. With Optimizely setting up an experiment like this is very easy and was up and running within a half an hour,” says Balazs Kereskenyi, Ustream’s product manager for content discovery.

Original

Variation

Results

The text and icon version was 12% more effective in achieving the desired result for the Ustream team.

Conclusion

WIth Optimizely, Ustream was able to quickly measure the impacts of two potential designs prior to rolling them out to the entire Ustream user community.  After running the test for four days, they were able to find a clear winner and ensure that the optimal design was being implemented. Rather than acting on hunches or feelings, the team could confidently assess and measure the value of their work.

About Jodie Ellis:

Jodie hails from the Land of Enchantment, commonly referred to as New Mexico, but has made San Francisco his home for many moons. As Optimizely's Director of Marketing, Jodie markets awesome products that allow businesses to make data-driven decisions.

Posted in Case Studies | Leave a reply

A/B Testing All-Star: John from IXL Learning

By on September 14, 2012

Name: John Barth
Company: IXL Learning 
Role: Senior Marketing Analyst
What’s the average age of students who use IXL Learning? Our most popular grade is 2nd grade, so, most of our students are around 8 years old.

John says, “Every problem I had with Google Website Optimizer (GWO) had been thought through and solved by Optimizely.  The difference between the two is like night and day.”

Optimizely: When did you start testing and why do you test?

John: We started testing about a year ago.  I had ideas for several different things I wanted to test for awhile, but without an A/B test framework, you can’t properly see the differences between the original page and the test variation, other than taking the plunge and implementing the changes.

We started out with Google Website Optimizer (GWO), but I was disappointed with its limited functionality and difficulty in test deployment, especially that it required the engineers to code the different variations and then put them under different URLs.  We were just starting to look at different A/B testing solutions when Julio reached out to us, and the rest is history!  Every problem I had with GWO had been thought through and solved by Optimizely.  The difference between the two is like night and day.

What is the testing culture and process like at IXL Learning?

Everyone on the marketing and development teams are encouraged to come up with ideas for tests.  When we all like the idea, we have our UI designer to come up with a few different variations.  Then I create the variations in the Optimizely editor (or get help from an engineer when there are larger UI changes), get approval from the product management team and the VP, and deploy the test.  On the long side, it can take a couple of weeks to get everything set up and approved for our more complicated tests.  On the short side, I’ve taken an idea from a concept to an actual running test in under an hour.

Where do your testing ideas come from?

I follow some blogs and read about tests other people have run for ideas.  I also try to think like a user and try to figure out what I can improve on our pages to improve engagement or continue through our subscription flows.

Tell us about a test that has yielded actionable results.

The membership area on our site directs users between our school subscriptions and our family subscriptions.  It’s also the first page in all of our subscription flows, so it’s critical that this page be easy to understand and navigate.  The original page had all the calls to action on the right side of the page, including some below the fold, and one column of text across the page.  Moving the calls to action above the text, and breaking the text into two columns saw gains across the board for all of our conversion types!

Before

 After

What’s your favorite Optimizely feature to use in your tests and what new feature would you like to see?

There are so many it’s hard to pick.  My absolute favorite is multiple goal tracking.  We have several different types of conversions on our site, so when I run tests, it’s critical for me to see the impact to all of them.  Sometimes the best variation will see some decline in one of the conversions at a larger gain in another.

One thing I would like to see that is more of a new product than a feature would be adding A/B testing for e-mails.

Share some best practices you’ve developed with testing.

  • Don’t make assumptions about what users will like or not like. Sometimes they’ll surprise you.
  • Some of the biggest gains can come from the smallest tests.  A lot of the higher impact tests I’ve run have been simple changes in button colors and text.

About Cara Harshman:

As a Content Marketer at Optimizely, Cara is constantly looking to tell innovative stories about how businesses are using A/B testing to become more successful. Have a data-driven story that deserves to be told? Email her at cara [at] optimizely [dot] com. Before joining the team in 2012 she was reporting the news in Madison, WI.

Judy’s Book Sees 198% Increase in Engagement with Simple Code Edit

By on September 12, 2012

Background

Judy’s Book is a reviews website that caters to a family audience. After reading an article in WIRED about Optimizely, Ali Alami, GM and Acting CEO of Judy’s Book got started with the tool because it allowed his small team to maximize their resources.

“At board meetings, there are many ideas on strategy and features like, ‘so and so is doing this, let’s try it,’” Ali says. “Rather than going off hunches and just doing it, our leadership understands the need to validate. We can quickly test ideas and let the numbers speak for themselves.”

Challenge and Goal

Paid business listings are a primary way Judy’s Book makes money. Businesses get better search positioning, more robust profiles with photos if they pay a monthly fee. While they also offer a free listing, Judy’s Book wanted to increase the number of businesses who sign up for paid listings. The form to sign up for a free listing was a bit hidden due to the thought that if people saw a free listing, they wouldn’t want to pay for one.

Ali and his team wanted to optimize the pricing page to drive more businesses to sign up for paid listings.

Hypothesis

Ali thought that positioning a column highlighting the features included with a free listing alongside the paid listings might lead more people to sign up for paid listings. Since the features included with a free listing are very limited, Ali had a feeling that visually showing the comparison of features that come with a free versus a paid listing would convince visitors to sign up for the paid listings.

Solution

One of the engineers at Judy’s Book appended the original code with a simple change to add space for a new column. Ali copied and pasted the new code into the “edit code” section in Optimizely’s editorand it immediately deployed the new version of the listing breakdowns.

For them, success meant increasing the overall engagement on the pricing page. Optimizely by default tracks engagement, or the number of clicks each variation page receives. By easily setting up click goals on every sign up button on the page, Judy’s Book measured engagement specifically on each button. They ran the experiment through Optimizely for a month to ensure statistically significant results.

Original

Variation

Results

Showing a free listing column increased clicks on the sign up button for the basic paid listing by a whopping 198.6%.

Conclusion

When in doubt, test. “People have lots of ideas, hunches and feelings. A best practice is to continue to A/B test and just see what happens,” says Ali.

With Optimizely, Judy’s Book was able to quickly deploy a simple code change and test the impact it had on sign-ups for paid listings before implementing this change on their live site. Ali validated a hunch he had about the pricing page through A/B testing that led to significant gains for the company.

About Cara Harshman:

As a Content Marketer at Optimizely, Cara is constantly looking to tell innovative stories about how businesses are using A/B testing to become more successful. Have a data-driven story that deserves to be told? Email her at cara [at] optimizely [dot] com. Before joining the team in 2012 she was reporting the news in Madison, WI.

Optimizely Open Sources Python Scripts for Amazon EC2

By on

Today we’re excited to release an open source grab bag of scripts for Amazon EC2, available at https://github.com/jeff-optimizely/grab-bag-for-ec2

There are two main purposes for the grab bag:

1. The Grab Bag contains useful scripts – such as a backup script – that you may want to use by themselves.
2. The Grab Bag scripts can be used as examples for creating your own scripts.

The repository contains Python scripts for creating instances, creating RAID arrays, backing up volumes, updating ELBs, and updating your /etc/hosts file with aliases for your EC2 instances.

To get started with the scripts, add your AWS access key and secret to the credentials.py. It’s a best practice to test third party code with a secondary account first, rather than executing it on your primary account.

Version 1.0 contains the following files:
backup_config.json – a config file for backup.py
backup.py – a backup script for backing up volumes on Amazon EC2
common.py – common functions that can be used by the other scripts
create_instance.py – a script for creating an instance and doing other first-time maintenance work, such as installing Yum packages
create_raid.py – a script for creating a RAID array and attaching it to an instance
credentials.py – your AWS credentials
README.md – a read me file
update_elb.py – a script for adding to and removing from an ELB
update_hosts.py – a script that adds the IP addresses of your EC2 instances to your /etc/hosts file, for convenience

We’re also inviting other developers to contribute to this collection of scripts. If you have ideas for new scripts, edits or improvements, please contribute. We’re looking forward to seeing what you come up with.

And Python developers, stay tuned. We have another project we’re planning to open source in the near future for efficiently working with MongoDB.

If this post interests you, we’re also looking to hire backend and dev ops engineers.  See our jobs page for more info.

 

About Optimizely:

Optimizely is the world's leading website optimization platform.

Posted in Inside Optimizely, News & Events | Leave a reply

Data is King

By on September 7, 2012
Data_is_king

“I have been testing the aphorism, “A watched pot never boils.” I have boiled the same amount of water in this kettle sixty-two times. In some cases I have ignored the kettle; in others, I have watched it intently. In every instance, the water reaches its boiling point in precisely 51.7 seconds. It appears I am not capable of perceiving time any differently than my internal chronometer.” 

 -Lt. Commander Data

Today is Star Trek’s 46th birthday. While we love all the Star Trek crew, we have a special place in our collective testing hearts for Data. Data resides at the core of our and our customers’ continued success. It steers us through the ominous skies of hunches and HIPPOs. We would all be lost without it. Let data be your guide. Boldly test where no one has tested before. A/B testing: engage. 

A bit about Data: 

  • He was activated on February 2, 2338
  • He weighs 220 pounds
  • Data has a storage capacity of 800 quadrillion bits (approx. 50 times the storage capacity of the human brain)
  • He graduated from the Starfleet Academy in 2341, just three years after activation
  • He tends to avoid contractions in his speech
  • Data has fake tear ducts
  • Data is incapable of getting drunk
  • To feed himself, Data occasional ingests a silicon-based liquid to lubricate his biofunctions
  • He has two older brothers, B-4 and Lore, and constructed a daughter, Lal
  • Data’s favorite book is Sherlock Holmes
  • He plays the oboe, violin and classical guitar
  • He is a poet, but his lack of emotion hinders a provocative, sensitive delivery
  • He died in 2379 when the Reman warbird Scimitar was destroyed

Source: http://en.memory-alpha.org/wiki/Data

 

About Jodie Ellis:

Jodie hails from the Land of Enchantment, commonly referred to as New Mexico, but has made San Francisco his home for many moons. As Optimizely's Director of Marketing, Jodie markets awesome products that allow businesses to make data-driven decisions.

Posted in Inside Optimizely | Leave a reply