Lessons on A/B Testing and Beyond from SXSW Interactive 2015

By

SXSW InteractiveSouth by Southwest Interactive was good to the growth marketer this year. Sessions at this mega-conference/party are a gamble — there are so many to pick from so some inevitably turn out to be fluff while others exceed your wildest expectations.

We launched the Growth Marketer Roadmap for SXSWi 2015 for people aspiring to learn about applying experimentation, data, and iterative learning to drive growth. As it goes at SXSW, you never know what you’re gonna get. The verdict: the 12 sessions we picked had many of actionable lessons for Growth Marketers. People even lined up for talks about AB testing strategy from Netflix hours in advance!

In this post, I’ll share insights from a few of the growth marketer sessions I attended. (Yes, this post is coming a few weeks after SXSW because I’ve been busy planning Opticon, Optimizely’s conference happening June 17 & 18 in San Francisco!)

Did you attend any of these sessions? What were your takeaways? Let’s discuss!

The Art and Science of Shareability 

Dao Nguyen Buzzfeed SXSW 2015

Dao Nguyen, the Publisher at Buzzfeed talked about what makes content shareable and used data to show that for Buzzfeed, it’s as much fearless art as it is a rigorous science. #shareable

Lessons:

  • For Buzzfeed, the absolute number of social shares is a misleading metric for what makes a piece of content shareable. Instead they use social lift as to measure sharability. Social Lift = the # of views referred from social media / # of seed views from Buzzfeed site.

What Marketers Can Learn From Political Campaigns

David Murphy, the president of ad agency, Team Detroit and a panel of other experts, Rich Mintz and Michelle Mullineaux of Blue State Digital, and Peter Bouchard of Civis Analytics, shared lessons on what brand managers and marketers can learn from political campaigns. #TDSXSW Lessons:

  • Storytelling creates relevancy and compassion for your brand.
  • Know your customer: “The Obama campaign knew exactly where their fans were, knew exactly who wasn’t going to buy what they were selling, and they knew where they could more efficiently advertise themselves to attract potential new followers.” (source: Entrepreneur.com)

A/B Testing at Scale: Minimizing UI Complexity 

Chris Saint-Amant, Manager of UI Engineering at Netflix spoke to a packed room about how to think about experimenting with many variables while minimizing chaos. He published the slides from his session on Slideshare. Check them out below. #ABtesting

A/B Testing at Scale: Minimizing UI Complexity (SXSW 2015) from Chris Saint-Amant

The Case for Design Thinking In Communications

Jason Schlossberg, the co-founder and CCO communications agency Kwittken spoke about design thinking, what it is and why it enables teams to do better work. #commdesign

Lessons:

  • Design thinking is rooted in brainstorming, prototyping, and experimenting.
  • In order for a test-and-learn culture to thrive, you need to create a environment where fearlessness and risk-taking are encouraged and judgement is discouraged.
  • The biggest barrier to design thinking is fear.

Everything Eric Ries Has Learned Since 2011

In 2011, Eric Ries published The Lean Startup that has sparked a movement around how products are built, and gave new meaning to the acronym MVP. Since publishing, Eric has traveled around the world visiting companies who are putting lean startup values to practice. Ted Greenwald, the Enterprise Technology Director at the Wall Street Journal interviewed Eric at SXSW about failure, cross-functional teams, and his new project, The Leaders Guide.

Everything Eric Ries Has Learned Since 2011 Imagine Think SXSW 2015

The Imagine Think whiteboard from the SXSWi 2015 talk, Everything Eric Ries Has Learned Since 2011.

Defining the Next Generation Retail Experience

Julie Bornstein, the former CMO at Sephora (now COO at Stitch Fix), moderated a fireside chat with Stitch Fix Founder and CEO Katrina Lake, and Jennifer Hyman, CEO and co-founder of Rent the Runway. Marketers in e-commerce are constnatly innovating and therefore have so much to gain from experimentation. It was exciting to hear these panelists discuss the role of data and personalization in their businesses. #sxstyle Lessons:

      • Retailers who will win are thinking of the customer experience as one holistic experience and the channel as wherever the customer wants to be.

Netflix Shares a Decade of A/B Test Learning 

Netflix was a familiar face on the #ABtesting conversation at SXSW this year, which makes sense because they do so much experimentation. In this packed session (seriously, people in line were offering cash to anyone who would give up their seat) Todd Yellin, the VP of Product Innovation shared why A/B testing matters and lessons they’ve learned in testing for 10 years.  Lessons:

Big Picture Testing: Beyond The One-Off Result

Another #ABtesting session! On this panel, Sarah Newhall of Blue State Digital, Toby Fallsgraff of Organizing for Action, and Kyle Rush from Optimizely discussed everything from why experimenting is essential for challenging assumptions to how to choose the right tools.

Great sketch notes from the session:

Notes from A/B testing session SXSW Interactive 2015

Notes by Katie Kelly

Notes from SXSW Interactive AB testing session

Notes by Katie Kelly

About Cara Harshman:

Cara is a storyteller and journalist. From 2012-2016 Cara was a Content Marketing Manager and Blog Editor at Optimizely. She wrote A/B Testing the book on behalf of Optimizely’s co-founders. Follow her @CaraHarshman.