Lessons on A/B Testing and Beyond from SXSW Interactive 2015

South by Southwest Interactive was good to the growth marketer this year. Sessions at this mega-conference/party are a gamble — there are so many to pick from so some inevitably turn out to be fluff while others exceed your wildest expectations.

We launched the Growth Marketer Roadmap for SXSWi 2015 for people aspiring to learn about applying experimentation, data, and iterative learning to drive growth. As it goes at SXSW, you never know what you’re gonna get. The verdict: the 12 sessions we picked had many of actionable lessons for Growth Marketers. People even lined up for talks about AB testing strategy from Netflix hours in advance!

In this post, I’ll share insights from a few of the growth marketer sessions I attended. (Yes, this post is coming a few weeks after SXSW because I’ve been busy planning Opticon, Optimizely’s conference happening June 17 & 18 in San Francisco!)

Did you attend any of these sessions? What were your takeaways? Let’s discuss!

The Art and Science of Shareability 

Dao Nguyen, the Publisher at Buzzfeed talked about what makes content shareable and used data to show that for Buzzfeed, it’s as much fearless art as it is a rigorous science. #shareable

Lessons:

• For Buzzfeed, the absolute number of social shares is a misleading metric for what makes a piece of content shareable. Instead they use social lift as to measure sharability. Social Lift = the # of views referred from social media / # of seed views from Buzzfeed site.

.@Buzzfeed‘s experimentation strategy: Give small teams autonomy and resources to go out and learn something new. #sxsw #shareable

— Lacy Baugher (@LacyMB) March 13, 2015

What you need to know if you missed @daozers #shareable talk at #SXSW via @Gerick pic.twitter.com/rvN5wrjufa — Christina DiRusso (@CDiRusso) March 13, 2015

What Marketers Can Learn From Political Campaigns

David Murphy, the president of ad agency, Team Detroit and a panel of other experts, Rich Mintz and Michelle Mullineaux of Blue State Digital, and Peter Bouchard of Civis Analytics, shared lessons on what brand managers and marketers can learn from political campaigns. #TDSXSW Lessons:

• Storytelling creates relevancy and compassion for your brand.

• Know your customer: “The Obama campaign knew exactly where their fans were, knew exactly who wasn’t going to buy what they were selling, and they knew where they could more efficiently advertise themselves to attract potential new followers.” (source: Entrepreneur.com)

• Digital teams running political campaigns experiment a lot (something we’ve learned from work on the Obama 2012, Romney 2012, and Obama 2008 campaigns).

“Enable the people who already care about the campaign and empower them to bring others on board.” – @richmintz #SXSW #TDSXSW — Missy Laney (@missmissylaney) March 13, 2015

Politics and Marketing: Data, data, data. Test, test, test. #SXSW #TDSXSW — Brad Spychalski (@bradspy) March 13, 2015

A/B Testing at Scale: Minimizing UI Complexity 

Chris Saint-Amant, Manager of UI Engineering at Netflix spoke to a packed room about how to think about experimenting with many variables while minimizing chaos. He published the slides from his session on Slideshare. Check them out below. #ABtesting

The Case for Design Thinking In Communications

Jason Schlossberg, the co-founder and CCO communications agency Kwittken spoke about design thinking, what it is and why it enables teams to do better work. #commdesign

Design thinking: empathize (what is the human need?), observe (how will humans use it?), brainstorm, prototype then test. #commdesign #sxsw — Creative Circle (@Creative_Circle) March 16, 2015

Lessons:

• Design thinking is rooted in brainstorming, prototyping, and experimenting.
• In order for a test-and-learn culture to thrive, you need to create a environment where fearlessness and risk-taking are encouraged and judgement is discouraged.
• The biggest barrier to design thinking is fear.

Most marketers still make decisions based on gut, but we need to feed our gut w/ observable facts #commdesign #SXSWi — Cara Harshman (@CaraHarshman) March 16, 2015

“Create a culture of rapid prototyping – you’re never going to reach perfection immediately.” – @JSchlossberg #commdesign #SXSW — Amanda Sevasti (@AmandaSevasti) March 16, 2015

Everything Eric Ries Has Learned Since 2011

In 2011, Eric Ries published The Lean Startup that has sparked a movement around how products are built, and gave new meaning to the acronym MVP. Since publishing, Eric has traveled around the world visiting companies who are putting lean startup values to practice. Ted Greenwald, the Enterprise Technology Director at the Wall Street Journal interviewed Eric at SXSW about failure, cross-functional teams, and his new project, The Leaders Guide.

Defining the Next Generation Retail Experience

Julie Bornstein, the former CMO at Sephora (now COO at Stitch Fix), moderated a fireside chat with Stitch Fix Founder and CEO Katrina Lake, and Jennifer Hyman, CEO and co-founder of Rent the Runway. Marketers in e-commerce are constnatly innovating and therefore have so much to gain from experimentation. It was exciting to hear these panelists discuss the role of data and personalization in their businesses. #sxstyle Lessons:

• Retailers who will win are thinking of the customer experience as one holistic experience and the channel as wherever the customer wants to be.

“Data is the most important aspect of personalized, competitive online retail.” @Jenn_RTR @kmlake #SXstyle #sxsw http://t.co/tSXB5lEsfv — Tradable Bits (@tradablebits) March 13, 2015

Netflix Shares a Decade of A/B Test Learning 

Netflix was a familiar face on the #ABtesting conversation at SXSW this year, which makes sense because they do so much experimentation. In this packed session (seriously, people in line were offering cash to anyone who would give up their seat) Todd Yellin, the VP of Product Innovation shared why A/B testing matters and lessons they’ve learned in testing for 10 years.  Lessons:

“It’s not what users tell you, but what they *do* that matters” -Netflix VP of Product #SXSW #SXSWInteractive #abtesting — Chris Saint-Amant (@csaintamant) March 14, 2015

Reacting to the loudest voices (or using instinct) can ignore the silent majority. #abtesting gives you the real story about behaviour #sxsw — Camilla Yates (@camilla_y) March 14, 2015 Read more tweets from this session.

Big Picture Testing: Beyond The One-Off Result

Another #ABtesting session! On this panel, Sarah Newhall of Blue State Digital, Toby Fallsgraff of Organizing for Action, and Kyle Rush from Optimizely discussed everything from why experimenting is essential for challenging assumptions to how to choose the right tools.

Great sketch notes from the session:

Did you attend any of these sessions? What were your takeaways? Let’s discuss!

Also check out this excellent SXSW round-ups from Marketo.