Lessons on A/B Testing and Beyond from SXSW Interactive 2015

We launched the Growth Marketer Roadmap for SXSWi 2015 for people aspiring to learn about applying experimentation, data, and iterative learning to drive growth. As it goes at SXSW, you never know what you’re gonna get. The verdict: the 12 sessions we picked had many of actionable lessons for Growth Marketers. People even lined up for talks about AB testing strategy from Netflix hours in advance!

In this post, I’ll share insights from a few of the growth marketer sessions I attended. (Yes, this post is coming a few weeks after SXSW because I’ve been busy planning Opticon, Optimizely’s conference happening June 17 & 18 in San Francisco!)

Did you attend any of these sessions? What were your takeaways? Let’s discuss!

The Art and Science of Shareability 

Dao Nguyen, the Publisher at Buzzfeed talked about what makes content shareable and used data to show that for Buzzfeed, it’s as much fearless art as it is a rigorous science. #shareable

Lessons:

• For Buzzfeed, the absolute number of social shares is a misleading metric for what makes a piece of content shareable. Instead they use social lift as to measure sharability. Social Lift = the # of views referred from social media / # of seed views from Buzzfeed site.

What Marketers Can Learn From Political Campaigns

David Murphy, the president of ad agency, Team Detroit and a panel of other experts, Rich Mintz and Michelle Mullineaux of Blue State Digital, and Peter Bouchard of Civis Analytics, shared lessons on what brand managers and marketers can learn from political campaigns. #TDSXSW Lessons:

• Storytelling creates relevancy and compassion for your brand.
• Know your customer: “The Obama campaign knew exactly where their fans were, knew exactly who wasn’t going to buy what they were selling, and they knew where they could more efficiently advertise themselves to attract potential new followers.” (source: Entrepreneur.com)
• Digital teams running political campaigns experiment a lot (something we’ve learned from work on the Obama 2012, Romney 2012, and Obama 2008 campaigns).

A/B Testing at Scale: Minimizing UI Complexity 

A/B Testing at Scale: Minimizing UI Complexity (SXSW 2015) from Chris Saint-Amant

The Case for Design Thinking In Communications

Jason Schlossberg, the co-founder and CCO communications agency Kwittken spoke about design thinking, what it is and why it enables teams to do better work. #commdesign

Lessons:

• Design thinking is rooted in brainstorming, prototyping, and experimenting.
• In order for a test-and-learn culture to thrive, you need to create a environment where fearlessness and risk-taking are encouraged and judgment is discouraged.
• The biggest barrier to design thinking is fear.

Everything Eric Ries Has Learned Since 2011

In 2011, Eric Ries published The Lean Startup that has sparked a movement around how products are built, and gave new meaning to the acronym MVP. Since publishing, Eric has traveled around the world visiting companies who are putting lean startup values to practice. Ted Greenwald, the Enterprise Technology Director at the Wall Street Journal interviewed Eric at SXSW about failure, cross-functional teams, and his new project, The Leaders Guide.

The Imagine Think whiteboard from the SXSWi 2015 talk, Everything Eric Ries Has Learned Since 2011.

Defining the Next Generation Retail Experience

Julie Bornstein, the former CMO at Sephora (now COO at Stitch Fix), moderated a fireside chat with Stitch Fix Founder and CEO Katrina Lake, and Jennifer Hyman, CEO and co-founder of Rent the Runway. Marketers in e-commerce are constnatly innovating and therefore have so much to gain from experimentation. It was exciting to hear these panelists discuss the role of data and personalization in their businesses. #sxstyle Lessons:

• Retailers who will win are thinking of the customer experience as one holistic experience and the channel as wherever the customer wants to be.

Netflix Shares a Decade of A/B Test Learning 

Netflix was a familiar face on the #ABtesting conversation at SXSW this year, which makes sense because they do so much experimentation. In this packed session (seriously, people in line were offering cash to anyone who would give up their seat) Todd Yellin, the VP of Product Innovation shared why A/B testing matters and lessons they’ve learned in testing for 10 years.  Lessons:

“It’s not what users tell you, but what they *do* that matters” -Netflix VP of Product #SXSW #SXSWInteractive #abtesting — Chris Saint-Amant (@csaintamant) March 14, 2015

Big Picture Testing: Beyond The One-Off Result

Another #ABtesting session! On this panel, Sarah Newhall of Blue State Digital, Toby Fallsgraff of Organizing for Action, and Kyle Rush from Optimizely discussed everything from why experimenting is essential for challenging assumptions to how to choose the right tools.

Great sketch notes from the session:

Notes by Katie Kelly

Notes by Katie Kelly

Did you attend any of these sessions? What were your takeaways? Let’s discuss!