How Vivid Seats Ensures Experimentation Success

During the hour long webinar, Bill covered everything from his own experience with experimentation to the Optimizely Full Stack implementation at Vivid Seats, and he even dropped in some fun analogies along the way. There were so many golden nuggets in the webinar that we couldn’t help but wrap some of them up to share more broadly with our blog readers.

Bill kicked off the webinar by openly recognizing that experimentation is hard, building an experimentation culture is not easy, but it certainly can be done, and it can be done well and very efficiently.

Having started his experience at Orbitz using a homegrown platform, he really learned how to use experimentation as a platform driver when he moved on to Expedia. With the newfound experience under his belt, he moved to a startup called Raise, built a testing practice on the engineering team, and had an opportunity to oversee product experimentation in a space outside of the travel industry. Bill said of this experience, “At a small company, testing is the only way to survive. Experimenting is doing the least in order to learn the most.“

Today, Bill leads experimentation at Vivid Seats, a secondary marketplace for tickets. It’s a market where customers know what events they want to go to, but they have no brand loyalty, high price sensitivity, and usually no idea what type of tickets they’re looking for. This customer profile leads to having to go beyond just testing design elements and instead, requires his team to focus on creating longer-term audience relationships and learning how to influence customer behavior.

For example, Bill walked through an example at Vivid Seats where the team had a hypothesis that “adding a ticket details screen between the tickets list and checkout screen will aid users to find a more desirable ticket, which will ultimately improve conversions.” This type of change goes beyond web testing and changing the color of a button or copy, but rather, updates have to be made on the server-side to pull in the right ticket, the section details, and the price. From this example, Bill shared a useful tip that “you should be testing verbs, not nouns.” That is, you should be testing measurable changes to user behaviors versus just whether you should build something. In this case, the behavior the team wanted to impact was conversions on the “buy ticket” page. He suggested deciding what the verb is before you run a test and to focus on behavioral changes.

Bill also shared a helpful acronym he uses before kicking off an experiment: MVMRC (yes, Bill pronounced this on the webinar and I’ve tried to do so myself, but haven’t been able to successfully).

What does it stand for, you ask?

• M: Measurable (is it driving more conversions?)
• V: Visible (is it visible to the test group or defined audience?)
• M: Meaningful (did this change actually matter to our customers and do they care?)
• R: Realistic (is this something that we’d like to roll out to everyone?)
• C: Cheap (is it cheaper to run a test or to get the test out and accidentally fail?)

After walking through numerous experimentation examples and best practices, Bill ended by demonstrating the value of building a center of excellence. This center of excellence consists of testing experts and every team’s foundational learnings will come from this group of people. The goal should be to have a center of excellence that spreads best practices and helps consult on tests, but recognize that the ownership of testing will become more decentralized and there will be more individual team ownership as an organization grows. If you’ve established a strong center of excellence, you shouldn’t be concerned about the decentralized ownership so spend quality time building it.

Throughout the hour, there were many more examples and interesting tidbits, so If you want to learn more about Bill’s experience building experimentation cultures at four separate companies, be sure to watch the on-demand webinar. Create your own center of excellence and integrate MVMRC into your testing approach today!