4 Questions That Helped Visa Create a Foundation for Experimentation

In a joint webinar with Optimizely: The Journey to a Culture of Experimentation, Ramkumar Ravichandran, Visa’s Director of A/B Testing, shared the moments that made Visa’s experimentation program successful—with 2.5X more successful tests than is their  industry standard: 

We’ll share four questions that Ramkumar and his team used to determine the right mix of product, teammates, and process to build Visa’s current experimentation team. 

1. To build or to buy? 
2. Who will be responsible for the experiments? 
3. What’s the most efficient experimentation process?
4. How to set KPIs to track success?

1. To build or to buy?

Ramkumar shared that Visa first had to decide whether they’d build or buy their solution—an often painstaking decision many enterprises face when they begin testing. 

“So the default decision that we used to hear from others was—let’s build something in-house, you know, because not any experimentation tool in the world can solve all our problems.” 

• Ramkumar Ravichandran, Visa

Ramkumar’s team, at first did default to build in-house. He’d built solutions internally on previous experimentation teams so that he could not only run the most impactful experiments, but also execute complex tests (multivariate, server-side, multi-factor, multiple A/B, or multipage) and customize metrics & analytics integrations. 

“It’s about being able to define custom metrics and have a streamlined integration with the analytics and customer experience management tool, and have all of that flow back into our data lake, for us to be able to do retrospective analysis. It’s about doing all of this at scale with performance across all kinds of environments.” 

Another reason it was tempting for Visa to build internally was because they had a large and talented team of engineers. Despite this great engineering team, building internally still would have been too expensive. Ramkumar said his team would have to dedicate too much time to not only building the solution but maintaining it through the life cycle of the product. This investment didn’t make sense given other strategic priorities for engineers.

“The time and financial burdens required to build in house put us in favor of getting the industry leader…who can integrate quickly—today. Optimizely is a platform where non-programmers can also interact…and catch up with the newer trends in the industry.” 

After deciding to partner with Optimizely rather than build an internal solution, Ramkumar and his coworkers had to sit down to define the members of their team, their new process, and the team’s KPIs.

2. Who is responsible for the experiments? 

Once partnered with Optimizely, Visa set up a team to manage experimentation. Ramkumar shared with us two key questions he asked himself while building the team:

• Who manages the experimentation program?
• Who owns which parts of the process?

“We realized that it’s not going to be one person doing it all. It’s not going to be a few people sitting in silos doing it all. It’s  going to be a giant structure, which is supposed to be fast, it’s supposed to be dynamic, and responsive to the changing trends.”

To settle on the right team framework, Visa tested different team structures before landing on the right combination of professionals for them. This framework can be expanded as teams scale. Visa’s team now includes:

A/B test analyst: drives and manages the program. They lead the team from ideation to handing off a successful test to the product owner. This person is an expert in Optimizely’nd owns the test setup, deployment, and analytics.

A product partner: brings the product team’s perspective on ideation and testing. They’re connected to different product owners and act as the liaison to the product team. 

QA Partner: ensures there aren’t bugs in the test setup from a usability standpoint. 

Technology Partner: consults on tests and assists in setting up advanced tests that require programming knowledge. 

Design Partner: gives the team visuals to work off of for a test. 

After Ramkumar had an experimentation team in place, he turned his focus to honing the most effective experimentation process.

3. What process will the experimentation team follow?

Rather than building a team around a predetermined process, Visa built team first. This way, the team was involved in process creation and provided valuable feedback for each process step. 

Visa used Kanban teams with their QA, tech, design, and product partners. Kanban is an agile development method used to balance a team’s bandwidth and handle bottlenecks. They thought through the entire process as a team—from ideation and setup to QA, deployment, and post-deployment. 

“So the Kanban team setup is the one that helped us because everyone was focusing on the same goal of making experimentation successful and driving product impact.”

The team starts with ideation and prioritization, brainstorming and deciding which tests are worth pursuing. Then, they move on to the setup and designing the test. Next, they QA that test from the user’s perspective before they deploy the test. After deployment, the team can analyze data and implement changes for improvement. 

Image from Visa and Optimizely webinar: The Journey to a Culture of Experimentation

4. How will the team set KPIs?

Ramkumar noted how important it was to define KPIs early to track the team’s success as the team was both large and responsible for improving many features and components of Visa’s product. He posed two questions to help define KPIs:

• Do we have any success criteria? 
• Do we have any metrics or framework that can define this test as successful? 

Ramkumar’s team came up with a two-phased KPI approach. The first phase was “operational,” tracking how successful the experiment’s operations and processes were. The second phase of KPIs was the impact and ROI of each experiment—tracking their impact on overall business goals (like yearly revenue) as Visa’s experimentation team wanted to be able to show they had made a sizeable contribution to Visa’s overall business goals. 

Key takeaways

If you’re looking to get the most out of experimentation, it’s vital you lay the right foundation. 

• Choose a platform that can handle the complexity you need.
• Define your experimentation teammates’ roles and expected contributions.
• Build an efficient and effective process for testing—from ideation to deployment and optimization.

Visa continues to build on the successful experimentation foundation that Ramkumar established four years ago. In the future, they plan to scale their experimentation program, continuing to invest in making data-based decisions.