10 Questions to ask optimization candidates

Finding the right person to optimize your website presents a unique challenge. This person will have the opportunity make a tremendous business impact, but must also navigate the intricacies of company processes and culture to do so. Depending upon your conversion rate optimization (CRO) needs, this person could be an experienced conversion rate expert or someone who has expressed an interest in learning more about testing with potential to grow into the role.

In conducting this search, you’ll need to seek out individuals with a specific set of core competencies to ensure that you have the right person(s) to lead testing across your company and yield fantastic results. We tapped into the collective wisdom of Optimizely’s own Ryan Lillis, Kyle Rush and Chrix Finne to help identify the key competencies and questions to ask when searching for this key team member.

With experiences ranging from the very small (5 person startups) to the very large (Google, the Obama for America Campaign and Fortune 100 companies), this team had a diverse range of perspectives to offer when it comes to hiring for optimization.

Core Skills to Look for:

• Creativity  Identify an individual that will find innovative ways to reimagine your website.

• Curiosity  This person should always be asking “why?” and be relentless in their search of answers through data.

• Analytical Thinking  A basic understanding of statistics and how to position results in context will be invaluable.

• People Skills  This person will need be able to bring key stakeholders onboard, and have the charisma to spread testing excitement internally while enabling others to test.

10 Interview Questions for a CRO Rockstar:

1. Where would you start your optimization efforts on our site? Look to see if the candidate can distill your high-level conversion goals, and propose some feasible introductory tests to get started.

2. What are your biggest pet peeves with websites? Learn about what the candidate finds the biggest hurdles are for visitors to websites. What stops them from converting? This demonstrates empathy for the end-user or customer they will be addressing.

3. What is your process for coming up with test ideas? Understand if the candidate has a few random ideas, or if they have more of a systematic way of approaching testing. Avoid bringing in a CRO that could run out of test ideas after a few months’ time.

4. What are the limitations of split testing? Sometimes, recognizing the issues A/B testing can’t solve is just as valuable as recognizing the issues it can solve. Look for answers elsewhere when A/B testing tools can’t tell you what your business objectives should be. 

5. Describe your experience with analyzing data. Your organization may look at A/B test results in the testing platform only. It’s also possible that you may have whole sets of other data (via analytics, data warehouse system or similar) that you’ll want the optimization specialist to cross reference and draw meaning from.

6. How do you determine when to stop an experiment? In the event that an experiment yields inconclusive results, you’ll need the CRO to interpret the conditions of the experiment to determine when to stop and how to avoid the same outcome in the future. They should demonstrate knowledge of margin of error, volatility, accuracy, and bias.

7. Describe the roles of qualitative and quantitative data on the optimization process. A big-picture understanding of how best to deploy experiments relies on a combination of these two types of data.

8. When do you employ a multivariate versus an A/B test? Demonstrates an understanding of two test types and the benefits of using each, as well as the constraints.

9. How do you prioritize experiment ideas? In the answer to this question, you’re looking for the candidate to show a strong understanding of which tests will yield the best ROI.

10. How do you share and create excitement within your organization? The CRO needs to be an evangelist for testing, and will be more successful with team and company support. Ask for concrete examples or ideas that show a desire to be successful in sharing the optimization process with others.

Download the full Roadmap to Building an Optimization Team for Four Bonus Questions