Download our FREE ebook of 43 A/B testing case studies from the world's leading companies for test ideas and inspiration. Download Now

Optimizely Blog

Tips & Tricks for Building Your Experimentation Program

X

Download our FREE Testing Toolkit for A/B testing ideas, planning worksheets, presentation templates, and more!

Get It Now


As part of our ongoing ‘Ship Happens’ series we recently held a fireside chat with Optimizely’s Chief Product Officer Claire Vo and Bill Press, our Senior VP of Engineering. Moderated by Senior Manager Product Marketing David Isquick, Claire and Bill talked through best practices on aligning the sometimes diverse and competing orgs, how things might sometimes be at odds, and what success looks like. 

Here are some of the top takeaways from that conversation.

Chief Product Office Claire Vo

Optimizely CPO Claire Vo

AS TO WHY THINGS MIGHT NOT WORK, A/K/A WHY THERE MIGHT BE TENSION

Claire Vo: The most common example I see is that product has the understanding of WHAT they want to build. And their goal is to build the WHAT. Engineering has sort of the HOW and the WHEN. And while those things seem close, they’re not close enough. 

There needs to be really a shared outcome and a shared goal for product engineering to really drive product forward in an aligned way.

Bill Press:  It can be that engineering wants to optimize too soon. So they want to try to make things faster, where they don’t need to be faster, or fix problems before they occur that aren’t going to be real problems with the customer. 

And product can, from their side, want to put a product out as fast as possible without a deeper understanding of where we need to build in quality proactively.

FURTHER WAYS IN WHICH PROBLEMS CAN ARISE

Claire: One of the most common sort of conflicting priorities you will see in product and engineering organizations is how much you want to invest in technical debt and infrastructure work

There’s feature development, the things that you build for your customers, the features that you release. And then there’s sort of technical debt and infrastructure work that support engineers in building those features and bringing up your technical platforms to be best in class, making sure your engineers are efficient. And as a product manager, you want as much of the feature delivery as possible and as little of the tech debt as possible. And I think, as an engineering leader…well, you tell us Bill…

Bill: As an engineering leader, you want to make sure that you’re building in a way that doesn’t make your life more difficult later, and doesn’t make life more difficult for the customers later. But oftentimes, that work isn’t necessarily apparent immediately. What the customer value is or what the long term value will be for the engineering team isn’t always clear while you’re building it. 

Bill Press Headshot

SVP Bill Press

OPTIMIZELY’S PORTFOLIO APPROACH 

Claire: So we will look at how much capacity we have in a given year to deliver things, whether they are features or more related to infrastructure. And then we’ll say this year, we think it’s really important to allocate 15% or 30%, or 50% to those specific tasks. We then protect the allocation pretty significantly. 

A WORD OR TWO ON STRUCTURE AT OPTIMIZELY

Claire:  We pair at every level for organization. So Bill and I are a pair, our engineering directors and PM directors pair, our PMs, and end product managers and engineering managers pair. And then there’s the product manager, and epic owner on any one feature. 

So almost at every level, you have these pairs. And in an ideal situation, you have designed as part of those pairs as well in some parts of our product we have that some parts we don’t, and that’s something we’re working to kind of fix. And so I think organizationally creating that pair is really important. 

We organize our teams by product area and not by technical practice. 

A WORD ABOUT GOALS 

Claire: We use OKRs here at Optimizely. We determine as an executive or leadership team, what our strategy is going to be for the year for the next three years. We really take a PM (Product Manager) as GM (General Manager) mindset here.

Bill: The main dimensions that I focus on with engineering are the following. Successful delivery is that there’s throughput, that it’s predictable and on time. and that it is high quality. High quality translates to performance, security, privacy, compliance, and reliability

WHAT DOES COLLABORATION LOOK LIKE WHEN IT WORKS 

Claire:  I would say Full Stack. It really came out of a customer problem with people wanting to do more server side experimentation.

BiIl: It first started as an idea that we shared between PM and engineering. So we have a hack week twice a year. And the entire week we set aside time for exploring new ideas. And anything is allowed. And we have some great product ideas come out of that. 

One of which was Full Stack. And between the very first hack week to deeper product discovery, and initial prototypes and putting a team around it and releasing in this just a little over a year. So I think that was incredibly fast and very successful. And it was fun to see engineering and product working together. Really exciting example. 

Again for the full transcript you can go here.

You also sign up for our next ‘Ship Happens’ webinar, “Failure is an Option: Scaling Resilient Feature Delivery,” that will take place on February 13th. 

Optimizely X