Build, Buy, or Hire? How to Pick the Testing Solution That’s Right for You

You have options when it comes to the path you take to testing. You can build your own testing tool in-house, buy a testing platform, or hire a consultant or agency to do the testing for you. There’s no “wrong” choice; each has its pros and cons. There are a couple things to consider when making your decision.

Option One: Build

Building a testing solution in-house makes sense for organizations that have significant engineering resources available. Most companies don’t decide to build a testing tool from scratch without an engineering team that’s closely tied to the process. Why? It requires substantial engineering effort and quality assurance. Typically, homegrown testing tools are additions to already established data-gathering and analytics machines.

Amazon has an extensive testing system native to their site. At their size and scale an in-house service makes sense for all the testing and customization the site does.

Besides size, you might choose to build a homegrown solution instead of buying one or hiring out an agency because of trust and team structure. If you don’t trust a third party platform to provide accurate test results, then a proprietary solution may be best. Also, many of the companies who run testing in house have the engineering and product teams execute all tests, while marketing offers test ideas.

If you do build, run A/A tests – two identical versions of your page – to ensure there are no major differences between them. If there are, then something fishy is happening with tool.

Option Two: Buy

Buying or subscribing to a testing Software-as-a-Service (“SaaS”) platform is your second option. This option makes sense for a range of businesses  – individuals, small companies, and enterprises. Here are some advantages of going with a SaaS platform:

• Built-in features – An obvious advantage of this buying a testing solution is that advanced testing features are included in your purchase. You are buying something teams have spent time building, optimizing and innovating on. Fancy capabilities like targeting, automatic traffic allocation, and multivariate testing come with the product. Each subscription tier will offer different features.
• Community & Support – As a customer of an A/B testing platform, you are part of the user community who is available to answer questions, give technical support and suggest best practices. You might get training videos, webinars, invites to user meetups, and other customer perks.
• Trustworthy reporting –  If you want to test the tool’s accuracy yourself, you can of course run an A/A test.

When vetting different SaaS companies, consider these things:

• Integrations –  For most businesses, an A/B testing tool will complement the other tools you’re already using – especially analytics. The more your efficiency, data collection and lead generation/nurture tools communicate to each other, the more the ROI you get out of each service.
• Budget – Paying for testing should be a revenue positive investment; by spending money on testing, you should make more money. Most testing solutions’ payment structure is based on number of visitors tested and features included. Generally, the fewer visitors you have (or want to test), the less you’ll pay. Think about the scale at which your testing will be operating.
• People – The brand’s personality, dedication to customer success and availability are critical elements to consider. Support on the platform comes from the platform provider so ensuring the help is there if you need it is important. This may indeed be the tipping point that helps make your decision, since what differentiates one testing tool from the next is, in large part, the people.

Option Three: Hire

The final option is to hire an agency or an optimization consultant to do testing for you. Since many digital marketing agencies are quickly adding A/B testing to the list of services they offer, you can be selective when hiring.

There are a few reasons you might outsource testing:

• Lack of internal resources –  you might choose to partner with an agency to take care of all strategy and test implementation for you.
• Lack of test ideas – you can partner with an agency just to come up for test ideas for you.
• Lack of technical knowledge – you might have the creative juices running but lack the technical know-how to execute them.

If you outsource any part of testing – either the creative or the actual execution – here are a few things to look for before you sign a contract:

• A good track record – Whoever you’re paying should have proven optimization chops.
• Technically savvy with the platform – The agency should be expert with whichever platform it’s using to test your site.
• Pay structure – Some agencies operate on a per-hour or per-experiment basis and some offer “unlimited” or “constant” programs for a fixed price.

It comes down to a trade off between investing time and training in building a team internally, or investing money in an agency. If you’re not ready to build an internal testing team, then your best option is to hire an outside service to handle your testing.

The Choice Is Yours (And Your Team’s)

The choice between buying, building or hiring a testing solution depends on what makes the most sense for your situation – based on a variety of factors that only you and your coworkers can assess. The most important thing is that you enable your team to start making data-driven decisions about the website instead of operating off intuition and assumption alone.

Secure buy-in for testing by inviting your coworkers to the decision-making process early. In order for A/B testing to stick, everyone needs to trust the data. If they don’t, the discussion will center around the validity of the data, rather than what the data means. If everyone feels good about the tool you’re using, then actually testing will be much easier.

The good news is that no matter which route you choose, it will lead to a better website and a better user experience.

What do you think about buying versus building versus hiring a testing solution? Do you have advice or a question for fellow decision makers out there? Let us know below!