A/B Testing

You can’t know what’s best until you test, right? That’s what A/B testing does for you. Good A/B tests split your traffic evenly between two versions of the same page, ad, or email. Then, you can view the data to determine which copy, design, and offer convert best for your audience.

< Back to Glossary

How does Coveo use A/B testing?

A/B testing is the foundation of Coveo’s Machine Learning technology. When users land on your website or mobile app, they’re added to one of two groups – control or variation. Coveo observes their behaviour as they make their way through the customer journey, and reports back with data for each of the two groups. By analyzing the data, you can make educated decisions that will help you optimize your content.

What’s the difference between a control and a variation?

“Control” is the standard version of your website or app. All of the data you have for time-on-page, clicks, and conversions is based on the “control.” Think of it like the baseline for all your tests.

“Variation” is the version with modifications you want to test. That could be the copy, the design, or even the way you present your product. If you want to make sure you’re performing A/B tests that give you actionable results, check out this free report

After receiving enough traffic to both your control and your variation, you’ll be able to determine which performs better. If it was the control, then you can leave things the way they are and come up with some new testing ideas. If it was the variation, make that your new control!

How many variations should I test?

It depends! You could run a simple A/B test with one control and one variation. Or, you could run an A/B/n test with one control and multiple variations. There are advantages and disadvantages to both.

A standard A/B test is great for websites and apps that don’t get much traffic, because traffic will be less diluted. This kind of test also requires less time to implement because you only need to create one variation. However, if you’re only running one test at a time, your results will come in more slowly than with an A/B/n test.

With an A/B/n test, it’s important to consider that each new variation adds a cost to your testing. Your team will invest more resources to build the variations, observe the data, and analyze it afterward. But if you can afford that investment, you’ll get accurate results much more quickly because you’re limiting the influence of variables like seasonality, marketing efforts, and economic factors.

We recommend smaller businesses focus on standard A/B testing, while larger organizations run A/B/n tests with a maximum of 3 variations.