A/B Testing - How good are your product recommendations?

After activating a product recommendation engine, the last step is using an A/B test to measure its efficiency. While this step is not mandatory, it is highly recommended.

⏳ Duration: 30 minutes 
🎚️ Complexity: medium

Why an A/B test?

One option is to compare the results of your shop (in terms of conversion rate and AOV) with the results in the previous period or the same period of the past year. However, this will be subject to biases, as it assumes that the externalities remain the same, which almost never happens (think of the COVID pandemic as an example). Moreover, this would only give you the big-picture metrics without giving you any indication of what part of your recommendations work better (e.g. similar items, cross-sell items - see how to set them up here).

An A/B test presents two (or more) versions of your website to random visitors during a set period of time and can be configured to evaluate more granular metrics (e.g., the number of clicks on the recommendations widget). For this reason, we recommend an A/B test to find out if your website’s number and value of transactions increase, and what exactly drives those results. The simplest method, which does not require advanced programming and can be implemented by digital specialists and marketing professionals, is using the Google Optimize platform.

Setting up your website for an A/B test

All necessary tools are free and (fairly) easy to install

The three needed tools are Google Analytics, Google Tag Manager, and Google Optimize. They are all free and have great support documentation.

💡 You will need admin access to your website, as snippets of code need to be added for all three tools!

Here are some valuable resources that take you through the process for each tool installation:

Add the product recommendation widgets

When installing the recommendation widgets, make sure you are NOT REPLACING anything in your previous website, and only ADDING the new widgets to what you already had. This might mean that there are now two recommendation widgets where there was only one before, but this will all make sense in a minute.

Google Optimize works by HIDING certain parts of your webpage to a fraction of the visitors. When setting up the experiment, you will hide the new recommendations for group A (we will call them the control group), and hide the old recommendations for group B (we will call them the test group).

💡 Please note that once the A/B test is completed, you should only keep one recommendations widget – the best-performing one.

Tracking the relevant events

To get an accurate perspective, we recommend setting up at least 3 events along the customer journey, as identified below. If you are running an eCommerce shop, you are most probably already tracking two of them (Visit and Transaction).

 Event  User journey (step)  Setting up the event
Visit 1. User visits your page Natively supported by Google Analytics
Click on recommendations 2. User interacts with your page and recommendations Needs setting up in Google Tag Manager (https://support.google.com/tagmanager/answer/6106716?hl=en, https://support.google.com/tagmanager/answer/9442095)
Transaction 3. User completes the purchase Needs setting up in Google Analytics (https://developers.google.com/analytics/devguides/collection/ua/gtm/enhanced-ecommerce, https://developers.google.com/analytics/devguides/collection/ga4/ecommerce?client_type=gtag)

Starting the A/B test

The hard part is over once you have set everything up. Now, you just have to configure the A/B test. Here is an overview of how these experiments generally work. Now, let’s go into exactly how you need to configure it.

1. Setting up page variants

For this A/B test, you will need to create two Variants, select custom weights and give each one 50%, and then click on “Edit” for the new version of the website (where you are placing the new product recommendations widget). Now, select the “<>” sign at the top right of the page, and click on “Edit CSS”. Next, add the code to hide the old widgets, and display the new ones. Here is an example:

#old-recommendations-substitutes {
  display : none;

#old-recommendations-cross-sell {
  display : none;

#new-recommendations-substitutes {
  display : block !important;

#new-recommendations-cross-sell {
  display : block !important;

2. Setting up objectives

As a primary objective, you should be able to select Revenue, if Google Analytics is correctly set up for eCommerce transactions.

As a secondary objective, you should add the “Click on Recommendations” event (or whatever other fancy name you gave it in GTM). If this has been set up as a Goal in Google Analytics (about goals), you should be able to select it directly. Otherwise, you can create a custom event to match the parameters.

▶️ That’s it – hit “Start”!

How long should the test run?

In principle, you should wait at least two weeks before looking for meaningful results in your Optimize Reports. Your aim should be a probability to be the best of more than 95%. More on outcomes of Optimize experiments here.

Evaluating the A/B test

Once the experiment is running or completed, you can check its results with Optimize Reports (more on Optimize Reports). You will be able to see the leading Variant as presented by Optimize, but let’s consider some sample results:

   Variant A (control)  Variant B (test)
Visits 10,000 10,000
Click on recommendations 1,500 2,500
Revenue EUR 2,000 EUR 3,000

It is clear that Variant B performs better, but let’s dive deeper into the numbers:

   Variant A (control)  Variant B (test)
Visits influenced by recommendations 1,500/10,000 = 15% 2,500/10,000 = 25%
Revenue improvement (3,000 / 2,000) - 1= +50%  

This hopefully helped you understand and quantify the real business impact of your new product recommendation engine. If this helped you or think of ways to improve this article, give us a shout-out on LinkedIn or Facebook.


Advanced tests

If you are a more experienced GTM, GA and Optimize user, you can add more granularity in the objectives you are tracking by defining more events. A more complete overview of the events would be:

 User journey (step)  Event  Setting up the event
1. User visits your page Visit Natively supported by Google Analytics
2. User interacts with your page and recommendations Click on similar item recommendations on Product Detail Pages Needs set up in Google Tag Manager
2. User interacts with your page and recommendations Click on cross-sell item recommendations on Product Detail Pages Needs set up in Google Tag Manager
2. User interacts with your page and recommendations Click on cross-sell item recommendations on Shopping Cart Page Needs set up in Google Tag Manager
3. User completes the purchase Transaction Needs set up in Google Analytics Ecommerce
3. User completes the purchase Average Order Value Needs set up in Google Analytics Ecommerce


Questions about getting started? We're excited to help: support@aqurate.ai