A/B Testing - How good are your product recommendations?

After activating a product recommendation engine, the last step is using an A/B test to measure its efficiency. While this step is not mandatory, it is highly recommended.

Expected duration: 30 minutes ⏳
Technical complexity: medium

Why an A/B test?

One option is to compare the results of your shop (in terms of conversion rate and AOV) with the results in the previous period or the same period of the past year. However, this will be subject to biases, as it assumes that the externalities remain the same, which almost never happens (think of the COVID pandemic as an example). Moreover, this would only give you the big-picture metrics without giving you any indication of what part of your recommendations work better (e.g. similar items, cross-sell items - see how to set them up here).

An A/B test presents two (or more) versions of your website to random visitors during a set period of time and can be configured to evaluate more granular metrics (e.g., the number of clicks on the recommendations widget). For this reason, we recommend an A/B test to find out if your website’s number and value of transactions increase, and what exactly drives those results. The simplest method, which does not require advanced programming and can be implemented by digital specialists and marketing professionals, is using the Google Optimize platform.

Setting up your website for an A/B test

All necessary tools are free and (fairly) easy to install

The three needed tools are Google Analytics, Google Tag Manager, and Google Optimize. They are all free and have great support documentation.

Here are some valuable resources that take you through the process for each tool installation:

• Installing Google Analytics
• Installing Google Tag Manager
• Installing Google Optimize

Add the product recommendation widgets

When installing the recommendation widgets, make sure you are NOT REPLACING anything in your previous website, and only ADDING the new widgets to what you already had. This might mean that there are now two recommendation widgets where there was only one before, but this will all make sense in a minute.

Google Optimize works by HIDING certain parts of your webpage to a fraction of the visitors. When setting up the experiment, you will hide the new recommendations for group A (we will call them the control group), and hide the old recommendations for group B (we will call them the test group).

Tracking the relevant events

To get an accurate perspective, we recommend setting up at least 3 events along the customer journey, as identified below. If you are running an eCommerce shop, you are most probably already tracking two of them (Visit and Transaction).

#old-recommendations-substitutes {
  display : none;
}

#old-recommendations-cross-sell {
  display : none;
}

#new-recommendations-substitutes {
  display : block !important;
}

#new-recommendations-cross-sell {
  display : block !important;
}

How long should the test run?

In principle, you should wait at least two weeks before looking for meaningful results in your Optimize Reports. Your aim should be a probability to be the best of more than 95%. More on outcomes of Optimize experiments here.

Evaluating the A/B test

Once the experiment is running or completed, you can check its results with Optimize Reports (more on Optimize Reports). You will be able to see the leading Variant as presented by Optimize, but let’s consider some sample results:

It is clear that Variant B performs better, but let’s dive deeper into the numbers:

This hopefully helped you understand and quantify the real business impact of your new product recommendation engine. If this helped you or think of ways to improve this article, give us a shout-out on LinkedIn or Facebook.

Advanced tests

If you are a more experienced GTM, GA and Optimize user, you can add more granularity in the objectives you are tracking by defining more events. A more complete overview of the events would be: