A/B Testing

Overview

The A/B Testing feature lets you run controlled experiments on your search configuration. You can test different ranking strategies, boost rules, or UI layouts, measure the impact on click-through rate (CTR) and conversion, and confidently roll out the winning configuration. Navigate to Merchandising > A/B Testing in your ExpertRec dashboard (route: /ecom/merchandise/ab-testing).

How Traffic Splitting Works

When an A/B test is active, ExpertRec splits incoming search traffic between two variants:

• Control (Variant A): The current live configuration your customers already see.
• Test (Variant B): A new configuration you want to evaluate.

You define the traffic split percentage (for example, 50/50 or 80/20 control/test). Each visitor is consistently bucketed into one variant for the duration of their session so they see a coherent experience.

Setting Up a Test

1. Go to Merchandising > A/B Testing and click New Test.
2. Name the test and set the traffic split.
3. Configure Variant B: adjust ranking weights, boost fields, synonym rules, or any other search settings you want to test.
4. Set a start date and an optional end date, then click Activate.
5. Monitor results in the test detail view: compare CTR, add-to-cart rate, and revenue per search for each variant.

Interpreting Results

• Wait for statistical significance before drawing conclusions. As a rough guide, aim for at least 200 to 500 searches per variant before comparing results.
• Focus on downstream metrics (conversion and revenue) rather than just CTR, as higher CTR does not always mean higher sales.
• If Variant B wins, click Apply Winner to promote the test configuration to production and end the test.

Common Questions

Can I run multiple tests at the same time?

It depends on your plan. Running overlapping tests can confound results, so it is generally recommended to run one test at a time or to segment tests to non-overlapping query sets.

What happens to test data after I end a test?

Test results remain visible in the A/B Testing history view after the test ends so you can review the outcome and use it for future decisions.