how to measure search relevancy

How to measure search relevancy

Rate this article

Share this article

The search relevance metric could change depending on your business.

CTR- Click-through rate-

This is the most simplest and common metric to measure search relevance. If for a given search term, the results you are showing is relevant, the click-through rate will be high for the

CTR=number of clicks/ number of times the search result was shown.

If you open your Google search console report, you will see that CTR is one of the main metrics affecting google search results rankings.

search relevancy /analytics

Conversion rate-

This is another interpretation of your CTR. Here conversion can be (buys, clicks, and any goal that defines the success of your business).

For e-commerce,

conversion rate=number of buys/number of searches


Mean reciprocal rank-

Here every click is given a different weight based on its position.

Position 1 click -weight 1

Position 2 click -weight 1/2

Position 3 click -weight 1/3

Position 4 click -weight 1/4

And so on …

Here is an example-

For the query “iPhone” imagine we have 3 results

measure search relevancy

Here are click data for these 3 results

Apple iPhone 6s plus gold 32 gb – 100 clicks

Apple iPhone 6s plus gold 16 gb – 80 clicks

Apple iPhone 6s plus space grey 32 GB -60 clicks

So MRR for the query “iPhone” is

(100*1 + 80*(1/2) +60 *(1/3))/240=(100+40+20)/240=160/240=0.667

This MRR is saved as base MRR. the next time the search algorithm is changed, the MRR for every search query is calculated again. If the MRR is lower for most of the queries, then the search algorithm change is not justified.

Normalized Discounted Cumulative Gain

In this method, a group of people is used to rank search relevance who rank searches on a scale (for simplicity, we will take the scale as (0, 1, 2, 3)

For example, if the human rates the first four results as 2,0,3,2  respectively, we calculate the discounted gain as 2/1 (rating divided by rank), 0/2, 3/3, 2/4 respectively. Cumulative gain is measured as the sum of previous gains (2, 2+0, 2+0+1, 2+0+0.5). (kind of a running total of the value calculated till now)

Here is a table that showcases these calculations for clarity.

how to measure search relevancy

Ideal gains are the best possible ratings for a given search query and are acquired based on our previous experience. We use them to find the normalized discounted gains. (cumulative gain/ideal discounted cumulative gain).

After calculating NDCG values across all queries, we can now compare the previous search algorithms mean average NDCG values and current search algorithms’ mean average NDCG values.

Add Search to your Website


Are you showing the right products, to the right shoppers, at the right time? Contact us to know more.
You may also like