The search relevance metric could change depending on your business.

CTR- Click through rate-

This is the most simplest and common metric to measure search relevance. If for a given search term, the results you are showing is relevant, the click through rate will be high for the

CTR=number of clicks/ number of times the search result was shown.

If you open your google search console report, you will see that CTR is one of the main metrics affecting google search results rankings.

Conversion rate-

This is another interpretation of your CTR. Here conversion can be (buys, clicks, and any goal that defines the success of your business).

For an e-commerce,

conversion rate=number of buys/number of searches

 

Mean reciprocal rank-

Here every click is given a different based on its position.

Position 1 click -weight 1

Position 2 click -weight 1/2

Position 3 click -weight 1/3

Position 4 click -weight 1/4

And so on …

Here is an example-

For the query “iphone” imagine we have 3 results

Here are click data for these 3 results

Apple iphone 6s plus gold 32 gb – 100 clicks

Apple iphone 6s plus gold 16 gb – 80 clicks

Apple iphone 6s plus space grey 32 GB -60 clicks

So MRR for the query “iphone” is

(100*1 + 80*(1/2) +60 *(1/3))/240=(100+40+20)/240=160/240=0.667

This MRR is saved as base MRR. the next time the search algorithm is changed, the MRR for every search query is calculated again. If the MRR is lower for most of the queries, then the search algorithm change is not justified.

Normalized Discounted Cumulative Gain

In this method a group of people are used to rank search relevance who rank searches on a scale (for simplicity,we will take the scale as (0, 1, 2, 3)

For example, if the human rates the first four results as 2,0,3,2  respectively, we calculate the discounted gain as 2/1 (rating divided by rank), 0/2, 3/3, 2/4 respectively. Cumulative gain is measured as the sum of previous gains (2, 2+0, 2+0+1, 2+0+0.5). (kind of a running total of value calculated till now)

Here is a table that showcases these calculations for clarity.

Ideal gains are the best possible ratings for a given search query and are acquired based on our previous experience. We use them to find the normalized discounted gains. (cumulative gain / ideal discounted cumulative gain).

After calculating NDCG values across all queries, we can now compare the previous search algorithms mean average NDCG values and current search algorithms’ mean average NDCG values.

 


muthali ganesh

Muthali loves writing about emerging technologies and easy solutions for complex tech issues. You can reach out to him through chat or by raising a support ticket on the left hand side of the page.