The search relevance metric could change depending on your business.
CTR- Click-through rate-
This is the most simplest and common metric to measure search relevance. If for a given search term, the results you are showing is relevant, the click-through rate will be high for the
CTR=number of clicks/ number of times the search result was shown.
If you open your Google search console report, you will see that CTR is one of the main metrics affecting google search results rankings.
Conversion rate-
This is another interpretation of your CTR. Here conversion can be (buys, clicks, and any goal that defines the success of your business).
For e-commerce,
conversion rate=number of buys/number of searches
Mean reciprocal rank-
Here every click is given a different weight based on its position.
Position 1 click -weight 1
Position 2 click -weight 1/2
Position 3 click -weight 1/3
Position 4 click -weight 1/4
And so on …
Here is an example-
For the query “iPhone” imagine we have 3 results
Here are click data for these 3 results
Apple iPhone 6s plus gold 32 gb – 100 clicks
Apple iPhone 6s plus gold 16 gb – 80 clicks
Apple iPhone 6s plus space grey 32 GB -60 clicks
So MRR for the query “iPhone” is
(100*1 + 80*(1/2) +60 *(1/3))/240=(100+40+20)/240=160/240=0.667
This MRR is saved as base MRR. the next time the search algorithm is changed, the MRR for every search query is calculated again. If the MRR is lower for most of the queries, then the search algorithm change is not justified.
Normalized Discounted Cumulative Gain
In this method, a group of people is used to rank search relevance who rank searches on a scale (for simplicity, we will take the scale as (0, 1, 2, 3)
For example, if the human rates the first four results as 2,0,3,2 respectively, we calculate the discounted gain as 2/1 (rating divided by rank), 0/2, 3/3, 2/4 respectively. Cumulative gain is measured as the sum of previous gains (2, 2+0, 2+0+1, 2+0+0.5). (kind of a running total of the value calculated till now)
Here is a table that showcases these calculations for clarity.
Ideal gains are the best possible ratings for a given search query and are acquired based on our previous experience. We use them to find the normalized discounted gains. (cumulative gain/ideal discounted cumulative gain).
After calculating NDCG values across all queries, we can now compare the previous search algorithms mean average NDCG values and current search algorithms’ mean average NDCG values.
