[Solved] Crawl error with Cloudflare website, Cloudflare is blocking crawler bot

This guide will walk you through the process of allowing or whitelisting crawler’s IP address in Cloudflare which needs to be done when Cloudflare connected websites produces crawling related issues in Expertrec.

Step 1: Log in to your Cloudflare account, select the website that you are working on, and navigate to the ‘Firewall’ Menu on the Cloudflare dashboard.

 

Step 2: Click on ‘Tools’ Tab.

 

Step 3: Get ExpertRec Bot’s IP Address by following this link (login to your Expertrec account if session expired) and toggle ‘Enable IP Whitelisting’. This will display a list of IPs you could whitelist to allow our crawler to have uninterrupted access to your Cloudflare connected website. Keep this option turned to be notified of any changes/additions to the IPs list.

 

Step 4: Enter ExpertRec Bot’s IP Address in the ‘Enter an IP’ field, select ‘Allow’, and click on ‘Add’.

 

Note: You may select ‘All Websites’ if you have multiple websites using Exertrec search.

 

Some more Important things to look at:

• properly setting up rules in robots.txt
• Bot properly following robots.txt rules
• low crawl rate
• Try toggling ‘Bot Fight Mode’. Cloudflare offers ‘Bot Fight mode’ under the ‘Bot’ tab of the firewall menu which will be turned off by default & available in their free plan. More info (https://www.cloudflare.com/products/bot-management/)

 

Useful information about Allowlists by Cloudflare.