In this article we will see how to make expertrec search crawl your website faster.
A prerequisite is that you have created a custom search engine using expertrec from https://cse.expertrec.com/?platform=cse
How to increase crawler speed in expertrec control panel-
- Go to https://cse.expertrec.com
- Go to crawl-> Advanced -> Crawl speed -> Crawl speed
- Under crawl speed you have 4 crawl speed options by moving the slider-
- Slow crawl
- Polite crawl
- Fast crawl
- Aggressive crawl.
- By default the slow crawl option is set, you can choose any one of the above options depending upon your crawl speed requirements.
- You can reduce the page load delay time to a smaller value for faster crawling.