- List of all supported fields
- [Solved] Crawl error with Cloudflare website, Cloudflare is blocking crawler bot
- Expertrec Crawler - Control your Index
- Take Screenshot
- Manual extraction
- Automatic extraction
- Boost Fields
- Filter fields
- Create search for e-commerce store
- Use Existing Search Box
- Wordpress plugin
You are here:
Last Updated On
Sitemaps– You can use this option to add your website’s sitemap to ExpertRec’s control panel so that the search engine can find URLs in a more efficient manner. Your sitemap can usually be found at yourwebsite.com/sitemap.xml.
If you have more than one sitemap, you can add them using the add more sitemaps button. Once you are done, click on update.
Below this, you will find three options-
- Apply filters on sitemap– The URL filters that you had applied under the filter tab will also be applied to the sitemap resulting in a reduced number of URLs being crawled as per the URL filter logic applied by you.
- Crawl only sitemap-This will result in only the URLs in the sitemap being crawled.
- Handle URL redirect-By default our crawlers are in an intelligent state which determines how to handle a redirect set in your website. If you want to enable this, our search crawler goes to the redirected page and crawls its contents.
Table of Contents