Sitemaps– You can use this option to add your website’s sitemap to expertrec’s control panel so that the search engine can find more URLs in an efficient manner. Your sitemap can be usually found at yourwebsite.com/sitemap.xml.
If you have more than one sitemap, you can add them using the add more sitemaps button. Once you are done, click on update.
Below this, you will find three options-
- Apply filter on sitemap– The URL filters that you had applied under the filter tab will also be applied to the sitemap resulting in reduced number of URLs being crawled as per the URL filter logic applied by you.
- Crawl only sitemap-This will result in only the URLs in the sitemap being crawled.
- Handle URL redirect-By default our crawlers are in an intelligent state which determines how to handle a redirect set in your website. If you want to enable, this our search crawler goes to the redirected page and crawls its contents. If you want to disable this – enabled the button and disable it. This will forecfully disable the expertrec’s crawler to go and visit the redirected page and crawl its contents.