Sitemap

Sitemaps– You can use this option to add your website’s sitemap to ExpertRec’s control panel so that the search engine can find URLs in a more efficient manner. Your sitemap can usually be found at yourwebsite.com/sitemap.xml.

If you have more than one sitemap, you can add them using the add more sitemaps button. Once you are done, click on update.

Below this, you will find three options-

1. Apply filters on sitemap– The URL filters that you had applied under the filter tab will also be applied to the sitemap resulting in a reduced number of URLs being crawled as per the URL filter logic applied by you.
2. Crawl only sitemap-This will result in only the URLs in the sitemap being crawled.
3. Handle URL redirect-By default our crawlers are in an intelligent state which determines how to handle a redirect set in your website. If you want to enable this, our search crawler goes to the redirected page and crawls its contents.