Domain settings

Here you will find three tabs that help override certain rules related to indexing –

1. Settings-
   1. Subdomain Crawl– By default ExpertRec crawls only the pages under the domain URL you have mentioned. To crawl subdomains of your site, turn on Subdomain crawl.
2. No Index settings–
   1. No Index –“noindex” suggests to search engines not to index a specific webpage. Certain pages have a no index attribute for certain links or pages. Example- <meta name=”robots” content=”noindex”>. These pages will not be indexed by ExpertRec’s crawler by default. You have to either change this noindex to index or you can ignore these no-index settings by turning on Ignore noindex option.
   2. No Follow-“nofollow” suggests to search engines not to follow through links on a webpage. This can be identified by the rel=”nofollow” .ExpertRec’s crawler will not follow nofollow links. To turn this off, enable Ignore nofollow links.
3. Ignore robots- If you would like to ignore the settings in your robots.txt you can select the domains you want to enable and update. robots can be found on the URL yourwebsite.com/robots.txt