Here you will find three tabs that help override certain rules related to indexing –
- Subdomain Crawl– By default ExpertRec crawls only the pages under the domain URL you have mentioned. To crawl subdomains of your site, turn on Subdomain crawl.
- No Index settings–
- No Index –“noindex” suggests to search engines not to index a specific webpage. Certain pages have a no index attribute for certain links or pages. Example- <meta name=”robots” content=”noindex”>. These pages will not be indexed by ExpertRec’s crawler by default. You have to either change this noindex to index or you can ignore these no-index settings by turning on Ignore noindex option.
- No Follow-“nofollow” suggests to search engines not to follow through links on a webpage. This can be identified by the rel=”nofollow” .ExpertRec’s crawler will not follow nofollow links. To turn this off, enable Ignore nofollow links.
- Ignore robots- If you would like to ignore the settings in your robots.txt you can select the domains you want to enable and update. robots can be found on the URL yourwebsite.com/robots.txt