Here you will find three tabs that help override certain rules related to indexing –
- SSL certificate– In case if you have a expired SSL certificate, expertrec’s search crawler will ignore these pages, but you can override this and crawl these pages as well by turning this on.
- Subdomain Crawl– By default expertrec crawls only the pages under the domain URL you have mentioned. To crawl subdomains of your site, turn on Subdomain crawl.
- No Index settings–
- No Index –“noindex” suggests to search engines not to index a specific webpage. Certain pages have a no index attribute for certain links or pages. Example- <meta name=”robots” content=”noindex”>. These pages will not be indexed by expertrec’s crawler by default. You have to either change these noindex to index or you can ignore these no-index settings by turning on Ignore noindex option.
- No Follow-“nofollow” suggests to search engines not to follow through links on a webpage. This can be identified by the rel=”nofollow” .Expertrec’s crawler will not follow nofollow links. To turn this off, enable Ignore nofollow links.
- Ignore robots- If you would like to ignore the settings in your robots.txt you can select the domains you want to enable and update. robots can be found on the URL yourwebsite.com/robots.txt