Domain settings

You are here:
< Back

Here you will find three tabs that help override certain rules related to indexing –

  1. Settings-
    1. SSL certificate– In case if you have a expired SSL certificate, expertrec’s search crawler will ignore these pages, but you can override this and crawl these pages as well by turning this on.
    2. Subdomain Crawl– By default expertrec crawls only the pages under the domain URL you have mentioned. To crawl subdomains of your site, turn on Subdomain crawl.
  2. No Index settings
    1. No Index“noindex” suggests to search engines not to index a specific webpage. Certain pages have a no index attribute for certain links or pages. Example- <meta name=”robots” content=”noindex”>. These pages will not be indexed by expertrec’s crawler by default. You have to either change these noindex to index or you can ignore these no-index settings by turning on Ignore noindex option.
    2. No Follow-“nofollow” suggests to search engines not to follow through links on a webpage. This can be identified by the rel=”nofollow” .Expertrec’s crawler will not follow nofollow links. To turn this off, enable Ignore nofollow links.
  3. Ignore robots- If you would like to ignore the settings in your robots.txt you can select the domains you want to enable and update. robots can be found on the URL yourwebsite.com/robots.txt