Protected Pages

You are here:
< Back

You can use this feature to crawl and index pages behind a login page. For this you have to create a dummy username and password for expertrec’s crawlers to access these pages.

Here is an example which tells how to accomplish this:

Steps-

  1. Login URL.
  2. Find and enter the login box/form id.
  3. Username for login box.
  4. Id for password box.
  5. Sample password for password box.

Here, we want to index the pages behind the login page , https://www.thornburg.com/registration/logon.aspxcrawling behind login pages

Here login URL is the page URL. In this case it is https://www.thornburg.com/registration/logon.aspx

Login formid– To find the login form ID, open developer tools in your chrome browser (control+shift+i) and select the form element in the developer mode. In this case it is p_logincrawling behind login pages

Enter the id/name for username box- Select the id/name of the input box. Here it is p_txtUsername

Enter the id/name for password box- In the inspect element mode, click on the password box for the password box idcrawling behind login pages

Enter a username and password for the crawler so that the crawler can pass through and press update.

Now go to the home page and press recrawl. Thats it. Now you can go to your demo page and check if your search results include the behind login pages or not.crawling behind login pages