Expertrec crawler allows some of the advanced options while crawling. These features mainly include:
- Protected pages
- Manual Extraction
- Automatic Extraction
- Crawl Speed
- File Size Limit
- Domain Settings
Protected Pages:
You may need to search web pages that are protected such as user needs to login to see the content. Expertrec crawler can be guided with credentials to index these pages.
As shown above, login_url will be the page url where user needs to put the login credentials. login_form_id is the html document id for this signin form( id given to the form).
