Recrawl frequency

Overview

The Recrawl Frequency setting controls how often ExpertRec’s crawler revisits your website to pick up new, updated, or removed pages. Keeping your search index fresh ensures customers always find current content.

How to Configure

1. Navigate to Crawl > ReCrawl Frequency in your dashboard.
2. Select a recrawl schedule from the available options.
3. Click Update to save.

Available Schedules

• Daily: Best for sites with frequent content updates (news sites, active blogs, stores with daily inventory changes).
• Weekly: Recommended for most websites. Balances freshness with crawl resource usage.
• Monthly: Suitable for sites with infrequent content changes.
• Yearly: For static sites that rarely change.

Best Practices

• For ecommerce stores, use Daily or Weekly to keep product availability and pricing current.
• Schedule full recrawls during off-peak hours (e.g., 2 to 4 AM) to minimize impact on your server.
• If you use a product feed (CSV/XML/JSON) or webhooks, the index updates automatically and recrawl frequency matters less.
• After making major site changes (new sections, URL structure changes), trigger a manual recrawl from the dashboard.