Crawler interface

Use the Algolia Crawler interface to manage crawl settings and monitor indexing. These settings are separate from your installed DocSearch frontend package version.

DocSearch tab

The Crawler UI now includes a dedicated DocSearch tab. This tab provides everything you need to implement DocSearch on your site, including:

• Implementation code: Starter snippets for integrating DocSearch into your frontend. For v5, compare the snippet with the current package guide.
• API keys: Your unique Application ID and Search API Key for connecting to your Algolia index.
• Documentation links: Access to review your records, explore documentation, and join the support Discord.

Trigger a new crawl

Head over to the Overview section to start, restart or pause your crawls and view a real-time summary.

Monitor your crawls

The Monitoring section helps you find crawl errors or improve your search results.

Update your config

You can update your crawler configuration in two ways:

Visual configuration: Edit common options without writing code in the visual configuration interface.

Code editor: For advanced configuration, use the live code editor to modify your configuration and test URLs with the URL tester.

URL tester

From the editor, you can use the URL tester to debug selectors or see how the crawler interprets your site.

Suggestions

The Suggestions section in the Crawler UI provides actionable feedback to help you improve your crawl and data extraction. After each crawl, you'll see recommendations for:

• Fixing redirect or domain issues
• Addressing ignored or failed URLs
• Adding missing sitemaps

Each suggestion includes a description, a solution, and links to relevant documentation or monitoring tools.