Help

Overview
Overview What to Expect Pre-Session Checklist Your Onboarding Session
Platform Mental Model Admin & Operator Essentials Guardrails and Governance What to Use When Roles and Responsibilities Post-Session Next Steps Optional Deep-Dives Deep-Dive: Agents and Prompts Deep-Dive: Search and Knowledge Deep-Dive: Functions and Secure Services Deep-Dive: Handoff and Escalation Deep-Dive: Voice Mode Deep-Dive: Analytics and Improvement Facilitator Guide
Overview Account Overview User Management Usage
Overview General User Interface Agent Messages Voice Mode
Admin Chat UI Design Curated Answers Search Settings
Overview Concepts Categories Index Fields Facets Result Cards Layout Website Integration Styling Testing Preview Funnelback Migration Troubleshooting
Conversational Intelligence Data Sync Upload Documents
Overview Creating an Agent Editing An Agent
Overview Editing a Prompt User Segmented Prompts
Overview Writing Your First Function AirScript Functions AirScript Reference
Overview Privacy Security
Analytics API Documents Ingestion API
Human Handoff
Overview Customer Service Insights Categories And Themes Search Insights
Overview Identity Provider Widget SharePoint Integration How It Works SharePoint Checklist Implementation Checklist
Admin Console Authorisation
Methods Script Tag AI Question Input Fields
Overview Optimiser Monitor
Contact Support

Crawler Settings Overview

Airgentic Help

The Web Crawl Configuration screen controls how your website is crawled and indexed for use by the AI.

UI Design screen

The screen is organised into four tabs. This section of the help documentation mirrors those tabs:

  • General — Enable/disable the crawler, politeness delay, user agent, and HTTP→HTTPS conversion
  • Crawl Scope — Seed URLs, maximum pages, include/exclude patterns, URL parameters, and robots.txt
  • Image Extraction — AI auto-detection, image URL patterns, and XPaths for page thumbnails
  • Standard Result Fields — How to extract and transform standard metadata (title, description, image, date, etc.)

After you save changes, the configuration is written to your ingest config. Depending on what you changed, you may be prompted to trigger an index or HTML-processing job so that updates take effect.

You have unsaved changes