> ## Documentation Index
> Fetch the complete documentation index at: https://docs.naturalead.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl Sitemap

> Starts a crawl job that fetches pages from a sitemap (or a provided URL
list) and ingests their content into the knowledge base. Each page is
optionally distilled using LLM before storage.
Requires `knowledge:upload` permission.




## OpenAPI

````yaml POST /api/knowledge-bases/{id}/crawl-sitemap
openapi: 3.1.0
info:
  title: Naturalead Knowledge Bases API
  version: 1.0.0
  description: >
    API for managing knowledge bases, documents, sitemap crawl jobs, and webhook

    sources in the Naturalead platform. All endpoints require API key
    authentication

    and are scoped to the caller's account.
servers:
  - url: '{baseUrl}'
    description: API server
    variables:
      baseUrl:
        default: http://localhost:3001
security:
  - BearerAuth: []
tags:
  - name: Knowledge Bases
    description: Knowledge base CRUD operations
  - name: Documents
    description: Document management within a knowledge base
  - name: Sitemap & Crawl Jobs
    description: Sitemap scanning and web crawl job management
  - name: Webhook Sources
    description: Webhook source management for auto-updating content
paths:
  /api/knowledge-bases/{id}/crawl-sitemap:
    post:
      tags:
        - Sitemap & Crawl Jobs
      summary: Crawl sitemap
      description: |
        Starts a crawl job that fetches pages from a sitemap (or a provided URL
        list) and ingests their content into the knowledge base. Each page is
        optionally distilled using LLM before storage.
        Requires `knowledge:upload` permission.
      operationId: crawlSitemap
      parameters:
        - $ref: '#/components/parameters/KnowledgeBaseId'
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                sitemapUrl:
                  type: string
                  description: Sitemap URL to crawl
                maxPages:
                  type: integer
                  description: Maximum number of pages to crawl
                  maximum: 5000
                  default: 100
                useLlmDistill:
                  type: boolean
                  description: Whether to use LLM to distill page content
                  default: true
                urls:
                  type: array
                  items:
                    type: string
                  description: Explicit list of URLs to crawl instead of using a sitemap
      responses:
        '201':
          description: Crawl job created
          content:
            application/json:
              schema:
                type: object
                properties:
                  jobId:
                    type: string
                    description: ID of the created crawl job
                  status:
                    type: string
                    description: Initial job status
                  urlsFound:
                    type: integer
                    description: Number of URLs discovered from sitemap
                  urlListLength:
                    type: integer
                    description: Number of URLs in the explicit URL list
                required:
                  - jobId
                  - status
        '400':
          $ref: '#/components/responses/BadRequest'
        '404':
          $ref: '#/components/responses/NotFound'
        '500':
          $ref: '#/components/responses/InternalError'
components:
  parameters:
    KnowledgeBaseId:
      name: id
      in: path
      required: true
      description: Knowledge base ID
      schema:
        type: string
  responses:
    BadRequest:
      description: Bad request
      content:
        application/json:
          schema:
            type: object
            properties:
              error:
                type: string
            required:
              - error
    NotFound:
      description: Resource not found
      content:
        application/json:
          schema:
            type: object
            properties:
              error:
                type: string
            required:
              - error
    InternalError:
      description: Internal server error
      content:
        application/json:
          schema:
            type: object
            properties:
              error:
                type: string
            required:
              - error
  securitySchemes:
    BearerAuth:
      type: http
      scheme: bearer
      description: Bearer token using an API key (format nl_live_* or nl_test_*)

````