AI Scraper Response

Learn about the response structure of the AI Scraper API and how to interpret the results.

The AI Scraper API returns responses with a consistent wrapper structure across all scraping operations. While the wrapper remains the same, the data.data field contains agent-specific data optimized for each scraping type.

Standard Response Wrapper

Every API response follows this structure:

{
  "message": "Successful operation!",
  "data": {
    "id": "result-unique-identifier",
    "createdAt": "2026-03-06T05:30:44.642Z",
    "userId": "user-id",
    "scraperId": "scraper-id",
    "type": "AI",
    "url": "https://example.com",
    "status": "Finished",
    "error": "",
    "tokenUsage": 5,
    "runtime": 0,
    "data": {
      // Agent-specific data structure (see below)
    },
    "htmlPath": "...",
    "htmlContent": "<html>...</html>",
    "recordingPath": null,
    "screenshotPath": "...",
    "dataPath": null
  }
}

Response Fields

Field	Type	Description
`message`	string	Operation status message
`data.id`	string	Unique identifier for this result
`data.createdAt`	string	ISO 8601 timestamp when the operation started
`data.userId`	string	Your user identifier
`data.scraperId`	string	Identifier of the scraper used
`data.type`	string	Agent type used for scraping
`data.url`	string	Target URL that was scraped
`data.status`	string	Operation status (`Finished`, `Processing`, `Failed`)
`data.error`	string	Error message if the operation failed
`data.tokenUsage`	number	Number of tokens consumed by this operation
`data.runtime`	number	Total runtime of the scraping operation in seconds
`data.data`	object	Extracted data (structure varies by agent type)
`data.htmlPath`	string	Path to the saved HTML content
`data.htmlContent`	string	The actual HTML content of the scraped page
`data.recordingPath`	string	Path to the scraping session recording (if available)
`data.screenshotPath`	string	Path to the page screenshot
`data.dataPath`	string	Path to the raw data file (if available)

Agent-Specific Response Formats

The data.data field structure varies depending on which agent you use. Each agent optimizes the data format for its specific use case.

General Agent

The General Agent provides two response formats depending on the page type you're scraping.

General Table Response Format

General agent returns a dictionary containing an array of structured items when scraping listing pages with multiple items (e.g., product listings, search results, directory pages)

{
  "books": [
    {
      "price": "£51.77",
      "title": "A Light in the Attic"
    },
    {
      "price": "£53.74",
      "title": "Tipping the Velvet"
    },
    {
      "price": "£50.10",
      "title": "Soumission"
    }
  ]
}

General Detail Response Format

General agent returns A flat dictionary with all relevant fields about a single item when scraping individual item pages with comprehensive information (e.g., product details, profile pages, article pages)

{
  "price": "£51.77",
  "title": "A Light in the Attic",
  "image_url": "../../media/cache/fe/72/fe72f0532301ec28892ae79a629a293c.jpg",
  "description": "It's hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition...",
  "availability": "In stock (22 available)",
  "product_information": {
    "UPC": "a897fe39b1053632",
    "Product Type": "Books",
    "Price (excl. tax)": "£51.77",
    "Price (incl. tax)": "£51.77",
    "Tax": "£0.00",
    "Availability": "In stock (22 available)",
    "Number of reviews": 0
  }
}

Listing Agent

Listing agent returns an array of page objects, each containing pagination info and extracted data from the page.

{
  "response": [
    {
      "page_num": 0,
      "data": {
        "mode": "direct",
        "pagination": {
          "current_page": 1,
          "total_pages": 50
        },
        "products": [
          {
            "title": "Product Name",
            "price": "£XX.XX",
            "availability": "In stock"
          },
          {
            "title": "Another Product",
            "price": "£YY.YY",
            "availability": "In stock"
          }
        ],
        "product_count": 20,
        "__counts__": {
          "products": 20
        }
      },
      "total_items": 20,
      "next_found": true
    },
    {
      "page_num": 1,
      "data": {
        "mode": "direct",
        "pagination": {
          "current_page": 2,
          "total_pages": 50
        },
        "products": [
          {
            "title": "Product Name Page 2",
            "price": "£ZZ.ZZ",
            "availability": "In stock"
          }
        ],
        "product_count": 20,
        "__counts__": {
          "products": 20
        }
      },
      "total_items": 20,
      "next_found": true
    }
  ],
  "link": "https://books.toscrape.com/"
}

Map Agent

Map agent returns a list of all discovered URLs with a total count.

{
  "count": 581,
  "urls": [
    "https://books.toscrape.com",
    "https://books.toscrape.com/catalogue/10-day-green-smoothie-cleanse-lose-up-to-15-pounds-in-10-days_581/index.html",
    "https://books.toscrape.com/catalogue/1000-places-to-see-before-you-die_1/index.html",
    "https://books.toscrape.com/catalogue/1491-new-revelations-of-the-americas-before-columbus_650/index.html"
  ]
}

Bulk Scraping Response

Bulk scraping returns aggregated results for multiple URLs in a single operation. The response keeps the same top-level wrapper, while data.data includes merged extraction output, per-URL statuses, and a summary.

{
  "mergedData": [
    {
      "products": [
        {
          "price": "£51.77",
          "title": "A Light in the Attic",
          "availability": "In stock"
        }
      ],
      "page_title": "A Light in the Attic",
      "total_results": null
    },
    {
      "products": [
        {
          "price": "£53.74",
          "title": "Tipping the Velvet",
          "availability": "In stock (20 available)"
        }
      ],
      "page_title": "Tipping the Velvet",
      "total_results": null
    }
  ],
  "urlDetails": [
    {
      "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
      "status": "Finished",
      "error": ""
    },
    {
      "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
      "status": "Finished",
      "error": ""
    }
  ],
  "summary": {
    "totalUrls": 2,
    "successfulUrls": 2,
    "failedUrls": 0,
    "scrapedCount": 2,
    "totalTokenUsage": 20,
    "estimatedFinishAt": null
  }
}

Bulk Data Fields

Field	Type	Description
`mergedData`	array	Combined extracted output grouped by processed page. The exact object shape depends on your prompt and target pages.
`urlDetails`	array	Per-URL execution result with status and error details for each submitted URL.
`summary.totalUrls`	number	Total number of URLs requested in the bulk job.
`summary.successfulUrls`	number	Number of URLs completed successfully.
`summary.failedUrls`	number	Number of URLs that failed.
`summary.scrapedCount`	number	Number of pages with extracted data.
`summary.totalTokenUsage`	number	Total token usage across all URLs in the bulk job.
`summary.estimatedFinishAt`	string	Estimated completion timestamp for running jobs. `null` when finished.

Response Status Values

The data.status field indicates the current state of your scraping operation:

Status	Description
`Finished`	Scraping completed successfully
`Processing`	Scraping is currently in progress
`Failed`	Scraping encountered an error (check `data.error` for details)

AI Scraper Response

On this page