AI Scraper Response

Learn about the response structure of the AI Scraper API and how to interpret the results.

The AI Scraper API returns responses with a consistent wrapper structure across all scraping operations. While the wrapper remains the same, the data.data field contains agent-specific data optimized for each scraping type.

Standard Response Wrapper

Every API response follows this structure:

{
  "message": "Successful operation!",
  "data": {
    "id": "unique-identifier",
    "createdAt": "2026-03-06T05:30:44.642Z",
    "updatedAt": "2026-03-06T05:33:01.855Z",
    "userId": "user-id",
    "scraperId": "scraper-id",
    "type": "Rerun-AI",
    "url": "https://example.com",
    "status": "Finished",
    "error": "",
    "tokenUsage": 5,
    "data": {
      // Agent-specific data structure (see below)
    },
    "htmlPath": "...",
    "recordingPath": null,
    "screenshotPath": "...",
    "dataPath": null
  }
}

Response Fields

Field	Type	Description
`message`	string	Operation status message
`data.id`	string	Unique identifier for this scraping operation
`data.createdAt`	string	ISO 8601 timestamp when the operation started
`data.updatedAt`	string	ISO 8601 timestamp when the operation completed
`data.userId`	string	Your user identifier
`data.scraperId`	string	Identifier of the scraper used
`data.type`	string	Agent type used for scraping
`data.url`	string	Target URL that was scraped
`data.status`	string	Operation status (`Finished`, `Processing`, `Failed`)
`data.error`	string	Error message if the operation failed
`data.tokenUsage`	number	Number of tokens consumed by this operation
`data.data`	object	Extracted data (structure varies by agent type)
`data.htmlPath`	string	Path to the saved HTML content
`data.recordingPath`	string	Path to the scraping session recording (if available)
`data.screenshotPath`	string	Path to the page screenshot
`data.dataPath`	string	Path to the raw data file (if available)

Agent-Specific Response Formats

The data.data field structure varies depending on which agent you use. Each agent optimizes the data format for its specific use case.

General Agent

The General Agent provides two response formats depending on the page type you're scraping.

General Table Response Format

General agent returns a dictionary containing an array of structured items when scraping listing pages with multiple items (e.g., product listings, search results, directory pages)

{
  "books": [
    {
      "price": "£51.77",
      "title": "A Light in the Attic"
    },
    {
      "price": "£53.74",
      "title": "Tipping the Velvet"
    },
    {
      "price": "£50.10",
      "title": "Soumission"
    }
  ]
}

General Detail Response Format

General agent returns A flat dictionary with all relevant fields about a single item when scraping individual item pages with comprehensive information (e.g., product details, profile pages, article pages)

{
  "price": "£51.77",
  "title": "A Light in the Attic",
  "image_url": "../../media/cache/fe/72/fe72f0532301ec28892ae79a629a293c.jpg",
  "description": "It's hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition...",
  "availability": "In stock (22 available)",
  "product_information": {
    "UPC": "a897fe39b1053632",
    "Product Type": "Books",
    "Price (excl. tax)": "£51.77",
    "Price (incl. tax)": "£51.77",
    "Tax": "£0.00",
    "Availability": "In stock (22 available)",
    "Number of reviews": 0
  }
}

Listing Agent

Listing agent returns an array of page objects, each containing pagination info and extracted data from the page.

{
  "response": [
    {
      "page_num": 0,
      "data": {
        "mode": "direct",
        "pagination": {
          "current_page": 1,
          "total_pages": 50
        },
        "products": [
          {
            "title": "Product Name",
            "price": "£XX.XX",
            "availability": "In stock"
          },
          {
            "title": "Another Product",
            "price": "£YY.YY",
            "availability": "In stock"
          }
        ],
        "product_count": 20,
        "__counts__": {
          "products": 20
        }
      },
      "total_items": 20,
      "next_found": true
    },
    {
      "page_num": 1,
      "data": {
        "mode": "direct",
        "pagination": {
          "current_page": 2,
          "total_pages": 50
        },
        "products": [
          {
            "title": "Product Name Page 2",
            "price": "£ZZ.ZZ",
            "availability": "In stock"
          }
        ],
        "product_count": 20,
        "__counts__": {
          "products": 20
        }
      },
      "total_items": 20,
      "next_found": true
    }
  ],
  "link": "https://books.toscrape.com/"
}

Map Agent

Map agent returns a list of all discovered URLs with a total count.

{
  "count": 581,
  "urls": [
    "https://books.toscrape.com",
    "https://books.toscrape.com/catalogue/10-day-green-smoothie-cleanse-lose-up-to-15-pounds-in-10-days_581/index.html",
    "https://books.toscrape.com/catalogue/1000-places-to-see-before-you-die_1/index.html",
    "https://books.toscrape.com/catalogue/1491-new-revelations-of-the-americas-before-columbus_650/index.html"
  ]
}

Response Status Values

The data.status field indicates the current state of your scraping operation:

Status	Description
`Finished`	Scraping completed successfully
`Processing`	Scraping is currently in progress
`Failed`	Scraping encountered an error (check `data.error` for details)

AI Scraper Response

On this page