Scrape JSON

Extract structured, strongly-typed JSON from single pages with the general agent, using natural language prompts or a JSON schema instead of selectors.

Overview

Selecting Scrape JSON in the Scraper API Use Case panel configures an AI-based scraper tailored for single-page structured data extraction. Powered by MrScraper's general AI agent (agent: "general"), it analyzes target web pages against your natural language prompt or optional JSON schema to return clean, strongly-typed JSON without writing fragile CSS selectors or custom XPath parsers.

The Scraper API pre-loads the prompt editor, schema builder, and general agent parameters with optimized defaults for single-page parsing.

Practical Applications

  • E-Commerce Product Detail Extraction: Extract structured product attributes—such as title, price, SKU, availability, variants, and image URLs—into a standardized schema.
  • Article & Editorial Metadata: Automatically pull publication dates, author bios, headline tags, body copy, and category tags from news sites and blogs.
  • Directory & Profile Scraping: Gather contact details, business metadata, addresses, and social links from company landing pages or user profiles.
  • Cross-Site Data Normalization: Map disparate target websites into a single, uniform JSON schema regardless of differences in site markup.
  • Automated Data Ingestion: Feed structured payload data directly into databases, webhooks, or downstream ETL pipelines without manual cleanup.

Advanced settings

SettingDescriptionDefault
Data typeSelect a predefined data schema to auto-fill the JSON Schema.General
PromptDescribe the data shape you want back (e.g. fields like title, price, rating). This prompt only parses the scraped content. It cannot instruct the scraper itself.Extract all entry data as complete as possible.
JSON SchemaBuild a JSON schema to shape the extraction output. It is appended to the prompt as a 'Json Schema:' section when the request is sent.None

Use cache

Serve a recent result when one exists. With Use cache on, the Scraper API returns a cached result of the same request when it is newer than the selected age. A cached result costs the same tokens as the original scrape.

Choose how old a cached result can be, and pass it as maxAge in milliseconds:

Cache agemaxAge
1 hour3600000
1 day86400000
3 days259200000
7 days604800000
Request with a 1-day cache
curl --location 'https://api.mrscraper.com?token={MRSCRAPER_API_TOKEN}&maxAge=86400000&geoCode=us&html=true&super=true&proxyCountry=us&url=https%3A%2F%2Fquotes.toscrape.com%2F&prompt=Extract%20all%20data%20as%20complete%20as%20possible.&agent=general' \
  -H 'x-api-token: {MRSCRAPER_API_TOKEN}'

Example

Request

Scrape https://quotes.toscrape.com/ with default settings.

Request
curl --location 'https://api.mrscraper.com?token={MRSCRAPER_API_TOKEN}&geoCode=us&html=true&super=true&proxyCountry=us&url=https%3A%2F%2Fquotes.toscrape.com%2F&prompt=Extract%20all%20data%20as%20complete%20as%20possible.&agent=general' \
  -H 'x-api-token: {MRSCRAPER_API_TOKEN}'

Every field goes in the query string, URL-encoded. If the URL would go over about 8 KB, for example with a long JSON schema in the prompt, send the same fields as a POST with url, prompt and agent in a JSON body.

Response

Trimmed: html is shortened, and data shows 2 of the 10 quotes and leaves out top_ten_tags and attribution.

Response
{
  "success": true,
  "data": {
    "site": {
      "name": "Quotes to Scrape",
      "url": "https://quotes.toscrape.com/",
      "navigation": [
        {
          "label": "Login",
          "url": "/login"
        }
      ]
    },
    "quotes": [
      {
        "text": "The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.",
        "author": "Albert Einstein",
        "author_url": "/author/Albert-Einstein",
        "tags": [
          {
            "name": "change",
            "url": "/tag/change/page/1/"
          },
          {
            "name": "deep-thoughts",
            "url": "/tag/deep-thoughts/page/1/"
          },
          {
            "name": "thinking",
            "url": "/tag/thinking/page/1/"
          },
          {
            "name": "world",
            "url": "/tag/world/page/1/"
          }
        ]
      },
      {
        "text": "It is our choices, Harry, that show what we truly are, far more than our abilities.",
        "author": "J.K. Rowling",
        "author_url": "/author/J-K-Rowling",
        "tags": [
          {
            "name": "abilities",
            "url": "/tag/abilities/page/1/"
          },
          {
            "name": "choices",
            "url": "/tag/choices/page/1/"
          }
        ]
      }
    ],
    "pagination": {
      "next": {
        "label": "Next →",
        "url": "/page/2/"
      }
    }
  },
  "thread_id": "01a1e8cb-0fb2-7d42-a67c-38c57e9e77eb",
  "token_usage": 10,
  "html": "<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n\t<meta charset=\"UTF-8\">\n\t<title>Quotes to Scrape</title>...</html>",
  "markdown": "",
  "screenshot": "",
  "status": "Results ready",
  "error": "",
  "runtime": 16.636244,
  "cookies": []
}

On this page