MrScraperMrScraper
Dashboard
DocumentationAPI ReferenceResidential ProxyChangelog

Scrape Sitemap

Crawl target sites with the Scrape Sitemap endpoint to discover endpoints, map URL architecture, and generate seed URL lists before running bulk extraction jobs.

Recursively traverse a site's link structure and return the discovered URLs. The map agent maps site architecture without downloading full page content payloads, which makes it ideal for generating seed URL lists before bulk extraction.

Note

  • Use the Playground API Server (https://api.mrscraper.com) host when calling this endpoint.
  • Control the crawl with maxDepth, maxPages, and limit.
  • Scope discovery with includePatterns and excludePatterns, one URL pattern per entry.
  • See Scrape Sitemap in the Playground guide for the full list of settings.
POST
/

Query Parameters

token*string

API token.

browserRendering?boolean

Render the page using a real Chromium browser. Required for JavaScript-heavy sites, SPAs, and pages that load content dynamically.

Defaultfalse
waitUntil?string

Defines when the browser considers navigation complete. networkidle0 waits until all network activity stops — the most thorough option, but the slowest.

Default"domcontentloaded"
Value in"domcontentloaded" | "load" | "networkidle0" | "networkidle2"
timeout?integer

Maximum seconds to wait for the page to fully load before the request times out. Increase for slow or complex pages.

Default300
screenshot?boolean

Captures a screenshot of the rendered page as a base64-encoded image. Useful for verifying what the scraper actually saw.

Defaultfalse
super?boolean

Uses additional resources to improve extraction accuracy on complex websites. Enable for sites protected by anti-bot systems or containing heavily dynamic content.

Defaultfalse
geoCode?string

Country to load the page from. Defaults to the United States.

Default"us"
proxyCountry?string

Country of the proxy server. Defaults to the United States.

Default"us"
saveResult?boolean

Saves the result to the Result tab.

Defaulttrue

Header Parameters

x-api-token*string

API token.

Request Body

application/json

url*string

The full URL of the page you want to scrape. Must include the protocol (https://).

Formaturi
agent*string

The AI agent used for discovery. Scrape Sitemap uses the map agent, which crawls link structures instead of extracting page content.

Default"map"
Value in"map"
maxDepth?integer

Maximum crawl depth from the starting URL.

Default2
Range1 <= value
maxPages?integer

Maximum number of pages to crawl.

Default3
Range1 <= value
limit?integer

Maximum number of URLs to collect.

Default1000
Range1 <= value
includePatterns?array<string>

Only URLs matching these patterns will be crawled. One pattern per entry.

excludePatterns?array<string>

URLs matching these patterns will be skipped. One pattern per entry.

tokenCap?integer

Maximum total tokens the scrape and its retries can use. Automatic retries are bounded by this cap, so the total tokens for a single run stay at or below the value you set. Omit for no limit.

Range1 <= value
maxRetries?integer

Maximum number of retries for the request. Omit for no limit.

Range0 <= value

Response Body

application/json

application/json

application/json

curl -X POST "https://api.mrscraper.com/?token=string" \  -H "x-api-token: string" \  -H "Content-Type: application/json" \  -d '{    "url": "https://quotes.toscrape.com/",    "agent": "map",    "maxDepth": 2,    "maxPages": 3,    "limit": 1000  }'
{
  "code": null,
  "screenshots": [],
  "recording_path": null,
  "extractions": "{}",
  "data_path": "results/6cef400f-ec7f-4a21-86df-0c7d46bfcf49/data.json",
  "html_path": "results/6cef400f-ec7f-4a21-86df-0c7d46bfcf49/page.html",
  "error": null,
  "residential_proxy_usage": {
    "received": 0.0009365081787109375,
    "sent": 0.0002651214599609375,
    "total": 0.001201629638671875
  },
  "runtime": 9.13,
  "token_usage": 6,
  "listen_network_data": {},
  "html": "",
  "markdown": "",
  "screenshot": "",
  "data": {
    "count": 47,
    "urls": [
      "https://quotes.toscrape.com",
      "https://quotes.toscrape.com/author/Albert-Einstein",
      "https://quotes.toscrape.com/login",
      "https://quotes.toscrape.com/page/2",
      "https://quotes.toscrape.com/tag/books"
    ]
  }
}
{
  "error": "Unauthorized",
  "message": "Invalid or missing API token",
  "statusCode": 401
}
{
  "error": "Internal Server Error",
  "message": "An unexpected error occurred",
  "statusCode": 500
}

Listing Page POST

Extract paginated item lists, catalog grids, and search results in one run with the Listing Page endpoint, handling numbered pages, scrolling, and Load More.

Verify API Token GET

Verify your API token and retrieve subscription account information, including token usage, limits, and billing details, with the MrScraper v3 API.