Scrape Sitemap
Crawl target sites with the Scrape Sitemap endpoint to discover endpoints, map URL architecture, and generate seed URL lists before running bulk extraction jobs.
Recursively traverse a site's link structure and return the discovered URLs. The map agent maps site architecture without downloading full page content payloads, which makes it ideal for generating seed URL lists before bulk extraction.
Note
- Use the Scraper API Server (
https://api.mrscraper.com) host when calling this endpoint. - Control the crawl with
maxDepth,maxPages, andlimit. - Scope discovery with
includePatternsandexcludePatterns, one URL pattern per entry. - See Scrape Sitemap in the Scraper API guide for the full list of settings.
Query Parameters
API token.
Controls whether the page is loaded in a browser. Leave it out to load the page without a browser first and switch to a browser automatically if the request is blocked. Set true to always load the page in a browser, which JavaScript-heavy sites, SPAs, and pages that load content dynamically need. Set false to never use a browser.
Defines when the browser considers navigation complete. networkidle0 waits until all network activity stops — the most thorough option, but the slowest.
"domcontentloaded""domcontentloaded" | "load" | "networkidle0" | "networkidle2"Maximum seconds to wait for the page to fully load before the request times out. Increase for slow or complex pages.
300Captures a screenshot of the rendered page as a base64-encoded image. Useful for verifying what the scraper actually saw.
falseUses additional resources to improve extraction accuracy on complex websites. Enable for heavily dynamic or JavaScript-rendered content.
falseCountry to load the page from. Defaults to the United States.
"us"Country of the proxy server. Defaults to the United States.
"us"Saves the result to the Result tab.
trueHeader Parameters
API token.
Request Body
application/json
The full URL of the page you want to scrape. Must include the protocol (https://).
uriThe AI agent used for discovery. Scrape Sitemap uses the map agent, which crawls link structures instead of extracting page content.
"map""map"Maximum crawl depth from the starting URL.
21 <= valueMaximum number of pages to crawl.
31 <= valueMaximum number of URLs to collect.
10001 <= valueOnly URLs matching these patterns will be crawled. One pattern per entry.
URLs matching these patterns will be skipped. One pattern per entry.
Maximum total tokens the scrape and its retries can use. Automatic retries are bounded by this cap, so the total tokens for a single run stay at or below the value you set. Omit for no limit.
1 <= valueMaximum number of retries for the request. Omit for no limit.
0 <= valueResponse Body
application/json
application/json
application/json
curl -X POST "https://api.mrscraper.com/?token=string" \ -H "x-api-token: string" \ -H "Content-Type: application/json" \ -d '{ "url": "https://quotes.toscrape.com/", "agent": "map", "maxDepth": 2, "maxPages": 3, "limit": 1000 }'{
"code": null,
"screenshots": [],
"recording_path": null,
"extractions": "{}",
"data_path": "results/6cef400f-ec7f-4a21-86df-0c7d46bfcf49/data.json",
"html_path": "results/6cef400f-ec7f-4a21-86df-0c7d46bfcf49/page.html",
"error": null,
"residential_proxy_usage": {
"received": 0.0009365081787109375,
"sent": 0.0002651214599609375,
"total": 0.001201629638671875
},
"runtime": 9.13,
"token_usage": 6,
"listen_network_data": {},
"html": "",
"markdown": "",
"screenshot": "",
"data": {
"count": 47,
"urls": [
"https://quotes.toscrape.com",
"https://quotes.toscrape.com/author/Albert-Einstein",
"https://quotes.toscrape.com/login",
"https://quotes.toscrape.com/page/2",
"https://quotes.toscrape.com/tag/books"
]
}
}{
"error": "Unauthorized",
"message": "Invalid or missing API token",
"statusCode": 401
}{
"error": "Internal Server Error",
"message": "An unexpected error occurred",
"statusCode": 500
}