Scrape Sitemap
Crawl target sites to map URL architecture, discover endpoints, and generate seed URL lists.
Overview
Selecting Scrape Sitemap in the Playground Use Case panel configures an AI-based discovery crawler for site mapping. Powered by MrScraper's map AI agent (agent: "map"), it recursively traverses website link structures to discover endpoints and map site architecture without downloading full page content payloads.
The Playground pre-configures crawler parameters for the map agent—exposing maximum crawl depth (maxDepth), page limits (maxPages), and URL regex filters (include/exclude patterns)—making it ideal for generating seed URL lists before running bulk data extraction tasks.
Practical Applications
- Pre-Crawl Site Topology Discovery: Analyze website hierarchy and URL distributions before triggering large-scale data extraction pipelines.
- Seed List Generation: Build comprehensive target URL lists to feed into bulk Scrape JSON, Scrape Markdown, or Scrape HTML runs.
- SEO Architecture Audits: Discover internal link structures, audit indexable routes, and detect orphan pages or broken URL paths.
- Scoped Endpoint Filtering: Restrict discovery to specific paths (e.g.,
/docs/*,/store/*) while ignoring unwanted routes like user dashboards or admin panels. - Site Change Detection: Periodically crawl target domains to monitor newly published pages or removed URL paths over time.
Advanced settings
| Setting | Description | Default |
|---|---|---|
| Max Pages | Maximum number of pages to crawl | 3 |
| Max Depth | Maximum crawl depth from the starting URL | 2 |
| Limit | Maximum number of URLs to collect | 1000 |
| Include Patterns | One URL pattern per line. Only URLs matching these patterns will be crawled. | None |
| Exclude Patterns | One URL pattern per line. URLs matching these patterns will be skipped. | None |
Example
Request
Scrape https://quotes.toscrape.com/ with default settings.
curl --location --request POST 'https://api.mrscraper.com?token={MRSCRAPER_API_TOKEN}&geoCode=us&proxyCountry=us' \
-H 'x-api-token: {MRSCRAPER_API_TOKEN}' \
-H 'Content-Type: application/json' \
-d '{
"url": "https://quotes.toscrape.com/",
"agent": "map",
"maxDepth": 2,
"maxPages": 3,
"limit": 1000
}'Response
{
"data": {
"count": 47,
"urls": [
"https://quotes.toscrape.com",
"https://quotes.toscrape.com/author/Albert-Einstein",
"https://quotes.toscrape.com/author/Andre-Gide",
"https://quotes.toscrape.com/author/Eleanor-Roosevelt",
"https://quotes.toscrape.com/author/J-K-Rowling",
"https://quotes.toscrape.com/author/Jane-Austen",
"https://quotes.toscrape.com/author/Marilyn-Monroe",
"https://quotes.toscrape.com/author/Steve-Martin",
"https://quotes.toscrape.com/author/Thomas-A-Edison",
"https://quotes.toscrape.com/login",
"https://quotes.toscrape.com/page/2",
"https://quotes.toscrape.com/tag/abilities/page/1",
"https://quotes.toscrape.com/tag/adulthood/page/1",
"https://quotes.toscrape.com/tag/aliteracy/page/1",
"https://quotes.toscrape.com/tag/be-yourself/page/1",
"https://quotes.toscrape.com/tag/books",
"https://quotes.toscrape.com/tag/books/page/1",
"https://quotes.toscrape.com/tag/change/page/1",
"https://quotes.toscrape.com/tag/choices/page/1",
"https://quotes.toscrape.com/tag/classic/page/1",
"https://quotes.toscrape.com/tag/deep-thoughts/page/1",
"https://quotes.toscrape.com/tag/edison/page/1",
"https://quotes.toscrape.com/tag/failure/page/1",
"https://quotes.toscrape.com/tag/friends",
"https://quotes.toscrape.com/tag/friendship",
"https://quotes.toscrape.com/tag/humor",
"https://quotes.toscrape.com/tag/humor/page/1",
"https://quotes.toscrape.com/tag/inspirational",
"https://quotes.toscrape.com/tag/inspirational/page/1",
"https://quotes.toscrape.com/tag/life",
"https://quotes.toscrape.com/tag/life/page/1",
"https://quotes.toscrape.com/tag/live/page/1",
"https://quotes.toscrape.com/tag/love",
"https://quotes.toscrape.com/tag/love/page/1",
"https://quotes.toscrape.com/tag/miracle/page/1",
"https://quotes.toscrape.com/tag/miracles/page/1",
"https://quotes.toscrape.com/tag/misattributed-eleanor-roosevelt/page/1",
"https://quotes.toscrape.com/tag/obvious/page/1",
"https://quotes.toscrape.com/tag/paraphrased/page/1",
"https://quotes.toscrape.com/tag/reading",
"https://quotes.toscrape.com/tag/simile",
"https://quotes.toscrape.com/tag/simile/page/1",
"https://quotes.toscrape.com/tag/success/page/1",
"https://quotes.toscrape.com/tag/thinking/page/1",
"https://quotes.toscrape.com/tag/truth",
"https://quotes.toscrape.com/tag/value/page/1",
"https://quotes.toscrape.com/tag/world/page/1"
]
},
"runtime": 5305,
"event": "done"
}