Scrape Sitemap

Crawl target sites with the map agent to discover endpoints, map URL architecture, and generate seed URL lists before running bulk extraction jobs.

Overview

Selecting Scrape Sitemap in the Playground Use Case panel configures an AI-based discovery crawler for site mapping. Powered by MrScraper's map AI agent (agent: "map"), it recursively traverses website link structures to discover endpoints and map site architecture without downloading full page content payloads.

The Playground pre-configures crawler parameters for the map agent—exposing maximum crawl depth (maxDepth), page limits (maxPages), and URL regex filters (include/exclude patterns)—making it ideal for generating seed URL lists before running bulk data extraction tasks.

Practical Applications

Pre-Crawl Site Topology Discovery: Analyze website hierarchy and URL distributions before triggering large-scale data extraction pipelines.
Seed List Generation: Build comprehensive target URL lists to feed into bulk Scrape JSON, Scrape Markdown, or Scrape HTML runs.
SEO Architecture Audits: Discover internal link structures, audit indexable routes, and detect orphan pages or broken URL paths.
Scoped Endpoint Filtering: Restrict discovery to specific paths (e.g., /docs/*, /store/*) while ignoring unwanted routes like user dashboards or admin panels.
Site Change Detection: Periodically crawl target domains to monitor newly published pages or removed URL paths over time.

Advanced settings

Setting	Description	Default
Max Pages	Maximum number of pages to crawl	`3`
Max Depth	Maximum crawl depth from the starting URL	`2`
Limit	Maximum number of URLs to collect	`1000`
Include Patterns	One URL pattern per line. Only URLs matching these patterns will be crawled.	None
Exclude Patterns	One URL pattern per line. URLs matching these patterns will be skipped.	None

Example

Request

Scrape https://quotes.toscrape.com/ with default settings.

Request

curl --location --request POST 'https://api.mrscraper.com?token={MRSCRAPER_API_TOKEN}&geoCode=us&proxyCountry=us' \
  -H 'x-api-token: {MRSCRAPER_API_TOKEN}' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "https://quotes.toscrape.com/",
  "agent": "map",
  "maxDepth": 2,
  "maxPages": 3,
  "limit": 1000
}'

Response

{
  "data": {
    "count": 47,
    "urls": [
      "https://quotes.toscrape.com",
      "https://quotes.toscrape.com/author/Albert-Einstein",
      "https://quotes.toscrape.com/author/Andre-Gide",
      "https://quotes.toscrape.com/author/Eleanor-Roosevelt",
      "https://quotes.toscrape.com/author/J-K-Rowling",
      "https://quotes.toscrape.com/author/Jane-Austen",
      "https://quotes.toscrape.com/author/Marilyn-Monroe",
      "https://quotes.toscrape.com/author/Steve-Martin",
      "https://quotes.toscrape.com/author/Thomas-A-Edison",
      "https://quotes.toscrape.com/login",
      "https://quotes.toscrape.com/page/2",
      "https://quotes.toscrape.com/tag/abilities/page/1",
      "https://quotes.toscrape.com/tag/adulthood/page/1",
      "https://quotes.toscrape.com/tag/aliteracy/page/1",
      "https://quotes.toscrape.com/tag/be-yourself/page/1",
      "https://quotes.toscrape.com/tag/books",
      "https://quotes.toscrape.com/tag/books/page/1",
      "https://quotes.toscrape.com/tag/change/page/1",
      "https://quotes.toscrape.com/tag/choices/page/1",
      "https://quotes.toscrape.com/tag/classic/page/1",
      "https://quotes.toscrape.com/tag/deep-thoughts/page/1",
      "https://quotes.toscrape.com/tag/edison/page/1",
      "https://quotes.toscrape.com/tag/failure/page/1",
      "https://quotes.toscrape.com/tag/friends",
      "https://quotes.toscrape.com/tag/friendship",
      "https://quotes.toscrape.com/tag/humor",
      "https://quotes.toscrape.com/tag/humor/page/1",
      "https://quotes.toscrape.com/tag/inspirational",
      "https://quotes.toscrape.com/tag/inspirational/page/1",
      "https://quotes.toscrape.com/tag/life",
      "https://quotes.toscrape.com/tag/life/page/1",
      "https://quotes.toscrape.com/tag/live/page/1",
      "https://quotes.toscrape.com/tag/love",
      "https://quotes.toscrape.com/tag/love/page/1",
      "https://quotes.toscrape.com/tag/miracle/page/1",
      "https://quotes.toscrape.com/tag/miracles/page/1",
      "https://quotes.toscrape.com/tag/misattributed-eleanor-roosevelt/page/1",
      "https://quotes.toscrape.com/tag/obvious/page/1",
      "https://quotes.toscrape.com/tag/paraphrased/page/1",
      "https://quotes.toscrape.com/tag/reading",
      "https://quotes.toscrape.com/tag/simile",
      "https://quotes.toscrape.com/tag/simile/page/1",
      "https://quotes.toscrape.com/tag/success/page/1",
      "https://quotes.toscrape.com/tag/thinking/page/1",
      "https://quotes.toscrape.com/tag/truth",
      "https://quotes.toscrape.com/tag/value/page/1",
      "https://quotes.toscrape.com/tag/world/page/1"
    ]
  },
  "runtime": 5305,
  "event": "done"
}

Scrape Sitemap

Overview

Practical Applications

Advanced settings

Example

Request

Response

On this page