Web Fetching

MrScraper's Web Fetching retrieves the HTML of public web pages, renders JavaScript-heavy websites, and returns region-specific content. It combines selectable non-browser or browser loading, proxy routing, optional real-device Super Mode, and intelligent retry mechanisms for reliable data extraction.

Key Features

Browser Rendering

Execute JavaScript and render dynamic content just like a real browser.

Use Cases:

• Single Page Applications (SPAs) built with React, Vue, or Angular
• Websites that load content via AJAX
• Pages with lazy-loaded images or infinite scroll
• Dynamic pricing or availability information
• Client-side rendered content

Super Mode

Route requests through real devices when a website needs a different routing path. Super Mode is independent of browser rendering: it can use either the non-browser loader or browser loading with JavaScript.

Use Cases:

• Websites that fail to load through standard routing
• Pages that return different content through real-device routing

Independent Loading Paths

Browser rendering and Super Mode are two independent switches. Every combination is supported, and the same URL can return different results in each:

Start with both off and change one switch at a time after inspecting the response. Browser rendering is not a strictly stronger option. Some websites fail or return worse content with browser rendering enabled but load correctly through the non-browser path. If that happens, retry with browser rendering off while preserving the current Super Mode value. Stop after a usable response unless you need to compare loading paths.

Geo-Targeting with Proxies

Retrieve region-specific content by routing requests through residential proxies in specific countries.

Use Cases:

• Scraping region-specific pricing
• Testing localized versions of websites
• Market research across different regions

Smart Waiting

Wait for specific elements to appear in the DOM before extracting content.

Use Cases:

• Waiting for lazy-loaded product information
• Ensuring dynamic filters have applied
• Waiting for search results to populate
• Synchronizing with page animations
• Handling delayed content rendering

Homepage Navigation

Navigate through a website's home page before visiting the target URL.

Use Cases:

• Websites that expect visitors to arrive from the homepage
• Pages that set required cookies or session state on the homepage

Intelligent Retry Control

Automatically retry failed scrapes while limiting retry costs with configurable retry settings.

Use Cases:

• Recovering from temporary network failures
• Handling intermittent server errors
• Preventing excessive token usage with retry limits
• Balancing reliability and scraping cost

Resource Blocking

Block non-essential resources such as images, fonts, and media to improve scraping speed and reduce bandwidth usage.

Use Cases:

• Faster page rendering
• Lower bandwidth consumption
• Scraping text-heavy pages
• Large-scale scraping jobs

Configurable Timeout

Control how long to wait for page loading and rendering.

Use Cases:

• Slow-loading websites
• Complex pages with many resources
• Sites with large media files
• Pages with extensive JavaScript execution
• Unreliable network conditions

How It Works

Web Fetching follows this process:

Example Request

Example request for scraping using the web fetching endpoint:

curl -X GET "https://api.mrscraper.com/?url=https%3A%2F%2Fwww.amazon.com%2Fstores%2Fluxurystores%2Fpage%2FB6BC6264-7221-424B-9191-DAE2BCF963A2&token=atk_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&timeout=120&geoCode=pt&browserRendering=true&super=true&waitForSelector=div%5Bdata-testid%3D%22grid-item-info%22%5D&homePage=false&blockResources=true&maxRetries=3&tokenCap=10" \
  -H "x-api-token: MRSCRAPER_API_KEY"

Parameters