Scraping Tips

Get reliable results from the Scraper API by choosing the right loading path, geo-targeting, waiting, retry, and timeout settings.

Use these tips to get reliable results from the Scraper API. They cover how browser rendering, real-device Super Mode, proxy routing, waiting, retries, and timeouts affect a request, and when to change each setting. For the Scrape HTML use case itself, see Scrape HTML.

Key Features

Browser Rendering

Execute JavaScript and render dynamic content just like a real browser.

Use Cases:

  • Single Page Applications (SPAs) built with React, Vue, or Angular
  • Websites that load content via AJAX
  • Pages with lazy-loaded images or infinite scroll
  • Dynamic pricing or availability information
  • Client-side rendered content

Super Mode

Route requests through real devices. Super Mode is on by default. It is independent of browser rendering: it can use either the non-browser loader or browser loading with JavaScript. Turn it off to use standard routing instead.

Use Cases:

  • Websites that fail to load through standard routing
  • Pages that return different content through real-device routing

Independent Loading Paths

Browser rendering and Super Mode are two independent switches. Every combination is supported, and the same URL can return different results in each:

Browser renderingSuper ModeLoading path
OffOffStandard routing with the non-browser loader.
OnOffStandard routing with browser loading and JavaScript.
OffOnReal-device routing with the non-browser loader.
OnOnReal-device routing with browser loading and JavaScript.

When browserRendering is left out, the request starts on the non-browser path and switches to the browser path automatically if it gets blocked. Set browserRendering=true to use only the browser, or browserRendering=false to use only the non-browser loader.

Start with the defaults (browserRendering left out, Super Mode on) and change one switch at a time after inspecting the response. Browser rendering is not a strictly stronger option. Some websites fail or return worse content with browser rendering enabled but load correctly through the non-browser path. If that happens, retry with browserRendering=false while preserving the current Super Mode value. Stop after a usable response unless you need to compare loading paths.

Geo-Targeting with Proxies

Retrieve region-specific content by routing requests through residential proxies in specific countries.

Use Cases:

  • Scraping region-specific pricing
  • Testing localized versions of websites
  • Market research across different regions

Supported Regions

View the full list of supported countries on the Country Codes page.

Smart Waiting

Wait for specific elements to appear in the DOM before extracting content.

Use Cases:

  • Waiting for lazy-loaded product information
  • Ensuring dynamic filters have applied
  • Waiting for search results to populate
  • Synchronizing with page animations
  • Handling delayed content rendering

Homepage Navigation

Navigate through a website's home page before visiting the target URL.

Use Cases:

  • Websites that expect visitors to arrive from the homepage
  • Pages that set required cookies or session state on the homepage

Intelligent Retry Control

Automatically retry failed scrapes while limiting retry costs with configurable retry settings.

Use Cases:

  • Recovering from temporary network failures
  • Handling intermittent server errors
  • Preventing excessive token usage with retry limits
  • Balancing reliability and scraping cost

Resource Blocking

Block non-essential resources such as images, fonts, and media to improve scraping speed and reduce bandwidth usage.

Use Cases:

  • Faster page rendering
  • Lower bandwidth consumption
  • Scraping text-heavy pages
  • Large-scale scraping jobs

Configurable Timeout

Control how long to wait for page loading and rendering.

Use Cases:

  • Slow-loading websites
  • Complex pages with many resources
  • Sites with large media files
  • Pages with extensive JavaScript execution
  • Unreliable network conditions

Recommended Settings

  • Simple pages: 30 seconds (default)
  • Standard e-commerce: 60 seconds
  • Heavy SPAs: 90-120 seconds
  • Very complex pages: Up to 180 seconds

How It Works

A Scraper API request follows this process:

Your request is routed through residential proxies in the specified region (if geo-targeting is enabled)

If Super Mode is enabled, the request is routed through a real device

The selected loader runs: a browser when browser rendering is enabled, or the non-browser loader when it is disabled

When browser rendering is enabled, the browser navigates to the target URL and begins rendering

When browser rendering is enabled, page JavaScript executes, including AJAX calls and dynamic content loading

If specified, the scraper waits for the target CSS selector to appear in the DOM

The process completes when the selector appears or the timeout is reached

The loaded HTML is extracted and returned

Loader resources are released

Example Request

Example request for scraping using the Scrape HTML endpoint:

curl -X GET "https://api.mrscraper.com/?url=https%3A%2F%2Fwww.amazon.com%2Fstores%2Fluxurystores%2Fpage%2FB6BC6264-7221-424B-9191-DAE2BCF963A2&token=atk_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&timeout=120&geoCode=pt&browserRendering=true&super=true&waitForSelector=div%5Bdata-testid%3D%22grid-item-info%22%5D&homePage=false&blockResources=true&maxRetries=3&tokenCap=10" \
  -H "x-api-token: MRSCRAPER_API_KEY"

Parameters

ParameterRequiredDefaultDescription
urlYes-Target website URL to scrape.
tokenYes-Your API authentication token.
browserRenderingNoAutomaticLeave it out to load the page without a browser first and switch to a browser automatically if the request is blocked. true always uses a browser. false never uses a browser.
superNotrueRoute the request through a real device. Set to false to use standard routing.
geoCodeNo-ISO 3166-1 alpha-2 country code used for proxy routing.
timeoutNo30Maximum number of seconds to wait for page loading.
waitForSelectorNo-CSS selector to wait for before returning the page content. Requires browserRendering=true.
homePageNofalseVisit the website's home page before navigating to the target URL.
blockResourcesNo-Block non-essential resources to improve performance. Supported only by compatible proxies.
maxRetriesNo3Maximum number of retry attempts if the scrape fails.
tokenCapNo-Maximum number of tokens that can be consumed across retry attempts. If the first scrape exceeds the cap, it still runs once but no retries are performed.

Parameter Dependencies

  • waitForSelector only works when browserRendering=true.
  • super is on by default. Set super=false to compare against standard routing when real-device routing does not return usable content.
  • blockResources is only supported by proxies that support resource blocking.
  • tokenCap only affects retry attempts. The initial scrape always runs, even if it exceeds the configured token cap.

On this page