Scraping Tips
Get reliable results from the Scraper API by choosing the right loading path, geo-targeting, waiting, retry, and timeout settings.
Use these tips to get reliable results from the Scraper API. They cover how browser rendering, real-device Super Mode, proxy routing, waiting, retries, and timeouts affect a request, and when to change each setting. For the Scrape HTML use case itself, see Scrape HTML.
Key Features
Browser Rendering
Execute JavaScript and render dynamic content just like a real browser.
Use Cases:
- Single Page Applications (SPAs) built with React, Vue, or Angular
- Websites that load content via AJAX
- Pages with lazy-loaded images or infinite scroll
- Dynamic pricing or availability information
- Client-side rendered content
Super Mode
Route requests through real devices. Super Mode is on by default. It is independent of browser rendering: it can use either the non-browser loader or browser loading with JavaScript. Turn it off to use standard routing instead.
Use Cases:
- Websites that fail to load through standard routing
- Pages that return different content through real-device routing
Independent Loading Paths
Browser rendering and Super Mode are two independent switches. Every combination is supported, and the same URL can return different results in each:
| Browser rendering | Super Mode | Loading path |
|---|---|---|
| Off | Off | Standard routing with the non-browser loader. |
| On | Off | Standard routing with browser loading and JavaScript. |
| Off | On | Real-device routing with the non-browser loader. |
| On | On | Real-device routing with browser loading and JavaScript. |
When browserRendering is left out, the request starts on the non-browser path and switches to the browser path automatically if it gets blocked. Set browserRendering=true to use only the browser, or browserRendering=false to use only the non-browser loader.
Start with the defaults (browserRendering left out, Super Mode on) and change one switch at a time after inspecting the response. Browser rendering is not a strictly stronger option. Some websites fail or return worse content with browser rendering enabled but load correctly through the non-browser path. If that happens, retry with browserRendering=false while preserving the current Super Mode value. Stop after a usable response unless you need to compare loading paths.
Geo-Targeting with Proxies
Retrieve region-specific content by routing requests through residential proxies in specific countries.
Use Cases:
- Scraping region-specific pricing
- Testing localized versions of websites
- Market research across different regions
Supported Regions
View the full list of supported countries on the Country Codes page.
Smart Waiting
Wait for specific elements to appear in the DOM before extracting content.
Use Cases:
- Waiting for lazy-loaded product information
- Ensuring dynamic filters have applied
- Waiting for search results to populate
- Synchronizing with page animations
- Handling delayed content rendering
Homepage Navigation
Navigate through a website's home page before visiting the target URL.
Use Cases:
- Websites that expect visitors to arrive from the homepage
- Pages that set required cookies or session state on the homepage
Intelligent Retry Control
Automatically retry failed scrapes while limiting retry costs with configurable retry settings.
Use Cases:
- Recovering from temporary network failures
- Handling intermittent server errors
- Preventing excessive token usage with retry limits
- Balancing reliability and scraping cost
Resource Blocking
Block non-essential resources such as images, fonts, and media to improve scraping speed and reduce bandwidth usage.
Use Cases:
- Faster page rendering
- Lower bandwidth consumption
- Scraping text-heavy pages
- Large-scale scraping jobs
Configurable Timeout
Control how long to wait for page loading and rendering.
Use Cases:
- Slow-loading websites
- Complex pages with many resources
- Sites with large media files
- Pages with extensive JavaScript execution
- Unreliable network conditions
Recommended Settings
- Simple pages: 30 seconds (default)
- Standard e-commerce: 60 seconds
- Heavy SPAs: 90-120 seconds
- Very complex pages: Up to 180 seconds
How It Works
A Scraper API request follows this process:
Your request is routed through residential proxies in the specified region (if geo-targeting is enabled)
If Super Mode is enabled, the request is routed through a real device
The selected loader runs: a browser when browser rendering is enabled, or the non-browser loader when it is disabled
When browser rendering is enabled, the browser navigates to the target URL and begins rendering
When browser rendering is enabled, page JavaScript executes, including AJAX calls and dynamic content loading
If specified, the scraper waits for the target CSS selector to appear in the DOM
The process completes when the selector appears or the timeout is reached
The loaded HTML is extracted and returned
Loader resources are released
Example Request
Example request for scraping using the Scrape HTML endpoint:
curl -X GET "https://api.mrscraper.com/?url=https%3A%2F%2Fwww.amazon.com%2Fstores%2Fluxurystores%2Fpage%2FB6BC6264-7221-424B-9191-DAE2BCF963A2&token=atk_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&timeout=120&geoCode=pt&browserRendering=true&super=true&waitForSelector=div%5Bdata-testid%3D%22grid-item-info%22%5D&homePage=false&blockResources=true&maxRetries=3&tokenCap=10" \
-H "x-api-token: MRSCRAPER_API_KEY"Parameters
| Parameter | Required | Default | Description |
|---|---|---|---|
url | Yes | - | Target website URL to scrape. |
token | Yes | - | Your API authentication token. |
browserRendering | No | Automatic | Leave it out to load the page without a browser first and switch to a browser automatically if the request is blocked. true always uses a browser. false never uses a browser. |
super | No | true | Route the request through a real device. Set to false to use standard routing. |
geoCode | No | - | ISO 3166-1 alpha-2 country code used for proxy routing. |
timeout | No | 30 | Maximum number of seconds to wait for page loading. |
waitForSelector | No | - | CSS selector to wait for before returning the page content. Requires browserRendering=true. |
homePage | No | false | Visit the website's home page before navigating to the target URL. |
blockResources | No | - | Block non-essential resources to improve performance. Supported only by compatible proxies. |
maxRetries | No | 3 | Maximum number of retry attempts if the scrape fails. |
tokenCap | No | - | Maximum number of tokens that can be consumed across retry attempts. If the first scrape exceeds the cap, it still runs once but no retries are performed. |
Parameter Dependencies
waitForSelectoronly works whenbrowserRendering=true.superis on by default. Setsuper=falseto compare against standard routing when real-device routing does not return usable content.blockResourcesis only supported by proxies that support resource blocking.tokenCaponly affects retry attempts. The initial scrape always runs, even if it exceeds the configured token cap.