mrscraper - Google Search

# Authentication import { Step, Steps } from 'fumadocs-ui/components/steps'; The MrScraper API uses API token authentication to secure all endpoints. Every request must include a valid API token in the HTTP request headers. ## Getting Your API Token API tokens are managed directly from your MrScraper dashboard. Sign in to your [MrScraper Dashboard](https://app.mrscraper.com). Click your **profile icon** in the top-right corner and select **API Tokens** (or go to [app.mrscraper.com/api-tokens](https://app.mrscraper.com/api-tokens)). Click **New Token**. Enter a descriptive **token name** and choose an **expiration date**. Click **Create**, then copy and securely store the generated token. Your API token grants access to your account and plan resources. Never expose your token in client-side code (like browsers or mobile apps), and store it securely in an environment variable or server configuration. ## Authenticating Requests Include your API token in the `x-api-token` HTTP header on every request: ```http x-api-token: ``` ### Marketplace Endpoints For [Marketplace endpoints](https://app.mrscraper.com/marketplace), authenticate using the standard RFC 6750 `Authorization` header with a Bearer token: ```http Authorization: Bearer ``` ## Verify Token To test your authentication and confirm that your token is valid, make a request to the [`/subscription-accounts`](/docs/api/v3/authentication/verify-token) endpoint: ```bash curl -X GET "https://api.app.mrscraper.com/api/v1/subscription-accounts" \ -H "accept: application/json" \ -H "x-api-token: " ``` ## Analytics Authentication [Analytics endpoints](/docs/api/v3/analytic/status) authenticate using the same `x-api-token` header as all other API endpoints. When querying analytics, you can also pass the `apiTokenName` query parameter to filter status data for a specific token by the name you assigned when creating it: ```bash curl -X GET "https://api.app.mrscraper.com/api/v1/analytic/statuses?domain=example.com&startDate=2026-04-12%2000:00:00&endDate=2026-04-13%2023:59:59&apiTokenName=YOUR_TOKEN_NAME" \ -H "accept: application/json" \ -H "x-api-token: " ``` The `apiTokenName` query parameter filters metrics by the token's name (e.g., `production-worker`). # Error Handling ## Overview The MrScraper API will return a JSON object containing error details when a request fails. This object includes an error code, message, and additional information to help diagnose the issue. This reference covers the most common errors. It is not exhaustive — an endpoint may occasionally return a transient error not listed here. When in doubt, inspect the `message` and `error` fields in the response body. ## Error response format When a request fails, the API returns a non-`2xx` HTTP status code and a JSON body with a consistent shape: ```json { "message": "Unauthorized access", "error": "Unauthorized" } ``` | Field | Type | Description | | --------- | ------ | ---------------------------------------------------------------- | | `message` | string | Human-readable description of what went wrong. | | `error` | string | Short error type or category (e.g. `Unauthorized`, `Not Found`). | ## HTTP status codes | Status | Error | Cause | Solution | Retryable | | ------ | ---------------------- | ------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- | ----------------- | | `400` | `Bad Request` | The request payload is malformed or a required parameter is missing or in the wrong format. | Fix the payload per the endpoint reference (e.g. use the `yyyy-MM-dd HH:mm:ss` date format where required). | No | | `401` | `Unauthorized` | The API token is missing or invalid. | Include a valid token in the `x-api-token` header. See [Authentication](/docs/api/authentication). | No | | `403` | `Forbidden` | The token is valid but does not have permission to access this resource or Token limit exceeded. | Verify your plan and that the token has access to the resource. [Contact support](mailto:support@mrscraper.com) if needed. | No | | `404` | `Not Found` | The requested resource (e.g. a scraper or result) does not exist. | Check that the `scraperId` / `resultId` is correct and belongs to your account. | No | | `422` | `Unprocessable Entity` | The request is well-formed but failed validation. | Review the `message` for the specific validation failure and correct the input. | No | | `429` | `Too Many Requests` | You have exceeded your rate limit. | Back off and retry after a short delay. Honor the `Retry-After` header if present. | Yes, with backoff | | `500` | Server errors | A temporary problem on MrScraper's side. | Retry with exponential backoff. If it persists, [contact support](mailto:support@mrscraper.com). | Yes, with backoff | ## Common errors ### 400 — Bad Request The request body could not be parsed, or a parameter is missing or incorrectly formatted. ```json { "message": "startDate is required and must be in `yyyy-MM-dd HH:mm:ss` format", "error": "Bad Request" } ``` ### 401 — Unauthorized The `x-api-token` header is missing or the token is invalid. ```json { "message": "Unauthorized access", "error": "Unauthorized" } ``` Ensure the token is included on every request: ```http x-api-token: MRSCRAPER_API_TOKEN ``` ### 403 — Forbidden The token is authenticated but lacks permission for the requested resource. ```json { "message": "You do not have permission to access this resource", "error": "Forbidden" } ``` ### 404 — Not Found The referenced scraper, result, or other entity could not be found. ```json { "message": "Resource not found", "error": "Not Found" } ``` ### 422 — Unprocessable Entity The request was understood but failed validation. ```json { "message": "Validation failed", "error": "Unprocessable Entity" } ``` ### 429 — Too Many Requests You have exceeded your rate limit. ```json { "message": "Too many requests. Please try again later." } ``` # API Overview import { Braces, Key, ListOrdered, CircleAlert, Repeat, Database, FileSearch, SquareTerminal, PanelTop, SlidersHorizontal } from 'lucide-react'; ## Introduction The MrScraper API provides a powerful and flexible way to programmatically scrape web pages and retrieve structured data. Built on REST principles, our API offers predictable resource-oriented URLs, accepts JSON-encoded request bodies, returns JSON-encoded responses, and uses standard HTTP response codes. ## API Features }> All requests with a payload require `Content-Type: application/json` header. All responses are returned in JSON format. }> All API endpoints require authentication using an API token passed in the `x-api-token` header. See the [Authentication](/docs/api/authentication) page for details. }> List endpoints support pagination to handle large datasets efficiently. See the [Pagination](/docs/api/pagination) page for details. }> Track token usage, bandwidth usage, execution runtime, and target status codes in response headers. See the [Response Headers](/docs/api/response-headers) page for details. }> The API uses standard HTTP status codes and returns detailed error messages. See the [Error Handling](/docs/api/error-handling) page for details. ## Available Endpoints } href="/docs/api/v3/scraper/web-fetching-scraping"> Scrape HTML, Markdown, screenshots, JSON, listings, and sitemaps from a single Scraper API endpoint. } href="/docs/api/v3/scraper/ai/ai-init"> Create, rerun, and manage scrapers programmatically with the Platform API. # Pagination ## Overview The MrScraper API uses cursor-based pagination for endpoints that return lists of resources. This approach ensures consistent results even when data is being added or modified. ## Pagination Parameters The `/api/v1/results` endpoint supports the following pagination parameters: ### Required Parameters | Parameter | Type | Description | Example | | ----------- | ------ | --------------------------------------- | ----------- | | `page` | number | The page number to retrieve (1-indexed) | `1` | | `pageSize` | number | Number of items per page | `10` | | `sortField` | string | Field name to sort by | `updatedAt` | | `sortOrder` | string | Sort direction (`ASC` or `DESC`) | `DESC` | * **Small datasets (\< 100 items):** Use `pageSize=50` or `pageSize=100` * **Large datasets:** Use `pageSize=10` or `pageSize=25` for faster responses * **Maximum recommended:** `pageSize=100` to avoid timeouts ### Optional Parameters | Parameter | Type | Description | Example | | ----------------- | ------ | ----------------------------------- | -------------------------- | | `search` | string | Search across all entity columns | `example.com` | | `dateRangeColumn` | string | Column name for date filtering | `createdAt` | | `startAt` | string | Start date for filtering (ISO 8601) | `2025-11-13T02:46:49.014Z` | | `endAt` | string | End date for filtering (ISO 8601) | `2025-11-14T02:46:49.014Z` | ## Making Paginated Requests ### Basic Example ```bash curl -X GET "https://api.app.mrscraper.com/api/v1/results?page=1&pageSize=10&sortField=updatedAt&sortOrder=DESC" \ -H "x-api-token: MRSCRAPER_API_TOKEN" \ -H "accept: application/json" ``` ### With Search ```bash curl -X GET "https://api.app.mrscraper.com/api/v1/results?page=1&pageSize=10&sortField=updatedAt&sortOrder=DESC&search=example.com" \ -H "x-api-token: MRSCRAPER_API_TOKEN" \ -H "accept: application/json" ``` ### With Date Range Filtering ```bash curl -X GET "https://api.app.mrscraper.com/api/v1/results?page=1&pageSize=10&sortField=updatedAt&sortOrder=DESC&dateRangeColumn=createdAt&startAt=2025-11-13T00:00:00.000Z&endAt=2025-11-14T23:59:59.999Z" \ -H "x-api-token: MRSCRAPER_API_TOKEN" \ -H "accept: application/json" ``` ## Response Format ### Successful Response ```json { "message": "Successful fetch", "data": [ { "id": "550e8400-e29b-41d4-a716-446655440000", "userId": "660e8400-e29b-41d4-a716-446655440000", "scraperId": "770e8400-e29b-41d4-a716-446655440000", "type": "AI", "url": "https://example.com", "status": "Finished", "error": null, "tokenUsage": 5, "runtime": 2.5, "data": "{\"title\":\"Example Page\"}", "createdAt": "2025-11-11T09:50:09.722Z" } ], "meta": { "page": 1, "pageSize": 10, "total": 100, "totalPage": 10 } } ``` ### Response Metadata The `meta` object provides pagination information: | Field | Type | Description | | ----------- | ------ | -------------------------------------- | | `page` | number | Current page number | | `pageSize` | number | Items per page | | `total` | number | Total number of items across all pages | | `totalPage` | number | Total number of pages available | # Response Headers When making requests via MrScraper's **Manual API** and **Web Fetching API**, essential metrics about your request execution are returned directly in the HTTP response headers. These response headers allow you to programmatically monitor resource consumption, response timing, and target page statuses without parsing separate metadata fields. Response header metrics (`token_usage`, `bandwidth_usage`, `runtime`, and `x-status-code`) are **only supported when using the Manual Scraper API and Web Fetching API**. ## Key Response Headers Every scraping response includes metadata headers detailing execution metrics: | Header Name | Type | Description | | :---------------- | :-------- | :------------------------------------------------------------------ | | `token_usage` | `number` | The total number of API tokens consumed by the scraping request. | | `bandwidth_usage` | `number` | The total data bandwidth consumed by the request (in Megabytes/MB). | | `runtime` | `float` | Total execution runtime for the request in seconds. | | `x-status-code` | `integer` | The HTTP status code returned by the target website being scraped. | Responses may also include `x-result-id` (a unique identifier for the execution result). ### Target Status Code (`x-status-code`) The `x-status-code` response header allows you to programmatically detect the status of the target page being scraped: * **Success (`2xx`)**: Indicates the target page was fetched successfully (e.g., `200 OK`). * **Forbidden / Blocked (`403`)**: Indicates access was blocked or denied by the target site. * **Page Not Found (`404`)**: Indicates the requested page was not found on the target site. * **Target Server Error (`500`)**: Indicates an internal error occurred on the target website's server. ## Example API Request & Response Headers ### cURL Example Below is an example request using the Manual / Web Fetching API: ```bash curl --location 'https://api.mrscraper.com?token=&geoCode=us&html=true&proxyCountry=us&url=https%3A%2F%2Fwww.scrapethissite.com' \ -H 'x-api-token: ' ``` ### Sample HTTP Response Headers When inspecting the headers returned by the server, you will receive output similar to the following: ```http HTTP/1.1 200 OK Server: nginx Content-Type: text/html; charset=utf-8 token_usage: 2 bandwidth_usage: 0.008867263793945312 runtime: 0.6612532138824463 x-status-code: 200 x-result-id: ``` ## Response Metrics in the Scraper API UI If you test your requests interactively inside **Scraper API**, these response header metrics are prominently displayed in the **Extracted Data** section toolbar: Response Headers in Scraper API UI

As highlighted in red on the screenshot above: * **`200 response`**: Corresponds to the target status code (`x-status-code`). * **`3,591 ms`**: Displays the request runtime (`runtime`). * **`2 tokens`**: Displays total tokens used for the request (`token_usage`). # Get Screenshot Render a page in a real browser and return a base64-encoded screenshot. Real browser rendering is enabled by default so client-side scripts and layout elements render before the capture. * Use the Scraper API Server (`https://api.mrscraper.com`) host when calling this endpoint. * Set `screenshot` to `full` for the entire scroll height, or to `top` for the visible viewport only. * Set `record` to `true` to also capture a full-page recording, returned in `recording_path`. * See [Get Screenshot](/docs/getting-started/scraper-api/get-screenshot) in the Scraper API guide for the full list of settings. # Listing Page Scrape a multi-page listing and return the repeating items from every page as one consolidated result. The `listing` agent navigates numbered pagination, infinite scroll, and "Load More" buttons up to `maxPages`. * Use the Scraper API Server (`https://api.mrscraper.com`) host when calling this endpoint. * `super` must be enabled for this use case. * For infinite scroll and "Load More", one page counts as one batch of loaded data. * For single-page extraction, use the [Scrape JSON](/docs/api/scraper-api/scrape-json) endpoint instead. * See [Listing Page](/docs/getting-started/scraper-api/listing-page) in the Scraper API guide for the full list of settings. # Scrape JSON Scrape a single page and return structured JSON. The `general` agent analyzes the page against your prompt and returns clean, strongly-typed data without fragile CSS selectors or custom XPath parsers. * Use the Scraper API Server (`https://api.mrscraper.com`) host when calling this endpoint. * The `prompt` only parses the scraped content. It cannot instruct the scraper itself. * To shape the output with a JSON schema, append it to the prompt as a `Json Schema:` section. * For multi-page catalogs, search results, and infinite-scroll listings, use the [Listing Page](/docs/api/scraper-api/listing-page) endpoint instead. * See [Scrape JSON](/docs/getting-started/scraper-api/scrape-json) in the Scraper API guide for the full list of settings. # Scrape Markdown Scrape a page and return its content as Markdown. Navigation menus, scripts, advertisements, and layout HTML are stripped while semantic structures such as headings, lists, code blocks, and links are preserved. * Use the Scraper API Server (`https://api.mrscraper.com`) host when calling this endpoint. * Set `markdown` to `true` to enable the conversion. * The output is typically 70-90% smaller than raw HTML, which makes it well suited for AI prompts, RAG pipelines, and document stores. * See [Scrape Markdown](/docs/getting-started/scraper-api/scrape-markdown) in the Scraper API guide for the full list of settings. # Scrape Sitemap Recursively traverse a site's link structure and return the discovered URLs. The `map` agent maps site architecture without downloading full page content payloads, which makes it ideal for generating seed URL lists before bulk extraction. * Use the Scraper API Server (`https://api.mrscraper.com`) host when calling this endpoint. * Control the crawl with `maxDepth`, `maxPages`, and `limit`. * Scope discovery with `includePatterns` and `excludePatterns`, one URL pattern per entry. * See [Scrape Sitemap](/docs/getting-started/scraper-api/scrape-sitemap) in the Scraper API guide for the full list of settings. # v3.4.14 import { Sparkles, TrendingUp } from 'lucide-react'; Version **v3.4.14** introduces a new **Agent Setup** page for AI workflows, and major enhancements to **Analytics** with per-use-case scoping, scraper type filtering, and token consumption tracking. ## New Features ### Agent Setup A new [**Agent Setup**](https://app.mrscraper.com/setup) page provides a centralized starting point for connecting MrScraper with supported AI coding agents and harnesses. It includes setup guidance and ready-to-use configuration details, making it easier to get an agent connected and start scraping from an AI-assisted workflow. ### Token Usage Analytics Chart [Analytics](https://app.mrscraper.com/analytic) now includes a dedicated **Token Usage** chart alongside scraping activity charts. It shows token consumption over time and respects the selected filters, including scraper type, domain, action, date range, and API token. The Token Usage chart has independent visualization and interval controls, making it easier to compare token-consumption patterns over time. ## Improvements & Updates ### Playground Analytics per Use Case Playground now includes an **Analytics** tab for each use case. Analytics automatically scopes data to the active scraper type, keeping metrics for Scrape HTML, Scrape JSON, Screenshot, and other use cases separate. The view focuses on the last 24 hours of activity and includes a link to the full [Analytics](https://app.mrscraper.com/analytic) page for deeper exploration. * [Scrape HTML](https://app.mrscraper.com/playground?useCase=unblocker) * [Scrape Markdown](https://app.mrscraper.com/playground?useCase=markdown) * [Get Screenshot](https://app.mrscraper.com/playground?useCase=screenshot) * [Scrape JSON](https://app.mrscraper.com/playground?useCase=extract) * [Listing Page](https://app.mrscraper.com/playground?useCase=listing) * [Scrape Sitemap](https://app.mrscraper.com/playground?useCase=sitemap) ### Scraper Type Filter in Analytics The [Analytics](https://app.mrscraper.com/analytic) dashboard now includes a **Scraper type** filter. Users can filter metrics and charts by the scraper type used, making it easier to analyze activity for a specific workflow. Available filter options are based on scraper result types present in the user’s account. ### New on Marketplace [Marketplace](https://app.mrscraper.com/marketplace) now have a **New on Marketplace** section on the bottom of the page for APIs published in the last 14 days. The new APIs also have a `NEW` tag on the top right of the card. # Workflow API Access import { Step, Steps } from 'fumadocs-ui/components/steps'; MrScraper lets you run any scraper you've created directly through an API endpoint. This is useful when you want to automate scraping, integrate it into your application, or test requests in tools like Postman. ## How to Run a Scraper Through the API ### Setup Steps Open your dashboard and select a scraper you've already created. If you don't have one yet, create it first. In the scraper window, click **Settings**. Select **AI Scraper API Access** and slide to activate it. This turns off the AI chat for this scraper and enables API mode. After activation, copy the example API request shown on the page. Paste the request into [Postman](https://www.postman.com/) (or another API tool). You can replace the URL with any URL you want to scrape. Hello

You can also run the scraper directly from [Rerun an AI Scraper endpoint](/docs/api/v3/scraper/ai/ai-rerun). ## API Request Parameters | Parameter | Type | Location | Required | Default | Description | | ------------- | ------- | -------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `x-api-token` | string | header | Yes | - | Your MrScraper API token | | `scraperId` | string | body | Yes | - | The unique ID of the scraper you activated | | `url` | string | body | Yes | - | The webpage URL you want to scrape | | `maxRetry` | integer | body | No | `3` | Number of retry attempts if the scraping fails | | `maxPages` | integer | body | No | 0 | Maximum number of pages to scrape (useful for listing/pagination). Use this to control how many pages you want to extract from multi-page listings | | `timeout` | integer | body | No | `None` | Request timeout in seconds. Increase this value if you're scraping many pages or complex websites | | `stream` | boolean | body | No | `false` | Enable streaming response mode. **Highly recommended when scraping multiple pages** that may take several minutes. If the connection is interrupted mid-process, you'll still receive the partial data that was already scraped from completed pages | If the website has fewer pages than maxPages, it will scrape all available pages. ## Example Request Body ### Basic Single-Page Scraper ```json { "scraperId": "1c9f4043-b572-462a-ab94-a330672f1af5", "url": "https://example.com/property", "maxRetry": 3, "maxPages": 1, "timeout": 300, "stream": false } ``` ### Listing Agent API (Multi-Page Scraper) When scraping listings or paginated content, use the `maxPages` parameter: ```json { "scraperId": "1c9f4043-b572-462a-ab94-a330672f1af5", "url": "https://www.zillow.com/ky/", "maxRetry": 3, "maxPages": 4, "timeout": 1100, "stream": false } ``` **Timeout Guidelines:** * For single-page scraping: 300-600 seconds is usually sufficient * For multi-page scraping: Increase timeout proportionally to `maxPages` (e.g., 1100+ seconds for 4+ pages) * If you encounter timeout errors, increase the `timeout` value ### Scraping with Streaming Response ```json { "scraperId": "your-scraper-id", "url": "https://example.com/search", "maxRetry": 3, "maxPages": 10, "timeout": 2000, "stream": true } ``` **When to Use `stream: true`:** * When scraping multiple pages (`maxPages` > 1) that may take several minutes * For long-running scraping jobs to prevent data loss * If connection interruptions occur mid-process, you'll still receive partial data from pages that were already scraped * Ensures you don't lose all progress if the request is interrupted before completion ## Common Use Cases ### Scraping a Single Product Page ```json { "scraperId": "your-scraper-id", "url": "https://shop.example.com/product/123", "maxRetry": 2 } ``` ### Scraping Real Estate Listings (Multiple Pages) ```json { "scraperId": "your-scraper-id", "url": "https://www.zillow.com/ky/", "maxRetry": 3, "maxPages": 10, "timeout": 2000, "stream": true } ``` For multi-page scraping, always set `stream: true` to ensure you receive data from completed pages even if the process is interrupted. ## Best Practices 1. **Start with lower `maxPages` values** and increase gradually to test performance 2. **Adjust timeout based on website complexity** and number of pages 3. **Use `stream: true` for multi-page scraping** to prevent data loss if the connection is interrupted 4. **Use appropriate retry values** (3 is recommended for most cases) 5. **Monitor your API usage** to stay within rate limits 6. **Test with a single page first** before running large batch operations ## Troubleshooting | Issue | Solution | | ----------------------------- | ------------------------------------------------------------------ | | Timeout errors | Increase the `timeout` parameter, especially when using `maxPages` | | Failed requests | Check `maxRetry` value and ensure your API token is valid | | Incomplete data from listings | Increase `maxPages` or verify the URL structure | | Rate limiting | Reduce concurrent requests or contact support for higher limits | If you encounter issues or need assistance with API integration, contact our support team or check the [API Reference](/docs/api/v3/scraper/ai/ai-rerun) for detailed endpoint documentation. # Bulk Scraping import { Step, Steps } from 'fumadocs-ui/components/steps'; MrScraper's Bulk Scraping feature allows you to extract data from multiple URLs in a single operation. Instead of scraping URLs one at a time, you can upload a list and apply the same scraping configuration to all of them, making it easy to gather large datasets efficiently. Scraping multiple pages with similar structures, such as product listings, article archives, user profiles, or directory entries. ## How to Configure Bulk Scraping ### Prepare Your URLs You have two options for providing URLs: #### Option A: Excel File Upload Create an Excel file (`.xlsx` or `.xls`) with a column header named `url` or `URL`. Each row should contain one URL. Your spreadsheet should look like this: | url | | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | [https://books.toscrape.com/catalogue/a-light-in-the-attic\_1000/index.html](https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html) | | [https://books.toscrape.com/catalogue/tipping-the-velvet\_999/index.html](https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html) | | [https://books.toscrape.com/catalogue/soumission\_998/index.html](https://books.toscrape.com/catalogue/soumission_998/index.html) | | [https://books.toscrape.com/catalogue/sharp-objects\_997/index.html](https://books.toscrape.com/catalogue/sharp-objects_997/index.html) | #### Option B: Direct Input Paste URLs directly into the text area, one URL per line: * [https://books.toscrape.com/catalogue/a-light-in-the-attic\_1000/index.html](https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html) * [https://books.toscrape.com/catalogue/tipping-the-velvet\_999/index.html](https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html) * [https://books.toscrape.com/catalogue/soumission\_998/index.html](https://books.toscrape.com/catalogue/soumission_998/index.html) * [https://books.toscrape.com/catalogue/sharp-objects\_997/index.html](https://books.toscrape.com/catalogue/sharp-objects_997/index.html) ### Upload Your URLs 1. Open a new or existing scraper in your MrScraper dashboard 2. Click the **Multiple URLs** button in the top section 3. Choose your input method: * Click **Upload File** to select an Excel file from your computer * Or paste URLs directly into the text area 4. Click **Save** to confirm your URL list ### Run the Bulk Scrape 1. Click **Run All** to start the bulk scraping process 2. Click Result to view the scraper’s progress and results in real time Each URL in your bulk scrape consumes tokens. Ensure you have sufficient tokens in your account before starting a large bulk operation. Check your token balance in your account settings. ### Monitor Progress The **Result** page displays real-time progress information: **While Scraping:** ```json title="In Progress" { "mergedData": null, "urlDetails": [], "summary": { "totalUrls": 4, "successfulUrls": 0, "failedUrls": 0, "scrapedCount": 0, "totalTokenUsage": 0, "estimatedFinishAt": null } } ``` **Progress Indicators:** | Field | Description | | ------------------- | -------------------------------------- | | `totalUrls` | Total number of URLs being scraped | | `successfulUrls` | Number of URLs scraped successfully | | `failedUrls` | Number of URLs that encountered errors | | `scrapedCount` | Current number of completed scrapes | | `totalTokenUsage` | Total tokens consumed so far | | `estimatedFinishAt` | Estimated completion time | Once scraping completes, the Result page displays all extracted data: ```json title="Completed Results" [ { "1": { "id": "a897fe39b1053632", "name": "A Light in the Attic", "price": "£51.77", "rating": null, "source": "product", "features": [ "Classic collection of poetry and drawings from Shel Silverstein", "20th anniversary special edition", "Humorous and creative verse" ], "return_policy": null, "shipping_info": { "availability": "In stock (22 available)", "shipping_arrival": null }, "specifications": { "UPC": "a897fe39b1053632", "Product Type": "Books", "Price (excl. tax)": "£51.77", "Price (incl. tax)": "£51.77", "Tax": "£0.00", "Availability": "In stock (22 available)", "Number of reviews": "0" } } }, { "1": { "id": "90fa61229261140a", "name": "Tipping the Velvet", "price": "£53.74", "rating": null, "source": "product", "features": [ "Erotic and absorbing...Written with starling power.", "Nan King, an oyster girl, is captivated by the music hall phenomenon Kitty Butler" ], "shipping_info": { "availability": "In stock (20 available)", "shipping_arrival": null }, "specifications": { "UPC": "90fa61229261140a", "Product Type": "Books", "Price (excl. tax)": "£53.74", "Price (incl. tax)": "£53.74", "Tax": "£0.00", "Availability": "In stock (20 available)", "Number of reviews": "0" } } } ] ``` For more details, refer to the [Bulk Scraping API](/docs/api/v3/scraper/ai/ai-rerun-bulk). ### Cancel Bulk Scraping You can cancel a bulk scraping job using either the **Scraper ID** or the **Result ID**. #### Option 1: Cancel by Scraper ID Cancel all pending URLs associated with a scraper. Use this when you want to stop the bulk operation immediately. ```bash curl -X PATCH "https://api.app.mrscraper.com/api/v1/scrapers-bulks/{scraperId}/cancel" \ -H "x-api-token: MRSCRAPER_API_KEY" ``` **Parameters:** * `{scraperId}` - The scraper ID used for the bulk operation #### Option 2: Cancel by Result ID Cancel a bulk scraping job using the bulk result ID returned from the bulk rerun response. This is useful when you have the result ID from the Result page and want to cancel that specific bulk operation. ```bash curl -X PATCH "https://api.app.mrscraper.com/api/v1/scrapers-bulks/result/{resultId}/cancel" \ -H "x-api-token: MRSCRAPER_API_KEY" ``` **Parameters:** * `{resultId}` - The bulk result ID from the bulk scraping response (e.g., `ec5d81b9-55fa-4949-bba9-cceb448cb950`) **Response:** Both endpoints return the bulk result ID, the original URL list, and which URLs were successfully canceled: ```json { "message": "Successful operation!", "data": { "id": "ec5d81b9-55fa-4949-bba9-cceb448cb950", "bulkUrls": [ "https://books.toscrape.com/catalogue/page-1.html", "https://books.toscrape.com/catalogue/page-2.html", "https://books.toscrape.com/catalogue/page-3.html", "https://books.toscrape.com/catalogue/page-4.html", "https://books.toscrape.com/catalogue/page-5.html" ], "canceledUrls": [ "https://books.toscrape.com/catalogue/page-3.html", "https://books.toscrape.com/catalogue/page-4.html", "https://books.toscrape.com/catalogue/page-5.html" ] } } ``` For detailed API documentation, see: * [Cancel Bulk Scraping by Scraper ID](/docs/api/v3/scraper/bulk/cancel-bulk-by-scraper-id) * [Cancel Bulk Scraping by Result ID](/docs/api/v3/scraper/bulk/cancel-bulk-by-result-id) # Cookie import { Step, Steps } from 'fumadocs-ui/components/steps'; Cookies are small pieces of data stored by your web browser that help websites remember your preferences and session information. In MrScraper, you can use cookies to access content that may be restricted or personalized based on your previous interactions with a website. Here's how to set up and use cookies in MrScraper: ## Obtaining Cookies You’ll first need to extract the cookies from your browser. There are a few ways to do this: ### Using Browser Developer Tools #### Chrome / Edge / Brave 1. Open your web browser and navigate to the website you want to scrape. 2. Open the developer tools (usually by pressing `F12` or right-clicking and selecting "Inspect"). 3. Go to the "Application" or "Storage" tab, then find the "Cookies" section. 4. Select the website's URL to view the cookies stored for that site. 5. Copy the cookies you need (usually as key-value pairs). #### Firefox 1. Open your web browser and navigate to the website you want to scrape. 2. Right click on the page and select "Inspect Element" to open the developer tools. 3. Go to the "Storage" tab, then find the "Cookies" section. 4. Copy the cookies you need (usually as key-value pairs). ### Using Browser Extensions You can also use browser extensions to quickly view, manage, and export cookies. Popular options include: * [EditThisCookie](https://chromewebstore.google.com/detail/editthiscookie-v3/ojfebgpkimhlhcblbalbfjblapadhbol) (Chrome) * [Cookie Quick Manager](https://addons.mozilla.org/en-US/firefox/addon/cookie-quick-manager/) (Firefox) * [Cookie Editor](https://cookie-editor.com/) ## Using Cookies in your Scraper 1. Navigate to your [MrScraper dashboard](https://app.mrscraper.com/). 2. click **Scraper** in the left sidebar, and click on the scraper you want to add cookies to. 3. Press the ellipsis (three dots) button on the top right corner of the scraper details page. 4. Select **Set Cookies Settings** from the dropdown menu. 5. Paste the JSON cookies you exported earlier into the provided field. ```json title="Example Cookie JSON" [ { "domain": "www.example.com", "expirationDate": 1764823423.823961, "hostOnly": true, "httpOnly": false, "name": "token", "path": "/", "sameSite": "strict", "secure": false, "session": false, "storeId": null, "value": "eyJhbGciOiJIUzI1NiJ9.eyJ1c2VybmFtZSI6ImtpemEiLCJleHAiOjE3NjQ4MjM0MjN9.qvviwvaTZDc9ey0iW638cdZr-EdiGHOfKsvTahZ_nIs" } ] ``` # Marketplace import { Step, Steps } from 'fumadocs-ui/components/steps'; import { Tab, Tabs } from 'fumadocs-ui/components/tabs'; The **MrScraper Marketplace** offers pre-configured, production-ready scrapers for popular websites like YouTube, Booking.com, Instagram, TikTok, LinkedIn, and Amazon. Selectors, proxies, and retries are handled automatically by MrScraper. ## Exploring the Scraper Interface Each scraper page in the [Marketplace](https://app.mrscraper.com/marketplace) includes three main tabs and a details sidebar: | UI Element | Description | | :-------------------- | :-------------------------------------------------------------------------------- | | **Run Tab** | Test and run the scraper directly in the browser with form inputs. | | **Output Schema Tab** | Preview the JSON schema and fields returned by the scraper. | | **Example Code Tab** | Copy ready-to-use request snippets in 8 languages with your API token pre-filled. | | **Details Sidebar** | View pricing (tokens/run), estimated latency, concurrency limits, and categories. | ## Run Directly from the Platform Run scrapers interactively from the dashboard without writing any code. ### Choose your scraper Go to the [Marketplace](https://app.mrscraper.com/marketplace) and select the scraper that fits your target website. ### Fill in input parameters In the **Run** tab, enter the required fields (such as target URLs, search terms, or date filters). You can check the **Output Schema** tab to review what data fields will be returned. ### Execute and view results Click **Run** to start the scrape. Once completed, inspect and copy the extracted JSON data directly in the dashboard. ## Run in Your App or Flow Every Marketplace scraper can be called directly via the Sync API (`https://sync.scraper.mrscraper.com`). You can copy pre-configured request snippets directly from the UI. ### Open the "Example Code" tab Navigate to the scraper page and switch to the **Example Code** tab. Your active API token (`atk_...`) is automatically injected into the code snippets and can also be copied from the token bar. ### Select your language and copy the snippet Select your language (**cURL**, **Python**, **JavaScript**, **PHP**, **Go**, **Java**, **Ruby**, or **C#**) and click copy. For example, a request to scrape a YouTube video: ```bash title="Request" curl --location --request POST 'https://sync.scraper.mrscraper.com/api/video/youtube/sync' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ "url": "https://www.youtube.com/watch?v=HeyIXwZyR8Y" }' ``` ```python title="Request" import requests response = requests.post( "https://sync.scraper.mrscraper.com/api/video/youtube/sync", headers={ "Authorization": "Bearer ", "Content-Type": "application/json" }, json={"url": "https://www.youtube.com/watch?v=HeyIXwZyR8Y"} ) print(response.json()) ``` ```javascript title="Request" const response = await fetch('https://sync.scraper.mrscraper.com/api/video/youtube/sync', { method: 'POST', headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' }, body: JSON.stringify({ url: 'https://www.youtube.com/watch?v=HeyIXwZyR8Y' }) }); const data = await response.json(); console.log(data); ``` ### Integrate into your application or workflow Paste the request into your backend service, script, or workflow tool. Replace the sample URL with your dynamic input variables. ### Handle the JSON response Marketplace scrapers run and return the parsed result: ```json title="Response" { "success": true, "message": "Successfully scraped", "data": { "ok": true, "video_id": "HeyIXwZyR8Y", "title": "MrScraper MCP is now Live", "author": "MrScraper", "author_url": "https://www.youtube.com/channel/UCYCwUmYcwIgCEUE9spiSV6A", "thumbnail": "https://i.ytimg.com/vi/HeyIXwZyR8Y/maxresdefault.jpg", "duration": 59, "download_url": "https://mrscraper-files.s3.us-west-2.amazonaws.com/public/videos/HeyIXwZyR8Y_480p.mp4", "thumbnails": { "low": "https://i.ytimg.com/vi/HeyIXwZyR8Y/default.jpg", "max": "https://i.ytimg.com/vi/HeyIXwZyR8Y/maxresdefault.jpg" }, "formats": [ { "type": "video", "format": "144p", "filesize": 1287199 }, { "type": "video", "format": "360p", "filesize": 1889026 }, { "type": "video", "format": "720p", "filesize": 5029353 }, { "type": "audio", "format": "audio", "filesize": 960275 } ] }, "tokenUsage": 5 } ``` ## Sync & Async Marketplace scrapers run synchronously by default: you send a request and the extracted data comes back in the same response. Some scrapers also have an asynchronous endpoint, such as [SERP API - Web Results](https://app.mrscraper.com/marketplace/91d619cd-3f19-4240-9540-f1adaf5185d7). Use the sync endpoint by default. Switch to async when a request takes too long to return in a single call. The async endpoint queues the job and responds immediately with a task ID, so your request never waits for the scrape to finish. | | Sync | Async | | ------------ | ------------------------------------ | --------------------------------------- | | **Host** | `https://sync.scraper.mrscraper.com` | `https://async.scraper.mrscraper.com` | | **Returns** | The extracted data | A `taskId` you use to retrieve the data | | **Best for** | Requests that complete quickly | Long-running requests | ### Run a scraper asynchronously ### Create the task Send your request to the async endpoint. Set `callbackUrl` to receive the result as a webhook once the scrape finishes, so you don't have to poll for it. ```bash title="Request" curl --location --request POST 'https://async.scraper.mrscraper.com/api/serp' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ "query": "mrscraper", "region": "us", "language": "en", "page": 1, "format": "html", "renderJs": false, "callbackUrl": "https://async.scraper.mrscraper.com/webhook" }' ``` The response returns the `taskId` to retrieve the result with, along with the parameters the task was created from. ```json title="Response" { "success": true, "message": "Task created successfully", "data": { "taskId": 603086110, "params": { "query": "mrscraper", "region": "us", "language": "en", "page": 1, "format": "html", "renderJs": false, "callbackUrl": "https://async.scraper.mrscraper.com/webhook" } } } ``` ### Retrieve the result Pass the `taskId` to the result endpoint. ```bash title="Request" curl --location --request GET 'https://async.scraper.mrscraper.com/api/scraper/result/603086110' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' ``` While the scrape is still running, the response reports `PENDING`. Keep requesting the result until the task completes. ```json title="Response (Pending)" { "success": false, "message": "PENDING", "data": null } ``` Once the task finishes, the response contains the scraped data. ```json title="Response (Finished)" { "success": true, "message": "Successfully scraped", "data": { "html": "mrscraper - Google Search..." }, "tokenUsage": 1 } ``` ## Operational Details ### Token Usage & Pricing * Token cost per run is shown in the **Details** sidebar (e.g., `5 tokens/run`). * Tokens are only deducted when a run completes successfully. ### Concurrency Limits * The **Max Concurrency** indicator defines how many parallel requests you can run at once. * Requests exceeding your limit may return HTTP `429 Too Many Requests`. Concurrency can be upgraded from your billing settings. # Proxy import { Step, Steps } from 'fumadocs-ui/components/steps'; **Proxy Settings** let you attach a proxy to a specific scraper so that every time that scraper runs — manually, via schedule, or through the API — its requests are routed through the configured proxy. This is useful when you need to: * Mask your scraper's IP address to avoid detection or rate-limiting * Access region-locked or geo-restricted content from a specific country * Route traffic through your own private proxy infrastructure **Proxy Settings** (this page) is a configuration option inside an existing scraper. You attach a proxy so the scraper's requests are routed through it. The scraper defines *what* to scrape; the proxy setting controls *how* the traffic is routed. The **Proxy Scraper** (covered in [Proxy Scraper](/docs/residential-proxy/getting-started/proxy-scraper)) is a standalone tool on the Proxies page. You provide a URL and a proxy country, and MrScraper handles both the scraping and the proxying together. ## How to Configure Proxy Settings ### Open Your Scraper Settings 1. Select the scraper you want to configure on the [**Workflow**](https://app.mrscraper.com/workflow) page. 2. Click the **Ellipsis** ( **⋮** ) button in the top right corner and click **Proxy Settings**. ### Enable and Configure Proxy In the **Proxy Settings** panel, you can select one of the following options: #### Option 1: Use MrScraper's Built-in Proxy 1. Toggle **Enable Proxy**. 2. Under **Proxy Type**, choose **MrScraper Proxy**. 3. Select your desired **location/region** from the dropdown list. * Each region provides a local IP address, ideal for geo-targeted scraping. 4. Click **Save Settings**. The scraper will now route traffic through the selected country's proxy network, allowing access to region-specific or restricted content. #### Option 2: Use Custom Proxy Only use proxies from trusted providers to ensure your data remains secure. 1. Toggle **Enable Proxy**. 2. Under **Proxy Type**, choose **Custom Proxy**. 3. Enter your proxy URL in one of the following formats: `http://username:password@host:port` or `https://host:port`. 4. Click **Save Settings** to apply your configuration. Use your own proxy if you have an existing provider, private IP pool, or dedicated proxy infrastructure. #### Route the Entire Request Through the Proxy 1. Toggle **Route entire request through proxy**. This routes page subresources—including CSS, JavaScript, and images—through the proxy so the page loads fully and scripts run reliably. 2. Click **Save Settings** to apply your configuration. Try scraping with this option turned off first to minimize bandwidth usage. If the scrape fails or the page does not load correctly, turn it on. Some websites block CSS, JavaScript, images, or other subresources unless they are also routed through the proxy. Enabling this option can improve reliability, but it uses more bandwidth. ## MrScraper Built-in Proxy Countries Below is the full list of supported countries and regions available for MrScraper's built-in proxies: | Country/Region Name | Country/Region Name | Country/Region Name | Country/Region Name | | --------------------------------- | -------------------------------------------- | -------------------------------------------- | ------------------------------------ | | Afghanistan | Albania | Algeria | American Samoa | | Andorra | Angola | Anguilla | Antarctica | | Antigua and Barbuda | Argentina | Armenia | Aruba | | Australia | Austria | Azerbaijan | Bahamas | | Bahrain | Bangladesh | Barbados | Belarus | | Belgium | Belize | Benin | Bermuda | | Bhutan | Bolivia (Plurinational State of) | Bonaire, Sint Eustatius and Saba | Bosnia and Herzegovina | | Botswana | Bouvet Island | Brazil | British Indian Ocean Territory | | Brunei Darussalam | Bulgaria | Burkina Faso | Burundi | | Cambodia | Cameroon | Canada | Cape Verde | | Cayman Islands | Central African Republic | Chad | Chile | | China | Christmas Island | Cocos (Keeling) Islands | Colombia | | Comoros | Congo | Congo (Democratic Republic of the) | Cook Islands | | Costa Rica | Croatia | Cuba | Curaçao | | Cyprus | Czech Republic | Côte d'Ivoire | Denmark | | Djibouti | Dominica | Dominican Republic | Ecuador | | Egypt | El Salvador | Equatorial Guinea | Eritrea | | Estonia | Eswatini | Ethiopia | Falkland Islands (Malvinas) | | Faroe Islands | Fiji | Finland | France | | French Guiana | French Polynesia | French Southern Territories | Gabon | | Gambia | Georgia | Germany | Ghana | | Gibraltar | Greece | Greenland | Grenada | | Guadeloupe | Guam | Guatemala | Guernsey | | Guinea | Guinea-Bissau | Guyana | Haiti | | Heard Island and McDonald Islands | Holy See (Vatican City State) | Honduras | Hong Kong | | Hungary | Iceland | India | Indonesia | | Iran (Islamic Republic of) | Iraq | Ireland | Isle of Man | | Israel | Italy | Jamaica | Japan | | Jersey | Jordan | Kazakhstan | Kenya | | Kiribati | Korea (Democratic People's Republic of) | Korea (Republic of) | Kuwait | | Kyrgyzstan | Lao People's Democratic Republic | Latvia | Lebanon | | Lesotho | Liberia | Libya | Liechtenstein | | Lithuania | Luxembourg | Macao | North Macedonia | | Madagascar | Malawi | Malaysia | Maldives | | Mali | Malta | Marshall Islands | Martinique | | Mauritania | Mauritius | Mayotte | Mexico | | Micronesia (Federated States of) | Moldova (Republic of) | Monaco | Mongolia | | Montenegro | Montserrat | Morocco | Mozambique | | Myanmar | Namibia | Nauru | Nepal | | Netherlands | New Caledonia | New Zealand | Nicaragua | | Niger | Nigeria | Niue | Norfolk Island | | Northern Mariana Islands | Norway | Oman | Pakistan | | Palau | Palestine, State of | Panama | Papua New Guinea | | Paraguay | Peru | Philippines | Pitcairn | | Poland | Portugal | Puerto Rico | Qatar | | Romania | Russian Federation | Rwanda | Réunion | | Saint Barthélemy | Saint Helena, Ascension and Tristan da Cunha | Saint Kitts and Nevis | Saint Lucia | | Saint Martin (French part) | Saint Pierre and Miquelon | Saint Vincent and the Grenadines | Samoa | | San Marino | Sao Tome and Principe | Saudi Arabia | Senegal | | Serbia | Seychelles | Sierra Leone | Singapore | | Sint Maarten (Dutch part) | Slovakia | Slovenia | Solomon Islands | | Somalia | South Africa | South Georgia and the South Sandwich Islands | South Sudan | | Spain | Sri Lanka | Sudan | Suriname | | Svalbard and Jan Mayen | Sweden | Switzerland | Syrian Arab Republic | | Taiwan (Province of China) | Tajikistan | Tanzania, United Republic of | Thailand | | Timor-Leste | Togo | Tokelau | Tonga | | Trinidad and Tobago | Tunisia | Turkey | Turkmenistan | | Turks and Caicos Islands | Tuvalu | Uganda | Ukraine | | United Arab Emirates | United Kingdom | United States | United States Minor Outlying Islands | | Uruguay | Uzbekistan | Vanuatu | Venezuela (Bolivarian Republic of) | | Viet Nam | Virgin Islands (British) | Virgin Islands (U.S.) | Wallis and Futuna | | Western Sahara | Yemen | Zambia | Zimbabwe | | Åland Islands | | | | Use built-in proxies when you need to scrape data from region-locked websites or compare results between countries (for example, e-commerce prices or job listings). # S3 Storage import { Step, Steps } from 'fumadocs-ui/components/steps'; **S3 Storage** saves your scraper results as files instead of returning all the data in each API response. When you enable it, MrScraper uploads each result to S3 storage and returns a signed URL you can use to download the file. ## Why Use S3 Storage Returning a large result directly in an API response can be slow and may exceed response size limits. S3 Storage helps you: * **Keep a durable copy.** Store each result as a file you can download or reprocess later. * **Fit results into a pipeline.** Pass the signed URL to another service, or queue results to process in a separate step. ## Enable S3 Storage To save your scraped data to an S3 bucket, follow these steps: Open your scraper detail page Click the **Ellipsis** ( **⋮** ) button in the top right corner and click **Result Storage**. Toggle the **Enable Result Storage** option. Results are stored as JSON files matching your scraper's output format. The signed URL is returned in the data field of each result. Save your settings. The scraper will returns a signed URL: ```json title="Example Result" { "value": "https://api.app.mrscraper.com/api/v1/attachments/results/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxxxx.json" } ``` # Schedule import { Step, Steps } from 'fumadocs-ui/components/steps'; Schedule allows you to automate the execution of your scrapers at predefined times or intervals. This ensures that your data is always up-to-date without manual intervention. Schedule is useful in scenarios such as: * **Regular Data Updates:** If you need to keep your data fresh, such as daily price monitoring or weekly news aggregation. * **Time-Sensitive Data:** For data that changes frequently, like stock prices. * **Resource Management:** To run scrapers during off-peak hours to optimize resource usage and reduce costs. ## Set up a schedule Follow these steps to schedule a scraper: On the left sidebar, select **Workflow** to open the **Workflow** page. Select the scraper you want to schedule. Click the **Ellipsis** ( **⋮** ) button in the top right corner and click **Schedule**. In the **Run Frequency** dropdown, choose how often the scraper should run: * **Every Hour** * **Daily** * **Weekly** * **Custom**: Build your own schedule from one or more intervals (see below). Select **Save Schedule** to apply your settings. Once you save the schedule, the scraper runs automatically. You can view its results on the **Results** page. To change or stop a schedule, return to the scraper's **Schedule** settings. ## Set a custom interval Choose **Custom** when the standard options (hourly, daily, weekly) don't match the frequency you need. A custom schedule is built from **schedule rows**. Each row holds one interval, and you can add several rows to run a scraper at uneven gaps. To set a custom interval: In the **Run Frequency** dropdown, select **Custom**. Under **Schedule Rows**, select a number in the first dropdown, then choose **Minutes**, **Hours**, or **Days** in the second. If you choose **Days**, also set the time of day the scraper should run, such as **13:16**. To add another interval, select **Add Row** and set its number and unit. Repeat for each interval you want in the cycle. Check the **Schedule Summary** below the rows. It shows how often the scraper will run. The time zone used for the schedule appears under the **Run Frequency** dropdown, for example *Times shown in Asia/Jakarta (UTC+7)*. Select **Save Schedule** to apply your settings. ### Use a single interval With one schedule row, the scraper runs once every interval you set. Use this method to calculate any custom interval: ``` Interval = Total minutes in the period ÷ Number of runs you want ``` For example, to run a scraper 3 times per hour: 60 ÷ 3 = 20. Set the interval to **20 Minutes**. 20 minutes interval

### Use multiple intervals With more than one schedule row, MrScraper cycles through the rows in order. After each run, it waits the next row's interval, then starts again from the first row once it reaches the end of the list. For example, with two rows set to **20 Minutes** and **10 Minutes**, the scraper runs at minutes 20, 30, 50, and 60 of each hour: | Run | Interval used | Runs at | | --- | ------------- | --------- | | 1 | 20 minutes | Minute 20 | | 2 | 10 minutes | Minute 30 | | 3 | 20 minutes | Minute 50 | | 4 | 10 minutes | Minute 60 | The cycle then repeats in the next hour. Use multiple rows when you want runs clustered around certain points in the hour or day instead of spread evenly. # Team import { Accordion, Accordions } from 'fumadocs-ui/components/accordion'; import { Step, Steps } from 'fumadocs-ui/components/steps'; The Team feature in MrScraper allows you to collaborate with others by sharing scrapers, results, subscription access, and token usage across your organization. ## What You Can Share When you create a team, all members have access to: * **Scrapers**: All configured scrapers in your account * **Results**: Scraping results and historical data * **Subscription**: Your plan features and limits * **Token Usage**: Shared token allocation across the team ## Creating a Team ### Create Your Team 1. Click your account profile in the top right corner of the MrScraper dashboard 2. Select **My Team** from the dropdown menu 3. Enter your team details: * **Team Name** (required) * **Description** (optional) 4. Click **Create Team** You'll be redirected to the team details page where you can manage members and settings. All scrapers and results are automatically shared with team members once they join. ### Invite Team Members 1. On the team details page, click **Invite Member** 2. Enter the member's information: * **Email address** (required) * **Role** (see [Role Capabilities](#role-capabilities) below) 3. Click **Send Invite** The invited member will receive an email invitation to join your team. All team members can access the same MrScraper features as the team owner. ## Role Capabilities Team roles determine what administrative actions members can perform. All roles have full access to MrScraper features for scraping and data collection. | Role | All Features Access | Invite Members | Change Member Roles | Remove Members | Delete Team | | --------------- | ------------------- | -------------- | ------------------- | -------------- | ----------- | | **Team Owner** | ✅ | ✅ | ✅ | ✅ | ✅ | | **Team Admin** | ✅ | ✅ | ✅ | ✅ | ❌ | | **Team Member** | ✅ | ❌ | ❌ | ❌ | ❌ | ### Role Details **Team Owner** * Full administrative control over the team * Can perform all actions including deleting the team * Only one owner per team (the account that created the team) **Team Admin** * Can manage team membership and permissions * Cannot delete the team * Ideal for trusted administrators who help manage the team **Team Member** * Can use all MrScraper features (scrapers, results, etc.) * Cannot perform administrative actions * Best for users who only need access to scrapers and data ## Managing Your Team ### Changing Member Roles Navigate to **My Team** from your account profile Find the member you want to update Click the role dropdown next to their name Select the new role Only Team Owners and Team Admins can change member roles. ### Removing Team Members Navigate to **My Team** from your account profile Find the member you want to remove Click the **Remove** button next to their name Confirm the removal Only Team Owners and Team Admins can remove members. ### Deleting a Team Navigate to **My Team** from your account profile Click **Team Settings** or **Delete Team** Confirm the deletion Deleting a team is permanent and cannot be undone. Only Team Owners can delete teams. ## Frequently Asked Questions Team ownership is tied to the account that created the team. Contact support if you need to transfer ownership. Scrapers remain with the team and are not affected when members leave. Only the member's access is revoked. Team size limits depend on your subscription plan. Check your plan details or contact support for more information. # Web Fetching import { Step, Steps } from 'fumadocs-ui/components/steps'; MrScraper's Web Fetching retrieves the HTML of public web pages, renders JavaScript-heavy websites, and returns region-specific content. It combines selectable non-browser or browser loading, proxy routing, optional real-device Super Mode, and intelligent retry mechanisms for reliable data extraction. ## Key Features ### Browser Rendering Execute JavaScript and render dynamic content just like a real browser. **Use Cases:** * Single Page Applications (SPAs) built with React, Vue, or Angular * Websites that load content via AJAX * Pages with lazy-loaded images or infinite scroll * Dynamic pricing or availability information * Client-side rendered content ### Super Mode Route requests through real devices when a website needs a different routing path. Super Mode is independent of browser rendering: it can use either the non-browser loader or browser loading with JavaScript. **Use Cases:** * Websites that fail to load through standard routing * Pages that return different content through real-device routing ### Independent Loading Paths Browser rendering and Super Mode are two independent switches. Every combination is supported, and the same URL can return different results in each: | Browser rendering | Super Mode | Loading path | | ----------------- | ---------- | -------------------------------------------------------- | | Off | Off | Standard routing with the non-browser loader. | | On | Off | Standard routing with browser loading and JavaScript. | | Off | On | Real-device routing with the non-browser loader. | | On | On | Real-device routing with browser loading and JavaScript. | Start with both off and change one switch at a time after inspecting the response. Browser rendering is not a strictly stronger option. Some websites fail or return worse content with browser rendering enabled but load correctly through the non-browser path. If that happens, retry with browser rendering off while preserving the current Super Mode value. Stop after a usable response unless you need to compare loading paths. ### Geo-Targeting with Proxies Retrieve region-specific content by routing requests through residential proxies in specific countries. **Use Cases:** * Scraping region-specific pricing * Testing localized versions of websites * Market research across different regions View the full list of supported countries on the [Country Codes](/docs/residential-proxy/configuration/country-codes) page. ### Smart Waiting Wait for specific elements to appear in the DOM before extracting content. **Use Cases:** * Waiting for lazy-loaded product information * Ensuring dynamic filters have applied * Waiting for search results to populate * Synchronizing with page animations * Handling delayed content rendering ### Homepage Navigation Navigate through a website's home page before visiting the target URL. **Use Cases:** * Websites that expect visitors to arrive from the homepage * Pages that set required cookies or session state on the homepage ### Intelligent Retry Control Automatically retry failed scrapes while limiting retry costs with configurable retry settings. **Use Cases:** * Recovering from temporary network failures * Handling intermittent server errors * Preventing excessive token usage with retry limits * Balancing reliability and scraping cost ### Resource Blocking Block non-essential resources such as images, fonts, and media to improve scraping speed and reduce bandwidth usage. **Use Cases:** * Faster page rendering * Lower bandwidth consumption * Scraping text-heavy pages * Large-scale scraping jobs ### Configurable Timeout Control how long to wait for page loading and rendering. **Use Cases:** * Slow-loading websites * Complex pages with many resources * Sites with large media files * Pages with extensive JavaScript execution * Unreliable network conditions - **Simple pages**: 30 seconds (default) - **Standard e-commerce**: 60 seconds - **Heavy SPAs**: 90-120 seconds - **Very complex pages**: Up to 180 seconds ## How It Works Web Fetching follows this process: Your request is routed through residential proxies in the specified region (if geo-targeting is enabled) If Super Mode is enabled, the request is routed through a real device The selected loader runs: a browser when browser rendering is enabled, or the non-browser loader when it is disabled When browser rendering is enabled, the browser navigates to the target URL and begins rendering When browser rendering is enabled, page JavaScript executes, including AJAX calls and dynamic content loading If specified, the scraper waits for the target CSS selector to appear in the DOM The process completes when the selector appears or the timeout is reached The loaded HTML is extracted and returned Loader resources are released ## Example Request Example request for scraping using the [web fetching](/docs/api/v3/scraper/web-fetching-scraping) endpoint: ```bash curl -X GET "https://api.mrscraper.com/?url=https%3A%2F%2Fwww.amazon.com%2Fstores%2Fluxurystores%2Fpage%2FB6BC6264-7221-424B-9191-DAE2BCF963A2&token=atk_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&timeout=120&geoCode=pt&browserRendering=true&super=true&waitForSelector=div%5Bdata-testid%3D%22grid-item-info%22%5D&homePage=false&blockResources=true&maxRetries=3&tokenCap=10" \ -H "x-api-token: MRSCRAPER_API_KEY" ``` ### Parameters | Parameter | Required | Default | Description | | ------------------ | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `url` | Yes | - | Target website URL to scrape. | | `token` | Yes | - | Your API authentication token. | | `browserRendering` | No | `false` | Render the page in a headless browser. | | `super` | No | `false` | Route the request through a real device. | | `geoCode` | No | - | ISO 3166-1 alpha-2 country code used for proxy routing. | | `timeout` | No | `30` | Maximum number of seconds to wait for page loading. | | `waitForSelector` | No | - | CSS selector to wait for before returning the page content. Requires `browserRendering=true`. | | `homePage` | No | `false` | Visit the website's home page before navigating to the target URL. | | `blockResources` | No | - | Block non-essential resources to improve performance. Supported only by compatible proxies. | | `maxRetries` | No | `3` | Maximum number of retry attempts if the scrape fails. | | `tokenCap` | No | - | Maximum number of tokens that can be consumed across retry attempts. If the first scrape exceeds the cap, it still runs once but no retries are performed. | * `waitForSelector` only works when `browserRendering=true`. * Start without `super` and enable it when standard routing does not return usable content. * `blockResources` is only supported by proxies that support resource blocking. * `tokenCap` only affects retry attempts. The initial scrape always runs, even if it exceeds the configured token cap. # Token Plan import { Step, Steps } from 'fumadocs-ui/components/steps'; MrScraper uses a **token-based system** for both usage and authentication. * **Plan Tokens** measure how much you can use MrScraper's features, such as running or rerunning scrapers. * **API Tokens** allow you to securely access the MrScraper API, for example, to rerun an existing scraper programmatically. ## Why Token Usage Varies Token consumption isn't fixed — a few key factors affect how many tokens any given scraper uses: * **Website Complexity**: Heavy JavaScript, dynamic content, or multiple requests increase runtime and token usage. * **Page Size and Content Density**: Larger pages with long articles or extensive metadata produce more input text for AI models to process. * **Data Volume**: Extracting large datasets (e.g., paginated listings) requires more compute and bandwidth. * **AI Processing Load**: Complex instructions or large expected outputs increase input and output token usage. Keep these in mind as you read the pricing tables below — they explain why two runs on the same plan can consume different amounts of tokens. ## Understanding Plan Tokens Each MrScraper billing plan includes a specific number of **tokens**. Tokens represent the compute resources your scraper consumes — similar to how minutes or data are counted in a phone plan. The number of tokens used depends on the type of scraper you run. ### AI Scraper Token Usage AI Scrapers consume more tokens due to AI processing and additional factors such as: | Usage Type | Description | Conversion Rate | | ----------------- | ----------------------------------------------------------------------------------------------------------------- | --------------------------------- | | **Run Tracking** | Each AI agent run consumes 5 tokens for trace tracking. Reruns and workflows consume 0 tokens for this component. | 0 or 5 tokens per run | | **Runtime** | Time spent processing your task. | 1 token per 30 seconds of runtime | | **Input Tokens** | Amount of text data (prompts or instructions) sent to the AI model. | 1 token per \~1,000 input tokens | | **Output Tokens** | Amount of text generated by the AI model (responses or extracted results). | 1 token per \~200 output tokens | #### Example If your AI scraper runs for **90 seconds**, processes **5,000 input tokens**, and generates **1,000 output tokens**, your total token usage would be: | Component | Calculation | Tokens | | --------- | ------------- | ------ | | Runtime | 90 ÷ 30 | 3 | | Input | 5,000 ÷ 1,000 | 5 | | Output | 1,000 ÷ 200 | 5 | | Run Trace | Fixed | 5 | | **Total** | | **18** | * Each run automatically includes trace tracking for debugging and performance monitoring. * Input and output tokens are based on how much text the AI model processes and generates. * AI Scrapers tend to consume more tokens due to model processing. * Runtime tokens are rounded up using the ceiling rule. For example, if your scraper runs for 40 seconds, it will count as 2 tokens (not 1). ### Workflow and Web Fetching Token Usage Workflows and Web Fetching consume tokens based on two factors: **runtime** and **bandwidth**. | Usage Type | Description | Conversion Rate | | ------------- | -------------------------------------- | ---------------------- | | **Runtime** | Time your scraper runs. | 1 token per 30 seconds | | **Bandwidth** | Amount of data downloaded or uploaded. | 1 token per 0.25 MB | #### Example If your workflow or web fetching runs for **40 seconds** and uses **1.1 MB** of bandwidth: | Component | Calculation | Tokens | | --------- | ----------------------------- | ------ | | Runtime | 40 ÷ 30 = 1.33 → rounded up | 2 | | Bandwidth | 1.1 ÷ 0.25 = 4.4 → rounded up | 5 | | **Total** | | **7** | Runtime tokens are rounded up using the ceiling rule. For example, if your scraper runs for 40 seconds, it will count as 2 tokens (not 1). ### Scraper API Token Cap The **Scraper API token cap** limits how many tokens a single scrape can use in the Scraper API. It applies only to **Workflow** and **Web Fetching**, using the same runtime and bandwidth calculations shown in [Workflow and Web Fetching Token Usage](#workflow-and-web-fetching-token-usage). When **Retry** is enabled, a failed scrape retries automatically. Two limits control how far retrying goes: * **Max retries**: The maximum number of retry attempts (3 by default). * **Token cap**: The maximum total tokens all attempts can use (Unlimited by default). Retrying stops at the first of these: the scrape succeeds, it reaches Max retries, or the running token total reaches the token cap. Each attempt's token cost is rounded up (ceiling rule) and added to a running total. The token cap takes effect only when Retry is enabled, since its purpose is to limit how many tokens retries can consume. If Retry is off, the scrape runs once and returns a result or an error. #### Example Max retries **3**, token cap **10**. | Attempt | Runtime + Bandwidth | Rounded-up Cost | Running Total | Result | | ----------- | ------------------- | --------------- | ------------- | ---------------- | | 1 (initial) | 2.4 tokens | 3 | 3 | Failed, retrying | | 2 (retry 1) | 1.6 tokens | 2 | 5 | Succeeded | The scrape succeeds on the second attempt, using 5 tokens. Because it succeeded, the Scraper API returns the result and stops even though max retries and tokens still remain (5 of 10 tokens left, and 2 retries unused). Max retries **2**, token cap **20**. | Attempt | Runtime + Bandwidth | Rounded-up Cost | Running Total | Result | | ----------- | ------------------- | --------------- | ------------- | --------------------------- | | 1 (initial) | 3.6 tokens | 4 | 4 | Failed, retrying | | 2 (retry 1) | 3.8 tokens | 4 | 8 | Failed, retrying | | 3 (retry 2) | 3.4 tokens | 4 | 12 | Failed, max retries reached | The scrape uses both retries without succeeding, so it stops even though 8 tokens still remain under the 20-token cap. Max retries **3**, token cap **12**, about 4 tokens per run. | Attempt | Runtime + Bandwidth | Rounded-up Cost | Running Total | Result | | ----------- | ------------------- | --------------- | ------------- | ------------------- | | 1 (initial) | 3.5 tokens | 4 | 4 | Failed, retrying | | 2 (retry 1) | 3.9 tokens | 4 | 8 | Failed, retrying | | 3 (retry 2) | 3.7 tokens | 4 | 12 | Failed, cap reached | The running total reaches the 12-token cap on the second retry, so the scrape stops before using all 3 retries. If the first attempt costs more than the cap (for example, a cap of 10 but a first scrape that needs 15 tokens), the scrape still runs once but doesn't retry. It returns either a result or an error. Set the token cap high enough to allow for retries, especially on pages with variable load times or bandwidth. If a scrape hits the cap before completing, try increasing the cap. ### Marketplace Scraper Token Usage Marketplace Scrapers use a custom token cost based on factors such as the target domain, workflow complexity, proxy usage, infrastructure requirements, and scraper configuration. Each scraper may use either a fixed token cost or a custom usage calculation depending on how the scraper is designed. For the complete list of Marketplace Scraper token costs, see the [Marketplace](/docs/features/marketplace) catalog. # Billing import { Accordion, Accordions } from 'fumadocs-ui/components/accordion'; import { Step, Steps } from 'fumadocs-ui/components/steps'; MrScraper offers flexible subscription plans designed for different usage levels, from individual developers testing the platform to enterprise teams running large-scale scraping operations. All plans include access to our AI-powered scraping agents, browser rendering, and residential proxies (on applicable plans). ## Available Plans | Feature | Scraper Free | Scraper Pro | Scraper Enterprise | | ----------------------------- | ------------ | ------------- | ------------------ | | **API Tokens** | 1,000 token | 200,000 token | Custom | | **Concurrent Requests** | 10 | 100 | Custom | | **Residential Proxies** | ❌ | ✅ | ✅ | | **Priority Email Support** | ❌ | ✅ | ✅ | | **Dedicated Account Manager** | ❌ | ✅ | ✅ | The Free plan includes **100 tokens per month** with no credit card required. Tokens reset at the beginning of each month. Once your allocation is exhausted, you'll need to upgrade to continue scraping. ## Transaction Page You can view your past transactions, by going to **Account** → **Subscription**. The transaction page displays a list of all your billing activities, including: * Product * Amount (Price) * Status * Date * Invoice and Receipt links ### Invoices and Receipts To access the invoices or receipts of your account: Go to **Account Profile** → **Subscription** You'll be shown with a list of your billing history, including the date, amount, and status of each transaction Under **Invoice**, and **Receipt** columns, you can click on the respective links to view or download the invoice or receipt for each transaction. ## Manage Billing In the Subscription page, you can click on the **Manage Billing** button to open the Stripe customer portal. From there, you can: * View your payment history * Cancel your subscription plan * Edit your billing information ## Billing Cycle ### How Token Allocation Works **Monthly Plans:** * Tokens reset on your monthly renewal date * Full token allotment available at the start of each cycle * Unused tokens do **not** roll over to the next month **Annual Plans:** * Billed once per year * Tokens still reset **monthly** on your renewal date * 20% discount on total cost * Unused tokens do **not** roll over ### Example Timeline If you subscribe to Scraper Pro on **January 15th**: | Date | Event | Token Balance | | ------ | -------------------- | ---------------------- | | Jan 15 | Subscription starts | 200,000 tokens | | Jan 31 | Used 60,000 tokens | 140,000 tokens | | Feb 15 | Billing cycle renews | 200,000 tokens (reset) | Unused tokens from the previous month are **lost** when your billing cycle renews. Plan your scraping operations accordingly to maximize your token allocation. ## Tracking Your Usage Monitor your token consumption and billing information through multiple channels: * Dashboard: View real-time usage in your [account API token dashboard](https://app.mrscraper.com/api-tokens) * API Endpoints: Track usage programmatically using our [Analytics API](/docs/api/v3/analytic/status) ## FAQs No, unused tokens do **not** roll over. Your token allocation resets to your plan amount at the start of each billing cycle. This is why it's important to choose a plan that matches your typical monthly usage. Token usage varies by: * Page complexity * Scraping mode (Cheap vs Super) * Agent type (General, Listing, Map) * Additional features (AI extraction, etc.) Your scrapers remain intact and accessible. However: * You'll have fewer tokens to run them * Concurrent request limits may decrease * Some features (like residential proxies) may become unavailable Yes! Contact [support@mrscraper.com](mailto:support@mrscraper.com) to switch billing cycles. When switching to annual billing: * You'll be charged for a full year upfront * Receive a 20% discount * Any remaining time on your monthly subscription will be credited The Free plan serves as a trial with 100 monthly tokens. Test MrScraper's features before committing to a paid plan. No credit card required. # CLI The MrScraper CLI provides terminal access to page retrieval, web search, and account usage. This guide covers installation, verification, authentication, output formats, environment variables, programmatic usage, and complete command details. ## Requirements * Node.js 20 or newer * A MrScraper account for browser login, or an API key for non-interactive use This page documents version **0.6.0**. See [@mrscraper/cli on npm](https://www.npmjs.com/package/@mrscraper/cli) for the current published version. ## Installation Install only the CLI globally: ```bash npm install -g @mrscraper/cli@latest mrscraper --version ``` Install the CLI and all four MrScraper agent skills for every detected harness: ```bash npx -y @mrscraper/cli@latest init --all ``` Install for one harness: ```bash npx -y @mrscraper/cli@latest init --agent codex ``` Supported IDs are `claude-code`, `cursor`, `codex`, `grok`, `hermes`, `opencode`, `openclaw`, `pi`, and `omp`. ### Copyable AI setup prompt ```text Read and follow https://github.com/mrscraper-com/cli/blob/main/skills/mrscraper/SKILL.md ``` For an agent-controlled, non-interactive setup, skip authentication during bootstrap and start login separately: ```bash npx -y @mrscraper/cli@latest init --agent codex --yes --skip-auth mrscraper auth status --json mrscraper login ``` `mrscraper init` installs the CLI and skill pack. It does not install or configure MCP. ### `init` parameters | Parameter | Default | Scope | Behavior | | ----------------- | ------- | ----- | --------------------------------------------------------------------------------------- | | `--api-key ` | — | Local | Saves the supplied key during bootstrap instead of browser login. | | `--all` | enabled | Local | Installs skills for every supported harness detected on this machine. | | `--agent ` | — | Local | Installs skills for one supported harness, even when its detection directory is absent. | | `-y, --yes` | off | Local | Keeps bootstrap non-interactive. Missing authentication is left for a later login. | | `--skip-install` | off | Local | Does not install the current CLI version globally. | | `--skip-auth` | off | Local | Performs no authentication step. | | `--skip-skills` | off | Local | Does not install the skill pack. | | `--dry-run` | off | Local | Prints intended actions without installing, authenticating, or copying skills. | Refresh only the skills with `mrscraper setup skills`. It accepts `--agent ` for one harness and `--dry-run`; without `--agent`, it targets detected harnesses. ### Native plugins (optional) The MrScraper CLI repository also provides skills-only native plugins for Codex, Claude Code, and Cursor. Plugin installation is an alternative to `mrscraper init`, not an additional setup step. If `npx ... init` already installed the four MrScraper skills for an agent, skip plugin installation for that agent. Enabling both methods can load duplicate copies of the same skills. Plugins do not install or authenticate the CLI. When using a plugin, install and authenticate the CLI separately: ```bash npm install -g @mrscraper/cli@latest mrscraper login ``` * **Claude Code:** run `claude plugin marketplace add mrscraper-com/cli`, then install `mrscraper-cli@mrscraper`. * **Cursor:** use `/add-plugin mrscraper-cli` after its public marketplace listing; until then, load the repository through Cursor's local plugin directory. * **Codex:** until its public plugin listing is available, use the [local marketplace example](https://github.com/mrscraper-com/cli/blob/main/examples/marketplace.json) from the CLI repository. ### Verifying installation After completing installation, verify that the CLI binary and commands are accessible in your environment: ```bash mrscraper --version mrscraper auth status --json mrscraper account --json ``` A successful `account` call confirms that the CLI can load its authentication and reach the MrScraper service. Restart an agent harness or open a new terminal session after installing skills so it reloads the skill pack properly. ## Authentication ### Browser login ```bash mrscraper login ``` The CLI opens the MrScraper login page, waits for the local browser callback, exchanges the returned code, and saves the provisioned API key in `~/.mrscraper/auth.json`. Set `MRSCRAPER_HOME` to change the containing directory. ### API-key login ```bash mrscraper login --api-key "$MRSCRAPER_API_KEY" ``` Prefer environment variables to literal keys in shell history: ```bash export MRSCRAPER_API_KEY="your-key" ``` Authentication precedence for API commands is: 1. Command `--token` 2. `MRSCRAPER_API_KEY` 3. `MRSCRAPER_API_TOKEN` 4. `~/.mrscraper/auth.json` ### `login` parameters | Parameter | Default | Scope | Behavior | | --------------------- | ------- | ----- | ----------------------------------------------------------------------------- | | `--api-key ` | — | Local | Saves this API key and does not open browser login. | | `--token ` | — | Local | Deprecated alias for login's `--api-key`. | | `--no-browser` | off | Local | Prompts a human for an API key; requires an interactive terminal. | | `--no-open` | off | Local | Prints the browser URL without launching it. The callback server still waits. | | `--timeout ` | `180` | Local | Maximum time to wait for the browser callback. | ### Auth status and logout `mrscraper auth status` reports the credential currently available to the CLI. It reads the local configuration without sending a test request. JSON output uses the `credential_configured` field: ```json { "credential_configured": true, "auth_type": "api_key", "path": "/home/user/.mrscraper/auth.json" } ``` `mrscraper logout` deletes local credential files. It does not revoke an API key remotely. ## Output contract `scraperAPI` and `webSearch` return a consistent JSON envelope: ```json { "status_code": 200, "data": {}, "headers": { "content-type": "application/json" } } ``` The fields are: * `status_code` contains the HTTP response code. * `headers` contains non-sensitive response headers. * `data` contains the response body. JSON responses are parsed; other bodies, such as HTML-only `scraperAPI` results, remain strings. * Known credential metadata and sensitive response headers are redacted. * HTTP and network failures remain JSON on stdout, set `error`, and exit non-zero. * Progress messages use stderr so stdout remains machine-readable. `account` presents account data as a normalized summary at `data.account`. ## Environment variables | Variable | Default | Purpose | | -------------------------- | -------------------------------------- | ------------------------------------------------------ | | `MRSCRAPER_API_KEY` | — | Preferred API-key environment override. | | `MRSCRAPER_API_TOKEN` | — | Alternative API-key environment override. | | `MRSCRAPER_HOME` | `~/.mrscraper` | Credential directory override. | | `MRSCRAPER_API_BASE_URL` | `https://api.app.mrscraper.com/api/v1` | Development override for `account` and login. | | `MRSCRAPER_FETCH_BASE_URL` | `https://api.mrscraper.com` | Development override for `scraperAPI` and `webSearch`. | The CLI also loads a `.env` file from the current working directory. ## Programmatic use The package exports the same three operations used by the commands: ```javascript import { scraperAPI, webSearch, account } from "@mrscraper/cli"; const token = process.env.MRSCRAPER_API_KEY; const page = await scraperAPI({ token, url: "https://www.scrapethissite.com/pages/simple/", markdown: true, }); console.log(page.data.markdown); const results = await webSearch({ token, query: "books", country: "us", numResults: 10, }); for (const item of results.data.data) console.log(item.rank, item.title, item.url); const status = await account({ token, timeout: 30 }); console.log(status.data.token_remaining); ``` The functions return the same response envelope used by the CLI commands. Programmatic calls require an explicit `token` and do not load saved CLI credentials or `.env` files. They return an error result for HTTP or network failures and throw for invalid arguments. Programmatic `account` results place the summary directly in `data`, not in `data.account`. Programmatic options use camelCase, such as `browserRendering`, `homePage`, `waitForSelector`, `waitUntil`, `maxRetries`, `tokenCap`, and `numResults`. ## CLI command list ### `scraperAPI` `scraperAPI` uses MrScraper's [Web Fetching](/docs/features/web-fetching) to retrieve a known page as HTML, Markdown, a screenshot, or a combination through: ```text GET https://api.mrscraper.com/ ``` The target URL, output flags, page options, and retry controls are all sent as query parameters. HTML is requested by default. Requesting only HTML returns the page as a string in `data`. Requesting Markdown or a screenshot returns an object in `data` with an entry for each requested output, such as `html`, `markdown`, and `screenshot`. At least one of `--html`, `--markdown`, or `--screenshot` must remain selected. ```bash mrscraper scraperAPI "https://www.scrapethissite.com/pages/simple/" | jq -r '.data' mrscraper scraperAPI "https://www.scrapethissite.com/pages/simple/" \ --markdown --no-html | jq -r '.data.markdown' mrscraper scraperAPI "https://www.scrapethissite.com/pages/simple/" --save page.html ``` A screenshot is a base64-encoded JPEG; the `x-status-code` response header carries the target page's status. Browser rendering is automatic unless you set it: the service falls back to a browser when needed. `--browser-rendering` forces a browser, `--no-browser-rendering` disables it, and `--screenshot` always uses one. Real-device routing (`--super`) is independent of browser rendering: ```bash mrscraper scraperAPI "https://www.scrapethissite.com/pages/ajax-javascript/#2015" \ --browser-rendering \ --wait-for-selector ".film" mrscraper scraperAPI "https://www.scrapethissite.com/pages/simple/" \ --super --geo-code ID --home-page ``` #### `scraperAPI` parameters | CLI parameter | Default | API mapping | Behavior | | ------------------------------------------------ | --------------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------- | | `` | required | Query `url` | Target page URL. | | `--html` / `--no-html` | HTML requested | Query `html` | Requests or skips HTML content. | | `--markdown` | off | Query `markdown=true` | Requests Markdown content. | | `--screenshot` | off | Query `screenshot=true` | Captures a screenshot and forces browser rendering. | | `--browser-rendering` / `--no-browser-rendering` | automatic | Query `browserRendering` | Forces or disables browser rendering. | | `--super` | off | Query `super=true` | Selects real-device routing. | | `--geo-code

`                              | `US`                  | Query `geoCode`                 | Proxy country code.                                                                                      |
| `--proxy-country `                         | omitted               | Query `proxyCountry`            | Compatibility country option.                                                                            |
| `--no-use-proxy`                                 | proxy allowed         | Query `useProxy=false`          | Does not use the service's configured proxy.                                                             |
| `--proxy `                                | omitted               | Query `proxy`                   | Custom proxy as `username:password@host:port`.                                                           |
| `--wait-for-selector `                 | omitted               | Query `waitForSelector`         | Waits for a CSS selector.                                                                                |
| `--wait-until `                           | omitted               | Query `waitUntil`               | Waits for a page lifecycle event, such as `domcontentloaded`.                                            |
| `--home-page`                                    | off                   | Query `warmStart=true`          | Visits the site root before loading the target page.                                                     |
| `--block-resources`                              | off                   | Query `blockResources=true`     | Blocks images, CSS, and fonts.                                                                           |
| `--bypass-proxy`                                 | off                   | Query `bypassProxy=true`        | Enables the proxy-bypass behavior.                                                                       |
| `--cookie-jar `                          | omitted               | Query `cookieJar`               | Cookie configuration string.                                                                             |
| `--headers `                               | omitted               | Query `headers`                 | Target page request headers as a JSON object.                                                            |
| `--max-retries `                              | `3`                   | Query `maxRetries`              | Maximum retries; zero disables retries.                                                                  |
| `--token-cap `                                | omitted               | Query `tokenCap`                | Running token total that stops further retries. The initial request always runs.                         |
| `--timeout `                            | `120`                 | Query `timeout`                 | Page timeout from 1 to 300 seconds. The command allows an additional 30 seconds to receive the response. |
| `--stream` / `--no-stream`                       | API default           | Query `stream`                  | Sets the streaming preference; the CLI still prints one JSON response.                                   |
| `--discard-token-secret `                | omitted               | Header `x-discard-token-secret` | Optional upstream header.                                                                                |
| `--save `                                  | omitted               | CLI output                      | Saves `data` to a file: HTML as text, combined outputs as JSON.                                          |
| `--token `                                  | configured credential | Request headers                 | Overrides authentication for this command.                                                               |

Web Fetching usage is based on runtime and bandwidth: one plan token per 30
seconds and one plan token per 0.25 MB, rounded up per component. Resource
blocking can reduce bandwidth for text-focused pages. Use `--max-retries` and
`--token-cap` to balance retry reliability with usage. See the
[Token Plan](/docs/getting-started/api-token) for the complete calculation.

### `webSearch`

`webSearch` searches the web to retrieve real-time search results, including
relevant website URLs, titles, and descriptions.

```bash
mrscraper webSearch "e-menessaptieka.lv linkedin" --country us --num-results 10
mrscraper webSearch "web scraping best practices"
```

The CLI sends this request:

```bash
curl --location 'https://api.mrscraper.com' \
--header 'x-api-token: {MRSCRAPER_API_TOKEN}' \
--header 'Content-Type: application/json' \
--data '{
    "action": "websearch",
    "query": "e-menessaptieka.lv linkedin",
    "country": "us",
    "numResults": 10
}'
```

`data` contains `success`, `message`, and `data`. Each item in `data.data` has
`rank`, `title`, `url`, `description`, and `source`.

#### `webSearch` parameters

| CLI parameter         | Default               | API mapping       | Behavior                                                                              |
| --------------------- | --------------------- | ----------------- | ------------------------------------------------------------------------------------- |
| ``             | required              | Body `query`      | The search query.                                                                     |
| `--country `    | omitted               | Body `country`    | Country code for localized results, such as `us`. Omitted to use the service default. |
| `--num-results `   | `10`                  | Body `numResults` | Number of results, an integer from 10 to 100.                                         |
| `--timeout ` | `600`                 | CLI request       | Sets the HTTP request deadline. Not sent in the API body.                             |
| `--token `       | configured credential | Request headers   | Overrides authentication for this command.                                            |

### `account`

`account` checks the remote account's identity, subscription, token quota,
and rate limits. It calls:

```text
GET https://api.app.mrscraper.com/api/v1/subscription-accounts
```

```bash
mrscraper account
mrscraper account --json
mrscraper account --pretty --no-color
```

Interactive terminals show a dashboard by default; redirected output is JSON.
API keys and billing identifiers are excluded, and remaining tokens are
clamped to zero for over-quota accounts.

```json
{
  "status_code": 200,
  "data": {
    "account": {
      "subscription_status": "active",
      "enterprise": false,
      "token_usage": 250,
      "token_limit": 1000,
      "token_remaining": 750,
      "usage_percent": 25,
      "rate_limit": 10,
      "rate_ttl": 60,
      "auto_renew": true,
      "ends_at": null,
      "user": {
        "name": "Ada",
        "email": "ada@example.com",
        "verified": true
      }
    }
  },
  "headers": {}
}
```

#### `account` parameters

| CLI parameter         | Default               | API mapping     | Behavior                                                         |
| --------------------- | --------------------- | --------------- | ---------------------------------------------------------------- |
| `--json`              | automatic when piped  | CLI output      | Prints the summary as JSON.                                      |
| `--pretty`            | automatic in TTY      | CLI output      | Prints the terminal dashboard. Cannot be combined with `--json`. |
| `--no-color`          | off                   | CLI output      | Disables ANSI color in the dashboard.                            |
| `--timeout ` | `30`                  | CLI request     | HTTP request timeout.                                            |
| `--token `       | configured credential | Request headers | Overrides authentication for this command.                       |

## Migrating from earlier versions

### From 0.5.x

Version 0.6.0 returns to three public operations: `scraperAPI`, `webSearch`,
and `account`. The 0.5.x format-specific commands and package exports
(`scraperHtml`, `scraperMarkdown`, `getScreenshot`, `scraperJson`,
`scraperMultiplePage`, `scraperSitemap`, `scrapeBrand`, `getBrand`, and their
snake-case aliases) are no longer public.

| 0.5.x                                       | 0.6.0                                                     |
| ------------------------------------------- | --------------------------------------------------------- |
| `mrscraper scraperHtml `               | `mrscraper scraperAPI `                              |
| `mrscraper scraperMarkdown `           | `mrscraper scraperAPI  --no-html --markdown`         |
| `mrscraper getScreenshot `             | `mrscraper scraperAPI  --no-html --screenshot`       |
| `mrscraper webSearch  --region us`   | `mrscraper webSearch  --country us`                |
| `scraperMarkdown({ token, url })`           | `scraperAPI({ token, url, html: false, markdown: true })` |
| `webSearch({ token, query, region: "us" })` | `webSearch({ token, query, country: "us" })`              |

### From 0.4.x

* **`scraperAPI`** now sends a `GET` request with every option in the query
  string, and adds `--wait-until` and `--save`. Requesting only HTML returns a
  string in `data`, and screenshots are returned as base64 in a JSON object.
* **`webSearch`** now calls `https://api.mrscraper.com` with
  `action: "websearch"` and accepts `--country` and `--num-results` (10–100).
  The `--region`, `--language`, `--page`, `--format`, `--raw`, `--render-js`,
  and `--client-timeout` flags and search URL input are removed, as are the
  `raw`, `format`, `region`, `language`, `page`, and `renderJs` programmatic
  options and the `url` input alias.
* The `MRSCRAPER_SYNC_BASE_URL` setting and the `SYNC_SCRAPER_BASE_URL` export
  are removed.
* **`account`** is unchanged.

### Removed commands

Version 0.4.0 removed the commands below. The CLI exposes only `scraperAPI`,
`webSearch`, and `account` as data commands; `login`, `logout`, `auth`, `init`,
and `setup` remain.

| Previous command                       | Replacement                                                                                             |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| `fetch`                                | `scraperAPI`; `--super-mode` is now `--super`, and browser rendering is automatic unless set.           |
| `serp`                                 | `webSearch`.                                                                                            |
| `status`                               | `account`; domain analytics are no longer included.                                                     |
| `scrape`, `rerun`, `results`, `result` | Manage AI scrapers, reruns, and stored results in the [MrScraper dashboard](https://app.mrscraper.com). |


# MCP Server





import { Accordion, Accordions } from 'fumadocs-ui/components/accordion';
import { Step, Steps } from "fumadocs-ui/components/steps";
import { CodeBlock } from "fumadocs-ui/components/codeblock";
import {
  Activity,
  FileSearch,
  GlobeLock,
  LayoutDashboard,
  Search,
  Server,
} from "lucide-react";

MrScraper MCP is the official Model Context Protocol server for MrScraper.
It connects AI clients to web page retrieval, web search, and your account
usage.

[MrScraper MCP server source code on GitHub](https://github.com/mrscraper-com/mrscraper-mcp)
· [Install the @mrscraper/mcp npm package](https://www.npmjs.com/package/@mrscraper/mcp)

The server exposes three tools:


  } href="#scraperapi">
    Retrieve a known page as HTML, Markdown, a screenshot, or a combination.
  

  } href="#websearch">
    Search the web and get ranked results with titles, links, and descriptions.
  

  } href="#status">
    Check subscription status, token usage, remaining tokens, and limits.
  


Use the hosted Streamable HTTP endpoint for a managed connection, or run the
published MCP package locally through Streamable HTTP or stdio.


  This page documents `@mrscraper/mcp` version **0.3.0**, which moved `webSearch` to MrScraper's new web search endpoint. Earlier versions exposed `fetch`, `scrape`, `serp`, `rerun`, `results`, and `result`. See [Migration from the previous tool set](#migration-from-the-previous-tool-set).


## Capabilities


  }>
    Retrieve a known URL with automatic browser fallback, real-device routing,
    proxy controls, selector waiting, and retry limits.
  

  }>
    Search the web for ranked results, for a chosen country and up to 100
    results at a time.
  

  }>
    Read the subscription, token limit, token usage, remaining tokens, and rate
    limit.
  

  }>
    Connect to the hosted MCP endpoint through Streamable HTTP with OAuth 2.1
    browser sign-in.
  

  }>
    Receive a consistent response envelope in MCP structured content.
  

  }>
    Render search results and an account usage meter in hosts that support MCP
    Apps.
  


## Requirements

* A [MrScraper account](https://app.mrscraper.com)
* An MCP client with Streamable HTTP support
* An [API key](/docs/getting-started/api-token) only when the client cannot use
  OAuth 2.1 or when running the server locally

## Quick start with the hosted server


  
    Add the hosted endpoint

    Configure a Streamable HTTP server named `mrscraper` with no credential:

    ```json
    {
      "mcpServers": {
        "mrscraper": {
          "type": "http",
          "url": "https://mcp.mrscraper.com/mcp"
        }
      }
    }
    ```

    Use this exact URL, including the `/mcp` path.
  

  
    Sign in

    Connect or reload the MCP server. Your client follows the OAuth 2.1
    challenge, opens MrScraper in your browser, and asks you to approve access.
  

  
    Reload and verify

    Reload the client or start a new session, then inspect its MCP tool list.
    The connection should expose `scraperAPI`, `webSearch`, and `status`.
  


## Copyable AI setup prompt


      
      Connect MrScraper MCP to this agent. Detect the current MCP client and use its native Streamable HTTP setup to add a server named mrscraper at exactly 

      [https://mcp.mrscraper.com/mcp](https://mcp.mrscraper.com/mcp)

       with no static credential. Complete the OAuth 2.1 browser sign-in; if it does not start automatically, use the client's MCP login or authenticate command. Only if the client does not support OAuth, fall back to a MrScraper API key from 

      [https://app.mrscraper.com/api-tokens](https://app.mrscraper.com/api-tokens)

       stored through the client's environment-variable or secret-storage mechanism and sent as Authorization: Bearer . Reload the MCP client when required, then list the tools and confirm that scraperAPI, webSearch, and status are available.
    
  


## Client configuration

Add the hosted URL without a static `Authorization` header, then complete
browser sign-in when the client prompts you.

### Codex

```bash
codex mcp add mrscraper \
  --url https://mcp.mrscraper.com/mcp
```

Codex uses OAuth by default for Streamable HTTP servers. If browser sign-in does
not start automatically, run `codex mcp login mrscraper`.

### Claude Code

```bash
claude mcp add \
  --transport http \
  --scope user \
  mrscraper https://mcp.mrscraper.com/mcp
```

Open `/mcp` inside Claude Code and authenticate, or run
`claude mcp login mrscraper` from your shell.

### Claude and Claude Desktop


  
    Open the custom connector form

    In a general Claude chat, select the **+** button below the input box, then
    choose **Connectors → Add connector → Add custom connector**.

        
  

  
    Add MrScraper

    Enter `MrScraper` as the name and
    `https://mcp.mrscraper.com/mcp` as the MCP server URL, then select
    **Continue**.

        
  

  
    Sign in

    Complete the MrScraper OAuth browser flow when Claude prompts you.
  


### Cursor

Open **Cursor Settings → Tools & MCP → Add Custom MCP**, then add:

```json
{
  "mcpServers": {
    "mrscraper": {
      "type": "http",
      "url": "https://mcp.mrscraper.com/mcp"
    }
  }
}
```

Refresh the MCP server list, then authenticate when prompted.

### VS Code

Add the server to your user settings:

```json
{
  "mcp": {
    "servers": {
      "mrscraper": {
        "type": "http",
        "url": "https://mcp.mrscraper.com/mcp"
      }
    }
  }
}
```

For a workspace configuration, place the server object under `servers` in
`.vscode/mcp.json`. Start the server and VS Code opens a browser for OAuth.

### Windsurf

Add the following entry to the Windsurf MCP configuration:

```json
{
  "mcpServers": {
    "mrscraper": {
      "serverUrl": "https://mcp.mrscraper.com/mcp"
    }
  }
}
```

Refresh the MCP server list, then complete OAuth when prompted.

### Other MCP clients

Use these connection values:

| Setting        | Value                                       |
| -------------- | ------------------------------------------- |
| Name           | `mrscraper`                                 |
| Transport      | Streamable HTTP                             |
| URL            | `https://mcp.mrscraper.com/mcp`             |
| Authentication | OAuth 2.1; do not configure a static header |

## API key fallback

If a client cannot complete OAuth, create an [API key](https://app.mrscraper.com/api-tokens),
store it through the client's secret mechanism, and send it as a bearer token:

```json
{
  "mcpServers": {
    "mrscraper": {
      "type": "http",
      "url": "https://mcp.mrscraper.com/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_MRSCRAPER_API_KEY"
      }
    }
  }
}
```

For Codex, pass the environment-variable name instead of embedding the key:

```bash
export MRSCRAPER_API_KEY="YOUR_MRSCRAPER_API_KEY"
codex mcp add mrscraper \
  --url https://mcp.mrscraper.com/mcp \
  --bearer-token-env-var MRSCRAPER_API_KEY
```

## Authentication

| Connection                  | Credential source                                                      |
| --------------------------- | ---------------------------------------------------------------------- |
| Hosted Streamable HTTP      | OAuth 2.1 browser sign-in, or an API key fallback                      |
| Self-hosted Streamable HTTP | OAuth 2.1 access token, or `Authorization: Bearer ` |
| Local stdio                 | `MRSCRAPER_API_KEY`, then `MRSCRAPER_API_TOKEN`                        |

The hosted server publishes OAuth 2.1 protected-resource metadata and challenges
unauthenticated clients to start browser sign-in. Access tokens are issued for
the exact `https://mcp.mrscraper.com/mcp` resource. A tool called without its
required scope returns `403 insufficient_scope`, and your client can offer to
grant the missing permission.

| Scope          | Tools        |
| -------------- | ------------ |
| `scrape:write` | `scraperAPI` |
| `scrape:read`  | `webSearch`  |
| `account:read` | `status`     |

API keys remain supported for clients without OAuth and carry full account
authority, so no scope is enforced against them. Authentication stays at the
transport layer; tools never accept credentials as arguments.

## Run the MCP server locally

Node.js 20 or newer is required.

### Published package over stdio

```json
{
  "mcpServers": {
    "mrscraper": {
      "command": "npx",
      "args": ["-y", "@mrscraper/mcp@latest"],
      "env": {
        "MRSCRAPER_API_KEY": "YOUR_MRSCRAPER_API_KEY"
      }
    }
  }
}
```

### Source checkout over Streamable HTTP


  
    Clone and build

    ```bash
    git clone https://github.com/mrscraper-com/mrscraper-mcp.git
    cd mrscraper-mcp
    npm ci
    npm run build
    ```
  

  
    Start HTTP transport

    ```bash
    TRANSPORT=http npm start
    ```

    The default endpoint is `http://127.0.0.1:8000/mcp`.
  

  
    Connect the client

    ```json
    {
      "mcpServers": {
        "mrscraper": {
          "type": "http",
          "url": "http://127.0.0.1:8000/mcp",
          "headers": {
            "Authorization": "Bearer YOUR_MRSCRAPER_API_KEY"
          }
        }
      }
    }
    ```
  


### Docker

```bash
docker build -f docker/Dockerfile -t mrscraper-mcp .
docker run --rm -p 8000:8000 mrscraper-mcp
```

The image binds to `0.0.0.0`. Apply network controls appropriate for the
deployment and keep bearer verification enabled.

## Environment variables

| Variable                         | Default                          | Purpose                                                          |
| -------------------------------- | -------------------------------- | ---------------------------------------------------------------- |
| `TRANSPORT`                      | `stdio`                          | Selects `stdio` or `http`.                                       |
| `HOST`                           | `127.0.0.1`                      | HTTP bind address; the Docker image uses `0.0.0.0`.              |
| `PORT`                           | `8000`                           | HTTP listen port.                                                |
| `MRSCRAPER_API_KEY`              | —                                | Primary stdio credential.                                        |
| `MRSCRAPER_API_TOKEN`            | —                                | Alternative stdio credential, also used by the Python SDK.       |
| `MRSCRAPER_HTTP_AUTH`            | `1`                              | Enables HTTP bearer verification.                                |
| `MRSCRAPER_OAUTH`                | `1`                              | Accepts OAuth 2.1 access tokens alongside API keys.              |
| `MRSCRAPER_MCP_PUBLIC_URL`       | `https://mcp.mrscraper.com`      | Public origin; `/mcp` is the required token audience.    |
| `MRSCRAPER_OAUTH_ISSUER`         | `https://api.app.mrscraper.com`  | Authorization server issuer.                                     |
| `MRSCRAPER_OAUTH_JWKS_URL`       | `/.well-known/jwks.json` | Key set used to verify access tokens.                            |
| `MRSCRAPER_ALLOWED_ORIGINS`      | —                                | Comma-separated browser origins allowed to call the HTTP server. |
| `MRSCRAPER_API_BASE_URL`         | MrScraper platform API           | Platform endpoint override used by `status`.                     |
| `MRSCRAPER_FETCH_BASE_URL`       | MrScraper Scraper API            | Endpoint override used by `scraperAPI` and `webSearch`.          |
| `MRSCRAPER_LOG_HTTP_PAYLOAD`     | off                              | Enables trusted-environment request-body diagnostics.            |
| `MRSCRAPER_LOG_HTTP_PAYLOAD_MAX` | `8192`                           | Maximum diagnostic payload length.                               |

## Security behavior

* HTTP authentication is enabled by default.
* Credential-bearing response headers are filtered from tool output.
* Parsed JSON credential metadata is redacted, and network errors redact signed
  query credentials.
* `status` copies only account usage fields, so the API key and billing
  identifiers in the account record never reach the client.
* Browser-origin requests are accepted from trusted local origins, from
  `claude.ai` and `chatgpt.com`, and from exact
  values configured through `MRSCRAPER_ALLOWED_ORIGINS`.
* Tool calls receive the caller's authenticated OAuth token or API key through
  the MCP transport rather than through tool arguments.

## Choosing the right tool

| Starting point     | Desired outcome                                | Tool         |
| ------------------ | ---------------------------------------------- | ------------ |
| A known URL        | HTML, Markdown, a screenshot, or a combination | `scraperAPI` |
| A topic or keyword | Search result discovery                        | `webSearch`  |
| An account         | Plan, token usage, and remaining tokens        | `status`     |

A discovery workflow commonly uses `webSearch` first, followed by `scraperAPI`
for the pages that will inform the answer. Agents produce summaries, tables,
and structured output from the retrieved content.

### Migration from the previous tool set

| Previous tool                          | Replacement                                                                                                                          |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `fetch`                                | `scraperAPI`; `super_mode` becomes `super`, `home_page` is sent as `warmStart`, and browser rendering is automatic unless set.       |
| `serp`                                 | `webSearch`; `query_or_url` becomes `url` and `client_timeout` becomes `timeout`.                                                    |
| `status`                               | `status` for account usage; the domain analytics inputs are removed.                                                                 |
| `scrape`, `rerun`, `results`, `result` | Removed from the MCP server. Manage AI scrapers, reruns, and stored results in the [MrScraper dashboard](https://app.mrscraper.com). |

Reconnect the MCP client after the change so it refreshes its tool list.

In 0.3.0, `webSearch` dropped the `raw`, `format`, `language`, `page`, and
`render_js` options because the new web search endpoint doesn't support them;
calls that still send them succeed and the options are ignored. It added
`num_results`.

## Common response envelope

`scraperAPI` and `webSearch` return the same envelope through MCP
`structuredContent` and a formatted JSON text block:

```json
{
  "status_code": 200,
  "data": {},
  "headers": {
    "content-type": "application/json"
  }
}
```

| Field         | Type                          | Description                                                                                                     |
| ------------- | ----------------------------- | --------------------------------------------------------------------------------------------------------------- |
| `status_code` | number or `null`              | HTTP status returned by the MrScraper service, or `null` when the request ends before an HTTP response arrives. |
| `data`        | JSON value, string, or `null` | Parsed JSON or response text supplied by the service.                                                           |
| `headers`     | object                        | Response headers with credential-bearing headers filtered out.                                                  |
| `error`       | string, when present          | Request failure summary.                                                                                        |

API failures preserve the envelope in `structuredContent` and set the MCP
result's `isError` flag. Input-contract errors are returned as MCP tool
errors. `status` returns the account summary described in its
[reference](#status).

## Tool reference

The options and defaults of `scraperAPI` follow the Python SDK's `scraper_api`
method. Tool names use camel case; argument names use snake case.

### `scraperAPI`

Use `scraperAPI` for a known URL. It calls:

```text
POST https://api.mrscraper.com/
```

The JSON body contains `url`, `maxRetries`, and optional `tokenCap`. Page
options are sent as query parameters.

#### Basic request

```json
{
  "url": "https://www.scrapethissite.com/pages/simple/"
}
```

#### HTML, Markdown, and a screenshot

```json
{
  "url": "https://www.scrapethissite.com/pages/simple/",
  "html": true,
  "markdown": true,
  "screenshot": true
}
```

#### Real-device routing from another country

```json
{
  "url": "https://www.scrapethissite.com/pages/simple/",
  "super": true,
  "geo_code": "ID",
  "home_page": true
}
```

Omit `browser_rendering`, or set it to `null`, to let the service try without a
browser and fall back to browser loading when needed. `true` forces browser
loading and `false` disables it. A screenshot always forces browser loading.
`super` selects real-device routing independently of browser rendering.

#### Response

The page content is in `data`:

```json
{
  "status_code": 200,
  "data": {
    "html": "...",
    "markdown": "...",
    "screenshot": "/9j/4AAQSkZJRgABAQAAAQAB...",
    "token_usage": 3,
    "runtime": 18.02,
    "retry_count": 0,
    "error": null
  },
  "headers": {
    "content-type": "application/json",
    "x-status-code": "200",
    "token_usage": "3"
  }
}
```

`screenshot` is a base64-encoded JPEG and can be large. `x-status-code` is the
target page's status. Usage is calculated from runtime and bandwidth: one plan
token per 30 seconds and one plan token per 0.25 MB, rounded up per component.

#### Parameters

| Parameter              | Required | Default   | Request mapping                      | Description                                                                             |
| ---------------------- | -------- | --------- | ------------------------------------ | --------------------------------------------------------------------------------------- |
| `url`                  | Yes      | —         | Body `url`                           | Absolute HTTP or HTTPS page URL.                                                        |
| `html`                 | No       | `true`    | Query `html`                         | Requests HTML content.                                                                  |
| `markdown`             | No       | `false`   | Query `markdown`                     | Requests Markdown content.                                                              |
| `screenshot`           | No       | `false`   | Query `screenshot`                   | Captures a screenshot and forces browser rendering.                                     |
| `browser_rendering`    | No       | automatic | Query `browserRendering`             | `true` forces and `false` disables browser rendering.                                   |
| `super`                | No       | `false`   | Query `super`                        | Real-device routing.                                                                    |
| `geo_code`             | No       | `"US"`    | Query `geoCode`                      | Proxy country code.                                                                     |
| `proxy_country`        | No       | omitted   | Query `proxyCountry`                 | Compatibility country option.                                                           |
| `use_proxy`            | No       | `true`    | Query `useProxy=false` when disabled | Allows the service's configured proxy.                                                  |
| `proxy`                | No       | omitted   | Query `proxy`                        | Custom proxy as `username:password@host:port`.                                          |
| `wait_for_selector`    | No       | omitted   | Query `waitForSelector`              | CSS selector to wait for.                                                               |
| `home_page`            | No       | `false`   | Query `warmStart`                    | Visits the site's root before the target URL.                                           |
| `block_resources`      | No       | `false`   | Query `blockResources`               | Blocks images, CSS, and fonts.                                                          |
| `bypass_proxy`         | No       | `false`   | Query `bypassProxy`                  | Enables the proxy-bypass behavior.                                                      |
| `cookie_jar`           | No       | omitted   | Query `cookieJar`                    | Cookie configuration, such as `domain=example.org&country=us&max_used=100&max_age=50m`. |
| `headers`              | No       | omitted   | Query `headers`                      | Target page request headers as an object or JSON string.                                |
| `max_retries`          | No       | `3`       | Body `maxRetries`                    | Maximum retries; `0` disables retries.                                                  |
| `token_cap`            | No       | omitted   | Body `tokenCap`                      | Running token total that stops further retries.                                         |
| `timeout`              | No       | `120`     | Query `timeout`                      | Page timeout from 1 to 300 seconds.                                                     |
| `stream`               | No       | omitted   | Query `stream`                       | Asks the API to stream; the server returns one buffered response.                       |
| `discard_token_secret` | No       | `""`      | Header `x-discard-token-secret`      | Optional upstream header.                                                               |

### `webSearch`

Use `webSearch` when no target URL is known. It calls MrScraper's web search
action on the Scraper API host:

```text
POST https://api.mrscraper.com
```

The request body carries `action: "websearch"`, the query, and the optional
country and result count. This asks for 20 results from the United Kingdom:

```json
{
  "url": "running shoes",
  "region": "gb",
  "num_results": 20
}
```

A search results URL also works as `url`; its `q` parameter becomes the query.

#### Results

The results are a ranked list in `data.data`:

```json
{
  "status_code": 200,
  "data": {
    "success": true,
    "message": "Search completed",
    "data": [
      {
        "rank": 1,
        "title": "Web Scraping API | MrScraper",
        "url": "https://mrscraper.com/",
        "description": "Extract structured data from any website with AI.",
        "source": "MrScraper"
      }
    ],
    "tokenUsage": 1
  },
  "headers": {
    "content-type": "application/json; charset=utf-8"
  }
}
```

#### Parameters

| Parameter     | Required | Default | Request mapping        | Description                                                     |
| ------------- | -------- | ------- | ---------------------- | --------------------------------------------------------------- |
| `url`         | Yes      | —       | Body `query`           | Search text, or a search results URL whose `q` value is used.   |
| `region`      | No       | `us`    | Body `country`         | Two-letter country code for the results.                        |
| `num_results` | No       | `10`    | Body `numResults`      | Number of results, from 10 to 100. The search may return fewer. |
| `timeout`     | No       | `600`   | Local request deadline | Upstream HTTP deadline in seconds.                              |

### `status`

Use `status` to check the connected account. It takes no input and calls:

```text
GET https://api.app.mrscraper.com/api/v1/subscription-accounts
```

#### Status summary

```json
{
  "kind": "mrscraper-cli-status-summary",
  "source_endpoints": ["/subscription-accounts"],
  "status_code": 200,
  "data": {
    "account": {
      "subscription_status": "active",
      "enterprise": false,
      "token_usage": 61240,
      "token_limit": 100000,
      "token_remaining": 38760,
      "usage_percent": 61.24,
      "rate_limit": 0,
      "rate_ttl": 0,
      "auto_renew": true,
      "ends_at": "2026-12-01T00:00:00.000Z",
      "user": {
        "name": "Ada",
        "email": "ada@example.com",
        "verified": true
      }
    }
  }
}
```

The summary selects account fields and calculates `token_remaining` and
`usage_percent`. An OAuth connection needs the `account:read` scope.

## Troubleshooting


  
    The MCP client may still be using a cached tool list.

    1. Reload the MCP client or start a new session.
    2. Inspect `tools/list`.
    3. Confirm that the server advertises these tools: `scraperAPI`, `webSearch`, and `status`.
  

  
    The MCP connection is not signed in or is using an invalid API key.

    1. For hosted OAuth connections, reconnect so the client starts browser sign-in again.
    2. For HTTP API-key connections, confirm that the request includes `Authorization: Bearer `.
    3. For stdio connections, confirm that `MRSCRAPER_API_KEY` or `MRSCRAPER_API_TOKEN` is available in the launched server environment.
  

  
    The OAuth grant does not include the tool's scope.

    1. `scraperAPI` needs `scrape:write`, `webSearch` needs `scrape:read`, and `status` needs `account:read`.
    2. Reconnect the server so the client requests the missing scope.
  

  
    The client's browser origin is not included in the server's allowed origins.

    1. Add the client's exact browser origin to `MRSCRAPER_ALLOWED_ORIGINS`.
    2. Restart the MCP server if required.
    3. Run the request again.

    Command-line and service-to-service MCP clients typically connect without an `Origin` header.
  

  
    The tool call may not match the parameters supported by the selected tool.

    1. Check the tool's parameter table.
    2. Use `scraperAPI` and `webSearch` in place of the removed `fetch` and `serp` tools.
    3. Run the request again with the corrected parameters.
  

  
    The upstream service may need more time to complete the request.

    1. Increase `scraperAPI.timeout` (up to 300 seconds) for a page request.
    2. Increase `webSearch.timeout` for a search request.
    3. Run the request again after updating the appropriate timeout.
  

  
    Use the following command to verify that a local HTTP MCP server is responding correctly:

    ```bash
    npm run test:mcp -- \
      --target http://127.0.0.1:8000/mcp \
      --token "$MRSCRAPER_API_KEY"
    ```

    Call one tool at a time against the live service:

    ```bash
    npm run test:manual -- scraperAPI --url "https://www.scrapethissite.com/pages/simple/"
    npm run test:manual -- webSearch --query "web scraping"
    npm run test:manual -- status
    ```

    For package development, run the following checks:

    ```bash
    npm run format:check
    npm run lint
    npm test
    npm run build
    npm pack --dry-run
    ```
  



# Overview



import { Bot, BotMessageSquare, TextSearch, Workflow, ShoppingCart, Briefcase, Star, SlidersHorizontal, Mail, Database, Webhook, ClipboardClock, Network, Layers, Map, Home, Plane, DollarSign, Store, Code2, Rocket, SquareTerminal, Key, Terminal, Server, ShieldCheck, Waypoints, Users, Cookie, ChartLine, SatelliteDish, HeartPulse, Zap, PackageOpen, Package2, Link2, PanelTop, Globe } from 'lucide-react';



Discover how our powerful scraping platform can transform the way you extract data. Whether you're scraping e-commerce product data, collecting job listings, gathering customer reviews for sentiment analysis, or extracting any website content, our AI scrapers and workflows are designed to automate and simplify the process, giving you a powerful edge in any project.

## Start Here

New to MrScraper? These pages get you from sign-up to your first scraped dataset.


  } href="/docs/getting-started/quickstart">
    Run your first scraper in a few minutes with a step-by-step walkthrough.
  

  } href="/docs/getting-started/scraper-api">
    Test API requests, explore features, and generate code without writing code.
  

  } href="/docs/getting-started/api-token">
    Learn how tokens work and create an API token for authentication.
  

  } href="/docs/getting-started/billing">
    Understand subscription plans, token usage, and billing cycles.
  


## AI Agents

Pick the agent that matches the page you're scraping, or chain several together.


  } href="/docs/features/ai-scraper">
    Automatically extract data with natural language instructions — no coding required.
  

  } href="/docs/features/ai-scraper/general">
    Extract structured data from a single web page using a natural language prompt.
  

  } href="/docs/features/ai-scraper/listing">
    Scrape listing pages with automated pagination, infinite scroll, and load-more handling.
  

  } href="/docs/features/ai-scraper/map">
    Discover and extract every URL from a website to understand its structure.
  

  } href="/docs/features/ai-scraper/multi-agent-flow">
    Combine Map, Listing, and General agents into end-to-end extraction workflows.
  


## Features

Explore MrScraper's key features and tools designed to make data extraction effortless.


  } href="/docs/features/workflow-scraper">
    Build and customize your own scraper workflow step by step for full control.
  

  } href="/docs/features/workflow-scraper/self-healing">
    Let AI generate, repair, and extend workflows when a site changes.
  

  } href="/docs/features/bulk-scraping">
    Scrape multiple URLs in one go by uploading a list or Excel file.
  

  } href="/docs/features/web-fetching">
    Retrieve page HTML with browser rendering and geo-targeting.
  

  } href="/docs/features/proxy">
    Route individual scrapers through proxies to mask your IP and improve reliability.
  

  } href="/docs/features/cookie">
    Use your own cookies to access content behind logins and restrictions.
  

  } href="/docs/features/schedule">
    Automate scraper runs on a schedule to keep your data fresh and up to date.
  

  } href="/docs/features/marketplace">
    Use ready-to-run scrapers for popular websites and common scraping use cases.
  

  } href="/docs/features/activating-api">
    Activate a scraper as an API endpoint and run it programmatically.
  

  } href="/docs/features/s3-storage">
    Save scraped data straight into your own S3 bucket.
  

  } href="/docs/features/team">
    Create a team to share scrapers and collaborate on projects.
  


## Use Cases

Discover how MrScraper can support different real-world scenarios.


  } href="/docs/guides/ecommerce">
    Scrape product details, pricing, and availability from major e-commerce sites.
  

  } href="/docs/guides/job-listing">
    Collect job postings and company data for recruitment or market analysis.
  

  } href="/docs/guides/sentiment-analysis">
    Extract and analyze customer feedback to gain insights into sentiment and satisfaction.
  

  } href="/docs/guides/real-estate">
    Gather property listings, prices, and market trends from real estate websites.
  

  } href="/docs/guides/travel-hospitality">
    Scrape flight prices, airlines, and schedules for price monitoring and booking analysis.
  

  } href="/docs/guides/optimizing-cost">
    Minimize token usage and maximize scraping efficiency without losing accuracy.
  

  } href="/docs/guides/programmatically">
    Create an AI scraper once, then run it from your own code via the API.
  


## Integrations

Integrate MrScraper with your favorite tools to automate your data flow.


  } href="/docs/integrations/email">
    Receive email notifications whenever a scraper finishes or encounters an error.
  

  } href="/docs/integrations/sql">
    Automatically store scraped data in your database for querying and analysis.
  

  } href="/docs/integrations/webhook">
    Send scraper results directly to external services through webhooks.
  

  } href="/docs/integrations/zapier">
    Connect your scrapers to thousands of apps with no-code Zapier automations.
  

  } href="/docs/integrations/n8n">
    Build automated workflows around your scrapers with the MrScraper n8n node.
  

  } href="/docs/integrations/dify">
    Fetch web pages and search Google from Dify agents, chatflows, and workflows.
  

  } href="/docs/integrations/apify">
    Run MrScraper as Apify Actors and pipe results into the Apify platform.
  

  } href="/docs/integrations/clawhub">
    Install the MrScraper skill to give your OpenClaw agent reliable web scraping.
  


## Developer Tools

Work with MrScraper from your terminal, your codebase, or your AI agent.
The [MrScraper MCP server source code](https://github.com/mrscraper-com/mrscraper-mcp)
is available on GitHub for local setup, contributions, and issue reports.


  } href="/docs/getting-started/cli">
    Fetch pages, search Google, and check account usage from your terminal.
  

  } href="/docs/getting-started/mcp-server">
    Let AI agents fetch pages and search Google through the Model Context Protocol.
  

  } href="/docs/integrations/python">
    Fetch pages, search Google, and check account usage with the async `mrscraper-sdk` package.
  

  } href="/docs/integrations/node">
    Fetch pages, search Google, and check account usage with the `@mrscraper/sdk` package.
  

  } href="/docs/integrations/langchain">
    Use MrScraper as LangChain tools inside your own agents.
  

  } href="/docs/api/overview">
    Explore every endpoint, authentication, pagination, and error handling.
  


## Residential Proxy

Route your own tools and browsers through our proxy network.


  } href="/docs/residential-proxy/getting-started/overview">
    Understand proxy types, global coverage, and how to get started.
  

  } href="/docs/residential-proxy/configuration/authentication">
    Configure username patterns, sessions, and country targeting.
  

  } href="/docs/residential-proxy/examples/overview">
    Ready-to-use snippets for Python, Node.js, Go, PHP, Playwright, and more.
  



# Quickstart

import { Play } from 'lucide-react';

This guide shows you how to make your first scraping request with MrScraper.

## Make Your First API Request


  Create a MrScraper account and obtain your API key from the [Scraper API](https://app.mrscraper.com/scraper-api) before following this guide.


In this quickstart, you'll scrape a Walmart product page using the General AI Agent to get the following data:

* Product name
* Price
* Rating
* Number of reviews
* Product link


  
    
      cURL
    
  

  
    ```shell
    curl --location --request POST 'https://api.mrscraper.com?token=&html=true&super=true' \
      -H 'x-api-token: ' \
      -H 'Content-Type: application/json' \
      -d '{
        "url": "https://www.walmart.com/ip/IdeaPad-Slim-3x-15-Laptop-Snapdragon-X-X1-26-100-16GB-RAM-512GB-SSD-Luna-Grey/19075520026",
        "prompt": "Extract the following information in JSON format:\n\nproduct_name\nprice\nrating\nnumber_of_reviews\nproduct_link",
        "agent": "general"
      }'
    ```
  


} href="https://app.mrscraper.com/scraper-api">
  Run this request instantly without writing any code.


The response contains the extracted data from the page.


  
    
      JSON
    
  

  
    ```json
    {
      "success": true,
      "data": {
        "product_name": "Lenovo IdeaPad Slim 3x 15.3\" Laptop AI Snapdragon X 16GB 512GB SSD Luna Grey",
        "price": 449,
        "rating": 4.5,
        "number_of_reviews": 103,
        "product_link": "https://www.walmart.com/ip/IdeaPad-Slim-3x-15-Laptop-Snapdragon-X-X1-26-100-16GB-RAM-512GB-SSD-Luna-Grey/19075520026"
      },
      "thread_id": "cebfaa37-f6de-45e7-8dd4-ec955824d81b",
      "token_usage": 16,
      "markdown": "",
      "screenshot": "",
      "status": "Converting data to JSON",
      "error": "",
      "runtime": 14852
    }
    ```
  


## Use a Predefined Marketplace API

MrScraper also provides ready-to-use scraping APIs through the [Marketplace](/docs/features/marketplace).

These APIs are designed for specific industries and websites. You do not need to create prompts or extraction logic yourself.

Available categories include:

* E-commerce
* News
* Real Estate
* Travel
* And more

} href="https://app.mrscraper.com/marketplace?page=1&pageSize=12">
  Browse and test ready-to-use scrapers.


For example, you can use the [Amazon Product Details API](https://app.mrscraper.com/marketplace/d08bcc7c-fa99-4886-9db2-3c274a9a1c2d) to extract product information from an Amazon product page.

```shell
curl --location --request POST 'https://sync.scraper.mrscraper.com/api/amazon/pdp/sync' \
  -H 'Authorization: bearer ' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://www.amazon.com/STANLEY-Flowstate-3-Position-Compatible-Insulated/dp/B0CP9YB3Q4/ref=zg_bs_c_kitchen_d_sccl_2/145-6361404-7555432?pd_rd_w=3BPkW&content-id=amzn1.sym.fef9af56-6177-46e9-8710-a5293a68dd39&pf_rd_p=fef9af56-6177-46e9-8710-a5293a68dd39&pf_rd_r=E598BMWHB9QEZ75X7ABP&pd_rd_wg=Z9psI&pd_rd_r=7e3be220-af38-4e7b-b45c-9a68470a3e1f&pd_rd_i=B0CP9YB3Q4&th=1"
}'
```

The response contains the extracted data from the page.


  
    
      JSON
    
  

  
    ```json
    {
      "success": true,
      "message": "Successfully scraped",
      "data": [
        {
          "pageNumber": 1,
          "payload": {
            "templates": [],
            "mods": {
              "filter": {
                "tItemType": "nt_filter",
                "filterItems": [
                  {
                    "name": "category",
                    "unfoldRow": "2",
                    "title": "Category",
                    "urlKey": "category",
                    "hidden": false,
                    "locked": false,
                    "type": "category",
                    "uniqueName": "category",
                    "value": "",
                    "displayValue": "",
                    "options": [
                      {
                        "url": "/beli-sabun-pembersih-wajah/?q=159947433",
                        "title": "Facial Cleanser",
                        "value": "beli-sabun-pembersih-wajah"
                      }
                    ]
                  },
                  {
                    "name": "brand",
                    "unfoldRow": "-1",
                    "title": "Brand",
                    "urlKey": "ppath",
                    "hidden": false,
                    "locked": false,
                    "type": "brand",
                    "uniqueName": "20000",
                    "options": [
                      {
                        "title": "Men's Biore",
                        "value": "men-s-biore"
                      }
                    ]
                  },
                  {
                    "name": "service",
                    "unfoldRow": "4",
                    "title": "Service & Promotion",
                    "urlKey": "service",
                    "hidden": false,
                    "locked": false,
                    "type": "multiple",
                    "uniqueName": "service",
                    "options": [
                      {
                        "title": "Fulfilled By Lazada",
                        "value": "FBL",
                        "activeIcon": "",
                        "normalIcon": ""
                      },
                      {
                        "title": "Cash On Delivery",
                        "value": "COD",
                        "activeIcon": "",
                        "normalIcon": ""
                      },
                      {
                        "title": "Coins",
                        "value": "coins",
                        "activeIcon": "",
                        "normalIcon": ""
                      },
                      {
                        "title": "6.6 Sale",
                        "value": "gcp_D6",
                        "activeIcon": "https://img.lazcdn.com/us/lazgcp/e0557f6e-169b-45ec-91c4-e9e30eb17160_ALL-60-60.png",
                        "normalIcon": "https://img.lazcdn.com/us/lazgcp/e0557f6e-169b-45ec-91c4-e9e30eb17160_ALL-60-60.png"
                      }
                    ]
                  },
                  {
                    "name": "location",
                    "unfoldRow": "3",
                    "title": "Shipped From",
                    "urlKey": "location",
                    "hidden": false,
                    "locked": false,
                    "type": "location",
                    "uniqueName": "location",
                    "value": [],
                    "options": [
                      {
                        "title": "East Java",
                        "value": "A-ID-4"
                      },
                      {
                        "title": "Jabodetabek",
                        "value": "A-ID-1"
                      },
                      {
                        "title": "North Sumatera",
                        "value": "A-ID-6"
                      },
                      {
                        "title": "Kota Depok",
                        "value": "R80010434"
                      },
                      {
                        "title": "Kab. Sidoarjo",
                        "value": "R80010303"
                      },
                      {
                        "title": "Kab. Deli Serdang",
                        "value": "R80010265"
                      }
                    ]
                  },
                  {
                    "showMin": "Min",
                    "showMax": "Max",
                    "name": "price",
                    "unfoldRow": "2",
                    "title": "Price",
                    "urlKey": "price",
                    "hidden": false,
                    "locked": false,
                    "type": "price",
                    "uniqueName": "price"
                  },
                  {
                    "name": "rating",
                    "unfoldRow": "2",
                    "title": "Rating",
                    "urlKey": "rating",
                    "hidden": false,
                    "locked": false,
                    "type": "rating",
                    "uniqueName": "rating",
                    "value": "0"
                  },
                  {
                    "name": "attribute",
                    "unfoldRow": "-1",
                    "title": "Skin Care Benefits",
                    "urlKey": "ppath",
                    "hidden": false,
                    "locked": false,
                    "type": "multiple",
                    "uniqueName": "40385",
                    "options": [
                      {
                        "title": "Oil Control",
                        "value": "40385:135942"
                      },
                      {
                        "title": "Pore Control",
                        "value": "40385:135951"
                      },
                      {
                        "title": "Brightening",
                        "value": "40385:228695"
                      }
                    ]
                  },
                  {
                    "name": "attribute",
                    "unfoldRow": "-1",
                    "title": "Skin Types",
                    "urlKey": "ppath",
                    "hidden": false,
                    "locked": false,
                    "type": "multiple",
                    "uniqueName": "30740",
                    "options": [
                      {
                        "title": "Normal",
                        "value": "30740:60807"
                      },
                      {
                        "title": "Oily",
                        "value": "30740:60839"
                      }
                    ]
                  },
                  {
                    "name": "attribute",
                    "unfoldRow": "-1",
                    "title": "Skin Concerns",
                    "urlKey": "ppath",
                    "hidden": false,
                    "locked": false,
                    "type": "multiple",
                    "uniqueName": "40391",
                    "options": [
                      {
                        "title": "Oiliness",
                        "value": "40391:71277"
                      },
                      {
                        "title": "Dullness",
                        "value": "40391:135957"
                      }
                    ]
                  },
                  {
                    "name": "attribute",
                    "unfoldRow": "-1",
                    "title": "Product Feature",
                    "urlKey": "ppath",
                    "hidden": false,
                    "locked": false,
                    "type": "multiple",
                    "uniqueName": "120024468",
                    "options": [
                      {
                        "title": "Paraben-free",
                        "value": "120024468:136834"
                      }
                    ]
                  },
                  {
                    "name": "attribute",
                    "unfoldRow": "-1",
                    "title": "Product Form",
                    "urlKey": "ppath",
                    "hidden": false,
                    "locked": false,
                    "type": "multiple",
                    "uniqueName": "30780",
                    "options": [
                      {
                        "title": "Foam",
                        "value": "30780:68564"
                      }
                    ]
                  },
                  {
                    "name": "attribute",
                    "unfoldRow": "-1",
                    "title": "Ingredient Preference",
                    "urlKey": "ppath",
                    "hidden": false,
                    "locked": false,
                    "type": "multiple",
                    "uniqueName": "120150202",
                    "options": [
                      {
                        "title": "Niacinamide",
                        "value": "120150202:123811818"
                      }
                    ]
                  },
                  {
                    "name": "attribute",
                    "unfoldRow": "-1",
                    "title": "Special Claims",
                    "urlKey": "ppath",
                    "hidden": false,
                    "locked": false,
                    "type": "multiple",
                    "uniqueName": "120023693",
                    "options": [
                      {
                        "title": "BPOM Registered",
                        "value": "120023693:127537215"
                      }
                    ]
                  }
                ],
                "filteredQuatity": "1",
                "filteredDoneText": "Product",
                "title": "Search Filter",
                "pos": 0
              },
              "listItems": [
                {
                  "name": "Men's Biore Double Scrub Facial Foam Cool Oil Clear 100gr",
                  "nid": "159947433",
                  "itemId": "159947433",
                  "icons": [
                    {
                      "domClass": "120014",
                      "text": "Voucher save 4%",
                      "type": "text",
                      "group": "2",
                      "showType": "0",
                      "bizType": "voucherApplied"
                    },
                    {
                      "domClass": "225188",
                      "type": "img",
                      "group": "3",
                      "showType": "0",
                      "bizType": "campaign"
                    },
                    {
                      "domClass": "68675",
                      "type": "img",
                      "group": "3",
                      "showType": "0",
                      "bizType": "lazMall"
                    }
                  ],
                  "image": "https://id-live-01.slatic.net/p/282e6a2fbf26571c3c5eab569e609852.jpg",
                  "isSmartImage": false,
                  "originalPriceShow": "",
                  "priceShow": "Rp32.200",
                  "discount": "4% Off",
                  "ratingScore": "4.864764860348532",
                  "review": "8378",
                  "location": "Kota Depok",
                  "thumbs": [],
                  "sellerName": "KAO Store",
                  "sellerId": "1000001053",
                  "brandName": "Men's Biore",
                  "brandId": "116687",
                  "cheapest_sku": "BI568HBACHV3ANID-136017",
                  "skuId": "181742191",
                  "sku": "BI568HBACHV3ANID",
                  "categories": [
                    3509,
                    3902,
                    18067,
                    18216
                  ],
                  "price": "32200",
                  "restrictedAge": 0,
                  "inStock": true,
                  "originalPrice": "33600",
                  "clickTrace": "query:159947433;nid:159947433;src:LazadaMainSrp;rn:223837b9b80f3a6303f7c1f263ef8577;region:id;sku:BI568HBACHV3ANID;price:32200;client:desktop;supplier_id:1000001053;session_id:;biz_source:h5_internal;slot:0;utlog_bucket_id:470687;asc_category_id:18216;item_id:159947433;sku_id:181742191;shop_id:16328;templateInfo:-1_A3_C#107879_E#",
                  "itemSoldCntShow": "41.5K sold",
                  "longImageDisplayable": false,
                  "skus": [
                    {
                      "id": "BI568HBACHV3ANID-136017"
                    }
                  ],
                  "promotionId": "",
                  "isSponsored": false,
                  "tItemType": "nt_product",
                  "skuType": "2",
                  "adFlag": "0",
                  "directSimilarUrl": "https://native.m.lazada.com/dynamicxresult?item_img=https%3A%2F%2Fid-live-01.slatic.net%2Fp%2F282e6a2fbf26571c3c5eab569e609852.jpg&src=srp_findsimilar&item_id=159947433&sku_id=181742191¶ms=%7B%22sub_src%22%3A%22long_press%22%2C%22src%22%3A%22srp_findsimilar%22%7D&m=tpp_findSimilar&q=159947433&similarType=findSimilarV1&price=32200.00&spuTriggerItem=0",
                  "gridTitleLine": "2",
                  "isFission": "0",
                  "isBadgeAutoScroll": false,
                  "showCart": false,
                  "showBackIcon": false,
                  "showUnitPrice": false,
                  "itemUrl": "//www.lazada.co.id/products/pdp-i159947433.html",
                  "querystring": "fs_ab=2&priceCompare=skuId%3A181742191%3Bsource%3Alazada-search-voucher%3Bsn%3A223837b9b80f3a6303f7c1f263ef8577%3BoriginPrice%3A3220000%3BdisplayPrice%3A3220000%3BisGray%3Afalse%3BsinglePromotionId%3A910000049169004%3BsingleToolCode%3ApromPrice%3BvoucherPricePlugin%3A0%3Btimestamp%3A1780565848388&c=&ratingscore=4.864764860348532&freeshipping=0&source=search&channelLpJumpArgs=&fuse_fs=&search=1&sale=41518&price=3.22E%204&review=8378&location=Kota%20Depok&stock=1&lang=en&request_id=223837b9b80f3a6303f7c1f263ef8577&clickTrackInfo=query%253A159947433%253Bnid%253A159947433%253Bsrc%253ALazadaMainSrp%253Brn%253A223837b9b80f3a6303f7c1f263ef8577%253Bregion%253Aid%253Bsku%253ABI568HBACHV3ANID%253Bprice%253A32200%253Bclient%253Adesktop%253Bsupplier_id%253A1000001053%253Bsession_id%253A%253Bbiz_source%253Ah5_internal%253Bslot%253A0%253Butlog_bucket_id%253A470687%253Basc_category_id%253A18216%253Bitem_id%253A159947433%253Bsku_id%253A181742191%253Bshop_id%253A16328%253BtemplateInfo%253A-1_A3_C%2523107879_E%2523"
                }
              ],
              "breadcrumb": [
                {
                  "url": "https://www.lazada.co.id",
                  "title": "Home"
                },
                {
                  "title": "Search Results"
                }
              ],
              "sortBar": {
                "tItemType": "nt_sortbar",
                "filter": "Filter",
                "style": "wf",
                "sortItems": [
                  {
                    "name": "Popularity",
                    "tip": "Best Match",
                    "isActive": "true",
                    "value": "popularity",
                    "key": "sort",
                    "tabName": ""
                  },
                  {
                    "name": "Price-ASC",
                    "tip": "Price low to high",
                    "value": "priceasc",
                    "key": "sort",
                    "tabName": ""
                  },
                  {
                    "name": "Price-DESC",
                    "tip": "Price high to low",
                    "value": "pricedesc",
                    "key": "sort",
                    "tabName": ""
                  }
                ],
                "hiddenLayoutBtn": false,
                "showFilterBtn": true,
                "hasFilter": false
              },
              "resultTips": {
                "tItemType": "nt_resulttips",
                "tips": "1 items found for \"{$0}\"",
                "keywords": [
                  {
                    "text": "159947433"
                  }
                ]
              },
              "linksInfo": []
            },
            "mainInfo": {
              "errorMsg": "",
              "bizCode": 0,
              "serverParams": "translatedEnQuery%3D159947433%26in_sufficient_relevant_items%3D1%26",
              "totalResults": "1",
              "pageSize": "40",
              "page": "1",
              "hiddenLayoutBtn": false,
              "layoutInfo": {
                "listHeader": [
                  "didYouMean",
                  "campaignBanner",
                  "categoryBar",
                  "brandBar",
                  "superStore",
                  "banner",
                  "resultTips",
                  "navCategory",
                  "hotDeal",
                  "emptyResult",
                  "recQuery",
                  "emptyResultBannerTips",
                  "limitedResult",
                  "recommendTitle"
                ],
                "stickyHeader": [
                  "reminderBanner"
                ],
                "sceneHeader": [],
                "halfStickyHeader": [
                  "sortBar",
                  "preposeFilter"
                ],
                "bottomHalfStickyHeader": []
              },
              "pageType": "searchList",
              "gridTitleLine": "2",
              "trackParams": {},
              "userId": "",
              "cate_id": "",
              "hyperspaceInfo": "",
              "currency": "Rp",
              "currencyOnRight": "false",
              "currencySpace": "false",
              "isShowFeedbackForm": "false",
              "showThumbs": "false",
              "allProductURL": "/all-products/",
              "addToCartURL": "//cart.lazada.co.id/cart/api/add",
              "cluster": "fiber2_os30",
              "column": 2,
              "bucketId": "mobile_v4",
              "auctionType": 1,
              "noMorePages": true,
              "srpName": "LazadaMainSrp",
              "params": "{\"src\":\"h5_internal\"}",
              "searchScenario": "keyword",
              "hiddenShopChangeBtn": true,
              "selectedFilters": {},
              "expParams": {
                "disableTileSimple": "1",
                "isBadgeAutoScroll": "0",
                "filterLazyLoad": "0",
                "funnelFilterBtnPosition": "sortBarContainerRight",
                "enableShortStyle": "0",
                "enableFeedbackAssistant": "1",
                "disableScreenSimple": "1",
                "enableCompactLineHeight": "1",
                "disableFspImagePreload": "0"
              },
              "filterLazyLoad": false,
              "RN": "223837b9b80f3a6303f7c1f263ef8577",
              "style": "wf",
              "title": "159947433",
              "lang": "en",
              "venture": "ID",
              "themes": [
                {
                  "font-size": "11",
                  "labelMarginLeft": "0",
                  "color": "#FF0066",
                  "key": "120014"
                },
                {
                  "img": "https://img.lazcdn.com/us/lzd-onepiece/e09da34c2fe4a880c5c286c8850d9fbd1755779562798.png",
                  "width": "117",
                  "labelMarginLeft": "0",
                  "key": "68675",
                  "height": "36"
                },
                {
                  "img": "https://img.lazcdn.com/us/lazgcp/a49c11fa-1ebb-468b-bcd6-ad3ba69ef827_ALL-112-52.png",
                  "width": "112",
                  "labelMarginLeft": "0",
                  "key": "225188",
                  "height": "52"
                }
              ],
              "q": "159947433",
              "rt": {
                "all": 179
              },
              "serverResponseTime": 1780565848,
              "reqParams": "{\"src\":\"h5_internal\",\"refer\":\"https://www.lazada.co.id/\",\"scenario\":\"keyword-dynamic\",\"ab_test\":\"BASE:0\"}",
              "pageTitle": "159947433 - Harga & Promo Terbaik | Lazada Indonesia"
            },
            "seoInfo": {
              "pageTitle": "159947433 - Harga & Promo Terbaik | Lazada Indonesia",
              "description": "Cari 159947433? Lazada Indonesia adalah tempat berkumpulnya toko lokal dan internasional tepercaya! Nikmati belanja lebih aman dan nyaman dengan produk asli, pengiriman cepat, harga terbaik. Belanja sekarang dengan kebijakan pengembalian mudah dan pembayaran aman!",
              "robotsContent": "index,follow",
              "canonicalHref": "/catalog/?q=159947433",
              "androidDeepLink": "android-app://com.lazada.android/lazada/id/page?url_key=%2Fcatalog%2F%3Fq%3D159947433&utm_campaign=%2Fcatalog%2F%3Fq%3D159947433&utm_medium=organic&utm_source=google_app_indexing&utm_from=search",
              "nextHref": "/catalog/?page=2&q=159947433",
              "internalLink": [],
              "h1": "",
              "itemListSchema": {
                "@context": "https://schema.org",
                "@type": "ItemList",
                "name": "Best Selling 159947433 Products",
                "itemListElement": [
                  {
                    "@type": "ListItem",
                    "position": 1,
                    "item": {
                      "@type": "Product",
                      "name": "Men's Biore Double Scrub Facial Foam Cool Oil Clear 100gr",
                      "image": "https://id-live-01.slatic.net/p/282e6a2fbf26571c3c5eab569e609852.jpg",
                      "url": "//www.lazada.co.id/products/pdp-i159947433.html",
                      "aggregateRating": {
                        "@type": "AggregateRating",
                        "ratingValue": "4.9",
                        "ratingCount": 8378
                      }
                    }
                  }
                ]
              },
              "breadcrumbSchema": {
                "@context": "https://schema.org",
                "@type": "BreadcrumbList",
                "itemListElement": [
                  {
                    "item": {
                      "@id": "https://www.lazada.co.id",
                      "name": "Home"
                    },
                    "@type": "ListItem",
                    "position": 1
                  }
                ]
              },
              "breadcrumbData": [
                {
                  "categoryName": "Home",
                  "categoryLink": "https://www.lazada.co.id"
                },
                {
                  "categoryName": "Search Results"
                }
              ],
              "productSchema": {
                "@type": "Product",
                "@context": "https://schema.org",
                "name": "Men's Biore Double Scrub Facial Foam Cool Oil Clear 100gr",
                "image": "https://id-live-01.slatic.net/p/282e6a2fbf26571c3c5eab569e609852.jpg",
                "sku": "BI568HBACHV3ANID",
                "mpn": "181742191",
                "aggregateRating": {
                  "@type": "AggregateRating",
                  "ratingValue": "4.9",
                  "ratingCount": 8378
                },
                "brand": {
                  "@type": "Organization",
                  "name": "Men's Biore"
                }
              }
            }
          }
        }
      ],
      "tokenUsage": 10
    }
    ```
  



# ClawHub

import { Step, Steps } from 'fumadocs-ui/components/steps';

## ClawHub Integration

[ClawHub](https://clawhub.ai/) is a marketplace of skills and plugins for OpenClaw agents. Skills extend an agent's capabilities across research, integrations, and automation, and are installed from a central registry.

The [**MrScraper skill**](https://clawhub.ai/ai-mrscraper/skills/mrscraper) lets your agent run AI-powered web scraping and data extraction using natural language, backed by the MrScraper API.

## Overview

Once installed, your agent can:

* **Fetch pages**: Retrieve rendered HTML from public pages, with browser rendering and geo-targeted proxies.
* **Create AI scrapers**: Turn a natural-language instruction into a working scraper.
* **Rerun scrapers**: Apply an existing scraper configuration to new URLs, one at a time or in bulk.
* **Run manual scrapers**: Execute browser workflows built from selectors.
* **Fetch results**: Retrieve scrape results, paginated or by ID.

The skill makes direct HTTPS calls to the MrScraper API. There are no bundled scripts and no local installation step beyond installing the skill itself.

## Prerequisites

* An **OpenClaw agent** with the `openclaw` CLI available.
* A [**MrScraper API token**](https://app.mrscraper.com/api-tokens).

## Installation


  
    ### Install the skill

    ```bash
    openclaw skills install @ai-mrscraper/mrscraper
    ```
  

  
    ### Create a MrScraper API token

    1. Open the [MrScraper dashboard](https://app.mrscraper.com/).
    2. Click your user profile in the top-right corner.
    3. Select **API Tokens**.
    4. Click **New Token**.
    5. Enter a name and an expiration date.
    6. Click **Create** and copy the token.
  

  
    ### Store the token

    Expose the token to your agent as the `MRSCRAPER_API_TOKEN` environment variable.

    ```bash
    export MRSCRAPER_API_TOKEN="your-token-here"
    ```
  



  Never expose your API token in client-side code, logs, or commits. Store it in an environment variable or a server-side secret manager.


## Authentication

Every request the skill makes carries these headers:

```http
x-api-token: 
accept: application/json
content-type: application/json
```

MrScraper exposes two base URLs:

| API          | Base URL                        | Purpose                                      |
| ------------ | ------------------------------- | -------------------------------------------- |
| Web Fetching | `https://api.mrscraper.com`     | Fetch HTML from public pages                 |
| Platform     | `https://api.app.mrscraper.com` | Create, rerun, and read scrapers and results |

## Available Endpoints

### Web Fetching

Fetches a public page with browser rendering and geo-targeted proxies, and returns the page content.

```bash
curl 'https://api.mrscraper.com?url=https%3A%2F%2Fexample.com&timeout=120&geoCode=SG' \
  -H "x-api-token: "
```

| Parameter        | Required | Default | Description                                      |
| ---------------- | -------- | ------- | ------------------------------------------------ |
| `url`            | Yes      | —       | The target URL, URL-encoded.                     |
| `timeout`        | No       | `60`    | Maximum seconds to wait for the page to load.    |
| `geoCode`        | No       | —       | ISO country code for proxy routing.              |
| `blockResources` | No       | —       | Block images, CSS, and fonts for faster loading. |

### Create AI Scraper

Creates a scraper from a natural-language instruction and runs it.

```bash
curl -X POST "https://api.app.mrscraper.com/api/v1/scrapers-ai" \
  -H "x-api-token: " \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "message": "Extract title, price, stocks, and rating",
    "agent": "general"
  }'
```

**Agent types:**

| Agent     | Description                                                                          |
| --------- | ------------------------------------------------------------------------------------ |
| `general` | Standard pages such as product details, articles, and profiles. This is the default. |
| `listing` | Product listings, job boards, search results, and other paginated collections.       |
| `map`     | Website crawling that follows links to discover pages.                               |

### Rerun AI Scraper

Applies an existing scraper configuration to a new URL.

```bash
curl -X POST "https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun" \
  -H "x-api-token: " \
  -H "Content-Type: application/json" \
  -d '{
    "scraperId": "6695bf87-aaa6-46b0-b1ee-88586b222b0b",
    "url": "https://shopee.sg/"
  }'
```

### Bulk Rerun AI Scraper

Runs one scraper configuration across multiple URLs in a single request.

```bash
curl -X POST "https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun/bulk" \
  -H "x-api-token: " \
  -H "Content-Type: application/json" \
  -d '{
    "scraperId": "6695bf87-aaa6-46b0-b1ee-88586b222b0b",
    "urls": [
      "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
      "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"
    ]
  }'
```

### Manual Scraper Rerun

Executes a browser workflow built from selectors, for cases where you want explicit extraction rules instead of AI.

```bash
curl -X POST "https://api.app.mrscraper.com/api/v1/scrapers-manual-rerun" \
  -H "x-api-token: " \
  -H "Content-Type: application/json" \
  -d '{
    "scraperId": "6695bf87-aaa6-46b0-b1ee-88586b222b0b",
    "url": "https://books.toscrape.com/",
    "workflow": [
      {
        "type": "extract",
        "data": {
          "extraction_type": "text",
          "name": "book",
          "selector": "h3 a"
        }
      }
    ]
  }'
```

### Fetch Results

Returns paginated scrape results.

```bash
curl -X GET "https://api.app.mrscraper.com/api/v1/results?sortField=updatedAt&sortOrder=DESC&page=1&pageSize=10" \
  -H "x-api-token: "
```

### Fetch Result by ID

Returns the detail of a single result.

```bash
curl -X GET "https://api.app.mrscraper.com/api/v1/results/497f6eca-6276-4993-bfeb-53cbbbba6f08" \
  -H "x-api-token: "
```

## Map Agent Configuration

When you use the `map` agent, these parameters control how the crawl behaves:

| Parameter         | Description                                                                 |
| ----------------- | --------------------------------------------------------------------------- |
| `maxDepth`        | How many levels of links to follow. A value of 1–2 is recommended.          |
| `maxPages`        | Maximum number of pages to crawl.                                           |
| `limit`           | Maximum number of records to extract.                                       |
| `includePatterns` | Regex patterns for URLs to include. Separate multiple patterns with `\|\|`. |
| `excludePatterns` | Regex patterns for URLs to exclude. Separate multiple patterns with `\|\|`. |


  Set `maxDepth` to 1 or 2 and cap `maxPages`. Depth grows the crawl exponentially, and an unbounded crawl burns tokens fast.


## Error Handling

The API uses standard HTTP status codes:

| Status | Meaning                       |
| ------ | ----------------------------- |
| `400`  | Invalid request payload.      |
| `401`  | Missing or invalid API token. |
| `404`  | Resource not found.           |
| `429`  | Rate limit exceeded.          |
| `500`  | Internal error.               |

Retry `429` responses with exponential backoff.

## Data Scope

Data is transmitted only to MrScraper servers. Responses contain the extracted page content and its metadata. Never expose your API token in logs or commits.


# Dify

import { Step, Steps } from 'fumadocs-ui/components/steps';

The [MrScraper plugin for Dify](https://marketplace.dify.ai/plugin/ai-mrscraper/mrscraper)
adds web retrieval and search to your Dify applications. Use its tools as workflow
nodes or let an agent choose when to call them.

## Available tools

| Tool            | Tool ID                 | Purpose                                                                                 |
| --------------- | ----------------------- | --------------------------------------------------------------------------------------- |
| **Account**     | `mrscraper_account`     | Check your MrScraper account, token usage, and limits.                                  |
| **Scraper API** | `mrscraper_scraper_api` | Retrieve a page as HTML, Markdown, a screenshot, or a combination.                      |
| **Web Search**  | `mrscraper_web_search`  | Search the web for real-time results, including website URLs, titles, and descriptions. |

This guide covers plugin version **0.2.0**, which requires **Dify 1.17.0 or later**.
See the [marketplace listing](https://marketplace.dify.ai/plugin/ai-mrscraper/mrscraper)
for releases and compatibility details.

## Prerequisites

* A Dify workspace with permission to install tool plugins.
* A [MrScraper API token](https://app.mrscraper.com/api-tokens).
* A model configured in Dify if you want an agent or LLM node to process the results.

## Installation and authentication


  
    ### Install the plugin

    Open [MrScraper in the Dify Marketplace](https://marketplace.dify.ai/plugin/ai-mrscraper/mrscraper),
    click **Install**, and select your Dify environment. You can also find MrScraper
    from the marketplace inside your workspace's **Integrations** page.
  

  
    ### Add your API token

    In Dify, open **Integrations > Tools > Tool Plugin**, select **MrScraper**, and
    add a credential. Paste your MrScraper token into **API Token** and save it.

    Dify supplies this credential whenever a MrScraper tool runs. You do not need to
    pass the token as a tool input or include it in an agent prompt.
  

  
    ### Verify the connection

    Add a MrScraper **Account** tool node to a workflow and select the credential you
    saved. Run the node without additional inputs and check that it returns your
    account information.
  


For help creating a token, see [Token Plan](/docs/getting-started/api-token).
Dify's [tools guide](https://docs.dify.ai/en/cloud/use-dify/workspace/tools)
explains workspace credentials and using tools across applications.

## Fetch a page with Scraper API

Use **Scraper API** when you already have a target URL. For example, configure a
tool node with these inputs to retrieve Markdown for an LLM:

```json
{
  "url": "https://example.com",
  "html": false,
  "markdown": true
}
```

### Common inputs

| Input               | Default  | Description                                                     |
| ------------------- | -------- | --------------------------------------------------------------- |
| `url`               | Required | The page URL to retrieve.                                       |
| `html`              | `true`   | Include HTML in the response.                                   |
| `markdown`          | `false`  | Include Markdown in the response.                               |
| `screenshot`        | `false`  | Include a screenshot; this also enables browser rendering.      |
| `browser_rendering` | `auto`   | Select `auto`, `true`, or `false` to control browser rendering. |
| `timeout`           | `120`    | Page timeout in seconds, from `1` to `300`.                     |
| `geo_code`          | `US`     | Geolocation country code.                                       |
| `max_retries`       | `3`      | Retry limit. Set to `0` to disable retries.                     |

At least one of `html`, `markdown`, or `screenshot` must be enabled.

The tool also exposes advanced options for proxy routing (`proxy_country`,
`use_proxy`, `proxy`, `bypass_proxy`, `super`), page loading (`home_page`,
`wait_for_selector`, `wait_until`, `block_resources`), request data
(`cookie_jar`, `headers` as a JSON object string), retry budgets
(`token_cap`), response streaming (`stream`: `auto`, `true`, or `false`; the
plugin buffers streamed responses before returning them), and the optional
`x-discard-token-secret` request header (`discard_token_secret`). Start
with the defaults and adjust them for the target page. See the
[Scraper API guide](/docs/getting-started/scraper-api) for retrieval use cases.

## Discover pages with Web Search

Use **Web Search** when you have a topic or keyword rather than a target page.
It always returns structured JSON results.

```json
{
  "query": "web scraping tutorials",
  "country": "us",
  "num_results": 10
}
```

| Input         | Default         | Description                                                           |
| ------------- | --------------- | --------------------------------------------------------------------- |
| `query`       | Required        | The search query.                                                     |
| `country`     | Service default | Optional two-letter country code for localized results, such as `us`. |
| `num_results` | `10`            | Number of results to return, from `10` to `100`.                      |

Each result includes its `rank`, `title`, `url`, `description`, and `source`.

## Example: summarize a web page

Build a workflow that accepts a URL, retrieves its content, and produces a summary:

1. Add a **User Input** node with a required text input named `page_url`.
2. Choose **Add Node > Tools > MrScraper > Scraper API**.
3. Bind the tool's `url` input to `page_url`, set `html` to `false`, and set
   `markdown` to `true`.
4. Test the tool node and inspect its output. Bind the returned page text or
   Markdown field to an **LLM** node with an instruction such as
   `Summarize this page in three bullet points.`
5. Connect the LLM response to an **Output** node, then test the workflow with
   `https://example.com`.

For research workflows, run **Web Search** first, select a result URL from its
output, and pass that URL to **Scraper API** before summarizing the page.
See Dify's [Tool node guide](https://docs.dify.ai/en/cloud/use-dify/nodes/tools)
for adding nodes and choosing credentials.

## Use MrScraper in an agent

Add **Web Search** and **Scraper API** to your agent's available tools and select
your MrScraper credential. The agent can fill in the search query and page URL
on its own.

An example agent instruction:

```text
Search the web for pages relevant to the user's question. Fetch the most useful
pages with Scraper API, then answer using the retrieved content and cite the
source URLs.
```

## Working with outputs

The plugin returns the API response data directly. Objects and arrays appear as
Dify JSON messages; strings such as HTML appear as text. HTTP status and response
headers are not included in the tool output.

Default timeouts are 30 seconds for **Account**, 120 seconds (plus 30 seconds of
transport) for **Scraper API**, and 600 seconds for **Web Search**. The plugin's
maximum request time is 660 seconds.

Scraper API can return text or an object containing multiple requested formats.
Inspect a test run before connecting its output to another node. Screenshot data
is returned as base64 inside the JSON response; the plugin does not create a
separate Dify file.

## Troubleshooting

| Issue                                          | What to check                                                                                             |
| ---------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| Plugin will not install                        | Confirm your Dify version meets the plugin requirement and your workspace permissions allow installation. |
| Authentication fails                           | Check the provider's **API Token** credential and verify it with **Account**.                             |
| Web Search rejects the result count            | Set `num_results` to an integer from `10` to `100`.                                                       |
| A downstream node receives an unexpected value | Inspect the tool's JSON or text output and bind the specific content your next node needs.                |
| A request times out                            | Check the tool timeout and any shorter limits imposed by your Dify deployment or reverse proxy.           |

For self-hosted Dify, allow outbound HTTPS access to `api.app.mrscraper.com` and
`api.mrscraper.com`.

## Upgrading

* **From 0.1.x:** Tool IDs are unchanged. Web Search now takes `query`, `country`,
  and `num_results`, and always returns JSON; the `url`, `raw`, `format`,
  `region`, `language`, and `page` inputs were removed. Update any Web Search
  nodes and their bindings. Scraper API adds new options, and Account is
  unchanged.
* **From 0.0.1:** The old tools were replaced by the three tools listed above.
  Recreate affected nodes and reconnect their bindings. Extraction, crawl,
  scraper creation, rerun, and saved-result operations have no equivalent.


# Email



import { Mail, Cog } from 'lucide-react';
import { Step, Steps } from 'fumadocs-ui/components/steps';

## What Are Email Connections?

Email connections let you receive an email notification when specific events happen in your scraper, such as when a scrape starts, completes, or fails.

## Key Features


  } title="Instant Notifications">
    Get real-time updates on your scraper's status directly in your inbox.
  

  } title="Customizable Alerts">
    Choose which events you want to be notified about, such as completion or errors.
  


## Create an Email Connection

There are two ways to create an email connection in MrScraper:

### 1. Scraper


  
    Select the **scraper** that you want to add an email connection.
  

  
    **Scraper page** -> **Ellipsis** ( **⋮** ) button -> **Email Notifications** -> **Add new Email Connection**.
  

  
    **Email Connection page** -> **+ icon** in the top-right corner.
  

  
    Fill in the email connection details:

    * **Email Address**: The email address to which the notification should be sent
  

  
    Click **Submit** to save your email connection.
  


### 2. Settings


  
    Open your **MrScraper dashboard**.
  

  
    Click the **gear icon** -> select **Email Connection**.
  

  
    **Email Connection page** -> + icon in the top-right corner.
  

  
    Fill in the email connection details:

    * **Email Address**: The email address to which the notification should be sent
  

  
    Click **Submit** to save your email connection.
  


## Connect Email Notification to Scraper

After creating the email connection, follow these steps to connect it to your scraper:


  
    Go to your chosen **Scraper** page.
  

  
    Click the **Ellipsis** ( **⋮** ) button and click **Email Notifications**.
  

  
    Select the email connection you just created from the dropdown menu.
  

  
    Click **Save Email Notifications**.
  


## Example Result

Once the scraper finished, you will receive an email notification. Depending on the email type you selected, the email will contain either the result in the email body or as an attachment. Here's an example of what the email might look like:




# LangChain SDK

The [langchain-mrscraper](https://pypi.org/project/langchain-mrscraper/) package exposes the MrScraper API as LangChain `BaseTool` instances, so AI agents can fetch web pages as HTML, Markdown, or screenshots, search the web, and check account usage. It wraps the [Python SDK](/docs/integrations/python) (`mrscraper-sdk`), which is installed as a dependency. MrScraper is also listed in LangChain's [web browsing tools](https://docs.langchain.com/oss/python/integrations/tools/index#web-browsing).


  This integration provides **tools** that agents can explicitly call—not a document loader. For deterministic "URL → documents" ingestion into vector stores, a document loader is typically a better fit. MrScraper may offer that in a separate package later.


## Requirements

* **Python 3.9 or higher**
* `mrscraper-sdk` 0.9.x and `langchain-core` 0.3 or later (installed as dependencies)


  This page documents version **0.5.0**. See [langchain-mrscraper on PyPI](https://pypi.org/project/langchain-mrscraper/) for the latest release, version history, and installation details.


## Installation

Install the package from PyPI using pip:

```bash
pip install -U langchain-mrscraper
```

This installs both the LangChain integration and the underlying MrScraper SDK client.

## Authentication

Set your API token from the [MrScraper dashboard](https://app.mrscraper.com) as an environment variable:

```python
import os

os.environ["MRSCRAPER_API_KEY"] = "MRSCRAPER_API_TOKEN"
```


  The tools resolve the token in this order:

  1. The `token` parameter, such as `MrScraperToolkit(token="...")`
  2. The `mrscraper_api_key` parameter, such as `load_mrscraper_tools(mrscraper_api_key="...")`
  3. The `MRSCRAPER_API_KEY` environment variable
  4. The `MRSCRAPER_API_TOKEN` environment variable

  You can also pass an existing `mrscraper.MrScraper` client with `client=...`.


For more information on generating API tokens, see the [Generate Token](/docs/getting-started/api-token) guide.

## Quick Start

### Using MrScraperToolkit

Load all three tools with the toolkit:

```python
from langchain_mrscraper import MrScraperToolkit

# Using environment variable
tools = MrScraperToolkit().get_tools()

# Or with explicit token
tools = MrScraperToolkit(token="MRSCRAPER_API_TOKEN").get_tools()
```

### Using Convenience Loader

Alternatively, use the convenience function. It accepts the same configuration as the toolkit:

```python
from langchain_mrscraper import load_mrscraper_tools

tools = load_mrscraper_tools()
```

### Using with AI Agents

```python
from langgraph.prebuilt import create_react_agent
from langchain_openai import ChatOpenAI
from langchain_mrscraper import MrScraperToolkit

# Load MrScraper tools
tools = MrScraperToolkit(token="MRSCRAPER_API_TOKEN").get_tools()

# Create agent with OpenAI
agent = create_react_agent(ChatOpenAI(model="gpt-4o-mini"), tools)

# Use the agent
response = agent.invoke({
    "messages": [{"role": "user", "content": "Search the web for MrScraper and summarize the top result."}]
})
```


  This example requires `langgraph` and `langchain-openai` packages:

  ```bash
  pip install langgraph langchain-openai
  ```


## Available Tools

| Class                 | Tool name               | SDK method    | Purpose                                                                                                        |
| --------------------- | ----------------------- | ------------- | -------------------------------------------------------------------------------------------------------------- |
| `MrScraperScraperAPI` | `mrscraper_scraper_api` | `scraper_api` | Fetch a public web page as rendered HTML, Markdown, screenshot, or a combination                               |
| `MrScraperWebSearch`  | `mrscraper_web_search`  | `web_search`  | Search the web to retrieve real-time search results, including relevant website URLs, titles, and descriptions |
| `MrScraperAccount`    | `mrscraper_account`     | `account`     | Read account identity, subscription, quota, token usage, and rate limits                                       |


  Every tool returns a pretty-printed JSON string containing the SDK's `status_code`, `data`, and `headers`.


## Core Methods

### Scraper API

Fetch a public web page as rendered HTML, Markdown, screenshot, or a combination.

```python
import base64
import json
from langchain_mrscraper import MrScraperScraperAPI

tool = MrScraperScraperAPI(token="MRSCRAPER_API_TOKEN")

# HTML only (the default) returns the HTML string in data
page = json.loads(tool.invoke({"url": "https://example.com"}))
print(page["data"][:200])

# Markdown or a screenshot returns an object with one entry per output
combined = json.loads(tool.invoke({
    "url": "https://example.com",
    "html": True,
    "markdown": True,
    "screenshot": True,
}))
print(combined["data"]["markdown"][:200])
image = base64.b64decode(combined["data"]["screenshot"])
```

At least one of `html`, `markdown`, or `screenshot` must be `True`. The screenshot is a base64 string. Setting `screenshot=True` enables browser rendering, so combining it with `browser_rendering=False` is rejected.

**Parameters:**

| Parameter              | Type           | Required | Default | Description                                                                             |
| ---------------------- | -------------- | -------- | ------- | --------------------------------------------------------------------------------------- |
| `url`                  | string         | Yes      | -       | HTTP or HTTPS URL to fetch                                                              |
| `html`                 | boolean        | No       | `True`  | Return the rendered page HTML                                                           |
| `markdown`             | boolean        | No       | `False` | Return the page as Markdown                                                             |
| `screenshot`           | boolean        | No       | `False` | Return a base64 screenshot. Enables browser rendering                                   |
| `timeout`              | integer        | No       | `120`   | Page timeout in seconds (1-300)                                                         |
| `geo_code`             | string         | No       | `"US"`  | Geographic country code                                                                 |
| `proxy_country`        | string         | No       | `None`  | Proxy country code                                                                      |
| `block_resources`      | boolean        | No       | `False` | Block images, CSS, and fonts                                                            |
| `browser_rendering`    | boolean        | No       | `None`  | `True` forces JavaScript rendering, `False` disables it, `None` lets the service decide |
| `super`                | boolean        | No       | `False` | Use real-device routing                                                                 |
| `use_proxy`            | boolean        | No       | `True`  | Use a proxy for this request                                                            |
| `proxy`                | string         | No       | `None`  | Custom proxy in `username:password@host:port` format                                    |
| `wait_for_selector`    | string         | No       | `None`  | CSS selector to wait for                                                                |
| `wait_until`           | string         | No       | `None`  | Page lifecycle event to wait for, such as `"domcontentloaded"`                          |
| `home_page`            | boolean        | No       | `False` | Visit the site's root page first                                                        |
| `bypass_proxy`         | boolean        | No       | `False` | Enable the API's proxy bypass behavior                                                  |
| `cookie_jar`           | string         | No       | `None`  | Cookie configuration string                                                             |
| `headers`              | dict \| string | No       | `None`  | Target request headers, as a dictionary or JSON string                                  |
| `max_retries`          | integer        | No       | `3`     | Maximum retries, 0 or greater                                                           |
| `token_cap`            | integer        | No       | `None`  | Maximum retry-token budget, 0 or greater                                                |
| `stream`               | boolean        | No       | `None`  | Request a streamed response                                                             |
| `discard_token_secret` | string         | No       | `""`    | Discard-token secret, if required                                                       |

For more examples of what you can fetch, see [Scrape HTML](/docs/getting-started/scraper-api/scrape-html), [Scrape Markdown](/docs/getting-started/scraper-api/scrape-markdown), and [Get Screenshot](/docs/getting-started/scraper-api/get-screenshot).

### Web Search

Search the web to retrieve real-time search results, including relevant website URLs, titles, and descriptions.

```python
import json
from langchain_mrscraper import MrScraperWebSearch

tool = MrScraperWebSearch(token="MRSCRAPER_API_TOKEN")

result = json.loads(tool.invoke({
    "query": "python web scraping",
    "country": "us",
    "num_results": 10,
}))
for item in result["data"]["data"]:
    print(item["rank"], item["title"], item["url"])
```

**Parameters:**

| Parameter     | Type    | Required | Default | Description                                                                         |
| ------------- | ------- | -------- | ------- | ----------------------------------------------------------------------------------- |
| `query`       | string  | Yes      | -       | Search query. Must not be blank                                                     |
| `country`     | string  | No       | `None`  | Country code for localized results, such as `"us"`. Omit to use the service default |
| `num_results` | integer | No       | `10`    | Number of results to return (10-100)                                                |
| `timeout`     | float   | No       | `600.0` | HTTP timeout in seconds                                                             |

The response `data` contains `success`, `message`, and `data`, a list of ranked results. Each result has `rank`, `title`, `url`, `description`, and `source`.

### Account

Read account identity, subscription status, token usage and limits, remaining quota, and rate limits.

```python
from langchain_mrscraper import MrScraperAccount

tool = MrScraperAccount(token="MRSCRAPER_API_TOKEN")
print(tool.invoke({}))
```

**Parameters:**

| Parameter | Type  | Required | Default | Description             |
| --------- | ----- | -------- | ------- | ----------------------- |
| `timeout` | float | No       | `30.0`  | HTTP timeout in seconds |

See [Account](/docs/integrations/python#account) in the Python SDK documentation for the fields in the account summary.

## Tool Loading Options

You can obtain tools in several ways:

**1. Toolkit (Recommended)**

```python
from langchain_mrscraper import MrScraperToolkit

tools = MrScraperToolkit(token="MRSCRAPER_API_TOKEN").get_tools()
```

**2. Convenience Loader**

```python
from langchain_mrscraper import load_mrscraper_tools

tools = load_mrscraper_tools(token="MRSCRAPER_API_TOKEN")
```

**3. Selective Tool Loading**

`tool_names` filters by the exact tool names. An empty list returns no tools.

```python
from langchain_mrscraper import MrScraperToolkit

tools = MrScraperToolkit(
    token="MRSCRAPER_API_TOKEN",
    tool_names=["mrscraper_scraper_api", "mrscraper_web_search"],
).get_tools()
```

**4. Per-Tool Constructors**

```python
from langchain_mrscraper import MrScraperScraperAPI

scraper_api = MrScraperScraperAPI(token="MRSCRAPER_API_TOKEN")
```

**5. Shared SDK Client**

```python
from mrscraper import MrScraper
from langchain_mrscraper import MrScraperToolkit

client = MrScraper(token="MRSCRAPER_API_TOKEN")
tools = MrScraperToolkit(client=client).get_tools()
```

## Async Usage

Every tool supports both `.invoke(...)` and `await .ainvoke(...)`:

```python
import asyncio
from langchain_mrscraper import MrScraperScraperAPI

async def async_scrape():
    tool = MrScraperScraperAPI(token="MRSCRAPER_API_TOKEN")
    output = await tool.ainvoke({
        "url": "https://example.com",
        "timeout": 120,
    })
    print(output)

asyncio.run(async_scrape())
```

Synchronous `.invoke(...)` calls also work inside notebooks that already have a running event loop.

## Best Practices

### Token Management

Store your API token securely:

```python
import os

# Load from .env file
from dotenv import load_dotenv
load_dotenv()

token = os.getenv("MRSCRAPER_API_KEY")
```

### Error Handling

Each tool validates its input before sending a request. Invalid URLs, out-of-range limits, and unsupported fields are rejected by the tool schema. API, authentication, and network errors are raised from the [Python SDK](/docs/integrations/python#error-handling).

```python
from langchain_mrscraper import MrScraperScraperAPI

try:
    tool = MrScraperScraperAPI(token="MRSCRAPER_API_TOKEN")
    output = tool.invoke({"url": "https://example.com"})
except Exception as e:
    print(f"Scraping error: {e}")
```

### Tool Selection

Only load the tools your agent needs:

```python
from langchain_mrscraper import MrScraperToolkit

# Good: Only load needed tools
tools = MrScraperToolkit(tool_names=["mrscraper_web_search"]).get_tools()

# Loads all three tools
tools = MrScraperToolkit().get_tools()
```

## Migrating from 0.3.x

Version 0.5.0 requires `mrscraper-sdk>=0.9.0,<0.10.0`, which exposes exactly three public methods. The toolkit now returns three tools instead of ten.

| 0.3.x tool                                                                                          | 0.5.0 replacement                                                                                                 |
| --------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `MrScraperHTML` (`mrscraper_scraper_html`)                                                          | `MrScraperScraperAPI` with `{"url": ...}`                                                                         |
| `MrScraperMarkdown` (`mrscraper_scraper_markdown`)                                                  | `MrScraperScraperAPI` with `{"url": ..., "html": False, "markdown": True}`                                        |
| `MrScraperScreenshot` (`mrscraper_get_screenshot`)                                                  | `MrScraperScraperAPI` with `{"url": ..., "html": False, "screenshot": True}`                                      |
| `MrScraperJSON`, `MrScraperMultiplePage`, `MrScraperSitemap`, `MrScraperBrand`, `MrScraperGetBrand` | Removed. Fetch pages with `MrScraperScraperAPI` and let your model extract data, or pin `langchain-mrscraper<0.4` |
| `MrScraperWebSearch` (`url`, `raw`, `format`, `region`, `language`, `page`, `render_js`)            | `MrScraperWebSearch` (`query`, `country`, `num_results`, `timeout`)                                               |
| `MrScraperAccount`                                                                                  | Unchanged                                                                                                         |
| `MrScraperFetchHTML` import alias                                                                   | Removed. Use `MrScraperScraperAPI`                                                                                |

* Screenshots now come back as a base64 string in `data["screenshot"]`, together with any other requested outputs. The 0.3.x screenshot `full_page` option and its `timeout=300` and `wait_until="domcontentloaded"` defaults no longer exist; pass `timeout` and `wait_until` explicitly when needed.
* Update `tool_names` filters to `mrscraper_scraper_api`, `mrscraper_web_search`, and `mrscraper_account`.
* The scraper and result tools from earlier versions (create, rerun, and bulk-rerun scrapers, and result listing) remain removed. Call the [REST API](/docs/api/overview) directly for scrapers and results.


# Node.js SDK

The official [`@mrscraper/sdk`](https://www.npmjs.com/package/@mrscraper/sdk) package exposes three functions: `scraper_api` to fetch page content (HTML, Markdown, or screenshots), `web_search` to retrieve real-time web search results, and `account` to check your subscription and token usage from your Node.js applications.

## Requirements

* **Node.js 18 or later**
* **ES Modules enabled** in your `package.json`

```json title="package.json"
{
  "type": "module"
}
```


  This page documents version **5.0.0**. See [@mrscraper/sdk on npm](https://www.npmjs.com/package/@mrscraper/sdk) for the latest release, version history, and installation details.


## Installation

Install the SDK using npm:

```bash
npm install @mrscraper/sdk
```

## Authentication

Set your API token as an environment variable. The SDK reads `MRSCRAPER_API_TOKEN` automatically. You can get your API token from the [MrScraper dashboard](https://app.mrscraper.com).

```bash
export MRSCRAPER_API_TOKEN="MRSCRAPER_API_TOKEN"
```


  All functions accept an optional `token` in their options object if you need to override `MRSCRAPER_API_TOKEN` for specific requests.


For more information on generating API tokens, see the [Generate Token](/docs/getting-started/api-token) guide.

## Quick Start

```ts
import { scraper_api, web_search, account, MrScraperError } from "@mrscraper/sdk";

try {
  // 1. Fetch page content
  const page = await scraper_api("https://example.com");
  console.log(page.status_code, page.data);

  // 2. Search the web
  const search = await web_search("python web scraping", { country: "us" });
  console.log(search.data.data);

  // 3. Check remaining tokens
  const status = await account();
  console.log(status.data.token_remaining);
} catch (err) {
  if (err instanceof MrScraperError) {
    console.error(`[${err.status ?? "network"}] ${err.message}`);
  } else {
    throw err;
  }
}
```

## Response Format

All functions are async and resolve to a `MrScraperResponse` object:

| Property      | Type                     | Description                                                                                                  |
| ------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------ |
| `status_code` | number                   | HTTP status code of the API response                                                                         |
| `data`        | `T` (default `unknown`)  | Parsed JSON for JSON responses, a string for HTML and other text, or a Node.js `Buffer` for binary responses |
| `headers`     | `Record` | Response headers                                                                                             |

You can type `data` with a generic parameter:

```ts
const result = await scraper_api("https://example.com");
```

## Core Methods

### Scraper API

Use `scraper_api` to fetch a page through the MrScraper [Scraper API](/docs/getting-started/scraper-api). The service handles JavaScript rendering, bot detection, and optional geolocation proxying.

```ts
import { writeFile } from "node:fs/promises";
import { scraper_api } from "@mrscraper/sdk";

// HTML only (the default) returns the page as a string
const page = await scraper_api("https://example.com");
console.log(page.data);

// Markdown or a screenshot returns an object with one entry per output
const result = await scraper_api("https://example.com", {
  html: true,
  markdown: true,
  screenshot: true,
  geo_code: "US",
  timeout: 120,
  // token: "optional_override_token"
});
if (typeof result.data !== "string") {
  console.log(result.data.markdown);
  if (result.data.screenshot) {
    await writeFile("page.jpg", Buffer.from(result.data.screenshot, "base64"));
  }
}
```

Requesting only HTML returns the page as a string. Requesting Markdown or a screenshot returns an object such as `{ html, markdown, screenshot }` with an entry for each requested output, typed as `ScraperApiData`. The screenshot is a base64-encoded JPEG. At least one of `html`, `markdown`, or `screenshot` must be `true`.

The SDK sends a `GET` request to `https://api.mrscraper.com`, with the target URL and every option in the query string.

**Parameters:**

The first argument is the target URL. All other parameters go in the optional options object.

| Parameter              | Type                     | Required | Default | Description                                                                                                            |
| ---------------------- | ------------------------ | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------- |
| `url`                  | string                   | Yes      | -       | Target page URL (first argument)                                                                                       |
| `token`                | string                   | No       | -       | Per-request token override                                                                                             |
| `html`                 | boolean                  | No       | `true`  | Return HTML content                                                                                                    |
| `markdown`             | boolean                  | No       | `false` | Return Markdown content                                                                                                |
| `timeout`              | number                   | No       | `120`   | Page timeout in seconds (1-300)                                                                                        |
| `geo_code`             | string                   | No       | `"US"`  | Two-letter country code for geolocation                                                                                |
| `proxy_country`        | string \| null           | No       | -       | Optional proxy country                                                                                                 |
| `browser_rendering`    | boolean \| null          | No       | -       | `true` forces browser rendering, `false` disables it, unset tries without a browser first and falls back automatically |
| `screenshot`           | boolean                  | No       | `false` | Capture a screenshot. Also forces browser rendering                                                                    |
| `super`                | boolean                  | No       | `false` | Enable real device routing                                                                                             |
| `use_proxy`            | boolean                  | No       | `true`  | Allow the service to use a proxy                                                                                       |
| `proxy`                | string \| null           | No       | -       | Custom proxy in `username:password@host:port` format                                                                   |
| `home_page`            | boolean                  | No       | `false` | Visit the site's root page before the target URL                                                                       |
| `wait_for_selector`    | string \| null           | No       | -       | CSS selector to wait for before returning content                                                                      |
| `wait_until`           | string \| null           | No       | -       | Page lifecycle event to wait for, such as `"domcontentloaded"`                                                         |
| `block_resources`      | boolean \| null          | No       | `false` | Block images, CSS, and fonts to speed up the request                                                                   |
| `bypass_proxy`         | boolean                  | No       | `false` | Enable the API's proxy bypass behavior                                                                                 |
| `cookie_jar`           | string \| null           | No       | -       | Cookie configuration, e.g. `"domain=example.com&country=us&max_used=100&max_age=50m"`                                  |
| `headers`              | object \| string \| null | No       | -       | Headers forwarded to the target site, as an object or JSON string                                                      |
| `max_retries`          | number                   | No       | `3`     | Maximum number of retries, a nonnegative integer                                                                       |
| `token_cap`            | number \| null           | No       | -       | Optional token budget for retries                                                                                      |
| `stream`               | boolean \| null          | No       | -       | Request a streamed response. The SDK buffers it into the normal return value                                           |
| `discard_token_secret` | string                   | No       | `""`    | Value sent in the `x-discard-token-secret` request header                                                              |


  Setting `screenshot: true` always enables browser rendering, even if you pass `browser_rendering: false`.


For more examples of what you can fetch, see [Scrape HTML](/docs/getting-started/scraper-api/scrape-html), [Scrape Markdown](/docs/getting-started/scraper-api/scrape-markdown), and [Get Screenshot](/docs/getting-started/scraper-api/get-screenshot).

### Web Search

Use `web_search` to search the web and retrieve real-time search results, including relevant website URLs, titles, and descriptions. The first argument is the search query.

```ts
import { web_search } from "@mrscraper/sdk";

const result = await web_search("python web scraping", {
  country: "us",
  num_results: 10,
  // token: "optional_override_token"
});
for (const item of result.data.data) {
  console.log(item.rank, item.title, item.url);
}
```

The SDK sends this request:

```bash
curl --location 'https://api.mrscraper.com' \
--header 'x-api-token: {MRSCRAPER_API_TOKEN}' \
--header 'Content-Type: application/json' \
--data '{
    "action": "websearch",
    "query": "python web scraping",
    "country": "us",
    "numResults": 10
}'
```

**Parameters:**

| Parameter     | Type           | Required | Default | Description                                                                            |
| ------------- | -------------- | -------- | ------- | -------------------------------------------------------------------------------------- |
| `query`       | string         | Yes      | -       | Search query (first argument). Must not be empty                                       |
| `token`       | string         | No       | -       | Per-request token override                                                             |
| `country`     | string \| null | No       | -       | Country code for localized results, such as `"us"`. Omitted to use the service default |
| `num_results` | number         | No       | `10`    | Number of results, an integer from 10 to 100. Sent as `numResults`                     |
| `timeout`     | number         | No       | `600`   | Request timeout in seconds                                                             |

**Response (`result.data`, typed as `WebSearchResponse`):**

| Property  | Type                | Description                                                                  |
| --------- | ------------------- | ---------------------------------------------------------------------------- |
| `success` | boolean             | Whether the search succeeded                                                 |
| `message` | string              | Status message                                                               |
| `data`    | `WebSearchResult[]` | Ranked results. Each has `rank`, `title`, `url`, `description`, and `source` |

### Account

Use `account` to check the identity, subscription, token quota, and rate limits of the account that owns the API token.

```ts
import { account } from "@mrscraper/sdk";

const result = await account({
  timeout: 30,
  // token: "optional_override_token"
});
console.log(result.data.subscription_status);
console.log(result.data.token_remaining);
console.log(result.data.user.email);
```

**Parameters:**

| Parameter | Type   | Required | Default | Description                                       |
| --------- | ------ | -------- | ------- | ------------------------------------------------- |
| `token`   | string | No       | -       | Per-request token override                        |
| `timeout` | number | No       | `30`    | HTTP request timeout in seconds. Must be positive |

**Account summary (`result.data`, typed as `AccountSummary`):**

| Property              | Type           | Description                                   |
| --------------------- | -------------- | --------------------------------------------- |
| `subscription_status` | string \| null | Subscription status, such as `"active"`       |
| `enterprise`          | boolean        | Whether the account is on an enterprise plan  |
| `token_usage`         | number         | Tokens used in the current period             |
| `token_limit`         | number         | Token limit for the current period            |
| `token_remaining`     | number         | Remaining tokens, never below 0               |
| `usage_percent`       | number         | Token usage as a percentage of the limit      |
| `rate_limit`          | number         | Request rate limit                            |
| `rate_ttl`            | number         | Rate-limit window                             |
| `auto_renew`          | boolean        | Whether the subscription renews automatically |
| `ends_at`             | string \| null | Subscription end date                         |
| `user`                | object         | `name`, `email`, and `verified`               |


  API keys, billing identifiers, and sensitive response headers are excluded from the result.


## TypeScript Types

The package ships its own type definitions:

```ts
import type {
  ScraperApiOptions,
  ScraperApiData,
  WebSearchOptions,
  WebSearchResponse,
  WebSearchResult,
  AccountOptions,
  AccountSummary,
  MrScraperResponse,
} from "@mrscraper/sdk";
```

## Error Handling

All functions reject with `MrScraperError` for API, authentication, network, and timeout failures. Invalid options throw a `TypeError` or `RangeError` before any request is sent.

### Exception Properties

| Property  | Type                | Description                                                              |
| --------- | ------------------- | ------------------------------------------------------------------------ |
| `message` | string              | Human-readable error message                                             |
| `status`  | number \| undefined | HTTP status code (401, 429, 500, etc.) or `undefined` for network errors |
| `name`    | string              | Always `"MrScraperError"`                                                |

### Example Error Handling

```ts
import { scraper_api, MrScraperError } from "@mrscraper/sdk";

try {
  await scraper_api("https://example.com");
} catch (err) {
  if (err instanceof MrScraperError) {
    console.error(`Error: ${err.message}`);
    console.error(`Status: ${err.status ?? "network error"}`);
  } else {
    // Handle unexpected errors
    throw err;
  }
}
```

### Best Practices

* Always wrap SDK calls in try-catch blocks
* Check `err instanceof MrScraperError` before accessing error properties
* Log errors with appropriate context for debugging
* Implement retry logic for network errors (when `status` is `undefined`)
* Verify your API token if you encounter status `401`

## Migrating from Earlier Versions

### From Version 4

Version 5.0.0 returns to three public functions. The 4.x format-specific functions (`scraper_html`, `scraper_markdown`, `get_screenshot`, `scraper_json`, `scraper_multiple_page`, `scraper_sitemap`, `scrape_brand`, and `get_brand`) and their option types are no longer exported. Use `scraper_api` with output flags instead:

| Version 4                                   | Version 5                                                       |
| ------------------------------------------- | --------------------------------------------------------------- |
| `scraper_html(url, options)`                | `scraper_api(url, options)`                                     |
| `scraper_markdown(url, options)`            | `scraper_api(url, { ...options, html: false, markdown: true })` |
| `get_screenshot(url, { full_page: false })` | `scraper_api(url, { html: false, screenshot: true })`           |
| `web_search(query, { region: "us" })`       | `web_search(query, { country: "us" })`                          |
| `ScraperOptions` type                       | `ScraperApiOptions`                                             |

A screenshot from `scraper_api` is a base64-encoded JPEG in `data.screenshot`, not a `Buffer`.

### From Version 3

* **`web_search`** now calls the MrScraper API with `action: "websearch"` and returns JSON. Its first argument is the search query, and it accepts `country` and `num_results` (10 to 100). The `raw`, `format`, `region`, `language`, `page`, and `render_js` options and search URL input were removed. Results are at `result.data.data`.
* **`scraper_api`** sends a `GET` request with every option in the query string, and adds the `wait_until` option. At least one of `html`, `markdown`, or `screenshot` must be `true`. HTML-only requests return a string, and requests with Markdown or a screenshot return an object with one entry per output.
* **`account`** is unchanged.

### From Version 3.0 to 3.1

Version 3.1.0 adds the [`account`](#account) function and the `AccountOptions` and `AccountSummary` types. No existing functions changed.

### From Version 2

Version 3 renames the public functions and their option types. Request options and the `{ status_code, data, headers }` response are unchanged.

| Version 2                     | Version 3                   |
| ----------------------------- | --------------------------- |
| `fetch_content(url, options)` | `scraper_api(url, options)` |
| `FetchContentOptions`         | `ScraperApiOptions`         |

The version 2 names are no longer exported.

### From Version 1

`fetchHtml` and all AI scraper, manual scraper, and results functions (`createAiScraper`, `rerunAiScraper`, `bulkRerunAiScraper`, `rerunManualScraper`, `bulkRerunManualScraper`, `getAllResults`, `getResultById`) were removed in version 2.

* Replace `fetchHtml` with [`scraper_api`](#scraper-api). It now returns a response wrapper instead of only the API payload, and options use `snake_case` (for example, `geoCode` is now `geo_code`).
* For scrapers and results, call the [REST API](/docs/api/overview) directly.

## Migration from Firecrawl

If you're migrating from the [Firecrawl Node SDK](https://docs.firecrawl.dev/sdks/node), use this mapping to find equivalent MrScraper functions.

| Firecrawl Method                                                                                 | MrScraper Equivalent                                                    |
| ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------- |
| [Scrape a URL](https://docs.firecrawl.dev/sdks/node#scraping-a-url) with HTML or Markdown output | [`scraper_api`](#scraper-api) with `html: true` and/or `markdown: true` |
| Scrape a URL with a screenshot                                                                   | [`scraper_api`](#scraper-api) with `screenshot: true`                   |
| [Search](https://docs.firecrawl.dev/sdks/node)                                                   | [`web_search`](#web-search)                                             |


# Python SDK

The [mrscraper-sdk](https://pypi.org/project/mrscraper-sdk/) is a typed Python client for the MrScraper API. It exposes three methods: `scraper_api` to fetch page content (HTML, Markdown, or screenshots), `web_search` to retrieve real-time web search results, and `account` to check your subscription and token usage. All client methods are **async** and must be used with `asyncio` or another async runtime.

## Requirements

* **Python 3.9 or later**
* Basic familiarity with async/await syntax in Python


  This page documents version **0.9.0**. See the [mrscraper-sdk on PyPI](https://pypi.org/project/mrscraper-sdk/) for the latest release, version history, and metadata.


## Installation

Install the SDK from PyPI using pip:

```bash
pip install --upgrade mrscraper-sdk
```

Import the client in your Python code:

```python
from mrscraper import MrScraper
```

## Authentication

The client reads your API token from the `MRSCRAPER_API_TOKEN` environment variable automatically. You can get your API token from the [MrScraper dashboard](https://app.mrscraper.com).

```bash
export MRSCRAPER_API_TOKEN="MRSCRAPER_API_TOKEN"
```

```python
from mrscraper import MrScraper

client = MrScraper()  # uses MRSCRAPER_API_TOKEN
```

You can also pass the token explicitly:

```python
client = MrScraper(token="MRSCRAPER_API_TOKEN")
```


  Pass `http_client=` with your own `httpx.AsyncClient` to reuse connections across requests. Otherwise, the SDK creates a new client for each request.

  ```python
  import httpx
  from mrscraper import MrScraper

  async with httpx.AsyncClient() as http:
      client = MrScraper(http_client=http)
      result = await client.scraper_api("https://example.com")
  ```


For more information on generating API tokens, see the [Generate Token](/docs/getting-started/api-token) guide.

## Quick Start

```python
import asyncio
from mrscraper import MrScraper

async def main():
    client = MrScraper()

    # 1. Fetch page content
    page = await client.scraper_api("https://example.com")
    print(page["data"])

    # 2. Search the web
    search = await client.web_search("python web scraping", country="us")
    print(search["data"]["data"])

    # 3. Check remaining tokens
    account = await client.account()
    print(account["data"]["token_remaining"])

asyncio.run(main())
```

## Response Format

All methods return a dictionary with the same structure:

| Key           | Type    | Description                                                                                  |
| ------------- | ------- | -------------------------------------------------------------------------------------------- |
| `status_code` | integer | HTTP status code of the API response                                                         |
| `data`        | any     | Python values for JSON responses, a string for text responses, or bytes for binary downloads |
| `headers`     | dict    | Response headers                                                                             |

## Core Methods

### Scraper API

Use `scraper_api` to fetch a page through the MrScraper [Scraper API](/docs/getting-started/scraper-api). The service handles JavaScript rendering, bot detection, and optional geolocation proxying.

```python
import asyncio
from mrscraper import MrScraper

async def main():
    client = MrScraper()

    # HTML only (the default) returns the page as a string
    page = await client.scraper_api("https://example.com/product")
    print(page["data"])

    # Markdown or a screenshot returns a dictionary with one entry per output
    result = await client.scraper_api(
        "https://example.com/product",
        html=True,
        markdown=True,
        screenshot=True,
        geo_code="US",
        timeout=120,
    )
    print(result["data"]["markdown"])

asyncio.run(main())
```

Requesting only HTML returns the page as a string. Requesting Markdown or a screenshot returns a dictionary with an entry for each requested output, such as `html`, `markdown`, and `screenshot`. At least one of `html`, `markdown`, or `screenshot` must be `True`.

**Parameters:**

| Parameter              | Type           | Required | Default | Description                                                                                                             |
| ---------------------- | -------------- | -------- | ------- | ----------------------------------------------------------------------------------------------------------------------- |
| `url`                  | string         | Yes      | -       | Target page URL                                                                                                         |
| `html`                 | boolean        | No       | `True`  | Return HTML content                                                                                                     |
| `markdown`             | boolean        | No       | `False` | Return Markdown content                                                                                                 |
| `timeout`              | integer        | No       | `120`   | Page timeout in seconds (1-300)                                                                                         |
| `geo_code`             | string         | No       | `"US"`  | Two-letter country code for geolocation (e.g., `"US"`, `"GB"`, `"SG"`)                                                  |
| `proxy_country`        | string         | No       | `None`  | Optional proxy country                                                                                                  |
| `browser_rendering`    | boolean        | No       | `None`  | `True` forces browser rendering, `False` disables it, `None` tries without a browser first and falls back automatically |
| `screenshot`           | boolean        | No       | `False` | Capture a screenshot. Also forces browser rendering                                                                     |
| `super`                | boolean        | No       | `False` | Enable real device routing                                                                                              |
| `use_proxy`            | boolean        | No       | `True`  | Allow the service to use a proxy                                                                                        |
| `proxy`                | string         | No       | `None`  | Custom proxy in `username:password@host:port` format                                                                    |
| `home_page`            | boolean        | No       | `False` | Visit the site's root page before the target URL                                                                        |
| `wait_for_selector`    | string         | No       | `None`  | CSS selector to wait for before returning content                                                                       |
| `wait_until`           | string         | No       | `None`  | Page lifecycle event to wait for, such as `"domcontentloaded"`                                                          |
| `block_resources`      | boolean        | No       | `False` | Block images, CSS, and fonts to speed up the request                                                                    |
| `bypass_proxy`         | boolean        | No       | `False` | Enable the API's proxy bypass behavior                                                                                  |
| `cookie_jar`           | string         | No       | `None`  | Cookie configuration, e.g. `"domain=example.com&country=us&max_used=100&max_age=50m"`                                   |
| `headers`              | dict \| string | No       | `None`  | Headers forwarded to the target site, as a dictionary or JSON string                                                    |
| `max_retries`          | integer        | No       | `3`     | Maximum number of retries                                                                                               |
| `token_cap`            | integer        | No       | `None`  | Optional token budget for retries                                                                                       |
| `stream`               | boolean        | No       | `None`  | Request a streamed response. The SDK buffers it into the normal return value                                            |
| `discard_token_secret` | string         | No       | `""`    | Value sent in the `x-discard-token-secret` request header                                                               |


  Setting `screenshot=True` always enables browser rendering, even if you pass `browser_rendering=False`.


For more examples of what you can fetch, see [Scrape HTML](/docs/getting-started/scraper-api/scrape-html), [Scrape Markdown](/docs/getting-started/scraper-api/scrape-markdown), and [Get Screenshot](/docs/getting-started/scraper-api/get-screenshot).

### Web Search

Use `web_search` to search the web and retrieve real-time search results, including relevant website URLs, titles, and descriptions. The first argument is the search query.

```python
import asyncio
from mrscraper import MrScraper

async def main():
    client = MrScraper()

    result = await client.web_search(
        "python web scraping",
        country="us",
        num_results=10,
    )
    for item in result["data"]["data"]:
        print(item["rank"], item["title"], item["url"])

asyncio.run(main())
```

The SDK sends this request:

```bash
curl --location 'https://api.mrscraper.com' \
--header 'x-api-token: {MRSCRAPER_API_TOKEN}' \
--header 'Content-Type: application/json' \
--data '{
    "action": "websearch",
    "query": "python web scraping",
    "country": "us",
    "numResults": 10
}'
```

**Parameters:**

| Parameter     | Type    | Required | Default | Description                                                                            |
| ------------- | ------- | -------- | ------- | -------------------------------------------------------------------------------------- |
| `query`       | string  | Yes      | -       | Search query. Must not be empty                                                        |
| `country`     | string  | No       | `None`  | Country code for localized results, such as `"us"`. Omitted to use the service default |
| `num_results` | integer | No       | `10`    | Number of results, an integer from 10 to 100. Sent as `numResults`                     |
| `timeout`     | float   | No       | `600.0` | Request timeout in seconds                                                             |

**Response (`result["data"]`):**

| Key       | Type    | Description                                                                  |
| --------- | ------- | ---------------------------------------------------------------------------- |
| `success` | boolean | Whether the search succeeded                                                 |
| `message` | string  | Status message                                                               |
| `data`    | list    | Ranked results. Each has `rank`, `title`, `url`, `description`, and `source` |

### Account

Use `account` to check the identity, subscription, token quota, and rate limits of the account that owns the API token.

```python
import asyncio
from mrscraper import MrScraper

async def main():
    client = MrScraper()

    result = await client.account(timeout=30)
    print(result["data"]["subscription_status"])
    print(result["data"]["token_remaining"])
    print(result["data"]["user"]["email"])

asyncio.run(main())
```

**Parameters:**

| Parameter | Type  | Required | Default | Description                                        |
| --------- | ----- | -------- | ------- | -------------------------------------------------- |
| `timeout` | float | No       | `30.0`  | Request timeout in seconds. Must be greater than 0 |

**Account summary (`result["data"]`):**

| Key                   | Type           | Description                                   |
| --------------------- | -------------- | --------------------------------------------- |
| `subscription_status` | string \| None | Subscription status, such as `"active"`       |
| `enterprise`          | boolean        | Whether the account is on an enterprise plan  |
| `token_usage`         | number         | Tokens used in the current period             |
| `token_limit`         | number         | Token limit for the current period            |
| `token_remaining`     | number         | Remaining tokens, never below 0               |
| `usage_percent`       | number         | Token usage as a percentage of the limit      |
| `rate_limit`          | number         | Request rate limit                            |
| `rate_ttl`            | number         | Rate-limit window                             |
| `auto_renew`          | boolean        | Whether the subscription renews automatically |
| `ends_at`             | string \| None | Subscription end date                         |
| `user`                | dict           | `name`, `email`, and `verified`               |


  API keys, billing identifiers, and sensitive response headers are excluded from the result.


## Error Handling

The SDK raises typed exceptions, which you can import from `mrscraper` or `mrscraper.exceptions`.

### Exception Types

| Exception             | When It's Raised                                  |
| --------------------- | ------------------------------------------------- |
| `MrScraperError`      | Base class for all SDK errors                     |
| `AuthenticationError` | Missing or invalid API token (HTTP 401)           |
| `APIError`            | Non-success API response (exposes `.status_code`) |
| `NetworkError`        | Network timeouts or connection failures           |

Invalid arguments, such as a `timeout` outside 1-300 or a `num_results` outside 10-100, raise `ValueError` before any request is sent.

### Example Error Handling

```python
from mrscraper import MrScraper, AuthenticationError, APIError, NetworkError

client = MrScraper()

try:
    result = await client.scraper_api("https://example.com")
except AuthenticationError:
    print("Authentication failed. Check your API token at https://app.mrscraper.com")
except APIError as e:
    print(f"API error {e.status_code}: {e}")
except NetworkError as e:
    print(f"Network problem: {e}")
```

### Best Practices

* Always wrap SDK calls in try-except blocks
* Log errors with appropriate context for debugging
* Implement retry logic for `NetworkError` exceptions
* Verify your API token if you encounter `AuthenticationError`

## Migrating from Earlier Versions

### From 0.7.x

Version 0.9.0 returns to three public methods. The 0.7.x format-specific methods are no longer public; use `scraper_api` with output flags instead:

| 0.7.x                                         | 0.9.0                                                  |
| --------------------------------------------- | ------------------------------------------------------ |
| `client.scraper_html(url)`                    | `client.scraper_api(url)`                              |
| `client.scraper_markdown(url)`                | `client.scraper_api(url, html=False, markdown=True)`   |
| `client.get_screenshot(url, full_page=False)` | `client.scraper_api(url, html=False, screenshot=True)` |
| `client.web_search(query, region="us")`       | `client.web_search(query, country="us")`               |

### From 0.6.0

* **`web_search`** now calls the MrScraper API with `action: "websearch"` and returns JSON. Its first argument is `query`, and it accepts `country` and `num_results` (10 to 100). The `raw`, `format`, `region`, `language`, `page`, and `render_js` options and search URL input were removed. Results are at `result["data"]["data"]`.
* **`scraper_api`** adds the `wait_until` option. At least one of `html`, `markdown`, or `screenshot` must be `True`. HTML-only requests return a string, and requests with Markdown or a screenshot return a dictionary with one entry per output.
* **`account`** is unchanged.

### From 0.5.0

Version 0.6.0 adds the [`account`](#account) method. No existing methods changed.

### From 0.4.0

Version 0.5.0 renames a public method. Arguments and response formats are unchanged.

| 0.4.0                       | 0.5.0                     |
| --------------------------- | ------------------------- |
| `client.fetch_content(...)` | `client.scraper_api(...)` |

The previous method names have been removed.

### From 0.3.x and Earlier

`fetch_html` and the AI scraper, manual scraper, results, and account methods (`create_scraper`, `rerun_scraper`, `bulk_rerun_ai_scraper`, `rerun_manual_scraper`, `bulk_rerun_manual_scraper`, `get_all_results`, `get_result_by_id`, `get_analytic_statuses`) are no longer part of the SDK. `get_subscription_account` is replaced by [`account`](#account).

* Replace `fetch_html` with [`scraper_api`](#scraper-api). It returns a dictionary with `status_code`, `data`, and `headers`.
* For scrapers, results, and analytics, call the [REST API](/docs/api/overview) directly.

## Migration from Firecrawl

If you're migrating from the [Firecrawl Python SDK](https://docs.firecrawl.dev/sdks/python), use this mapping to find equivalent MrScraper methods.

| Firecrawl Method                                                                                   | MrScraper Equivalent                                                  |
| -------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| [Scrape a URL](https://docs.firecrawl.dev/sdks/python#scraping-a-url) with HTML or Markdown output | [`scraper_api`](#scraper-api) with `html=True` and/or `markdown=True` |
| Scrape a URL with a screenshot                                                                     | [`scraper_api`](#scraper-api) with `screenshot=True`                  |
| [Search](https://docs.firecrawl.dev/sdks/python)                                                   | [`web_search`](#web-search)                                           |


# SQL/Database

import { SquaresUnite, ChartBar, Cog } from 'lucide-react';
import { Step, Steps } from 'fumadocs-ui/components/steps';

## What Are SQL Connections?

SQL connections allow your MrScraper scrapers to send data directly to an external SQL database. This is useful for storing scraped data in a structured format and enables further analysis and reporting.

## Key Features


  } title="Centralized Data Storage">
    Store all your scraped data in one place for easy access and management.
  

  } title="Data Analysis">
    Use SQL queries to analyze and manipulate your scraped data.
  

  } title="Automation">
    Automatically insert scraped data into your database without manual intervention.
  


## Create an SQL Connection

Follow these steps to create an SQL connection in MrScraper:


  
    Open your **MrScraper dashboard**.
  

  
    Click the **gear icon** on the left panel and select **SQL Connection**.
  

  
    On the **SQL Connection page**, click the + icon in the top-right corner.
  

  
    Fill in the SQL connection details:

    * **SQL Driver**: Choose `PostgreSQL`, `MySQL`, `MariaDB`, `SQL Server`, `SQLite`, or `Oracle`
    * **Host**: Enter the hostname or IP address of your SQL server
    * **Port**: Enter the port number for your SQL server
    * **Username**: Enter your SQL database username
    * **Password**: Enter your SQL database password
    * **Database Name**: Enter the name of the database you want to connect to
    * **Default Table**: Enter the name of the table where the data will be inserted
  

  
    Click **Submit** to save your SQL connection.
  


## Connect SQL Connection to Scraper

After creating the SQL connection, follow these steps to connect it to your scraper:


  
    Go to your chosen **Scraper** page.
  

  
    Click the **Ellipsis** ( **⋮** ) button and click **Database Integration**.
  

  
    Enable the **Database Integration** toggle.
  

  
    Select the SQL connection you just created from the dropdown menu.
  

  
    Click **Save Database Settings**.
  


## Example Result

Once the scraper finishes, the scraped data will be automatically inserted into the specified table in your SQL database. You can then query and analyze the data using your preferred SQL tools. Here's an example of what the data might look like in a PostgreSQL database:

```sql title="Table containing scraping results"
scraping=# select * from books;
                 title
---------------------------------------
 A Light in the ...
 Tipping the Velvet
 Soumission
 Sharp Objects
 Sapiens: A Brief History ...
 The Requiem Red
 The Dirty Little Secrets ...
 The Coming Woman: A ...
 The Boys in the ...
 The Black Maria
 Starving Hearts (Triangular Trade ...
 Shakespeare’s Sonnets
 Set Me Free
 Scott Pilgrim’s Precious Little ...
 Rip it Up and ...
 Our Band Could Be ...
 Olio
 Mesaerion: The Best Science ...
 Libertarianism for Beginners
 It’s Only the Himalayas
(20 rows)
```


# Vercel AI SDK

The [`@mrscraper/ai-sdk`](https://www.npmjs.com/package/@mrscraper/ai-sdk) package
provides MrScraper tools for the [Vercel AI SDK](https://ai-sdk.dev). Pass them
to `generateText`, `streamText`, or an agent, and the model can retrieve web
pages, search the web, and check your MrScraper usage while it works.

## Requirements

* AI SDK 7 and Node.js 18 or later.
* A MrScraper API key from [app.mrscraper.com/api-tokens](https://app.mrscraper.com/api-tokens).

## Install

```bash
npm install @mrscraper/ai-sdk ai zod
```

Set your key as an environment variable:

```bash
export MRSCRAPER_API_KEY="your_mrscraper_api_key"
```

The tools read `MRSCRAPER_API_KEY`, then `MRSCRAPER_API_TOKEN`. To pass the key
in code instead, use `mrscraperTools({ apiKey })`.

## Quick start

```ts
import { generateText, isStepCount } from "ai";
import { mrscraperTools } from "@mrscraper/ai-sdk";

const { text } = await generateText({
  model: "openai/gpt-5",
  prompt:
    "Read https://www.scrapethissite.com/pages/simple/ and list the five countries with the largest area.",
  tools: mrscraperTools(),
  stopWhen: isStepCount(5),
});
```

`mrscraperTools()` returns `scraperAPI`, `webSearch`, and `status`, the same
names the [MrScraper MCP server](/docs/getting-started/mcp-server) uses. To add
only some of them, use the individual factories:

```ts
import { scraperApiTool, webSearchTool } from "@mrscraper/ai-sdk";

const tools = {
  scraperAPI: scraperApiTool(),
  webSearch: webSearchTool(),
};
```

## Tools

| Tool         | Inputs                                                                                                                                        | Returns                                                             |
| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| `scraperAPI` | `url`; `formats` (`markdown`, `html`, `screenshot`; default `markdown`); optional `browserRendering`, `waitForSelector`, `geoCode`, `timeout` | The requested formats, the API status code, and the tokens used     |
| `webSearch`  | `query` (search terms or a search results URL); optional `region` (two-letter country, default `us`) and `numResults` (10 to 100, default 10) | Ranked results (`rank`, `title`, `link`, `description`, `source`)   |
| `status`     | None                                                                                                                                          | Subscription status, token limit, tokens used, and tokens remaining |

Screenshots are sent to the model as images rather than as base64 text.
`scraperAPI` and `webSearch` use tokens from your plan; `status` uses none.

## Errors

API errors, such as an invalid key or an exhausted plan, are thrown from the
tool. The AI SDK reports them to the model as tool errors, so the model can
explain the problem or try another approach. The API key is never included in
an error message.

## Related

* [MCP server](/docs/getting-started/mcp-server): connect MCP clients such as Claude, Cursor, or ChatGPT instead.
* [Node.js SDK](/docs/integrations/node): call the MrScraper API directly.
* [Source on GitHub](https://github.com/mrscraper-com/mrscraper-ai-sdk)


# Webhook

import { Webhook, Cog } from 'lucide-react';
import { Step, Steps } from 'fumadocs-ui/components/steps';

## What Are Webhooks?

Webhooks let you automatically send HTTP requests to another API when specific events happen in your scraper — such as when a scrape starts, completes, or fails.

## Webhook Integration Benefits


  } title="Real-Time Notifications">
    Get instant updates on your scraper's status by sending data to your preferred endpoint.
  

  } title="Customizable Events">
    Choose which events trigger the webhook, allowing for tailored integrations.
  


## Setting Up a Webhook

Follow these steps to create and connect a webhook in MrScraper:


  
    Open your **MrScraper dashboard**.
  

  
    Click the **gear icon** on the left panel and select **Webhooks**.
  

  
    On the **Webhooks page**, click the + icon in the top-right corner.
  

  
    Fill in the webhook details:

    * **Request Type**: Choose `POST` or `PUT`
    * **Event**: Select when the webhook should trigger (e.g., *Scraping Started*, *Scraping Completed*, *Scraping Failed*)
    * **Name**: Give your webhook a descriptive name
    * **URL**: Enter the endpoint of your external webhook app
    * **Headers**: (Optional) Add any custom headers
  

  
    Click **Submit** to save your webhook.
  

  
    Go to your **Scraper** page, Click**Ellipsis** ( **⋮** ) button and click **Webhooks**.
  

  
    Under **Webhook Settings**, select the webhook you just created and click **Save Webhook Settings**.
  

  
    After connecting your scraper to a webhook, you'll receive a webhook payload each time a scrape runs. For example:

    ```json title="Webhook Example"
    {
      "event": "scrape_completed",
      "message": "Scraper completed successfully",
      "scraperId": "53542173-0b75-4f86-b001-15b7dbbc2ee4",
      "scraperName": "All-in-One E-Commerce Laptop Test Site",
      "timestamp": "2025-10-29T10:10:29.134Z",
      "data": {
        "userId": "0e264d60-fe1c-4033-a323-d2e5de5f7e0b",
        "scraperId": "53542173-0b75-4f86-b001-15b7dbbc2ee4",
        "type": "AI",
        "url": "https://webscraper.io/test-sites/e-commerce/allinone/computers/laptops",
        "status": "Finished",
        "error": null,
        "tokenUsage": 4,
        "runtime": null,
        "data": "{\"1\":{\"id\":\"1\",\"source\":\"table\",\"headers\":[\"Price\",\"Product Name\",\"Description\",\"Reviews\"],\"rows\":[[\"$295.99\",\"Asus VivoBook...\",\"Asus VivoBook X441NA-GA190 Chocolate Black, 14\\\", Celeron N3450, 4GB, 128GB SSD, Endless OS, ENG kbd\",\"14 reviews\"],[\"$299\",\"Prestigio Smar...\",\"Prestigio SmartBook 133S Dark Grey, 13.3\\\" FHD IPS, Celeron N3350 1.1GHz, 4GB, 32GB, Windows 10 Pro + Office 365 1 gadam\",\"8 reviews\"],[\"$299\",\"Prestigio Smar...\",\"Prestigio SmartBook 133S Gold, 13.3\\\" FHD IPS, Celeron N3350 1.1GHz, 4GB, 32GB, Windows 10 Pro + Office 365 1 gadam\",\"12 reviews\"],[\"$306.99\",\"Aspire E1-510\",\"15.6\\\", Pentium N3520 2.16GHz, 4GB, 500GB, Linux\",\"2 reviews\"],[\"$321.94\",\"Lenovo V110-15...\",\"Lenovo V110-15IAP, 15.6\\\" HD, Celeron N3350 1.1GHz, 4GB, 128GB SSD, Windows 10 Home\",\"5 reviews\"],[\"$356.49\",\"Lenovo V110-15...\",\"Asus VivoBook 15 X540NA-GQ008T Chocolate Black, 15.6\\\" HD, Pentium N4200, 4GB, 500GB, Windows 10 Home, En kbd\",\"6 reviews\"],[\"$364.46\",\"Hewlett Packar...\",\"Hewlett Packard 250 G6 Dark Ash Silver, 15.6\\\" HD, Celeron N3060 1.6GHz, 4GB, 128GB SSD, DOS\",\"12 reviews\"],[\"$372.7\",\"Acer Aspire 3...\",\"Acer Aspire 3 A315-31 Black, 15.6\\\" HD, Celeron N3350 1.1GHz, 4GB, 128GB SSD, Windows 10 Home\",\"2 reviews\"],[\"$379.94\",\"Acer Aspire A3...\",\"Acer Aspire A315-31-C33J Black 15.6\\\", HD, Celeron N3350, 4GB DDR3L, 128GB, Windows 10 Home, ENG\",\"0 reviews\"],[\"$379.95\",\"Acer Aspire ES...\",\"Acer Aspire ES1-572 Black, 15.6\\\" HD, Core i3-6006U, 4GB, 128GB SSD, Linux\",\"9 reviews\"],[\"$391.48\",\"Acer Aspire 3...\",\"Acer Aspire 3 A315-31 Black, 15.6\\\" HD, Pentium N4200 1.1GHz, 4GB, 128GB SSD, Windows 10 Home\",\"10 reviews\"],[\"$393.88\",\"Acer Aspire 3...\",\"Acer Aspire 3 A315-21, 15.6\\\", AMD A4-9120. 4GB. 128GB SSD, Linux\",\"9 reviews\"],[\"$399\",\"Asus VivoBook...\",\"Asus VivoBook Max X541NA-GQ041 Black Chocolate, 15.6\\\" HD, Pentium N4200 1.1GHz, 4GB, 500GB, Windows 10 Home\",\"4 reviews\"],[\"$399.99\",\"Asus VivoBook...\",\"Asus VivoBook E502NA-GO022T Dark Blue, 15.6\\\" HD, Pentium N4200 1.1GHz, 4GB, 128GB SSD, Windows 10 Home, En/Ru kbd\",\"3 reviews\"],[\"$404.23\",\"Lenovo ThinkPa...\",\"Lenovo ThinkPad E31-80, 13.3\\\" HD, Celeron 3855U 1.6GHz, 4GB, 128GB SSD, Windows 10 Home\",\"12 reviews\"],[\"$408.98\",\"Acer Aspire 3...\",\"Acer Aspire 3 A315-31 Black, 15.6\\\" HD, Pentium N4200 1.1GHz, 4GB, 128GB SSD, Windows 10 Home\",\"10 reviews\"],[\"$409.63\",\"Lenovo V110-15...\",\"Lenovo V110-15ISK, 15.6\\\" HD, Core i3-6006U, 8GB, 128GB SSD, Windows 10 Home\",\"9 reviews\"],[\"$410.46\",\"Acer Aspire ES...\",\"Acer Aspire ES1-732 Black, 17.3\\\" HD+, Celeron, N3350, 4GB, 1TB, Windows 10 Home\",\"14 reviews\"],[\"$410.66\",\"Asus VivoBook...\",\"Asus VivoBook 15 X540NA-GQ026T Chocolate Black, 15.6\\\" HD, Pentium N4200, 4GB, 128GB SSD, Windows 10 Home, En/Ru kbd\",\"4 reviews\"],[\"$416.99\",\"Packard 255 G2\",\"15.6\\\", AMD E2-3800 1.3GHz, 4GB, 500GB, Windows 8.1\",\"2 reviews\"],[\"$433.3\",\"Asus EeeBook R...\",\"Asus EeeBook R416NA-FA014T, 14\\\" FHD, Pentium N4200, 4GB, 128GB eMMC, Windows 10 Home, Eng kbd\",\"1 reviews\"],[\"$436.29\",\"Acer Aspire 3...\",\"Acer Aspire 3 A315-51, 15.6\\\" HD, Core i3-6006U, 4GB, 1TB, Windows 10 Home\",\"1 reviews\"],[\"$436.29\",\"Acer Aspire ES...\",\"Acer Aspire ES1-572 Black, 15.6\\\" HD, Core i3-6006U, 4GB, 500GB, Windows 10 Home\",\"2 reviews\"],[\"$439.73\",\"Acer Extensa 1...\",\"Acer Extensa 15 (2540) Black, 15.6\\\" HD, Core i5-7200U, 4GB, 500GB, Linux\",\"6 reviews\"],[\"$454.62\",\"Acer Aspire ES...\",\"Acer Aspire ES1-572 Black, 15.6\\\" HD, Core i5-7200U, 4GB, 500GB, Linux\",\"9 reviews\"],[\"$454.73\",\"Lenovo V110-15...\",\"Lenovo V110-15ISK, 15.6\\\" HD, Core i3-6006U, 4GB, 128GB SSD, Windows 10 Pro\",\"2 reviews\"],[\"$457.38\",\"Acer Aspire A3...\",\"Acer Aspire A315-51-33TG, Black 15.6\\\" HD, Core i3-7100U, 4GB DDR4, 128GB SSD, Windows 10 Home, ENG\",\"9 reviews\"],[\"$465.95\",\"Lenovo V110-15...\",\"Lenovo V110-15IKB, 15.6\\\" HD, Core i5-7200U, 4GB, 500GB, DOS\",\"7 reviews\"],[\"$468.56\",\"Asus VivoBook...\",\"Asus VivoBook 15 X540UA-DM260 Chocolate Black, 15.6\\\" FHD, Core i3-6006U, 4GB, 256GB SSD, Endless OS, En kbd\",\"1 reviews\"],[\"$469.1\",\"Acer Aspire ES...\",\"Acer Aspire ES1-572 Black, 15.6\\\" HD, Core i3-6006U, 4GB, 128GB SSD, Windows 10 Home\",\"5 reviews\"],[\"$484.23\",\"Lenovo V510 Bl...\",\"Lenovo V510 Black, 14\\\" HD, Core i3-6006U, 4GB, 128GB SSD, Windows 10 Home\",\"8 reviews\"],[\"$485.9\",\"Acer Aspire ES...\",\"Acer Aspire ES1-572 Black, 15.6\\\" HD, Core i5-7200U, 4GB, 128GB SSD, Linux\",\"6 reviews\"],[\"$487.8\",\"Lenovo V510 Bl...\",\"Lenovo V510 Black, 15.6\\\" HD, Core i3-6006U, 4GB, 128GB SSD, Windows 10 Home\",\"9 reviews\"],[\"$488.64\",\"Acer Swift 1 S...\",\"Acer Swift 1 SF113-31 Silver, 13.3\\\" FHD, Pentium N4200, 4GB, 128GB SSD, Windows 10 Home\",\"4 reviews\"],[\"$488.78\",\"Dell Vostro 15\",\"Dell Vostro 15 (3568) Black, 15.6\\\" FHD, Core i5-7200U, 4GB, 128GB SSD, Radeon R5 M420 2GB, Linux\",\"14 reviews\"],[\"$494.71\",\"Acer Aspire 3...\",\"Acer Aspire 3 A315-51 Black, 15.6\\\" FHD, Core i3-7100U, 4GB, 500GB + 128GB SSD, Windows 10 Home\",\"2 reviews\"],[\"$497.17\",\"Dell Vostro 15...\",\"Dell Vostro 15 (3568) Red, 15.6\\\" HD, Core i5-7200U, 4GB, 1TB, Radeon R5 M420 2GB, Linux\",\"9 reviews\"],[\"$498.23\",\"Lenovo V510 Bl...\",\"Lenovo V510 Black, 15.6\\\" FHD, Core i3-7100U, 4GB, 128GB SSD, Windows 10 Pro\",\"5 reviews\"],[\"$520.99\",\"HP 250 G3\",\"15.6\\\", Core i5-4210U, 4GB, 500GB, Windows 8.1\",\"13 reviews\"],[\"$564.98\",\"Acer Spin 5\",\"Acer Spin 5 SP513-51 Black, 13.3\\\" FHD Touch, Core i3-7100U, 4GB, 128GB SSD, Windows 10 Home\",\"0 reviews\"],[\"$577.99\",\"HP 350 G1\",\"15.6\\\", Core i5-4200U, 4GB, 750GB, Radeon HD8670M 2GB, Windows\",\"10 reviews\"],[\"$581.99\",\"Aspire E1-572G\",\"15.6\\\", Core i5-4200U, 8GB, 1TB, Radeon R7 M265, Windows 8.1\",\"2 reviews\"],[\"$609.99\",\"Pavilion\",\"15.6\\\", Core i5-4200U, 6GB, 750GB, Windows 8.1\",\"4 reviews\"],[\"$679\",\"Acer Aspire A5...\",\"Acer Aspire A515-51-5654, Black, 15.6\\\", FHD, Core i5-8250U, 8GB DDR4, 256GB SSD, Windows 10 Home, ENG\",\"9 reviews\"],[\"$679\",\"Dell Inspiron...\",\"Dell Inspiron 15 (5567) Fog Gray, 15.6\\\" FHD, Core i5-7200U, 8GB, 1TB, Radeon R7 M445 4GB, Linux\",\"7 reviews\"],[\"$729\",\"Asus VivoBook...\",\"Asus VivoBook S14 (S406UA-BV041T) Starry Grey, 14\\\", Core i5-8250U, 8GB, 256GB SSD, Windows 10 Home, Eng kbd\",\"2 reviews\"],[\"$739.99\",\"ProBook\",\"14\\\", Core i5 2.6GHz, 4GB, 500GB, Win7 Pro 64bit\",\"8 reviews\"],[\"$745.99\",\"Inspiron 15\",\"Moon Silver, 15.6\\\", Core i7-4510U, 8GB, 1TB, Radeon HD R7 M265 2GB,\",\"12 reviews\"],[\"$799\",\"Asus ROG STRIX...\",\"Asus ROG STRIX GL553VD-DM256, 15.6\\\" FHD, Core i5-7300HQ, 8GB, 1TB, GeForce GTX 1050 2GB, No OS\",\"7 reviews\"],[\"$809\",\"Acer Nitro 5 A...\",\"Acer Nitro 5 AN515-51, 15.6\\\" FHD IPS, Core i5-7300HQ, 8GB, 1TB, GeForce GTX 1050 2GB, Windows 10 Home\",\"0 reviews\"],[\"$899\",\"Asus ROG STRIX...\",\"Asus ROG STRIX GL553VD-DM256, 15.6\\\" FHD, Core i5-7300HQ, 8GB, 1TB, GeForce GTX 1050 2GB, No OS + Windows 10 Home\",\"7 reviews\"],[\"$999\",\"Lenovo ThinkPa...\",\"Lenovo ThinkPad L570, 15.6\\\" FHD, Core i7-7500U, 8GB, 256GB SSD, Windows 10 Pro\",\"11 reviews\"],[\"$1033.99\",\"ThinkPad Yoga\",\"12.5\\\" Touch, Core i3-4010U, 4GB, 500GB + 16GB SSD Cache,\",\"13 reviews\"],[\"$1096.02\",\"Lenovo ThinkPa...\",\"Lenovo ThinkPad L460, 14\\\" FHD IPS, Core i7-6600U, 8GB, 256GB SSD, Windows 10 Pro\",\"14 reviews\"],[\"$1098.42\",\"Dell Inspiron...\",\"Dell Inspiron 15 (7567) Black, 15.6\\\" FHD, Core i5-7300HQ, 8GB, 256GB SSD, GeForce GTX 1050 4GB, Windows 10 Home\",\"7 reviews\"],[\"$1099\",\"MSI GL72M 7RDX\",\"MSI GL72M 7RDX, 17.3\\\" FHD, Core i5-7300HQ, 8GB, 1TB + 128GB SSD, GeForce GTX 1050 2GB, Windows 10 Home\",\"1 reviews\"],[\"$1099\",\"MSI GL72M 7RDX\",\"Asus ROG Strix GL553VD-DM535T, 15.6\\\" FHD, Core i7-7700HQ, 8GB, 1TB + 128GB SSD, GeForce GTX 1050 2GB, Windows 10 Home, Eng kbd\",\"9 reviews\"],[\"$1101.83\",\"Asus ROG Strix...\",\"Apple MacBook Air 13.3\\\", Core i5 1.8GHz, 8GB, 128GB SSD, Intel HD 4000, RUS\",\"4 reviews\"],[\"$1102.66\",\"Dell Latitude...\",\"Dell Latitude 5280, 12.5\\\" HD, Core i5-7300U, 8GB, 256GB SSD, Windows 10 Pro\",\"8 reviews\"],[\"$1110.14\",\"Dell Latitude...\",\"Dell Latitude 5480, 14\\\" FHD, Core i5-7300U, 8GB, 500GB, Linux + Windows 10 Home\",\"4 reviews\"],[\"$1112.91\",\"Lenovo Legion...\",\"Lenovo Legion Y520-15IKBM, Black, 15.6\\\" FHD IPS, Core i5-7300HQ, 8 GB, 128GB SSD + 2 TB HDD, NVIDIA GeForce GTX 1060 6 GB, FreeDOS + Windows 10 Home\",\"1 reviews\"],[\"$1114.55\",\"Toshiba Porteg...\",\"Toshiba Portege Z30-C-16J Grey, 13.3\\\" FHD, Core i5-6200U, 8GB, 256GB SSD, Windows 10 Pro\",\"0 reviews\"],[\"$1115.87\",\"Acer Predator...\",\"Acer Predator Helios 300 (PH317-51), 17.3\\\" FHD IPS, Core i5-7300HQ, 8GB, 1TB + 128GB SSD, GeForce GTX 1050Ti 4GB, Windows 10 Home\",\"1 reviews\"],[\"$1115.87\",\"Acer Aspire 7...\",\"Acer Aspire 7 A715-71G, 15.6\\\" FHD IPS, Core i7-7700HQ, 8GB, 128GB SSD + 1TB HDD, GTX 1050 Ti 4GB, Windows 10 Home\",\"4 reviews\"],[\"$1116.99\",\"Dell Inspiron...\",\"Dell Inspiron 17 2in1 (7779) Silver, 17.3\\\" FHD Touch, Core i5-7200U, 12GB, 1TB, GeForce GT940MX 2GB, Windows 10 Home\",\"10 reviews\"],[\"$1117.99\",\"Dell Latitude...\",\"Dell Latitude 5480, 14\\\" FHD, Core i5-7300U, 8GB, 256GB SSD, Windows 10 Pro\",\"14 reviews\"],[\"$1118.99\",\"Lenovo Legion...\",\"Lenovo Legion Y520, 15.6\\\" FHD, Core i7-7700HQ, 8GB, 128 GB SSD + 1TB HDD, GTX 1050 4GB, Windows 10 Home\",\"13 reviews\"],[\"$1119.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702VM-GC146T, 17.3\\\" FHD, Core i7-7700HQ, 8GB, 1TB + 128GB SSD, GeForce GTX 1060 3GB, Windows 10 Home, Eng kbd\",\"10 reviews\"],[\"$1120.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC154T, 17.3\\\" FHD, Ryzen 7 1700, 16GB, 256GB + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"7 reviews\"],[\"$1121.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC209T, 17.3\\\" FHD IPS, Ryzen 7 1700, 16GB, 256GB SSD + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"8 reviews\"],[\"$1122.99\",\"Asus ROG Strix...\",\"Asus ROG Strix SCAR Edition GL503VM-ED115T, 15.6\\\" FHD 120Hz, Core i7-7700HQ, 16GB, 256GB SSD + 1TB SSHD, GeForce GTX 1060 6GB, Windows 10 Home\",\"8 reviews\"],[\"$1123.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC154T, 17.3\\\" FHD, Ryzen 7 1700, 16GB, 256GB + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"7 reviews\"],[\"$1124.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC209T, 17.3\\\" FHD IPS, Ryzen 7 1700, 16GB, 256GB SSD + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"8 reviews\"],[\"$1125.99\",\"Asus ROG Strix...\",\"Asus ROG Strix SCAR Edition GL503VM-ED115T, 15.6\\\" FHD 120Hz, Core i7-7700HQ, 16GB, 256GB SSD + 1TB SSHD, GeForce GTX 1060 6GB, Windows 10 Home\",\"8 reviews\"],[\"$1126.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC154T, 17.3\\\" FHD, Ryzen 7 1700, 16GB, 256GB + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"7 reviews\"],[\"$1127.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC209T, 17.3\\\" FHD IPS, Ryzen 7 1700, 16GB, 256GB SSD + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"8 reviews\"],[\"$1128.99\",\"Asus ROG Strix...\",\"Asus ROG Strix SCAR Edition GL503VM-ED115T, 15.6\\\" FHD 120Hz, Core i7-7700HQ, 16GB, 256GB SSD + 1TB SSHD, GeForce GTX 1060 6GB, Windows 10 Home\",\"8 reviews\"],[\"$1129.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC154T, 17.3\\\" FHD, Ryzen 7 1700, 16GB, 256GB + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"7 reviews\"],[\"$1130.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC209T, 17.3\\\" FHD IPS, Ryzen 7 1700, 16GB, 256GB SSD + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"8 reviews\"],[\"$1131.99\",\"Asus ROG Strix...\",\"Asus ROG Strix SCAR Edition GL503VM-ED115T, 15.6\\\" FHD 120Hz, Core i7-7700HQ, 16GB, 256GB SSD + 1TB SSHD, GeForce GTX 1060 6GB, Windows 10 Home\",\"8 reviews\"],[\"$1132.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC154T, 17.3\\\" FHD, Ryzen 7 1700, 16GB, 256GB + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"7 reviews\"],[\"$1133.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC209T, 17.3\\\" FHD IPS, Ryzen 7 1700, 16GB, 256GB SSD + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"8 reviews\"],[\"$1134.99\",\"Asus ROG Strix...\",\"Asus ROG Strix SCAR Edition GL503VM-ED115T, 15.6\\\" FHD 120Hz, Core i7-7700HQ, 16GB, 256GB SSD + 1TB SSHD, GeForce GTX 1060 6GB, Windows 10 Home\",\"8 reviews\"],[\"$1135.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC154T, 17.3\\\" FHD, Ryzen 7 1700, 16GB, 256GB + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"7 reviews\"],[\"$1136.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC209T, 17.3\\\" FHD IPS, Ryzen 7 1700, 16GB, 256GB SSD + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"8 reviews\"],[\"$1137.99\",\"Asus ROG Strix...\",\"Asus ROG Strix SCAR Edition GL503VM-ED115T, 15.6\\\" FHD 120Hz, Core i7-7700HQ, 16GB, 256GB SSD + 1TB SSHD, GeForce GTX 1060 6GB, Windows 10 Home\",\"8 reviews\"],[\"$1138.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC154T, 17.3\\\" FHD, Ryzen 7 1700, 16GB, 256GB + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"7 reviews\"],[\"$1139.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC209T, 17.3\\\" FHD IPS, Ryzen 7 1700, 16GB, 256GB SSD + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"8 reviews\"]],\"name\":\"Laptops\"}}",
        "htmlPath": "results/d287adc4-9d11-41ed-8245-de05d577c286/page.html",
        "recordingPath": null,
        "screenshotPath": "results/d287adc4-9d11-41ed-8245-de05d577c286/screenshot.jpg",
        "dataPath": null,
        "createdById": null,
        "updatedById": null,
        "deletedById": null,
        "id": "68d3e07c-fab3-4b4a-9c54-b796cd64fe73",
        "createdAt": "2025-10-29T10:10:29.036Z",
        "updatedAt": "2025-10-29T10:10:29.036Z",
        "deletedAt": null
      }
    }
    ```
  



# Zapier





import { Step, Steps } from 'fumadocs-ui/components/steps';

Zapier is a powerful automation tool that allows you to connect different apps and services together. By integrating MrScraper with Zapier, you can automate workflows and streamline your data extraction processes.

## What is the MrScraper Zapier Integration?

The MrScraper Zapier integration connects MrScraper with Zapier, allowing you to trigger scraping jobs, automate data collection, and build workflows using MrScraper actions and triggers.

### Available Triggers

* **New Result**: Triggers when a new scraping result (succeeded or failed) is available.
* **Scrape Completed**: Triggers when a scrape job completes successfully.
* **Scrape Failed**: Triggers when a scrape job fails.
* **Scrape Started**: Triggers when a scrape job starts.
* **Specific Scraper New Result**: Triggers when a new scraping result (succeeded or failed) is available for a specific scraper.

### Available Actions

* **Get Latest Result**: Fetch the most recent result from a specified MrScraper scraper.
* **Run AI Scraper**: Trigger an AI scraper run with optional URL overrides.
* **Run Manual Scraper**: Trigger a manual scraper run with optional URL overrides.

## Why Use Zapier with MrScraper?

Integrating MrScraper with Zapier offers several benefits:

* **Automation**: Automate repetitive tasks such as data extraction and processing.
* **Flexibility**: Connect MrScraper with over 3,000 apps available on Zapier.
* **Efficiency**: Save time by eliminating manual data handling.

## Prerequisites

Before you start, make sure you have:

* [A **MrScraper API token**](https://app.mrscraper.com/api-tokens).
* [A **MrScraper scraper** with API access enabled](https://app.mrscraper.com/workflow).
* Access to **Zapier**.

## Integration Steps


  
    ### Add MrScraper Node

    1. Open the **zapier editor**.
    2. Click on **Add a Step** and select **MrScraper** from the list of available apps.
  

  
    ### Configure Trigger (Optional)

    1. Add an account by entering your **MrScraper API token**.
    2. Select one of the available trigger events:
       * **New Result**
       * **Scrape Completed**
       * **Scrape Failed**
       * **Scrape Started**
       * **Specific Scraper New Result**
    3. For most triggers, no additional fields are required.
    4. If you choose **Specific Scraper New Result**, configure the required field below:

    | Parameter      | Required | Description                                                                                      |
    | -------------- | -------- | ------------------------------------------------------------------------------------------------ |
    | **Scraper ID** | Yes      | The ID of the scraper you want to listen to. You can find this on the scraper page in MrScraper. |
  

  
    ### Configure the MrScraper Action Node

    1. Choose the desired action (e.g., **Get Latest Result**).
    2. Configure the action parameters as needed.

    #### Get Latest Result

    | Parameter      | Required | Description                                                                                   |
    | -------------- | -------- | --------------------------------------------------------------------------------------------- |
    | **Scraper ID** | Yes      | The ID of the AI scraper you want to run. You can find this on the scraper page in MrScraper. |

    #### Run AI Scraper

    | Parameter      | Required | Description                                                                                       |
    | -------------- | -------- | ------------------------------------------------------------------------------------------------- |
    | **Scraper ID** | Yes      | The ID of the manual scraper you want to run. You can find this on the scraper page in MrScraper. |
    | **URL**        | Yes      | Override the default target URL for this run.                                                     |

    #### Run Manual Scraper

    | Parameter      | Required | Description                                                                                |
    | -------------- | -------- | ------------------------------------------------------------------------------------------ |
    | **Scraper ID** | Yes      | The ID of the scraper you want to run. You can find this on the scraper page in MrScraper. |
    | **URL**        | Yes      | Override the default target URL for this run.                                              |
  


## Example Workflows

### Scheduled Scraping to Google Sheets



This workflow automatically runs a scraper on a schedule and saves the results to Google Sheets. Perfect for tracking prices, monitoring listings, or collecting data over time.


  
    #### Set Up the Schedule

    1. Add a **Schedule** node.
    2. Configure the interval (e.g., every hour, daily, weekly).
  

  
    #### Run the Scraper

    1. Add the **MrScraper** node.
    2. Select **Run AI/Manual Scraper** as the Action event.
    3. Enter your **Scraper ID** and **URL**.
  

  
    #### Fetch the Results

    1. Add another **MrScraper** node.
    2. Select **Get Latest Result** as the resource.
    3. Enter the same **Scraper ID**.
  

  
    #### Save to Google Sheets

    1. Add a **Google Sheets** node.
    2. Select **Create Spreadsheet Row** action.
  


### Multi-URL Scraping Loop



This workflow scrapes multiple URLs from a list and processes each result.


  This flow is ideal for batch scraping tasks.



  
    #### Prepare the URL List

    1. Add a **Schedule** node.
    2. Add a **Line Items From CSV** node to generate a list of URLs.:

    ```csv
    https://example.com/page1
    https://example.com/page2
    https://example.com/page3
    ```
  

  
    #### Loop Through URLs

    1. Add a **Create Loop From Text** node to iterate through each URL.
    2. Set **Values to Loop** to the output of the previous node (Line Items From CSV in Files > Line Items).
  

  
    #### Scrape Each URL

    1. Add the **MrScraper** node inside the loop.
    2. Select **Run AI/Manual Scraper** as the Action event.
    3. Use the URL from the previous step.
  

  
    #### Save Results

    1. Add a **Google Sheets** node.
    2. Select **Create Spreadsheet Row** action.
  



# Authentication



import { Info, AlertCircle } from 'lucide-react';

## Authentication Formats

Residential Proxy uses a flexible username-based authentication system where your username pattern determines the proxy configuration. All proxies connect through the same endpoint but behave differently based on your username structure.

## Connection Format

All Residential Proxy connections use the following format:

```
username:password@proxy.mrscraper.com:10000
```

The **username** controls:

* Proxy type
* Country targeting
* Session persistence
* Session duration
* IP rotation behavior

## Authentication Patterns

### 1. Default (United States)

The simplest format uses just your username and password without any modifiers.

**Format:**

```
username:password@proxy.mrscraper.com:10000
```

**Behavior:**

* Residential Proxy falls back to **United States (`us`)** IPs
* IP rotates on each new request
* No session persistence

**Example:**

```bash
curl -x "user123:pass456@proxy.mrscraper.com:10000" https://api.ipify.org
```


  **Note:** The UI always requires a country selection; only manual username configuration lets you omit it. Omitting it is not random assignment — it resolves to the US, so state the country explicitly whenever the target region matters.


### 2. Country-Specific Proxy

Target a specific country using ISO 3166 country codes.

**Format:**

```
username-country-{iso3166}:password@proxy.mrscraper.com:10000
```

**Parameters:**

* `{iso3166}`: 2-letter ISO 3166 country code (e.g., `US`, `GB`, `DE`, `JP`)

**Behavior:**

* All requests use IPs from the specified country
* IP rotates on each new request
* No session persistence

**Examples:**


  
    
      United States
    

    
      Great Britain
    

    
      Germany
    

    
      Japan
    
  

  
    ```bash
    # Target US proxies
    user123-country-us:pass456@proxy.mrscraper.com:10000
    ```
  

  
    ```bash
    # Target GB proxies
    user123-country-gb:pass456@proxy.mrscraper.com:10000
    ```
  

  
    ```bash
    # Target German proxies
    user123-country-de:pass456@proxy.mrscraper.com:10000
    ```
  

  
    ```bash
    # Target Japanese proxies
    user123-country-jp:pass456@proxy.mrscraper.com:10000
    ```
  


**Use Cases:**

* Accessing geo-restricted content
* Testing localized websites
* Market research in specific regions
* Price comparison across countries

### 3. Session-Based Static Proxy

Maintain the same IP across multiple requests using session IDs.

**Format:**

```
username-country-{iso3166}-sessid-{session_id}:password@proxy.mrscraper.com:10000
```

**Parameters:**

* `{iso3166}`: 2-letter ISO 3166 country code
* `{session_id}`: Any alphanumeric string (e.g., `session1`, `user_alpha`, `x9`)

**Behavior:**

* Same IP for all requests with the same `session_id`
* **Default session duration: 10 minutes**
* After expiration, a new IP is assigned
* Different `session_id` values get different IPs

**Examples:**


  
    
      Session Alpha
    

    
      Session Beta
    

    
      Session Custom
    
  

  
    ```bash
    # Session ID: alpha1
    user123-country-us-sessid-alpha1:pass456@proxy.mrscraper.com:10000
    ```
  

  
    ```bash
    # Session ID: beta2
    user123-country-gb-sessid-beta2:pass456@proxy.mrscraper.com:10000
    ```
  

  
    ```bash
    # Session ID: my_custom_session
    user123-country-de-sessid-my_custom_session:pass456@proxy.mrscraper.com:10000
    ```
  


**Python Example:**

```python
import requests

proxy = {
    'http': 'http://user123-country-us-sessid-session1:pass456@proxy.mrscraper.com:10000',
    'https': 'http://user123-country-us-sessid-session1:pass456@proxy.mrscraper.com:10000'
}

# All these requests use the same IP
for i in range(5):
    response = requests.get('https://api.ipify.org', proxies=proxy)
    print(f"Request {i+1}: {response.text}")  # Same IP for all
```

**Use Cases:**

* Account management and login sessions
* Shopping cart operations
* Dashboard interactions
* Form submissions requiring consistent IP

### 4. Custom Session Duration

Control exactly how long a session IP persists.

**Format:**

```
username-country-{iso3166}-sessid-{session_id}-sesstime-{minutes}:password@proxy.mrscraper.com:10000
```

**Parameters:**

* `{iso3166}`: 2-letter ISO 3166 country code
* `{session_id}`: Alphanumeric session identifier
* `{minutes}`: Session duration in minutes (integer)

**Behavior:**

* Same IP for all requests with the same `session_id`
* Session expires after `{minutes}` minutes
* New IP assigned after expiration
* Maximum flexibility for session management



**Examples:**


  
    
      30 Minutes
    

    
      60 Minutes
    

    
      5 Minutes
    
  

  
    ```bash
    # Session lasts 30 minutes
    user123-country-jp-sessid-x9-sesstime-30:pass456@proxy.mrscraper.com:10000
    ```
  

  
    ```bash
    # Session lasts 1 hour
    user123-country-ca-sessid-long1-sesstime-60:pass456@proxy.mrscraper.com:10000
    ```
  

  
    ```bash
    # Short 5-minute session
    user123-country-fr-sessid-quick-sesstime-5:pass456@proxy.mrscraper.com:10000
    ```
  


**Node.js Example:**

```javascript
const axios = require('axios');

// 20-minute session proxy
const proxy = {
  host: 'proxy.mrscraper.com',
  port: 10000,
  auth: {
    username: 'user123-country-us-sessid-node_session-sesstime-20',
    password: 'pass456'
  }
};

async function testProxy() {
  for (let i = 0; i < 10; i++) {
    const response = await axios.get('https://api.ipify.org', { proxy });
    console.log(`Request ${i+1}: ${response.data}`);
    
    // Wait 2 minutes between requests
    await new Promise(resolve => setTimeout(resolve, 120000));
  }
}

testProxy();
```

**Use Cases:**

* Long scraping sessions
* Extended browsing sessions
* Multi-step workflows
* Testing with specific time constraints

## Authentication Summary

| Format                                               | Country      | Session | Duration | Rotation      |
| ---------------------------------------------------- | ------------ | ------- | -------- | ------------- |
| `username:password`                                  | US (default) | No      | N/A      | Every request |
| `username-country-{code}`                            | Specific     | No      | N/A      | Every request |
| `username-country-{code}-sessid-{id}`                | Specific     | Yes     | 10 min   | After expiry  |
| `username-country-{code}-sessid-{id}-sesstime-{min}` | Specific     | No      | Custom   | After expiry  |

## Best Practices

### Session ID Naming

* Use descriptive names: `user_john`, `checkout_session`, `scraper_1`
* Avoid special characters (stick to alphanumeric and underscores)
* Keep IDs under 50 characters for compatibility

### Security Considerations

* Never hardcode credentials in source code
* Use environment variables for username/password
* Rotate session IDs regularly
* Monitor proxy usage for anomalies

## Testing Your Configuration

Test your proxy configuration with a simple cURL command:

```bash
curl -x "username-country-us:password@proxy.mrscraper.com:10000" https://api.ipify.org
```

This returns your current proxy IP address. Run it multiple times to verify rotation behavior:

```bash
# Test rotation
for i in {1..5}; do
  echo "Request $i:"
  curl -x "user-country-us:pass@proxy.mrscraper.com:10000" https://api.ipify.org
  echo ""
done
```

For session-based proxies, you should see the same IP across all requests:

```bash
# Test session persistence
for i in {1..5}; do
  echo "Request $i:"
  curl -x "user-country-us-sessid-test1:pass@proxy.mrscraper.com:10000" https://api.ipify.org
  echo ""
done
```

## Common Errors

### Authentication Failed

* Verify your username and password are correct
* Check for typos in the format pattern
* Ensure no extra spaces in the username

### Country Not Supported

* Verify the country code is valid ISO 3166 format
* Check that the country is in Residential Proxy's coverage area
* Try a different country code

### Session Expired

* Increase `sesstime` value if sessions expire too quickly
* Implement automatic reconnection in your code
* Use the same `session_id` to maintain the session


# Country Codes

import { MapPin, Search, AlertCircle } from 'lucide-react';

## Country Codes (ISO 3166)

Residential Proxy uses **ISO 3166-1 alpha-2** country codes to target specific geographic locations. This standardized format ensures compatibility and consistency across all proxy configurations.

## What are ISO 3166 Country Codes?

ISO 3166-1 alpha-2 is an international standard that uses **two-letter codes** to represent countries and territories. Residential Proxy adopts this standard for country targeting in proxy configurations.

**Format:**

* Always **2 letters**
* Case-insensitive (but lowercase is recommended)
* Based on English country names

**Examples:**

* `us` → United States
* `gb` → United Kingdom
* `de` → Germany (Deutschland)
* `jp` → Japan

## How to Use Country Codes

Country codes are inserted into your proxy username using the `-country-` parameter.

**Basic Format:**

```bash
username-country-{code}:password@proxy.mrscraper.com:10000
```

**Examples:**


  
    
      United States
    

    
      United Kingdom
    

    
      Germany
    

    
      Japan
    
  

  
    ```bash
    user123-country-us:pass456@proxy.mrscraper.com:10000
    ```
  

  
    ```bash
    user123-country-gb:pass456@proxy.mrscraper.com:10000
    ```
  

  
    ```bash
    user123-country-de:pass456@proxy.mrscraper.com:10000
    ```
  

  
    ```bash
    user123-country-jp:pass456@proxy.mrscraper.com:10000
    ```
  


## Popular Country Codes

### North America

| Country       | Code | Example                                          |
| ------------- | ---- | ------------------------------------------------ |
| United States | `us` | `user-country-us:pass@proxy.mrscraper.com:10000` |
| Canada        | `ca` | `user-country-ca:pass@proxy.mrscraper.com:10000` |
| Mexico        | `mx` | `user-country-mx:pass@proxy.mrscraper.com:10000` |

### Europe

| Country        | Code | Example                                          |
| -------------- | ---- | ------------------------------------------------ |
| United Kingdom | `gb` | `user-country-gb:pass@proxy.mrscraper.com:10000` |
| Germany        | `de` | `user-country-de:pass@proxy.mrscraper.com:10000` |
| France         | `fr` | `user-country-fr:pass@proxy.mrscraper.com:10000` |
| Spain          | `es` | `user-country-es:pass@proxy.mrscraper.com:10000` |
| Italy          | `it` | `user-country-it:pass@proxy.mrscraper.com:10000` |
| Netherlands    | `nl` | `user-country-nl:pass@proxy.mrscraper.com:10000` |
| Poland         | `pl` | `user-country-pl:pass@proxy.mrscraper.com:10000` |
| Sweden         | `se` | `user-country-se:pass@proxy.mrscraper.com:10000` |
| Switzerland    | `ch` | `user-country-ch:pass@proxy.mrscraper.com:10000` |
| Belgium        | `be` | `user-country-be:pass@proxy.mrscraper.com:10000` |
| Austria        | `at` | `user-country-at:pass@proxy.mrscraper.com:10000` |
| Norway         | `no` | `user-country-no:pass@proxy.mrscraper.com:10000` |
| Denmark        | `dk` | `user-country-dk:pass@proxy.mrscraper.com:10000` |
| Finland        | `fi` | `user-country-fi:pass@proxy.mrscraper.com:10000` |
| Ireland        | `ie` | `user-country-ie:pass@proxy.mrscraper.com:10000` |
| Portugal       | `pt` | `user-country-pt:pass@proxy.mrscraper.com:10000` |
| Greece         | `gr` | `user-country-gr:pass@proxy.mrscraper.com:10000` |
| Czech Republic | `cz` | `user-country-cz:pass@proxy.mrscraper.com:10000` |
| Romania        | `ro` | `user-country-ro:pass@proxy.mrscraper.com:10000` |
| Hungary        | `hu` | `user-country-hu:pass@proxy.mrscraper.com:10000` |
| Bulgaria       | `bg` | `user-country-bg:pass@proxy.mrscraper.com:10000` |
| Croatia        | `hr` | `user-country-hr:pass@proxy.mrscraper.com:10000` |
| Serbia         | `rs` | `user-country-rs:pass@proxy.mrscraper.com:10000` |
| Slovakia       | `sk` | `user-country-sk:pass@proxy.mrscraper.com:10000` |
| Slovenia       | `si` | `user-country-si:pass@proxy.mrscraper.com:10000` |

### Asia

| Country              | Code | Example                                          |
| -------------------- | ---- | ------------------------------------------------ |
| Japan                | `jp` | `user-country-jp:pass@proxy.mrscraper.com:10000` |
| China                | `cn` | `user-country-cn:pass@proxy.mrscraper.com:10000` |
| India                | `in` | `user-country-in:pass@proxy.mrscraper.com:10000` |
| South Korea          | `kr` | `user-country-kr:pass@proxy.mrscraper.com:10000` |
| Singapore            | `sg` | `user-country-sg:pass@proxy.mrscraper.com:10000` |
| Indonesia            | `id` | `user-country-id:pass@proxy.mrscraper.com:10000` |
| Thailand             | `th` | `user-country-th:pass@proxy.mrscraper.com:10000` |
| Vietnam              | `vn` | `user-country-vn:pass@proxy.mrscraper.com:10000` |
| Malaysia             | `my` | `user-country-my:pass@proxy.mrscraper.com:10000` |
| Philippines          | `ph` | `user-country-ph:pass@proxy.mrscraper.com:10000` |
| Hong Kong            | `hk` | `user-country-hk:pass@proxy.mrscraper.com:10000` |
| Taiwan               | `tw` | `user-country-tw:pass@proxy.mrscraper.com:10000` |
| Pakistan             | `pk` | `user-country-pk:pass@proxy.mrscraper.com:10000` |
| Bangladesh           | `bd` | `user-country-bd:pass@proxy.mrscraper.com:10000` |
| Israel               | `il` | `user-country-il:pass@proxy.mrscraper.com:10000` |
| United Arab Emirates | `ae` | `user-country-ae:pass@proxy.mrscraper.com:10000` |
| Saudi Arabia         | `sa` | `user-country-sa:pass@proxy.mrscraper.com:10000` |
| Turkey               | `tr` | `user-country-tr:pass@proxy.mrscraper.com:10000` |

### Oceania

| Country     | Code | Example                                          |
| ----------- | ---- | ------------------------------------------------ |
| Australia   | `au` | `user-country-au:pass@proxy.mrscraper.com:10000` |
| New Zealand | `nz` | `user-country-nz:pass@proxy.mrscraper.com:10000` |

### South America

| Country   | Code | Example                                          |
| --------- | ---- | ------------------------------------------------ |
| Brazil    | `br` | `user-country-br:pass@proxy.mrscraper.com:10000` |
| Argentina | `ar` | `user-country-ar:pass@proxy.mrscraper.com:10000` |
| Chile     | `cl` | `user-country-cl:pass@proxy.mrscraper.com:10000` |
| Colombia  | `co` | `user-country-co:pass@proxy.mrscraper.com:10000` |
| Peru      | `pe` | `user-country-pe:pass@proxy.mrscraper.com:10000` |
| Venezuela | `ve` | `user-country-ve:pass@proxy.mrscraper.com:10000` |

### Africa

| Country      | Code | Example                                          |
| ------------ | ---- | ------------------------------------------------ |
| South Africa | `za` | `user-country-za:pass@proxy.mrscraper.com:10000` |
| Egypt        | `eg` | `user-country-eg:pass@proxy.mrscraper.com:10000` |
| Nigeria      | `ng` | `user-country-ng:pass@proxy.mrscraper.com:10000` |
| Kenya        | `ke` | `user-country-ke:pass@proxy.mrscraper.com:10000` |
| Morocco      | `ma` | `user-country-ma:pass@proxy.mrscraper.com:10000` |

## Finding Country Codes

If you need a country code not listed above, you can:

1. **Search Online**: Use "ISO 3166-1 alpha-2 \[country name]"
2. **Check Wikipedia**: [ISO 3166-1 alpha-2 on Wikipedia](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2)
3. **Common Patterns**: Usually the first two letters of the English country name

**Common Code Notes:**

* 🇬🇧 United Kingdom: `gb` (Great Britain) - this is the official ISO 3166 code
* 🇨🇭 Switzerland: `ch` (from Latin *Confoederatio Helvetica*)
* 🇬🇷 Greece: `gr` (from Greek)
* 🇰🇷 South Korea: `kr` (from **K**o**r**ea)

## Country Code Best Practices

### Case Sensitivity

Country codes are **case-insensitive**, but lowercase is recommended for consistency:

```bash
# All valid, but lowercase is preferred
user-country-us:pass@proxy.mrscraper.com:10000    # ✅ Recommended
user-country-US:pass@proxy.mrscraper.com:10000    # ✅ Valid
user-country-Us:pass@proxy.mrscraper.com:10000    # ✅ Valid but inconsistent
```

### Validation

Always verify your country code is correct before deployment:

```python
import requests

def test_country_proxy(country_code):
    """Test if a country code works with Residential Proxy"""
    proxy = {
        'http': f'http://user-country-{country_code}:pass@proxy.mrscraper.com:10000',
        'https': f'http://user-country-{country_code}:pass@proxy.mrscraper.com:10000'
    }
    
    try:
        response = requests.get('https://api.ipify.org', proxies=proxy, timeout=10)
        print(f"✅ Country '{country_code}' works! IP: {response.text}")
        return True
    except Exception as e:
        print(f"❌ Country '{country_code}' failed: {e}")
        return False

# Test multiple countries
countries = ['us', 'gb', 'de', 'jp', 'au']
for country in countries:
    test_country_proxy(country)
```

## Omitting the Country Code

The country parameter is optional. When you **don't specify** one, requests fall back to **United States (`us`)** IPs:

```bash
# No country specified = United States IPs
username:password@proxy.mrscraper.com:10000

# Equivalent to
username-country-us:password@proxy.mrscraper.com:10000
```


  **Note:** The Residential Proxy UI always requires a country selection; only manual username configuration lets you omit it. Because omitting it still resolves to the US, set the country explicitly whenever the target region matters — relying on the default hides that intent from your code.


## Code Examples

### Python: Multi-Country Scraping

```python
import requests
from concurrent.futures import ThreadPoolExecutor

countries = ['us', 'gb', 'de', 'fr', 'jp', 'au']

def scrape_with_country(country_code):
    """Scrape using a specific country proxy"""
    proxy = {
        'http': f'http://user-country-{country_code}:pass@proxy.mrscraper.com:10000',
        'https': f'http://user-country-{country_code}:pass@proxy.mrscraper.com:10000'
    }
    
    response = requests.get('https://example.com', proxies=proxy)
    print(f"Scraped from {country_code.upper()}: {len(response.text)} bytes")
    return response.text

# Scrape from all countries in parallel
with ThreadPoolExecutor(max_workers=len(countries)) as executor:
    results = list(executor.map(scrape_with_country, countries))
```

### Node.js: Dynamic Country Selection

```javascript
const axios = require('axios');

const countries = ['us', 'gb', 'de', 'fr', 'jp'];

async function scrapeFromCountry(countryCode) {
  const proxy = {
    host: 'proxy.mrscraper.com',
    port: 10000,
    auth: {
      username: `user-country-${countryCode}`,
      password: 'pass123'
    }
  };

  try {
    const response = await axios.get('https://example.com', { proxy });
    console.log(`Scraped from ${countryCode.toUpperCase()}: ${response.data.length} bytes`);
    return response.data;
  } catch (error) {
    console.error(`Error with ${countryCode}:`, error.message);
  }
}

// Scrape from multiple countries
async function scrapeAll() {
  for (const country of countries) {
    await scrapeFromCountry(country);
  }
}

scrapeAll();
```

## Complete Country Code Reference

Below is the complete list of all ISO 3166-1 alpha-2 country codes supported by Residential Proxy:


  
    **View All Country Codes (240+ countries)**

     \- Click to expand
  

  | Country                        | Code |   | Country                   | Code |
  | ------------------------------ | ---- | - | ------------------------- | ---- |
  | Afghanistan                    | `af` |   | Liechtenstein             | `li` |
  | Åland Islands                  | `ax` |   | Lithuania                 | `lt` |
  | Albania                        | `al` |   | Luxembourg                | `lu` |
  | Algeria                        | `dz` |   | Macao                     | `mo` |
  | American Samoa                 | `as` |   | North Macedonia           | `mk` |
  | Andorra                        | `ad` |   | Madagascar                | `mg` |
  | Angola                         | `ao` |   | Malawi                    | `mw` |
  | Anguilla                       | `ai` |   | Malaysia                  | `my` |
  | Antarctica                     | `aq` |   | Maldives                  | `mv` |
  | Antigua and Barbuda            | `ag` |   | Mali                      | `ml` |
  | Argentina                      | `ar` |   | Malta                     | `mt` |
  | Armenia                        | `am` |   | Marshall Islands          | `mh` |
  | Aruba                          | `aw` |   | Martinique                | `mq` |
  | Australia                      | `au` |   | Mauritania                | `mr` |
  | Austria                        | `at` |   | Mauritius                 | `mu` |
  | Azerbaijan                     | `az` |   | Mayotte                   | `yt` |
  | Bahamas                        | `bs` |   | Mexico                    | `mx` |
  | Bahrain                        | `bh` |   | Micronesia                | `fm` |
  | Bangladesh                     | `bd` |   | Moldova                   | `md` |
  | Barbados                       | `bb` |   | Monaco                    | `mc` |
  | Belarus                        | `by` |   | Mongolia                  | `mn` |
  | Belgium                        | `be` |   | Montenegro                | `me` |
  | Belize                         | `bz` |   | Montserrat                | `ms` |
  | Benin                          | `bj` |   | Morocco                   | `ma` |
  | Bermuda                        | `bm` |   | Mozambique                | `mz` |
  | Bhutan                         | `bt` |   | Myanmar                   | `mm` |
  | Bolivia                        | `bo` |   | Namibia                   | `na` |
  | Bonaire                        | `bq` |   | Nauru                     | `nr` |
  | Bosnia and Herzegovina         | `ba` |   | Nepal                     | `np` |
  | Botswana                       | `bw` |   | Netherlands               | `nl` |
  | Bouvet Island                  | `bv` |   | New Caledonia             | `nc` |
  | Brazil                         | `br` |   | New Zealand               | `nz` |
  | British Indian Ocean Territory | `io` |   | Nicaragua                 | `ni` |
  | Brunei                         | `bn` |   | Niger                     | `ne` |
  | Bulgaria                       | `bg` |   | Nigeria                   | `ng` |
  | Burkina Faso                   | `bf` |   | Niue                      | `nu` |
  | Burundi                        | `bi` |   | Norfolk Island            | `nf` |
  | Cambodia                       | `kh` |   | Northern Mariana Islands  | `mp` |
  | Cameroon                       | `cm` |   | Norway                    | `no` |
  | Canada                         | `ca` |   | Oman                      | `om` |
  | Cape Verde                     | `cv` |   | Pakistan                  | `pk` |
  | Cayman Islands                 | `ky` |   | Palau                     | `pw` |
  | Central African Republic       | `cf` |   | Palestine                 | `ps` |
  | Chad                           | `td` |   | Panama                    | `pa` |
  | Chile                          | `cl` |   | Papua New Guinea          | `pg` |
  | China                          | `cn` |   | Paraguay                  | `py` |
  | Christmas Island               | `cx` |   | Peru                      | `pe` |
  | Cocos Islands                  | `cc` |   | Philippines               | `ph` |
  | Colombia                       | `co` |   | Pitcairn                  | `pn` |
  | Comoros                        | `km` |   | Poland                    | `pl` |
  | Congo                          | `cg` |   | Portugal                  | `pt` |
  | Congo (DRC)                    | `cd` |   | Puerto Rico               | `pr` |
  | Cook Islands                   | `ck` |   | Qatar                     | `qa` |
  | Costa Rica                     | `cr` |   | Réunion                   | `re` |
  | Côte d'Ivoire                  | `ci` |   | Romania                   | `ro` |
  | Croatia                        | `hr` |   | Russian Federation        | `ru` |
  | Cuba                           | `cu` |   | Rwanda                    | `rw` |
  | Curaçao                        | `cw` |   | Saint Barthélemy          | `bl` |
  | Cyprus                         | `cy` |   | Saint Helena              | `sh` |
  | Czech Republic                 | `cz` |   | Saint Kitts and Nevis     | `kn` |
  | Denmark                        | `dk` |   | Saint Lucia               | `lc` |
  | Djibouti                       | `dj` |   | Saint Martin              | `mf` |
  | Dominica                       | `dm` |   | Saint Pierre and Miquelon | `pm` |
  | Dominican Republic             | `do` |   | Saint Vincent             | `vc` |
  | Ecuador                        | `ec` |   | Samoa                     | `ws` |
  | Egypt                          | `eg` |   | San Marino                | `sm` |
  | El Salvador                    | `sv` |   | Sao Tome and Principe     | `st` |
  | Equatorial Guinea              | `gq` |   | Saudi Arabia              | `sa` |
  | Eritrea                        | `er` |   | Senegal                   | `sn` |
  | Estonia                        | `ee` |   | Serbia                    | `rs` |
  | Eswatini                       | `sz` |   | Seychelles                | `sc` |
  | Ethiopia                       | `et` |   | Sierra Leone              | `sl` |
  | Falkland Islands               | `fk` |   | Singapore                 | `sg` |
  | Faroe Islands                  | `fo` |   | Sint Maarten              | `sx` |
  | Fiji                           | `fj` |   | Slovakia                  | `sk` |
  | Finland                        | `fi` |   | Slovenia                  | `si` |
  | France                         | `fr` |   | Solomon Islands           | `sb` |
  | French Guiana                  | `gf` |   | Somalia                   | `so` |
  | French Polynesia               | `pf` |   | South Africa              | `za` |
  | French Southern Territories    | `tf` |   | South Georgia             | `gs` |
  | Gabon                          | `ga` |   | South Sudan               | `ss` |
  | Gambia                         | `gm` |   | Spain                     | `es` |
  | Georgia                        | `ge` |   | Sri Lanka                 | `lk` |
  | Germany                        | `de` |   | Sudan                     | `sd` |
  | Ghana                          | `gh` |   | Suriname                  | `sr` |
  | Gibraltar                      | `gi` |   | Svalbard and Jan Mayen    | `sj` |
  | Greece                         | `gr` |   | Sweden                    | `se` |
  | Greenland                      | `gl` |   | Switzerland               | `ch` |
  | Grenada                        | `gd` |   | Syria                     | `sy` |
  | Guadeloupe                     | `gp` |   | Taiwan                    | `tw` |
  | Guam                           | `gu` |   | Tajikistan                | `tj` |
  | Guatemala                      | `gt` |   | Tanzania                  | `tz` |
  | Guernsey                       | `gg` |   | Thailand                  | `th` |
  | Guinea                         | `gn` |   | Timor-Leste               | `tl` |
  | Guinea-Bissau                  | `gw` |   | Togo                      | `tg` |
  | Guyana                         | `gy` |   | Tokelau                   | `tk` |
  | Haiti                          | `ht` |   | Tonga                     | `to` |
  | Heard Island                   | `hm` |   | Trinidad and Tobago       | `tt` |
  | Holy See                       | `va` |   | Tunisia                   | `tn` |
  | Honduras                       | `hn` |   | Turkey                    | `tr` |
  | Hong Kong                      | `hk` |   | Turkmenistan              | `tm` |
  | Hungary                        | `hu` |   | Turks and Caicos Islands  | `tc` |
  | Iceland                        | `is` |   | Tuvalu                    | `tv` |
  | India                          | `in` |   | Uganda                    | `ug` |
  | Indonesia                      | `id` |   | Ukraine                   | `ua` |
  | Iran                           | `ir` |   | United Arab Emirates      | `ae` |
  | Iraq                           | `iq` |   | United Kingdom            | `gb` |
  | Ireland                        | `ie` |   | United States             | `us` |
  | Isle of Man                    | `im` |   | US Minor Outlying Islands | `um` |
  | Israel                         | `il` |   | Uruguay                   | `uy` |
  | Italy                          | `it` |   | Uzbekistan                | `uz` |
  | Jamaica                        | `jm` |   | Vanuatu                   | `vu` |
  | Japan                          | `jp` |   | Venezuela                 | `ve` |
  | Jersey                         | `je` |   | Vietnam                   | `vn` |
  | Jordan                         | `jo` |   | Virgin Islands (British)  | `vg` |
  | Kazakhstan                     | `kz` |   | Virgin Islands (U.S.)     | `vi` |
  | Kenya                          | `ke` |   | Wallis and Futuna         | `wf` |
  | Kiribati                       | `ki` |   | Western Sahara            | `eh` |
  | North Korea                    | `kp` |   | Yemen                     | `ye` |
  | South Korea                    | `kr` |   | Zambia                    | `zm` |
  | Kuwait                         | `kw` |   | Zimbabwe                  | `zw` |
  | Kyrgyzstan                     | `kg` |   |                           |      |
  | Laos                           | `la` |   |                           |      |
  | Latvia                         | `lv` |   |                           |      |
  | Lebanon                        | `lb` |   |                           |      |
  | Lesotho                        | `ls` |   |                           |      |
  | Liberia                        | `lr` |   |                           |      |
  | Libya                          | `ly` |   |                           |      |



  **Note:** Not all countries may have proxy coverage. Availability depends on Residential Proxy's network infrastructure in each region. Contact support to verify coverage for specific countries.


## Common Errors

### Invalid Country Code

**Error:** Connection fails or times out
**Cause:** Country code doesn't exist or is misspelled
**Solution:** Verify the code against ISO 3166-1 alpha-2 standard

```bash
# ❌ Wrong
user-country-usa:pass@proxy.mrscraper.com:10000  # Should be 'us', not 'usa'

# ✅ Correct
user-country-us:pass@proxy.mrscraper.com:10000
```

### Country Not Supported

**Error:** Connection fails or times out
**Cause:** Country code is valid but not in Residential Proxy's coverage
**Solution:** Try a nearby country or contact support

### Wrong Country Format

**Error:** Authentication fails
**Cause:** Incorrect parameter name or position
**Solution:** Use `-country-` exactly as shown

```bash
# ❌ Wrong formats
user-c-us:pass@proxy.mrscraper.com:10000         # Wrong parameter name
user-us-country:pass@proxy.mrscraper.com:10000   # Wrong order

# ✅ Correct format
user-country-us:pass@proxy.mrscraper.com:10000
```


# Proxy Status Codes

Proxy error responses identify failures that occur while processing or forwarding a request. Use the HTTP status code and the `Proxy-Status` response header to distinguish proxy-generated errors from responses returned by the destination server.

For SOCKS5 connections, inspect the protocol reply codes instead of HTTP status codes or headers.

## Response Format

The following response fields provide status information:

| Field                 | Location         | Purpose                                                                                                      |
| --------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------ |
| HTTP status code      | Status line      | Identifies the response category. Common proxy errors include `403`, `407`, `502`, `503`, and `504`.         |
| `Proxy-Status` header | Response headers | Identifies the reporting proxy and error type, following [RFC 9209](https://www.rfc-editor.org/rfc/rfc9209). |

For example:

```text
HTTP/1.1 504 Gateway Timeout
Proxy-Status: mrscraper; error=connection_timeout
```

This response indicates that the proxy gateway could not connect to the destination website through a proxy exit IP within the allowed time.

## Identifying the Response Source

```http
HTTP/1.1 200 Connection Established
```

* **Source:** MrScraper proxy — success.
* **Meaning:** The HTTPS CONNECT tunnel is open. This is the proxy's confirmation, not the target website's response.

```http
HTTP/1.1 502 Bad Gateway
Proxy-Status: mrscraper; error=connection_terminated
Content-Type: text/plain; charset=utf-8

Bad Gateway
```

* **Source:** MrScraper proxy — failure.
* **Meaning:** The `mrscraper; error=...` entry identifies a proxy-classified failure. Here, the connection closed or reset before the proxy could complete the response.

```http
HTTP/1.1 200 OK
Content-Type: text/html

The requested page.
```

* **Source:** Target website — success.
* **Meaning:** The target returned the requested page, which the proxy relayed to the client.

```http
HTTP/1.1 403 Forbidden
Content-Type: text/html

Access denied by the target website.
```

* **Source:** Target website — failure.
* **Meaning:** The target rejected the request. The proxy relayed its response rather than generating a proxy error.

The target-response examples show typical forwarded responses. Body content and the absence of a `mrscraper` error entry alone do not prove target origin: some proxy-generated failures can also lack that entry.

## HTTPS CONNECT Responses

HTTPS connections through the proxy begin with a `CONNECT` request. Before tunnel establishment, the proxy generates the response:

| Status | `error=`                        | What it means                                                                                                               |
| -----: | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
|  `200` | None                            | `Connection Established` — the tunnel is open. Subsequent encrypted traffic cannot include a proxy-generated status header. |
|  `403` | `http_request_denied_by_policy` | The request was blocked by policy.                                                                                          |
|  `407` | None                            | Client authentication or account access was rejected.                                                                       |
|  `502` | `dns_error`                     | A hostname needed to complete the request could not be resolved.                                                            |
|  `502` | `connection_terminated`         | The connection closed or reset before the tunnel could be established.                                                      |
|  `502` | None                            | The proxy gateway could not open the connection. No more specific error code is available.                                  |
|  `503` | `destination_unavailable`       | No suitable proxy exit IP is available, or the selected proxy exit IP cannot reach the destination website.                 |
|  `504` | `connection_timeout`            | The proxy gateway could not connect to the destination website through a proxy exit IP within the allowed time.             |
|  `504` | `connection_read_timeout`       | Waiting for connection data timed out before the tunnel could be established.                                               |
|  `504` | `connection_write_timeout`      | Sending data required to establish the tunnel timed out.                                                                    |
|  `504` | `http_response_timeout`         | The proxy timed out waiting for a response while establishing the tunnel.                                                   |

A CONNECT failure does not return the destination server's application error page because the tunnel has not been established. After `200 Connection Established`, the proxy cannot inspect encrypted application traffic or send a replacement HTTP error. Certificate validation failures, authentication pages, and application errors within the tunnel cannot be reported through `Proxy-Status`. Later transport failures close the connection instead.

`destination_unavailable` means no suitable proxy exit IP is available to complete your request, or the selected proxy exit IP cannot reach the destination website. It does not necessarily mean the website is down. Try again after a short delay or broaden your location targeting.

## SOCKS5

If you use SOCKS5, your proxy client reports SOCKS5 reply codes instead of HTTP status codes. Check your client's error message or logs for the code, then use the tables below to find the next step.

Errors can occur when your client signs in or when it asks the proxy to connect to a website. Check which step failed: the same code can mean different things at each step.

### Sign-In Responses

During sign-in, your client first agrees on an authentication method with the proxy gateway, then sends credentials if required.

| Reply  | What it means                                                          | Client action                                                      |
| ------ | ---------------------------------------------------------------------- | ------------------------------------------------------------------ |
| `0x00` | Sign-in succeeded after your client sent a username and password.      | Continue with the request.                                         |
| `0x02` | The proxy gateway selected username/password authentication.           | Send credentials.                                                  |
| `0xFF` | Your client did not offer a supported authentication method.           | Enable username/password authentication.                           |
| `0x01` | The proxy gateway rejected sign-in after your client sent credentials. | Check your username, password, and account access before retrying. |

### SOCKS5 Connection Responses

After sign-in, your client asks the proxy to connect to the destination website. The proxy returns a code indicating whether that request was accepted or why it failed. Some clients label this code `REP`, which means “reply.”

Codes are shown in hexadecimal, a number format commonly used in client logs. For example, `0x04` is code 4. `0x00` means the request was accepted; other values indicate an error.

| Reply code (`REP`) | Name                              | What it means                                                                                                                                                                                                | Client action                                                                              |
| -----------------: | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------ |
|             `0x00` | Succeeded                         | The request was accepted. For hostname requests, the tunnel is established. For IP-address requests, connection establishment may still be in progress.                                                      | Start application traffic. A later failure can close the connection without another reply. |
|             `0x02` | Connection not allowed by ruleset | Destination policy, account restrictions, or request-stage authentication rejected the request. `BIND` is also denied.                                                                                       | Resolve the restriction before retrying.                                                   |
|             `0x03` | Network unreachable               | A network needed to establish the requested connection could not be reached. This does not establish that the destination network is unreachable.                                                            | Retry a limited number of times with backoff.                                              |
|             `0x04` | Host unreachable                  | The proxy could not connect. Possible causes include an unavailable proxy exit IP, a connection timeout, or a website address that could not be resolved. This code alone does not mean the website is down. | Check the hostname and targeting options. Retry transient failures with bounded backoff.   |

You may also see these client errors:

| Code                                | What it means                                                                                | What to do                                                                  |
| ----------------------------------- | -------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| `0x05` — Connection refused         | The proxy gateway encountered a refused connection while connecting through a proxy exit IP. | Follow the same steps as for `0x04`.                                        |
| `0x07` — Command not supported      | Your client used an unsupported SOCKS5 command or HTTP-only scraper credentials.             | Check your client's SOCKS5 settings and use residential proxy credentials.  |
| `0x08` — Address type not supported | Your client sent an unsupported destination address format.                                  | Configure your client to use an IPv4 address, IPv6 address, or domain name. |

Retry a few times, waiting longer after each attempt. Avoid retrying requests that could cause duplicate actions. The website may have already processed the first request.

## Plain HTTP Responses

For unencrypted HTTP requests, the proxy can return the following errors before sending response headers to the client. Classified failures include a `mrscraper; error=...` entry in `Proxy-Status`.

| Status | `error=`                        | What it means                                                                                                   |
| -----: | ------------------------------- | --------------------------------------------------------------------------------------------------------------- |
|  `403` | `http_request_denied_by_policy` | The request was blocked by policy.                                                                              |
|  `407` | None                            | Client authentication or account access was rejected.                                                           |
|  `502` | `dns_error`                     | A hostname needed to complete the request could not be resolved.                                                |
|  `502` | `connection_terminated`         | The connection closed or reset before response headers could be sent to the client.                             |
|  `502` | None                            | The proxy gateway could not complete the request. No more specific error code is available.                     |
|  `503` | `destination_unavailable`       | No suitable proxy exit IP is available, or the selected proxy exit IP cannot reach the destination website.     |
|  `504` | `connection_timeout`            | The proxy gateway could not connect to the destination website through a proxy exit IP within the allowed time. |
|  `504` | `connection_read_timeout`       | Waiting for response data timed out before response headers could be sent to the client.                        |
|  `504` | `connection_write_timeout`      | Sending request data timed out before the response could start.                                                 |
|  `504` | `http_response_timeout`         | Waiting for the response to start timed out.                                                                    |

Once HTTP response headers have been sent, a later read timeout or connection termination cannot replace the response with a new `502`, `504`, or `Proxy-Status` header. The client observes a truncated response or a closed connection instead.

## When to Retry

For temporary connection failures, limit the number of retries and wait progressively longer between attempts. Add a small random delay to avoid sending all retries at once.

### HTTP and HTTPS CONNECT

The following guidance applies to plain HTTP responses and HTTPS CONNECT responses before the tunnel opens.

| Response                                                              | Client retry guidance                                                                                                                                                                                                  |
| --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Proxy-generated `403`                                                 | Do not retry the identical request automatically. Resolve the policy restriction or change the destination.                                                                                                            |
| `407`                                                                 | Do not retry unchanged credentials indefinitely. Check credentials and account access. If those are correct and the rejection persists, contact support.                                                               |
| Proxy-generated `502`                                                 | Retry a limited number of times for transient connection failures, including `connection_terminated` or an unclassified failure. If errors persist, check the destination and targeting options, then contact support. |
| Proxy-generated `503`                                                 | Retry with backoff if availability may recover. If no proxy exit IP matches your location targeting, broaden the targeting instead of repeatedly sending the identical request.                                        |
| Proxy-generated `504`                                                 | Retry with backoff. A proxy timeout does not prove that the destination did not receive or process the request.                                                                                                        |
| A destination response, including `403`, `429`, or `5xx`              | Follow the destination's retry policy and any `Retry-After` header. Do not treat it as a proxy failure solely because of the HTTP status.                                                                              |
| Connection closure after CONNECT success or a truncated HTTP response | Reconnect and retry only if replaying the application request is safe. There may be no replacement HTTP error or proxy status header.                                                                                  |

### SOCKS5

Check whether the code occurred during sign-in or while connecting to the destination.

| Response                                                                                | Client retry guidance                                                                                                                          |
| --------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| Sign-in rejected (`0x01`) or no supported authentication method (`0xFF`)                | Check your username, password, account access, and client authentication settings before retrying.                                             |
| Connection not allowed (`0x02`)                                                         | Resolve the destination or account restriction before retrying.                                                                                |
| Network unreachable (`0x03`), host unreachable (`0x04`), or connection refused (`0x05`) | Retry a limited number of times with increasing delays. If errors persist, check the destination and location targeting, then contact support. |
| Unsupported command (`0x07`) or address type (`0x08`)                                   | Correct your client settings before retrying.                                                                                                  |
| Connection closes after a successful reply (`0x00`)                                     | Reconnect and retry only if the application request is safe to repeat. The destination may already have received it.                           |

Only replay requests that are safe to repeat, such as idempotent operations or requests protected by an application-supported idempotency key. Ensure request bodies can be replayed. If the response was lost after the destination processed a request, retrying a non-idempotent operation can duplicate its effects.


# Proxy Types

import { RefreshCw, Pin } from 'lucide-react';

Residential Proxy offers **Static** and **Rotating** proxies, each designed for different use cases. Choose the right type based on whether you need consistent IP sessions or maximum distribution.

## Proxy Protocols

Residential Proxy supports both HTTP and SOCKS5 connections:

| Protocol   | Proxy URL prefix | Port    | Best for                                                                 |
| ---------- | ---------------- | ------- | ------------------------------------------------------------------------ |
| **HTTP**   | `http://`        | `10000` | HTTP requests, browser automation, and standard scraping clients         |
| **SOCKS5** | `socks5://`      | `10001` | Applications that require SOCKS5 support or proxying traffic beyond HTTP |

The full proxy URL uses this format:

```text
{protocol}://username:password@proxy.mrscraper.com:{port}
```

For example:

```text
http://user-country-us:pass@proxy.mrscraper.com:10000
socks5://user-country-us:pass@proxy.mrscraper.com:10001
```

## Static vs Rotating Proxies

### Static Proxy

A static proxy maintains the same IP address throughout a session. By using a session ID, all requests within that session are routed through the same IP until the session expires.

**Key Features:**

* Same IP address for all requests within the session
* Configurable session duration (default: 10 minutes)
* Consistent identity across requests
* Ideal for stateful operations

**Authentication Format:**

```bash
http://username-country-{code}-sessid-{id}-sesstime-{minutes}:password@proxy.mrscraper.com:10000
socks5://username-country-{code}-sessid-{id}-sesstime-{minutes}:password@proxy.mrscraper.com:10001
```

**Example:**

```python
import requests

# Static proxy with 30-minute session
proxy = {
    'http': 'http://user-country-us-sessid-stable1-sesstime-30:pass@proxy.mrscraper.com:10000',
    'https': 'http://user-country-us-sessid-stable1-sesstime-30:pass@proxy.mrscraper.com:10000'
}

# All requests use the same IP
for i in range(10):
    response = requests.get('https://api.ipify.org', proxies=proxy)
    print(f"Request {i+1}: {response.text}")  # Same IP every time
```

**Perfect For:**

| **Account Operations**           | **Stable Scraping Sessions**              | **Financial Transactions**          |
| -------------------------------- | ----------------------------------------- | ----------------------------------- |
| ✅ Login and authentication flows | ✅ Multi-page form submissions             | ✅ Payment processing                |
| ✅ Session-based interactions     | ✅ Workflows requiring consistent identity | ✅ Banking operations                |
| ✅ Account management dashboards  | ✅ APIs that track IP changes              | ✅ Any flow requiring IP consistency |
| ✅ Shopping cart operations       | ✅ Testing session persistence             |                                     |


  **Tip:** Start with a 10-minute session for testing. Increase duration for longer workflows, but keep it under 60 minutes for optimal IP freshness.


### Rotating Proxy

A rotating proxy automatically changes IP addresses either on every request or when the session expires, assigning a new IP to each request or session.

**Key Features:**

* Different IP for each request (or after session expiry)
* Maximum distribution across IP pool
* Reduces detection risk
* Ideal for high-volume operations

**Authentication Format (No Session):**

```bash
http://username-country-{code}:password@proxy.mrscraper.com:10000
socks5://username-country-{code}:password@proxy.mrscraper.com:10001
```

**Example:**

```python
import requests

# Rotating proxy - new IP each request
proxy = {
    'http': 'http://user-country-us:pass@proxy.mrscraper.com:10000',
    'https': 'http://user-country-us:pass@proxy.mrscraper.com:10000'
}

# Each request gets a different IP
for i in range(10):
    response = requests.get('https://api.ipify.org', proxies=proxy)
    print(f"Request {i+1}: {response.text}")  # Different IP each time
```

**Perfect For:**

| **Large-Scale Scraping**      | **Avoiding Rate Limits**           | **Reducing Bans**                |
| ----------------------------- | ---------------------------------- | -------------------------------- |
| ✅ Crawling thousands of pages | ✅ Spreading load across IPs        | ✅ Appearing as different users   |
| ✅ High-volume data extraction | ✅ Distributing load across IPs     | ✅ Avoiding IP-based blacklists   |
| ✅ Distributed scraping tasks  | ✅ Reducing ban risk                | ✅ Region-specific content checks |
| ✅ Avoiding rate limits        | ✅ Testing rate limiting mechanisms | ✅ Market research at scale       |


  **Note:** Rotating proxies are not suitable for workflows requiring session persistence, such as login flows or multi-step forms.


## Comparison

| Feature              | Static Proxy                     | Rotating Proxy                             |
| -------------------- | -------------------------------- | ------------------------------------------ |
| **IP Behavior**      | Same IP per session              | New IP per request                         |
| **Session Support**  | ✅ Yes (via session ID)           | ❌ No session needed                        |
| **Use Case**         | Login flows, stateful operations | High-volume scraping, rate limit avoidance |
| **Session Duration** | Configurable (default 10 min)    | N/A                                        |
| **Best For**         | Consistency required             | Maximum distribution                       |

## Which Type Should I Use?


  }>
    * Login and authentication flows
    * Shopping carts and multi-step forms
    * Sites that track IP changes
    * Session-based operations
  

  }>
    * Large-scale scraping (1000+ pages)
    * Spreading requests across IPs
    * Maximum IP distribution
    * No session persistence needed
  



# Target Blocklist

## How Target Blocklist Works

Target Blocklist lets you choose which targets your Residential Proxy should block, helping you keep unwanted requests from consuming your bandwidth.

Manage your rules in the **Save your bandwidth** tab on the Residential Proxy page.

**Behavior:**

* Add up to **50 target entries** to your User Blocklist
* Start blocking without a schedule or choose a future start time
* Changes apply to your account and everyone on your team after saving
* This customer tab shows user entries only; global blocklist entries are not
  shown or editable here
* Adding, editing, or removing an entry does not save it automatically


  **Shared rules:** If you are using a team account, you edit the team's shared
  blocklist, not a separate list for yourself. Review changes with your team
  before saving.


## Add a Target

1. Open the **Residential Proxy** page.
2. Select the **Save your bandwidth** tab.
3. In **User Blocklist**, click **Add target**.
4. Enter a target: a hostname, IP address, or CIDR range.
5. Leave **Starts at (UTC)** blank to apply the block as soon as your saved changes take effect, or select a start date and time.
6. Click **Save changes**.

After a successful save, the page shows **All changes saved**.


  Your changes will be propagated into our system within around 2 minutes. If it
  takes longer, contact our support.



  **Save before leaving:** Your changes only apply after you click **Save
  changes**. If saving fails, your edits remain on the page so you can try
  again.


## Accepted Target Formats

Each target must be a **hostname**, an **IP address**, or a **CIDR range**.

### Hostname

Include the domain extension, and use at least one dot. A hostname also blocks its subdomains.

**Format:**

```
abc.com
```

**Example matches:**

* `www.abc.com`
* `sub.abc.com`
* `dev.sub.abc.com`

### IP Address

Enter a single IPv4 address without a port.

**Formats:**

```
1.2.3.4
```

### CIDR Range

Enter an IPv4 address with a prefix length from 0 to 32.

**Formats:**

```
10.0.0.0/8
```

### Not Accepted

These values are rejected when you save:

```
https://abc.com
abc.com/path
abc.com:443
*.abc.com
abc
10.0.0.0/33
```


  **Enter the target on its own:** Do not include a scheme, path, port,
  wildcard, or spaces. Wildcards are never needed because subdomains are already
  blocked, and a hostname needs at least one dot.


## Schedule When Blocking Starts

Each entry has an optional **Starts at (UTC)** field.

| Start time           | Behavior                                            |
| -------------------- | --------------------------------------------------- |
| Blank                | Blocking starts once your saved changes take effect |
| Future date and time | Blocking starts at the scheduled time               |

**Example:**

To start blocking `abc.com` at a future time:

1. Add `abc.com` to your User Blocklist.
2. Set the **Starts at (UTC)** value on the entry.
3. Click **Save changes** before the scheduled time.


  **Use UTC:** All start times are in UTC. Convert your local time to UTC before
  choosing a start time.


## Edit or Remove a Target

### Edit an Entry

1. Find the entry in **User Blocklist**.
2. Update its **Target** or **Starts at (UTC)** value.
3. Click **Save changes**.

### Remove an Entry

1. Click the delete button beside the entry.
2. Click **Save changes**.

## Blocklist Limits

* You can save up to **50 user target entries**.
* Every entry must use an accepted format before you can save.
* Scheduled entries count toward the limit, even before their start time.
* At the limit, remove an existing entry before adding another.
* Every entry must have a target value before you can save.

## Best Practices

* Enter a hostname with at least one dot (`abc.com`), an IP address (`1.2.3.4`), or a CIDR range (`10.0.0.0/8`).
* Add one hostname, IP address, or CIDR range per entry.
* Start with targets you know your workflow does not need.
* Review shared rules with your team to avoid interrupting their requests.
* Check your workflow after saving to make sure required targets remain accessible.

## Common Issues

### Only Some Targets Are Blocked

* Check that the entry uses an accepted format: a hostname with a dot, an IP address, or a CIDR range.
* Compare the target with the accepted formats above.
* Check for typos in each entry.
* Make sure you clicked **Save changes** and allow around 2 minutes for your changes to take effect.
* If you set a start time, confirm that it has passed in UTC.

### Save Changes Is Disabled

* Check whether there are any unsaved changes.
* Fix any target marked as invalid below its field.
* Fill in every empty **Target** field, or remove unused entries.
* Make sure the list contains no more than 50 entries.
* Wait for the blocklist to finish loading or saving.

### The Blocklist Could Not Load

* Click **Retry** to load your saved rules again.
* If your team account could not be resolved, reload the page and try again.
* Editing is disabled while the blocklist is unavailable to avoid overwriting rules you cannot see.

### Changes Could Not Be Saved

* Your edits remain on the page after a save error.
* Keep the page open, check your connection, and try **Save changes** again.
* Wait for **All changes saved** before leaving the page.


# CloakBrowser

CloakBrowser is an open source Chromium-based browser developed by CloakHQ, with Playwright-like APIs for browser automation.

CloakBrowser code examples divided by programming language (JavaScript and Python) for browser automation with Residential Proxy.

## JavaScript (Node.js)

### Prerequisites

```bash
npm install cloakbrowser proxy-chain
npx cloakbrowser install
```

* `cloakbrowser`: Node.js browser automation library built on Chromium with Playwright-like APIs.

### Proxy Format

In JavaScript, CloakBrowser accepts a `proxy` object during `launch()`:

```javascript
const proxy = {
  server: 'http://proxy.mrscraper.com:10000',
  username: 'user-country-us-sessid-session1',
  password: 'pass123',
}
```

### SOCKS5 Proxy Setup

For SOCKS5 proxies, use `proxy-chain` to anonymize the authenticated SOCKS5 connection before passing the local proxy server to CloakBrowser:

```javascript
import { launch } from 'cloakbrowser'
import { anonymizeProxy, closeAnonymizedProxy } from 'proxy-chain'

const username = 'user-country-us'
const password = 'pass123'

const socks5Proxy =
  `socks5://${encodeURIComponent(username)}:${encodeURIComponent(password)}` +
  `@proxy.mrscraper.com:10001`

const localProxy = await anonymizeProxy(socks5Proxy)

const browser = await launch({
  proxy: {
    server: localProxy,
  },
})

try {
  const page = await browser.newPage()

  await page.goto('https://ipinfo.io/json')

  console.log(await page.textContent('body'))
} finally {
  await browser.close()
  await closeAnonymizedProxy(localProxy, true)
}
```

Use port `10001` for SOCKS5. Always close the browser before calling `closeAnonymizedProxy()`.

### 1. Basic Request

Demonstrates launching CloakBrowser with an authenticated residential proxy in JavaScript. It navigates to an IP lookup endpoint and prints the verified proxy IP.

```javascript
import { launch } from 'cloakbrowser'

async function testProxy() {
  // Launch the browser with proxy options
  const browser = await launch({
    headless: true,
    proxy: {
      server: 'http://proxy.mrscraper.com:10000',
      username: 'user-country-us',
      password: 'pass123',
    },
  })

  const page = await browser.newPage()
  await page.goto('https://api.ipify.org?format=json')

  const content = await page.textContent('body')
  console.log('Your Proxy IP:', content)

  await browser.close()
}

testProxy().catch(console.error)
```

**Use Case:** Baseline proxy connectivity check and browser launch in Node.js.

### 2. Sticky Session Navigation

Maintains a static IP address across sequential page navigations by reusing a session ID (`sessid`) and extending session time (`sesstime-30`).

```javascript
import { launch } from 'cloakbrowser'

async function runStickySession() {
  const browser = await launch({
    headless: true,
    proxy: {
      server: 'http://proxy.mrscraper.com:10000',
      // 30-minute Canadian static session
      username: 'user-country-ca-sessid-session1-sesstime-30',
      password: 'pass123',
    },
  })

  const page = await browser.newPage()

  const targetUrls = [
    'https://example.com/step1',
    'https://example.com/step2',
    'https://example.com/step3',
  ]

  for (const url of targetUrls) {
    await page.goto(url)
    console.log(`Visited ${url} -> Title:`, await page.title())
  }

  await browser.close()
}

runStickySession().catch(console.error)
```

**Use Case:** Executing multi-page JavaScript workflows that need a consistent IP across navigations.

### 3. Parallel Workers

Demonstrates spawning multiple concurrent JavaScript CloakBrowser instances, each configured with a unique sticky session ID for isolated parallel operations.

```javascript
import { launch } from 'cloakbrowser'

async function runParallelWorkers() {
  const workerIds = ['worker_1', 'worker_2', 'worker_3']

  const workerPromises = workerIds.map(async (workerId) => {
    const browser = await launch({
      headless: true,
      proxy: {
        server: 'http://proxy.mrscraper.com:10000',
        username: `user-country-us-sessid-${workerId}`,
        password: 'pass123',
      },
    })

    const page = await browser.newPage()
    await page.goto('https://api.ipify.org?format=json')
    const ipData = await page.textContent('body')

    console.log(`Worker [${workerId}] IP:`, ipData)
    await browser.close()
  })

  await Promise.all(workerPromises)
}

runParallelWorkers().catch(console.error)
```

**Use Case:** High-concurrency Node.js web scraping where each worker thread requires its own browser instance and IP isolation.

### 4. Production Error Handling

Production-ready error handling pattern for JavaScript CloakBrowser, providing navigation timeout configurations, response status validation, and safe browser closing routines.

```javascript
import { launch } from 'cloakbrowser'

async function safeScrape(targetUrl) {
  let browser = null
  try {
    browser = await launch({
      headless: true,
      proxy: {
        server: 'http://proxy.mrscraper.com:10000',
        username: 'user-country-us-sessid-prod1',
        password: 'pass123',
      },
    })

    const page = await browser.newPage()

    // Set 30s timeout
    const response = await page.goto(targetUrl, { timeout: 30000 })

    if (!response || !response.ok()) {
      throw new Error(`HTTP Error ${response?.status()}`)
    }

    console.log('Successfully loaded page title:', await page.title())
  } catch (error) {
    console.error(`CloakBrowser execution failed:`, error.message)
  } finally {
    if (browser) {
      await browser.close()
    }
  }
}

safeScrape('https://example.com')
```

**Use Case:** Deploying Node.js scrapers that handle timeouts and site exceptions safely.

***

## Python

### Prerequisites

```bash
pip install cloakbrowser
python -m cloakbrowser install
```

* `cloakbrowser`: Python package for Chromium browser automation.
* Install `sing-box` separately and make sure the `sing-box` command is available on your `PATH` for SOCKS5 proxy bridging.

### Proxy Format

In Python, CloakBrowser receives a dictionary with proxy parameters:

```python
proxy = {
    "server": "http://proxy.mrscraper.com:10000",
    "username": "user-country-us-sessid-session1",
    "password": "pass123"
}
```

### SOCKS5 Proxy Setup

CloakBrowser's Python integration uses `sing-box` to expose the authenticated SOCKS5 proxy as a local HTTP proxy:

```python
import json
import os
import subprocess
import tempfile
import time

from cloakbrowser import launch

username = "user-country-us"
password = "pass123"

local_proxy = "http://127.0.0.1:8080"

singbox_config = {
    "inbounds": [
        {
            "type": "http",
            "tag": "http-in",
            "listen": "127.0.0.1",
            "listen_port": 8080,
        }
    ],
    "outbounds": [
        {
            "type": "socks",
            "tag": "mrscraper",
            "server": "proxy.mrscraper.com",
            "server_port": 10001,
            "username": username,
            "password": password,
            "version": "5",
        }
    ],
    "route": {
        "final": "mrscraper"
    },
}

with tempfile.NamedTemporaryFile(
    mode="w",
    suffix=".json",
    delete=False,
) as config_file:
    json.dump(singbox_config, config_file)
    config_path = config_file.name

singbox_process = subprocess.Popen([
    "sing-box",
    "run",
    "-c",
    config_path,
])

try:
    time.sleep(1)

    browser = launch(
        headless=True,
        proxy={
            "server": local_proxy,
        },
    )

    try:
        page = browser.new_page()
        page.goto("https://ipinfo.io/json")
        print(page.text_content("body"))
    finally:
        browser.close()
finally:
    singbox_process.terminate()
    singbox_process.wait()
    os.remove(config_path)
```

Use port `10001` for SOCKS5 and always terminate `sing-box` after closing the browser.

### 1. Basic Request

Demonstrates launching CloakBrowser using its Python API to route automated browser requests through a residential proxy.

```python
from cloakbrowser import launch

def test_proxy():
    # Launch browser with proxy server and credentials
    browser = launch(
        headless=True,
        proxy={
            "server": "http://proxy.mrscraper.com:10000",
            "username": "user-country-us",
            "password": "pass123"
        }
    )

    page = browser.new_page()
    page.goto("https://api.ipify.org?format=json")

    print("Your Proxy IP:", page.text_content("body"))
    browser.close()

if __name__ == "__main__":
    test_proxy()
```

**Use Case:** Quick proxy verification script in Python.

### 2. Sticky Session Navigation

Maintains a static Python browser session across multiple page navigations using a session ID (`sessid`).

```python
from cloakbrowser import launch

def run_sticky_session():
    browser = launch(
        headless=True,
        proxy={
            "server": "http://proxy.mrscraper.com:10000",
            "username": "user-country-us-sessid-python1-sesstime-20",
            "password": "pass123"
        }
    )

    page = browser.new_page()

    urls = [
        "https://example.com/step1",
        "https://example.com/step2",
        "https://example.com/step3"
    ]

    for url in urls:
        page.goto(url)
        print(f"Scraped {url} -> Title: {page.title()}")

    browser.close()

if __name__ == "__main__":
    run_sticky_session()
```

**Use Case:** Python multi-step workflows that require session IP continuity.

### 3. Geotargeting

Demonstrates routing Python CloakBrowser traffic through a German residential proxy.

```python
from cloakbrowser import launch

def geotargeted_session():
    browser = launch(
        headless=True,
        proxy={
            "server": "http://proxy.mrscraper.com:10000",
            "username": "user-country-de-sessid-shop1",
            "password": "pass123"
        }
    )

    page = browser.new_page()
    page.goto("https://httpbin.org/headers")

    print("Browser Headers:\n", page.text_content("body"))
    browser.close()

if __name__ == "__main__":
    geotargeted_session()
```

**Use Case:** Retrieving the version of an international website served to a specific country.

### 4. Production Error Handling & Lifecycle Management

Provides robust Python error handling, custom navigation timeout configuration, response validation, and guaranteed teardown.

```python
from cloakbrowser import launch

def safe_scrape(target_url):
    browser = None
    try:
        browser = launch(
            headless=True,
            proxy={
                "server": "http://proxy.mrscraper.com:10000",
                "username": "user-country-us-sessid-prod1",
                "password": "pass123"
            }
        )

        page = browser.new_page()
        page.set_default_timeout(30000)

        page.goto(target_url)
        print("Successfully loaded page title:", page.title())
    except Exception as error:
        print(f"CloakBrowser Python execution failed: {error}")
    finally:
        if browser:
            browser.close()

if __name__ == "__main__":
    safe_scrape("https://example.com")
```

**Use Case:** Production Python automation scripts requiring clean error logging and reliable browser process cleanup.


  * CloakBrowser follows standard Playwright page and navigation APIs in both JavaScript and Python.
  * Run `npx cloakbrowser install` or `python -m cloakbrowser install` prior to execution to download the browser binaries.
  * Keep sensitive proxy credentials in environment variables (`process.env.PROXY_PASSWORD` or `os.environ.get("PROXY_PASSWORD")`).



# cURL

Command-line examples for testing Residential Proxy configurations quickly.

## SOCKS5 Proxy Setup

Every example on this page uses the HTTP endpoint on port `10000`. SOCKS5 is supported too, on port `10001` — add the scheme to `-x`:

```bash
# SOCKS5 with DNS resolved by the proxy
curl -x "socks5h://user-country-us:pass123@proxy.mrscraper.com:10001" https://api.ipify.org

# Equivalent using the dedicated flag
curl --socks5-hostname proxy.mrscraper.com:10001 \
  --proxy-user "user-country-us:pass123" \
  https://api.ipify.org
```

Use port `10001` for SOCKS5. Prefer `socks5h://` (or `--socks5-hostname`) so hostnames are resolved by the proxy instead of locally. Username parameters such as `-country-` and `-sessid-` behave identically on both protocols.

## 1. Basic Request

The simplest way to test Residential Proxy connectivity. This command routes your request through a US proxy and returns the proxy's IP address.

```bash
# Test proxy connection
curl -x "user-country-us:pass123@proxy.mrscraper.com:10000" https://api.ipify.org
```

**Use Case:** Quick connectivity testing and verifying your proxy credentials are working correctly.

## 2. Static Session Request

Demonstrates session-based proxying where the same session ID (`sessid-test1`) maintains the same IP address across multiple requests.

```bash
# Same IP across multiple requests
curl -x "user-country-us-sessid-test1:pass123@proxy.mrscraper.com:10000" https://example.com

# Verify same IP
curl -x "user-country-us-sessid-test1:pass123@proxy.mrscraper.com:10000" https://api.ipify.org
```

**Use Case:** Testing scenarios where you need to maintain session state, like logging into websites or simulating a single user's browsing session.

## 3. Rotating Proxy Test

This loop demonstrates IP rotation by making multiple requests without a session ID. Each request will come from a different IP address.

```bash
# Test IP rotation - each request gets different IP
for i in {1..5}; do
  echo "Request $i:"
  curl -x "user-country-gb:pass123@proxy.mrscraper.com:10000" https://api.ipify.org
  echo ""
done
```

**Use Case:** Verifying that IP rotation is working correctly and understanding the different IPs available in your proxy pool.

## 4. Custom Headers

Shows how to combine proxy usage with custom HTTP headers to appear more like a real browser from a specific region.

```bash
# Add custom user agent and headers
curl -x "user-country-de:pass123@proxy.mrscraper.com:10000" \
  -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)" \
  -H "Accept-Language: de-DE" \
  https://example.com
```

**Use Case:** Scraping geo-specific content or avoiding detection by matching your headers to your proxy's location.

## 5. Save Response

Captures the complete HTTP response body and saves it to a local file for later analysis or processing.

```bash
# Save scraped content to file
curl -x "user-country-jp-sessid-save1:pass123@proxy.mrscraper.com:10000" \
  -o output.html \
  https://example.com
```

**Use Case:** Batch downloading of pages, creating local copies of websites, or saving responses for offline analysis.

## 6. Verbose Output

Enables detailed logging to troubleshoot connection issues, see SSL handshakes, and understand the complete request/response flow.

```bash
# Debug proxy connection with verbose output
curl -v -x "user-country-fr:pass123@proxy.mrscraper.com:10000" \
  https://api.ipify.org
```

**Use Case:** Debugging proxy connection problems, understanding SSL certificate issues, or verifying that requests are properly routed through the proxy.

## 7. POST Request

Demonstrates sending POST requests with JSON data through a proxy, useful for API interactions and form submissions.

```bash
# POST request through proxy
curl -x "user-country-ca-sessid-post1:pass123@proxy.mrscraper.com:10000" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"key":"value"}' \
  https://api.example.com/endpoint
```

**Use Case:** Testing APIs through proxies, submitting forms, or sending data to web services from a specific region.

## 8. Custom Session Duration

Controls how long your proxy session remains active using the `sesstime` parameter, extending beyond the default 10-minute session time.

```bash
# 30-minute session
curl -x "user-country-au-sessid-long1-sesstime-30:pass123@proxy.mrscraper.com:10000" \
  https://example.com
```

**Use Case:** Long-running scripts that need to maintain the same IP for extended periods, such as multi-step authentication flows or comprehensive site crawling.

## 9. Multiple Requests Script

A practical bash script that demonstrates bulk scraping with status monitoring and rate limiting. Each URL gets a different IP due to proxy rotation.

```bash
#!/bin/bash

# Script to scrape multiple URLs with rotating proxy
PROXY="user-country-us:pass123@proxy.mrscraper.com:10000"

urls=(
  "https://example.com/page1"
  "https://example.com/page2"
  "https://example.com/page3"
  "https://example.com/page4"
  "https://example.com/page5"
)

for url in "${urls[@]}"; do
  echo "Scraping: $url"
  curl -x "$PROXY" -s "$url" -o /dev/null -w "Status: %{http_code}\n"
  sleep 1  # Delay between requests
done
```

**Use Case:** Batch processing multiple URLs with different IPs, monitoring success rates, and implementing respectful scraping practices with delays.

## 10. Error Handling

Production-ready error handling with timeouts and status code checking. Essential for reliable automated scraping operations.

```bash
# Check connection and handle errors
if curl -x "user-country-nl:pass123@proxy.mrscraper.com:10000" \
   --connect-timeout 30 \
   --max-time 60 \
   -f -s -o /dev/null https://example.com; then
  echo "Success"
else
  echo "Failed with exit code: $?"
fi
```

**Use Case:** Building robust automation scripts that can detect and handle network failures, timeouts, and HTTP errors gracefully.

## Useful cURL Options

| Option              | Description                       |
| ------------------- | --------------------------------- |
| `-x, --proxy`       | Specify proxy server              |
| `-H, --header`      | Add custom header                 |
| `-o, --output`      | Save output to file               |
| `-v, --verbose`     | Show detailed connection info     |
| `-s, --silent`      | Silent mode (no progress bar)     |
| `-f, --fail`        | Fail silently on HTTP errors      |
| `--connect-timeout` | Maximum time for connection       |
| `--max-time`        | Maximum time for entire operation |
| `-L, --location`    | Follow redirects                  |
| `-A, --user-agent`  | Set user agent string             |


# Go

Practical Go examples for integrating Residential Proxy into your scraping applications.

## SOCKS5 Proxy Setup

Every example on this page uses the HTTP endpoint on port `10000`. SOCKS5 is supported too, on port `10001`. `net/http` handles it natively — only the scheme and port change:

```go
package main

import (
	"fmt"
	"io"
	"net/http"
	"net/url"
)

func main() {
	// Same credentials, SOCKS5 scheme and port
	proxyURL, _ := url.Parse("socks5://user-country-us:pass123@proxy.mrscraper.com:10001")

	client := &http.Client{
		Transport: &http.Transport{
			Proxy: http.ProxyURL(proxyURL),
		},
	}

	resp, err := client.Get("https://api.ipify.org")
	if err != nil {
		fmt.Println("Error:", err)
		return
	}
	defer resp.Body.Close()

	body, _ := io.ReadAll(resp.Body)
	fmt.Println("Your IP:", string(body))
}
```

Use port `10001` for SOCKS5. `http.Transport` recognises the `socks5` scheme and passes the username and password from the URL as SOCKS authentication. Username parameters such as `-country-` and `-sessid-` behave identically on both protocols.

## 1. Basic HTTP Request

This example demonstrates the fundamental setup for using Residential Proxy with Go's `net/http` package. It configures a static proxy session and makes a simple GET request.

```go
package main

import (
	"fmt"
	"io"
	"net/http"
	"net/url"
)

func main() {
	// Static proxy
	proxyURL, _ := url.Parse("http://user-country-us-sessid-go1:pass123@proxy.mrscraper.com:10000")
	
	client := &http.Client{
		Transport: &http.Transport{
			Proxy: http.ProxyURL(proxyURL),
		},
	}
	
	resp, err := client.Get("https://api.ipify.org")
	if err != nil {
		fmt.Println("Error:", err)
		return
	}
	defer resp.Body.Close()
	
	body, _ := io.ReadAll(resp.Body)
	fmt.Println("Your IP:", string(body))
}
```

**Use Case:** Perfect for simple proxy testing, IP verification, and single-request scenarios where you need a consistent IP address through a static session.

## 2. Rotating Proxy for Scraping

Shows how to create a reusable scraping function that uses rotating proxies (no session ID). Each call gets a fresh IP address, ideal for avoiding rate limits.

```go
package main

import (
	"fmt"
	"io"
	"net/http"
	"net/url"
)

func scrapeWithProxy(targetURL string) error {
	// Rotating proxy - new IP per request
	proxyURL, _ := url.Parse("http://user-country-gb:pass123@proxy.mrscraper.com:10000")
	
	client := &http.Client{
		Transport: &http.Transport{
			Proxy: http.ProxyURL(proxyURL),
		},
	}
	
	resp, err := client.Get(targetURL)
	if err != nil {
		return fmt.Errorf("failed to fetch %s: %w", targetURL, err)
	}
	defer resp.Body.Close()
	
	body, _ := io.ReadAll(resp.Body)
	fmt.Printf("Scraped %s: %d bytes (status: %d)\n", targetURL, len(body), resp.StatusCode)
	
	return nil
}

func main() {
	urls := []string{
		"https://example.com/page1",
		"https://example.com/page2",
		"https://example.com/page3",
	}
	
	for _, url := range urls {
		if err := scrapeWithProxy(url); err != nil {
			fmt.Println("Error:", err)
		}
	}
}
```

**Use Case:** High-volume scraping where IP diversity is crucial for avoiding detection and rate limiting. Each page request appears to come from a different visitor.

## 3. Concurrent Scraping

Demonstrates parallel scraping using goroutines with different static proxy sessions. Each goroutine maintains its own consistent IP address for the duration of its work.

```go
package main

import (
	"fmt"
	"io"
	"net/http"
	"net/url"
	"sync"
)

func scrapeWithProxy(targetURL string, proxyAuth string, wg *sync.WaitGroup) {
	defer wg.Done()
	
	proxyURL, _ := url.Parse(fmt.Sprintf("http://%s@proxy.mrscraper.com:10000", proxyAuth))
	
	client := &http.Client{
		Transport: &http.Transport{
			Proxy: http.ProxyURL(proxyURL),
		},
	}
	
	resp, err := client.Get(targetURL)
	if err != nil {
		fmt.Printf("Error scraping %s: %v\n", targetURL, err)
		return
	}
	defer resp.Body.Close()
	
	body, _ := io.ReadAll(resp.Body)
	fmt.Printf("Scraped %s: %d bytes\n", targetURL, len(body))
}

func main() {
	urls := []string{
		"https://example.com/page1",
		"https://example.com/page2",
		"https://example.com/page3",
	}
	
	var wg sync.WaitGroup
	
	for i, targetURL := range urls {
		wg.Add(1)
		proxyAuth := fmt.Sprintf("user-country-fr-sessid-worker%d:pass123", i+1)
		go scrapeWithProxy(targetURL, proxyAuth, &wg)
	}
	
	wg.Wait()
	fmt.Println("All scraping completed")
}
```

**Use Case:** High-performance scraping that needs to process multiple URLs quickly while maintaining separate identities. Great for scraping sites with per-IP rate limits.**Use Case:** High-performance scraping that needs to process multiple URLs quickly while maintaining separate identities. Great for scraping sites with per-IP rate limits.

## 4. Error Handling & Retries

Implements robust error handling with exponential backoff retry logic. Essential for production scraping applications that need to handle network failures gracefully.

```go
package main

import (
	"fmt"
	"io"
	"math"
	"net/http"
	"net/url"
	"time"
)

func scrapeWithRetry(targetURL string, maxRetries int) ([]byte, error) {
	proxyURL, _ := url.Parse("http://user-country-de:pass123@proxy.mrscraper.com:10000")
	
	client := &http.Client{
		Transport: &http.Transport{
			Proxy: http.ProxyURL(proxyURL),
		},
		Timeout: 30 * time.Second,
	}
	
	var lastErr error
	
	for attempt := 0; attempt < maxRetries; attempt++ {
		resp, err := client.Get(targetURL)
		if err != nil {
			lastErr = err
			backoff := time.Duration(math.Pow(2, float64(attempt))) * time.Second
			fmt.Printf("Attempt %d failed: %v. Retrying in %v...\n", attempt+1, err, backoff)
			time.Sleep(backoff)
			continue
		}
		defer resp.Body.Close()
		
		if resp.StatusCode >= 200 && resp.StatusCode < 300 {
			body, _ := io.ReadAll(resp.Body)
			return body, nil
		}
		
		lastErr = fmt.Errorf("HTTP %d", resp.StatusCode)
		backoff := time.Duration(math.Pow(2, float64(attempt))) * time.Second
		fmt.Printf("Attempt %d returned %d. Retrying in %v...\n", attempt+1, resp.StatusCode, backoff)
		time.Sleep(backoff)
	}
	
	return nil, fmt.Errorf("failed after %d attempts: %w", maxRetries, lastErr)
}

func main() {
	body, err := scrapeWithRetry("https://example.com", 3)
	if err != nil {
		fmt.Println("Failed:", err)
	} else {
		fmt.Printf("Success: %d bytes\n", len(body))
	}
}
```

**Use Case:** Production-ready scraping that must handle temporary network issues, server errors, and proxy failures without crashing or losing data.

## 5. Custom Headers

Shows how to add realistic browser headers to your requests when using proxies. This helps avoid detection by making requests appear more legitimate.

```go
package main

import (
	"fmt"
	"io"
	"net/http"
	"net/url"
)

func scrapeWithHeaders(targetURL string) error {
	proxyURL, _ := url.Parse("http://user-country-jp:pass123@proxy.mrscraper.com:10000")
	
	client := &http.Client{
		Transport: &http.Transport{
			Proxy: http.ProxyURL(proxyURL),
		},
	}
	
	req, _ := http.NewRequest("GET", targetURL, nil)
	
	// Add custom headers
	req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)")
	req.Header.Set("Accept-Language", "ja-JP,ja;q=0.9")
	req.Header.Set("Accept", "text/html,application/xhtml+xml")
	
	resp, err := client.Do(req)
	if err != nil {
		return err
	}
	defer resp.Body.Close()
	
	body, _ := io.ReadAll(resp.Body)
	fmt.Printf("Scraped %d bytes with status %d\n", len(body), resp.StatusCode)
	
	return nil
}

func main() {
	if err := scrapeWithHeaders("https://example.com"); err != nil {
		fmt.Println("Error:", err)
	}
}
```

**Use Case:** Scraping sites that check for browser-like behavior or when you need to match your headers to your proxy's geographic location for consistency.

## 6. Context with Timeout

Demonstrates using Go's context package for proper timeout handling and request cancellation. This prevents requests from hanging indefinitely.

```go
package main

import (
	"context"
	"fmt"
	"io"
	"net/http"
	"net/url"
	"time"
)

func scrapeWithContext(targetURL string) error {
	proxyURL, _ := url.Parse("http://user-country-ca-sessid-ctx1:pass123@proxy.mrscraper.com:10000")
	
	client := &http.Client{
		Transport: &http.Transport{
			Proxy: http.ProxyURL(proxyURL),
		},
	}
	
	// Create context with 30-second timeout
	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()
	
	req, _ := http.NewRequestWithContext(ctx, "GET", targetURL, nil)
	
	resp, err := client.Do(req)
	if err != nil {
		return fmt.Errorf("request failed: %w", err)
	}
	defer resp.Body.Close()
	
	body, _ := io.ReadAll(resp.Body)
	fmt.Printf("Scraped %d bytes\n", len(body))
	
	return nil
}

func main() {
	if err := scrapeWithContext("https://example.com"); err != nil {
		fmt.Println("Error:", err)
	}
}
```

**Use Case:** Applications that need precise control over request timeouts and graceful cancellation, especially in concurrent scenarios or when dealing with unreliable target sites.

## 7. Rate Limiting

Implements respectful scraping with controlled request rates using Go's time.Ticker. This prevents overwhelming target servers while maintaining steady throughput.

```go
package main

import (
	"fmt"
	"io"
	"net/http"
	"net/url"
	"time"
)

func scrapeWithRateLimit(urls []string, requestsPerMinute int) {
	proxyURL, _ := url.Parse("http://user-country-au:pass123@proxy.mrscraper.com:10000")
	
	client := &http.Client{
		Transport: &http.Transport{
			Proxy: http.ProxyURL(proxyURL),
		},
	}
	
	delay := time.Minute / time.Duration(requestsPerMinute)
	ticker := time.NewTicker(delay)
	defer ticker.Stop()
	
	for _, targetURL := range urls {
		<-ticker.C // Wait for next tick
		
		resp, err := client.Get(targetURL)
		if err != nil {
			fmt.Printf("Error scraping %s: %v\n", targetURL, err)
			continue
		}
		
		body, _ := io.ReadAll(resp.Body)
		resp.Body.Close()
		
		fmt.Printf("Scraped %s: %d bytes\n", targetURL, len(body))
	}
}

func main() {
	urls := []string{
		"https://example.com/page1",
		"https://example.com/page2",
		"https://example.com/page3",
	}
	
	scrapeWithRateLimit(urls, 20) // 20 requests per minute
}
```

**Use Case:** Ethical scraping that respects server resources and terms of service. Essential for long-running scraping operations that need to avoid being blocked.


# Node.js

Practical Node.js examples for integrating Residential Proxy into your scraping and automation projects.

## Prerequisites

```bash
npm install axios cheerio puppeteer
```

* `axios`: Promise-based HTTP client for making requests with built-in proxy support and interceptors.
* `cheerio`: Server-side jQuery implementation for parsing and manipulating HTML responses.
* `puppeteer`: Headless Chrome automation library for full browser simulation with proxy support.

## SOCKS5 Proxy Setup

Every example on this page uses the HTTP endpoint on port `10000`. SOCKS5 is supported too, on port `10001`. The axios `proxy` option only speaks HTTP, so SOCKS5 goes through an agent instead:

```bash
npm install socks-proxy-agent
```

```javascript
const axios = require('axios');
const { SocksProxyAgent } = require('socks-proxy-agent');

// Same credentials, SOCKS5 scheme and port
const agent = new SocksProxyAgent(
  'socks5://user-country-us-sessid-session1:pass123@proxy.mrscraper.com:10001'
);

const response = await axios.get('https://api.ipify.org', {
  httpAgent: agent,
  httpsAgent: agent,
  proxy: false,
});

console.log('Your IP:', response.data);
```

Use port `10001` for SOCKS5. Keep `proxy: false` so axios does not layer its own HTTP proxy on top of the agent. Username parameters such as `-country-` and `-sessid-` behave identically on both protocols.

## 1. Basic Static Proxy

Demonstrates the simplest proxy setup with axios using a static session. The same IP address is maintained across requests using the session ID parameter.

```javascript
const axios = require('axios');

// Static proxy configuration
const proxy = {
  host: 'proxy.mrscraper.com',
  port: 10000,
  auth: {
    username: 'user-country-us-sessid-session1',
    password: 'pass123'
  }
};

// Make a request
async function testProxy() {
  try {
    const response = await axios.get('https://api.ipify.org', { proxy });
    console.log('Your IP:', response.data);
  } catch (error) {
    console.error('Error:', error.message);
  }
}

testProxy();
```

**Use Case:** Simple proxy testing and scenarios where you need consistent IP addresses for maintaining login sessions or avoiding duplicate detection.

## 2. Rotating Proxy with Axios

Shows how to combine rotating proxies with Cheerio for HTML parsing. Each request gets a fresh IP address, perfect for scraping multiple pages without detection.

```javascript
const axios = require('axios');
const cheerio = require('cheerio');

// Rotating proxy - new IP per request
const rotatingProxy = {
  host: 'proxy.mrscraper.com',
  port: 10000,
  auth: {
    username: 'user-country-gb',
    password: 'pass123'
  }
};

async function scrapePage(url) {
  try {
    const response = await axios.get(url, { 
      proxy: rotatingProxy,
      timeout: 30000
    });
    
    const $ = cheerio.load(response.data);
    const title = $('h1').first().text();
    
    console.log(`Scraped: ${title}`);
    return response.data;
    
  } catch (error) {
    console.error(`Error scraping ${url}:`, error.message);
    return null;
  }
}

// Scrape multiple pages (each with different IP)
async function scrapeMultiple() {
  const urls = [
    'https://example.com/page1',
    'https://example.com/page2',
    'https://example.com/page3'
  ];
  
  for (const url of urls) {
    await scrapePage(url);
  }
}

scrapeMultiple();
```

**Use Case:** High-volume web scraping where IP rotation helps avoid rate limits and detection. Ideal for scraping e-commerce sites, job boards, or news sites.

## 3. Multiple Sessions (Promise.all)

Demonstrates parallel scraping using Promise.all with different static proxy sessions. Each concurrent request maintains its own consistent IP address.

```javascript
const axios = require('axios');

// Create array of proxy configs with different sessions
const proxies = Array.from({ length: 5 }, (_, i) => ({
  host: 'proxy.mrscraper.com',
  port: 10000,
  auth: {
    username: `user-country-de-sessid-worker${i + 1}`,
    password: 'pass123'
  }
}));

async function scrapeWithProxy(proxy, url) {
  try {
    const response = await axios.get(url, { proxy, timeout: 30000 });
    return {
      status: response.status,
      length: response.data.length,
      url
    };
  } catch (error) {
    return { error: error.message, url };
  }
}

async function parallelScraping() {
  const urls = Array.from({ length: 10 }, (_, i) => 
    `https://example.com/page${i + 1}`
  );
  
  // Scrape in parallel using different sessions
  const promises = urls.map((url, index) => {
    const proxy = proxies[index % proxies.length];
    return scrapeWithProxy(proxy, url);
  });
  
  const results = await Promise.all(promises);
  console.log('Results:', results);
}

parallelScraping();
```

**Use Case:** High-performance scraping that needs maximum throughput while maintaining multiple distinct identities. Great for scraping large datasets quickly.

## 4. Puppeteer with Proxy

Shows how to use Residential Proxy with Puppeteer for full browser automation. This enables JavaScript rendering, screenshot capture, and interaction with dynamic content.

```javascript
const puppeteer = require('puppeteer');

async function scrapeWithPuppeteer() {
  // Proxy URL format for Puppeteer
  const proxyUrl = 'http://user-country-jp-sessid-pup1:pass123@proxy.mrscraper.com:10000';
  
  const browser = await puppeteer.launch({
    headless: true,
    args: [
      `--proxy-server=${proxyUrl}`,
      '--no-sandbox',
      '--disable-setuid-sandbox'
    ]
  });
  
  try {
    const page = await browser.newPage();
    
    // Navigate to page
    await page.goto('https://example.com', { waitUntil: 'networkidle2' });
    
    // Extract data
    const title = await page.title();
    console.log('Page title:', title);
    
    // Take screenshot
    await page.screenshot({ path: 'screenshot.png' });
    
  } finally {
    await browser.close();
  }
}

scrapeWithPuppeteer();
```

**Use Case:** Scraping Single Page Applications (SPAs), taking screenshots for monitoring, or automating complex user interactions through a proxy.

## 5. Error Handling & Retries

Implements robust error handling with exponential backoff retry logic. Essential for production applications that need to handle network failures gracefully.

```javascript
const axios = require('axios');

const proxy = {
  host: 'proxy.mrscraper.com',
  port: 10000,
  auth: {
    username: 'user-country-fr',
    password: 'pass123'
  }
};

async function scrapeWithRetry(url, maxRetries = 3) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      const response = await axios.get(url, { 
        proxy, 
        timeout: 30000 
      });
      return response.data;
      
    } catch (error) {
      console.log(`Attempt ${attempt + 1} failed: ${error.message}`);
      
      if (attempt < maxRetries - 1) {
        const delay = Math.pow(2, attempt) * 1000;  // Exponential backoff
        await new Promise(resolve => setTimeout(resolve, delay));
      }
    }
  }
  
  console.log(`Failed after ${maxRetries} attempts`);
  return null;
}

// Use with retry logic
scrapeWithRetry('https://example.com')
  .then(data => console.log('Success:', data ? 'Got data' : 'Failed'));
```

**Use Case:** Production-ready scraping that must handle temporary network issues, server errors, and proxy failures without losing data or crashing.

## 6. Environment Variables

Demonstrates secure credential management using environment variables. This keeps sensitive proxy credentials out of your source code.

```javascript
const axios = require('axios');
require('dotenv').config();

// Use environment variables for credentials
const proxy = {
  host: 'proxy.mrscraper.com',
  port: 10000,
  auth: {
    username: process.env.MRPROXY_USERNAME,
    password: process.env.MRPROXY_PASSWORD
  }
};

async function secureScrape(url) {
  try {
    const response = await axios.get(url, { proxy });
    return response.data;
  } catch (error) {
    console.error('Error:', error.message);
    return null;
  }
}

// .env file:
// MRPROXY_USERNAME=user-country-us-sessid-session1
// MRPROXY_PASSWORD=pass123
```

**Use Case:** Production deployments where credentials need to be kept secure and separate from code. Essential for CI/CD pipelines and team development.


# Overview

import { Code2, Terminal, FileCode, Theater, Bot, Zap, FlaskConical, ShieldCheck, Globe, LineChart, Bug } from 'lucide-react';

Ready-to-use code examples for integrating Residential Proxy into your applications. Choose your programming language to get started.

## Quick Reference

### Example Proxy Configurations


  
    
      Default (US)
    

    
      US Static
    

    
      GB Rotating
    

    
      Custom Session
    

    
      SOCKS5
    
  

  
    ```bash
    # No country specified, falls back to US, rotating IP
    user:password@proxy.mrscraper.com:10000
    ```
  

  
    ```bash
    # US proxy with 10-minute session
    user-country-us-sessid-a1:password@proxy.mrscraper.com:10000
    ```
  

  
    ```bash
    # GB rotating proxy
    user-country-gb:password@proxy.mrscraper.com:10000
    ```
  

  
    ```bash
    # 30-minute session in Japan
    user-country-jp-sessid-x9-sesstime-30:password@proxy.mrscraper.com:10000
    ```
  

  
    ```bash
    # Same username format, SOCKS5 scheme and port
    socks5://user-country-us:password@proxy.mrscraper.com:10001
    ```
  



  The snippets on this page and in most language guides use the HTTP endpoint on port `10000` because it needs no extra dependencies. SOCKS5 is fully supported on port `10001` with the `socks5://` prefix, and every username parameter works the same on both. Each language page has a **SOCKS5 Proxy Setup** section with the client-specific setup, and [Proxy Types](/docs/residential-proxy/configuration/proxy-types) compares the two protocols.


## Examples by Language & Framework


  } href="/docs/residential-proxy/examples/python">
    Examples with requests, BeautifulSoup, and asyncio
  

  } href="/docs/residential-proxy/examples/nodejs">
    Examples with axios, Puppeteer, and native http
  

  } href="/docs/residential-proxy/examples/curl">
    Command-line examples for quick testing
  

  } href="/docs/residential-proxy/examples/php">
    Examples with cURL and Guzzle
  

  } href="/docs/residential-proxy/examples/ruby">
    Examples with Net::HTTP and HTTParty
  

  } href="/docs/residential-proxy/examples/go">
    Examples with net/http package
  

  } href="/docs/residential-proxy/examples/playwright">
    Browser automation in JavaScript & Python
  

  } href="/docs/residential-proxy/examples/puppeteer">
    Headless Chrome automation with page authentication
  

  } href="/docs/residential-proxy/examples/selenium">
    Browser testing and scraping with selenium-wire
  

  } href="/docs/residential-proxy/examples/cloakbrowser">
    Chromium browser automation in JS & Python
  


## Best Practices

### 1. Error Handling

Always implement proper error handling and retry logic:

* Catch connection timeouts
* Handle proxy authentication failures
* Retry failed requests with exponential backoff

### 2. Session Management

* Use unique session IDs for parallel operations
* Set appropriate session durations (10-60 minutes)
* Reuse sessions for related requests

### 3. Rate Limiting

* Respect target website's rate limits
* Use rotating proxies for high-volume scraping
* Add delays between requests when appropriate

### 4. Security

* Never hardcode credentials in source code
* Use environment variables for sensitive data
* Rotate credentials regularly

## Common Use Cases


  }>
    Use rotating proxies with country targeting to scrape large amounts of data while avoiding rate limits and IP bans.
  

  }>
    Use static proxies with custom session duration to maintain consistent IPs for login flows and account operations.
  

  }>
    Use static or rotating proxies with country targeting to view localized content and pricing across different regions.
  

  }>
    Use static proxies with various country codes to test your application's behavior across different IP addresses and geographic locations.
  



# PHP

Practical PHP examples for integrating Residential Proxy into your web scraping projects.

## SOCKS5 Proxy Setup

Every example on this page uses the HTTP endpoint on port `10000`. SOCKS5 is supported too, on port `10001` — set the proxy type explicitly with cURL, or use the `socks5h://` scheme with Guzzle:

```php
get('https://api.ipify.org', [
    'proxy' => 'socks5h://user-country-us:pass123@proxy.mrscraper.com:10001',
]);
?>
```

Use port `10001` for SOCKS5. `CURLPROXY_SOCKS5_HOSTNAME` and `socks5h://` resolve hostnames through the proxy; `CURLPROXY_SOCKS5` and `socks5://` resolve them locally. Username parameters such as `-country-` and `-sessid-` behave identically on both protocols.

## 1. Basic cURL Request

Demonstrates the fundamental cURL setup for using Residential Proxy with a static session. This example shows essential cURL options for proxy configuration and basic error handling.

```php

```

**Use Case:** Simple proxy testing and basic scraping tasks where you need a consistent IP address. Perfect for beginners learning PHP proxy integration.

## 2. Rotating Proxy for Scraping

Shows how to create a reusable scraping function with rotating proxies. Each request gets a fresh IP address, and the function returns structured data for easy processing.

```php
 $httpCode,
        'html' => $html
    ];
}

// Scrape multiple pages
$urls = [
    'https://example.com/page1',
    'https://example.com/page2',
    'https://example.com/page3'
];

foreach ($urls as $url) {
    $result = scrapeWithProxy($url);
    echo "Scraped {$url}: Status {$result['status']}\n";
}
?>
```

**Use Case:** High-volume scraping where IP diversity helps avoid detection and rate limiting. Ideal for scraping product catalogs, news sites, or job boards.

## 3. Multiple Sessions

Implements an object-oriented approach with multiple static proxy sessions. The ProxyScraper class encapsulates proxy configuration and provides clean session management.

```php
username = $username;
        $this->password = $password;
    }
    
    public function scrape($url) {
        $ch = curl_init($url);
        curl_setopt($ch, CURLOPT_PROXY, $this->proxy);
        curl_setopt($ch, CURLOPT_PROXYUSERPWD, "{$this->username}:{$this->password}");
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($ch, CURLOPT_TIMEOUT, 30);
        
        $response = curl_exec($ch);
        $info = curl_getinfo($ch);
        curl_close($ch);
        
        return [
            'status' => $info['http_code'],
            'content' => $response
        ];
    }
}

// Create multiple scrapers with different sessions
$scrapers = [];
for ($i = 1; $i <= 3; $i++) {
    $scrapers[] = new ProxyScraper("user-country-gb-sessid-worker{$i}", 'pass123');
}

// Use different sessions
$urls = ['https://example.com/page1', 'https://example.com/page2', 'https://example.com/page3'];
foreach ($urls as $index => $url) {
    $scraper = $scrapers[$index % count($scrapers)];
    $result = $scraper->scrape($url);
    echo "Scraped {$url}: {$result['status']}\n";
}
?>
```

**Use Case:** Organized scraping operations that need multiple consistent identities. Great for maintaining separate sessions for different data sources or user profiles.

## 4. Error Handling & Retries

Implements robust retry logic with exponential backoff to handle network failures gracefully. This pattern is essential for production scraping applications.

```php
= 200 && $httpCode < 300) {
            return $response;
        }
        
        echo "Attempt {$attempt} returned HTTP {$httpCode}\n";
        sleep(pow(2, $attempt - 1));
    }
    
    return false;
}

$result = scrapeWithRetry('https://example.com');
if ($result) {
    echo "Success!\n";
} else {
    echo "Failed after retries\n";
}
?>
```

**Use Case:** Production-ready scraping that must handle temporary network issues, server errors, and proxy failures without losing data or crashing the application.

## 5. Guzzle HTTP Client

Demonstrates using the popular Guzzle library for more elegant HTTP requests. Guzzle provides better error handling and a more intuitive API compared to raw cURL.

```php
 'http://user-country-jp-sessid-guzzle1:pass123@proxy.mrscraper.com:10000',
    'timeout' => 30
]);

try {
    $response = $client->get('https://api.ipify.org');
    echo 'Your IP: ' . $response->getBody();
} catch (RequestException $e) {
    echo 'Error: ' . $e->getMessage();
}
?>
```

**Use Case:** Modern PHP applications that prefer object-oriented HTTP clients with better exception handling and cleaner syntax than raw cURL.

## 6. Async Requests with Guzzle

Shows how to make concurrent HTTP requests using Guzzle's promise-based async functionality. This dramatically improves performance for bulk scraping operations.

```php
 'http://user-country-au:pass123@proxy.mrscraper.com:10000',
    'timeout' => 30
]);

// Create array of promises
$promises = [
    'page1' => $client->getAsync('https://example.com/page1'),
    'page2' => $client->getAsync('https://example.com/page2'),
    'page3' => $client->getAsync('https://example.com/page3'),
];

// Wait for all requests to complete
$results = Promise\Utils::settle($promises)->wait();

// Process results
foreach ($results as $key => $result) {
    if ($result['state'] === 'fulfilled') {
        echo "{$key}: Success\n";
    } else {
        echo "{$key}: Failed - {$result['reason']}\n";
    }
}
?>
```

**Use Case:** High-performance scraping that needs maximum throughput. Perfect for bulk data collection where you need to process many URLs simultaneously.

## 7. Environment Variables

Demonstrates secure credential management by loading proxy credentials from environment variables instead of hardcoding them in your source code.

```php

```

**Use Case:** Production deployments where credentials must be kept secure and separate from code. Essential for team development and CI/CD pipelines.


# Playwright

Practical Playwright examples for integrating Residential Proxy into browser automation and web scraping projects, separated into JavaScript (Node.js) and Python sections.

## JavaScript (Node.js)

### Prerequisites

```bash
npm install playwright proxy-chain
npx playwright install chromium
```

* `playwright`: Node.js library for browser automation supporting Chromium, Firefox, and WebKit with native proxy authentication.

### Proxy Format

Playwright accepts proxy settings via the `proxy` launch option in JavaScript:

```javascript
const proxy = {
  server: 'http://proxy.mrscraper.com:10000',
  username: 'user-country-us-sessid-session1',
  password: 'pass123',
}
```

### SOCKS5 Proxy Setup

For SOCKS5 proxies, use `proxy-chain` to anonymize the authenticated SOCKS5 connection before passing the local proxy server to Playwright:

```javascript
import { chromium } from 'playwright'
import { anonymizeProxy, closeAnonymizedProxy } from 'proxy-chain'

const username = 'user-country-us'
const password = 'pass123'

const socks5Proxy =
  `socks5://${encodeURIComponent(username)}:${encodeURIComponent(password)}` +
  `@proxy.mrscraper.com:10001`

const localProxy = await anonymizeProxy(socks5Proxy)

const browser = await chromium.launch({
  proxy: {
    server: localProxy,
  },
})

try {
  const page = await browser.newPage()

  await page.goto('https://ipinfo.io/json')

  console.log(await page.textContent('body'))
} finally {
  await browser.close()
  await closeAnonymizedProxy(localProxy, true)
}
```

Use port `10001` for SOCKS5. Do not pass SOCKS5 credentials directly to Playwright; `proxy-chain` handles the authenticated upstream connection.

### 1. Basic Proxy Setup

Demonstrates launching Headless Chromium with authenticated proxy credentials in JavaScript, inspecting the resulting external proxy IP.

```javascript
import { chromium } from 'playwright'

async function testProxy() {
  // Launch browser with proxy settings
  const browser = await chromium.launch({
    headless: true,
    proxy: {
      server: 'http://proxy.mrscraper.com:10000',
      username: 'user-country-us',
      password: 'pass123',
    },
  })

  const page = await browser.newPage()

  // Navigate to IP inspection endpoint
  await page.goto('https://api.ipify.org?format=json')
  const content = await page.textContent('body')
  console.log('Your Proxy IP:', content)

  await browser.close()
}

testProxy().catch(console.error)
```

**Use Case:** Fundamental proxy connectivity check and baseline browser launch in Node.js.

### 2. Multi-Context Sticky Sessions

Shows how to create isolated browser contexts on a single browser instance. Each context maintains its own consistent IP address using unique session IDs (`sessid`).

```javascript
import { chromium } from 'playwright'

async function runParallelSessions() {
  const browser = await chromium.launch({ headless: true })
  const workerSessions = ['worker1', 'worker2', 'worker3']

  // Create isolated context per session
  for (const session of workerSessions) {
    const context = await browser.newContext({
      proxy: {
        server: 'http://proxy.mrscraper.com:10000',
        username: `user-country-us-sessid-${session}`,
        password: 'pass123',
      },
    })

    const page = await context.newPage()
    await page.goto('https://api.ipify.org?format=json')
    const ipData = await page.textContent('body')

    console.log(`Session [${session}]:`, ipData)
    await context.close()
  }

  await browser.close()
}

runParallelSessions().catch(console.error)
```

**Use Case:** Parallel web scraping where multiple workers maintain independent sticky sessions on a single browser process.

### 3. Geotargeting & Custom Session Duration

Demonstrates routing Playwright traffic through a German proxy with a 30-minute static session duration (`sesstime-30`), configuring regional locale settings.

```javascript
import { chromium } from 'playwright'

async function geotargetedSession() {
  const browser = await chromium.launch({
    headless: true,
    proxy: {
      server: 'http://proxy.mrscraper.com:10000',
      // German IP with 30-minute static duration
      username: 'user-country-de-sessid-shop1-sesstime-30',
      password: 'pass123',
    },
  })

  const context = await browser.newContext({
    userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    locale: 'de-DE',
  })

  const page = await context.newPage()
  await page.goto('https://httpbin.org/headers')
  console.log(await page.textContent('body'))

  await browser.close()
}

geotargetedSession().catch(console.error)
```

**Use Case:** Testing localized e-commerce pricing, geo-restricted page content, and multi-step checkouts requiring static IPs.

### 4. Production Error Handling

Production-ready error handling pattern for JavaScript Playwright operations, featuring timeout settings, response checks, and guaranteed resource cleanup.

```javascript
import { chromium } from 'playwright'

async function robustScraping(targetUrl) {
  let browser = null
  try {
    browser = await chromium.launch({
      headless: true,
      proxy: {
        server: 'http://proxy.mrscraper.com:10000',
        username: 'user-country-us-sessid-task1',
        password: 'pass123',
      },
    })

    const page = await browser.newPage()
    page.setDefaultTimeout(30000) // 30s timeout

    const response = await page.goto(targetUrl, { waitUntil: 'networkidle' })
    if (!response || !response.ok()) {
      throw new Error(`Failed to load page: ${response?.status()}`)
    }

    console.log('Page loaded successfully:', await page.title())
  } catch (error) {
    console.error(`Scraping failed for ${targetUrl}:`, error.message)
  } finally {
    if (browser) {
      await browser.close()
    }
  }
}

robustScraping('https://example.com')
```

**Use Case:** Deploying enterprise JavaScript Playwright scrapers that handle timeouts and failures gracefully.

***

## Python

### Prerequisites

```bash
pip install playwright
playwright install chromium
```

* `playwright`: Python package offering synchronous and asynchronous browser automation bindings.
* Install `sing-box` separately and make sure the `sing-box` command is available on your `PATH` for SOCKS5 proxy bridging.

### Proxy Format

Playwright for Python passes proxy options via a dictionary to `chromium.launch()`:

```python
proxy = {
    "server": "http://proxy.mrscraper.com:10000",
    "username": "user-country-us-sessid-session1",
    "password": "pass123"
}
```

### SOCKS5 Proxy Setup

Playwright's Python integration uses `sing-box` to expose the authenticated SOCKS5 proxy as a local HTTP proxy:

```python
import json
import os
import subprocess
import tempfile
import time

from playwright.sync_api import sync_playwright

username = "user-country-us"
password = "pass123"

local_proxy = "http://127.0.0.1:8080"

singbox_config = {
    "inbounds": [
        {
            "type": "http",
            "tag": "http-in",
            "listen": "127.0.0.1",
            "listen_port": 8080,
        }
    ],
    "outbounds": [
        {
            "type": "socks",
            "tag": "mrscraper",
            "server": "proxy.mrscraper.com",
            "server_port": 10001,
            "username": username,
            "password": password,
            "version": "5",
        }
    ],
    "route": {
        "rules": [
            {
                "action": "route",
                "outbound": "mrscraper",
            }
        ]
    },
}

with tempfile.NamedTemporaryFile(
    mode="w",
    suffix=".json",
    delete=False,
) as config_file:
    json.dump(singbox_config, config_file)
    config_path = config_file.name

print("Starting sing-box...")

singbox_process = subprocess.Popen([
    "sing-box",
    "run",
    "-c",
    config_path,
])

try:
    time.sleep(1)

    with sync_playwright() as playwright:
        browser = playwright.chromium.launch(
            proxy={
                "server": local_proxy,
            }
        )

        try:
            page = browser.new_page()
            page.goto(
                "https://ipinfo.io/json",
                wait_until="domcontentloaded",
                timeout=30000,
            )
            print(page.text_content("body"))
        finally:
            browser.close()
finally:
    singbox_process.terminate()
    singbox_process.wait()
    os.remove(config_path)
```

Use port `10001` for SOCKS5. The browser connects to the local HTTP proxy at `127.0.0.1:8080`; `sing-box` handles the authenticated SOCKS5 upstream.

### 1. Basic Proxy Setup (Sync API)

Demonstrates the Python synchronous Playwright API for initializing an authenticated proxy connection and inspecting page output.

```python
from playwright.sync_api import sync_playwright

def test_proxy():
    with sync_playwright() as p:
        # Launch browser with proxy server and credentials
        browser = p.chromium.launch(
            headless=True,
            proxy={
                "server": "http://proxy.mrscraper.com:10000",
                "username": "user-country-us",
                "password": "pass123"
            }
        )

        page = browser.new_page()
        page.goto("https://api.ipify.org?format=json")

        print("Your Proxy IP:", page.text_content("body"))
        browser.close()

if __name__ == "__main__":
    test_proxy()
```

**Use Case:** Quick proxy integration for Python automation scripts, test suites, and data collection utilities.

### 2. Sticky Session Management (Sync API)

Maintains a static Python Playwright browser session with a stable IP address across multi-step page navigations.

```python
from playwright.sync_api import sync_playwright

def run_sticky_session():
    with sync_playwright() as p:
        # Static session IP in Canada
        browser = p.chromium.launch(
            headless=True,
            proxy={
                "server": "http://proxy.mrscraper.com:10000",
                "username": "user-country-ca-sessid-auth1-sesstime-20",
                "password": "pass123"
            }
        )

        context = browser.new_context()
        page = context.new_page()

        for url in ["https://example.com/step1", "https://example.com/step2"]:
            page.goto(url)
            print(f"Visited {url} -> Title: {page.title()}")

        browser.close()

if __name__ == "__main__":
    run_sticky_session()
```

**Use Case:** Simulating multi-step user interactions and login sequences in Python requiring a consistent residential IP.

### 3. Rotating Proxy Scraping (Async API)

Uses an asynchronous Python Playwright setup with rotating proxies to visit multiple target URLs, ensuring each request receives a fresh IP address.

```python
import asyncio
from playwright.async_api import async_playwright

async def scrape_target(url, proxy_config):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True, proxy=proxy_config)
        page = await browser.new_page()

        try:
            await page.goto(url, wait_until="domcontentloaded", timeout=30000)
            title = await page.title()
            print(f"Scraped '{url}' -> Title: {title}")
        except Exception as e:
            print(f"Error scraping {url}: {e}")
        finally:
            await browser.close()

async def main():
    # Rotating proxy (no sessid parameter)
    proxy_config = {
        "server": "http://proxy.mrscraper.com:10000",
        "username": "user-country-gb",
        "password": "pass123"
    }

    urls = [
        "https://example.com/page1",
        "https://example.com/page2",
        "https://example.com/page3",
    ]

    for url in urls:
        await scrape_target(url, proxy_config)

if __name__ == "__main__":
    asyncio.run(main())
```

**Use Case:** High-volume automated Python browser scraping where rotating IPs prevent target site rate limits.

### 4. Production Error Handling & Async Teardown

Implements defensive Python async error trapping, default page timeout limits, and safe browser closing routines.

```python
import asyncio
from playwright.async_api import async_playwright

async def safe_scrape(url):
    proxy_config = {
        "server": "http://proxy.mrscraper.com:10000",
        "username": "user-country-us-sessid-prod1",
        "password": "pass123"
    }

    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True, proxy=proxy_config)
        try:
            page = await browser.new_page()
            page.set_default_timeout(30000)

            response = await page.goto(url, wait_until="networkidle")
            if not response or not response.ok:
                raise RuntimeError(f"HTTP error {response.status if response else 'No response'}")

            print(f"Successfully scraped {url} - Length: {len(await page.content())}")
        except Exception as err:
            print(f"Failed to scrape {url}: {err}")
        finally:
            await browser.close()

if __name__ == "__main__":
    asyncio.run(safe_scrape("https://example.com"))
```

**Use Case:** Enterprise Python async web scrapers operating in production environments with strict reliability requirements.


  * Define proxy credentials when calling `chromium.launch()` or `browser.newContext()` / `browser.new_context()`.
  * Use browser contexts to handle multiple proxy configurations within a single browser process.
  * Keep sensitive proxy credentials in environment variables (`process.env.PROXY_PASSWORD` or `os.environ.get("PROXY_PASSWORD")`).



# Puppeteer

Practical Puppeteer code examples for routing browser automation traffic through Residential Proxies using `page.authenticate()`.

## Prerequisites

```bash
npm install puppeteer proxy-chain
```

* `puppeteer`: Node.js library providing a high-level API to control Headless Chrome or Chromium over the DevTools Protocol.

## Proxy Format

Puppeteer passes the proxy server address via Chromium launch flags (`--proxy-server`), then authenticates using `page.authenticate()`.

```text
username:password@proxy.mrscraper.com:10000
```

For sticky sessions or targeted countries, format the username with parameters:

```text
user-country-us-sessid-session1:password@proxy.mrscraper.com:10000
```

### SOCKS5 Proxy Setup

For SOCKS5 proxies, use `proxy-chain` to anonymize the authenticated SOCKS5 connection before passing the local proxy server to Puppeteer:

```javascript
import puppeteer from 'puppeteer'
import { anonymizeProxy, closeAnonymizedProxy } from 'proxy-chain'

const username = 'user-country-us'
const password = 'pass123'

const socks5Proxy =
  `socks5://${encodeURIComponent(username)}:${encodeURIComponent(password)}` +
  `@proxy.mrscraper.com:10001`

const localProxy = await anonymizeProxy(socks5Proxy)

const browser = await puppeteer.launch({
  args: [`--proxy-server=${localProxy}`],
})

try {
  const page = await browser.newPage()

  await page.goto('https://ipinfo.io/json', {
    waitUntil: 'networkidle2',
  })

  console.log(await page.$eval('body', (el) => el.innerText))
} finally {
  await browser.close()
  await closeAnonymizedProxy(localProxy, true)
}
```

Use port `10001` for SOCKS5. Always close the browser before calling `closeAnonymizedProxy()`.

## 1. Basic Proxy Setup

Demonstrates the standard setup for launching Headless Chrome with proxy flags and authenticating page traffic.

```javascript
import puppeteer from 'puppeteer'

const proxyHost = 'proxy.mrscraper.com'
const proxyPort = '10000'
const proxyUsername = 'user-country-us'
const proxyPassword = 'pass123'

async function testProxy() {
  const browser = await puppeteer.launch({
    headless: true,
    args: [`--proxy-server=http://${proxyHost}:${proxyPort}`],
  })

  const page = await browser.newPage()

  // Authenticate page before navigating
  await page.authenticate({
    username: proxyUsername,
    password: proxyPassword,
  })

  await page.goto('https://api.ipify.org?format=json', {
    waitUntil: 'networkidle2',
  })
  const content = await page.evaluate(() => document.body.innerText)
  console.log('Your Proxy IP:', content)

  await browser.close()
}

testProxy().catch(console.error)
```

**Use Case:** Quick proxy connectivity check and baseline authentication testing in Puppeteer.

## 2. Sticky Session Management

Maintains a stable residential IP address across multi-step browser workflows using a consistent session ID (`sessid`) and explicit session duration (`sesstime-20`).

```javascript
import puppeteer from 'puppeteer'

async function runStickySession() {
  const browser = await puppeteer.launch({
    headless: true,
    args: ['--proxy-server=http://proxy.mrscraper.com:10000'],
  })

  const page = await browser.newPage()

  // 20-minute static session in Canada
  await page.authenticate({
    username: 'user-country-ca-sessid-checkout1-sesstime-20',
    password: 'pass123',
  })

  const urls = [
    'https://example.com/step1',
    'https://example.com/step2',
    'https://example.com/step3',
  ]

  for (const url of urls) {
    await page.goto(url, { waitUntil: 'domcontentloaded' })
    console.log(`Visited ${url} - Title:`, await page.title())
  }

  await browser.close()
}

runStickySession().catch(console.error)
```

**Use Case:** Form submissions, login workflows, and multi-page checkouts where maintaining the same IP address is mandatory.

## 3. Rotating Proxy Scraping

Demonstrates rotating proxies with Puppeteer. Each request uses rotating proxy credentials to automatically fetch content with a different residential IP address.

```javascript
import puppeteer from 'puppeteer'

async function scrapeWithRotation(urls) {
  const browser = await puppeteer.launch({
    headless: true,
    args: ['--proxy-server=http://proxy.mrscraper.com:10000'],
  })

  for (const url of urls) {
    const page = await browser.newPage()

    // Rotating proxy - new IP per request
    await page.authenticate({
      username: 'user-country-gb',
      password: 'pass123',
    })

    try {
      await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 })
      const h1Text = await page
        .$eval('h1', (el) => el.textContent)
        .catch(() => 'No H1')
      console.log(`Scraped [${url}]: ${h1Text}`)
    } catch (err) {
      console.error(`Failed to scrape ${url}:`, err.message)
    } finally {
      await page.close()
    }
  }

  await browser.close()
}

scrapeWithRotation([
  'https://example.com/page1',
  'https://example.com/page2',
  'https://example.com/page3',
])
```

**Use Case:** High-volume scraping across multiple target pages without triggering IP rate limits or blocklists.

## 4. Geotargeting & Location Testing

Demonstrates routing Puppeteer traffic through specific countries (Japan) while customizing HTTP headers and viewport to match target geographic personas.

```javascript
import puppeteer from 'puppeteer'

async function testLocalizedContent() {
  const browser = await puppeteer.launch({
    headless: true,
    args: ['--proxy-server=http://proxy.mrscraper.com:10000', '--lang=ja-JP'],
  })

  const page = await browser.newPage()

  // Target Japanese residential proxy
  await page.authenticate({
    username: 'user-country-jp',
    password: 'pass123',
  })

  await page.setExtraHTTPHeaders({
    'Accept-Language': 'ja-JP,ja;q=0.9',
  })

  await page.goto('https://httpbin.org/headers')
  console.log(await page.evaluate(() => document.body.innerText))

  await browser.close()
}

testLocalizedContent().catch(console.error)
```

**Use Case:** Auditing localized e-commerce pricing, geo-specific ad placement, and regional content availability.

## 5. Parallel Isolated Contexts

Uses Incognito browser contexts (`createBrowserContext()`) to isolate multiple concurrent worker sessions, each authenticated with a unique proxy session ID.

```javascript
import puppeteer from 'puppeteer'

async function parallelWorkers() {
  const browser = await puppeteer.launch({
    headless: true,
    args: ['--proxy-server=http://proxy.mrscraper.com:10000'],
  })

  const workers = ['worker_a', 'worker_b', 'worker_c']

  const tasks = workers.map(async (workerId) => {
    // Isolated incognito context
    const context = await browser.createBrowserContext()
    const page = await context.newPage()

    await page.authenticate({
      username: `user-country-us-sessid-${workerId}`,
      password: 'pass123',
    })

    await page.goto('https://api.ipify.org?format=json')
    const ip = await page.evaluate(() => document.body.innerText)
    console.log(`Worker ${workerId} IP:`, ip)

    await context.close()
  })

  await Promise.all(tasks)
  await browser.close()
}

parallelWorkers().catch(console.error)
```

**Use Case:** Scaling automated browser tasks in parallel with dedicated IP addresses per thread without launching multiple browser instances.

## 6. Production Error Handling & Cleanup

Provides robust exception handling, custom navigation timeouts, response validation, and safe browser shutdown routines.

```javascript
import puppeteer from 'puppeteer'

async function safeScrape(url) {
  let browser = null
  try {
    browser = await puppeteer.launch({
      headless: true,
      args: ['--proxy-server=http://proxy.mrscraper.com:10000'],
    })

    const page = await browser.newPage()
    page.setDefaultNavigationTimeout(30000)

    await page.authenticate({
      username: 'user-country-us-sessid-prod1',
      password: 'pass123',
    })

    const response = await page.goto(url, { waitUntil: 'networkidle2' })

    if (!response || !response.ok()) {
      throw new Error(`HTTP Error ${response?.status()}`)
    }

    console.log('Successfully fetched:', await page.title())
  } catch (error) {
    console.error('Scraping error:', error.message)
  } finally {
    if (browser) {
      await browser.close()
    }
  }
}

safeScrape('https://example.com')
```

**Use Case:** Deploying production Puppeteer bots that handle network glitches, authentication errors, and timeouts gracefully.


  * Call `page.authenticate()` before invoking `page.goto()`.
  * Use `--proxy-server=http://...` in `launch.args` to direct Puppeteer traffic.
  * Use `createBrowserContext()` for parallel isolated sessions with unique proxy usernames.



# Python

Practical Python examples for integrating Residential Proxy into your scraping and automation projects.

## Prerequisites

```bash
pip install requests beautifulsoup4
```

* `requests` : Human-friendly HTTP client used to send requests through Residential Proxy, control timeouts, and access response data.
* `beautifulsoup4` : HTML parser used together with `requests` to turn raw HTML into a searchable object model for extracting elements like titles, links, and product data.

## SOCKS5 Proxy Setup

Every example on this page uses the HTTP endpoint on port `10000`. SOCKS5 is supported too, on port `10001`. `requests` routes SOCKS traffic only when the `socks` extra is installed:

```bash
pip install "requests[socks]"
```

```python
import requests

# Same credentials, SOCKS5 scheme and port
proxy = {
    'http': 'socks5h://user-country-us-sessid-session1:pass123@proxy.mrscraper.com:10001',
    'https': 'socks5h://user-country-us-sessid-session1:pass123@proxy.mrscraper.com:10001'
}

response = requests.get('https://api.ipify.org', proxies=proxy)
print(f"Your IP: {response.text}")
```

Use port `10001` for SOCKS5. Prefer `socks5h://` so hostnames are resolved by the proxy instead of locally. Username parameters such as `-country-` and `-sessid-` behave identically on both protocols.

## 1. Basic Static Proxy

This example demonstrates how to use a static proxy with a consistent session. The same IP address will be maintained throughout your requests as long as you use the same session ID.

```python
import requests

# Static proxy with session
proxy = {
    'http': 'http://user-country-us-sessid-session1:pass123@proxy.mrscraper.com:10000',
    'https': 'http://user-country-us-sessid-session1:pass123@proxy.mrscraper.com:10000'
}

# Make a request
response = requests.get('https://api.ipify.org', proxies=proxy)
print(f"Your IP: {response.text}")
```

**Use Case:** Ideal for scenarios where you need to maintain the same IP across multiple requests, such as logging into websites or maintaining session state.

## 2. Rotating Proxy for Web Scraping

Rotating proxies automatically change your IP address with each request. This is perfect for scraping multiple pages without being detected or rate-limited.

```python
import requests
from bs4 import BeautifulSoup

# Rotating proxy - new IP each request
proxy = {
    'http': 'http://user-country-us:pass123@proxy.mrscraper.com:10000',
    'https': 'http://user-country-us:pass123@proxy.mrscraper.com:10000'
}

def scrape_page(url):
    """Scrape a page using rotating proxy"""
    try:
        response = requests.get(url, proxies=proxy, timeout=30)
        soup = BeautifulSoup(response.text, 'html.parser')
        
        # Extract data
        title = soup.find('h1').text if soup.find('h1') else 'No title'
        print(f"Scraped: {title}")
        
        return soup
    except Exception as e:
        print(f"Error: {e}")
        return None

# Scrape multiple pages (each with different IP)
urls = [
    'https://example.com/page1',
    'https://example.com/page2',
    'https://example.com/page3',
]

for url in urls:
    scrape_page(url)
```

**Use Case:** Best for high-volume scraping where you need to avoid IP-based rate limits or blocks. Each page appears to come from a different visitor.

## 3. Multiple Static Sessions (Parallel)

This advanced example shows how to scrape multiple pages simultaneously using different static proxy sessions. Each worker thread maintains its own consistent IP address.

```python
import requests
from concurrent.futures import ThreadPoolExecutor, as_completed

# Create multiple session-based proxies
proxies = [
    {
        'http': f'http://user-country-us-sessid-worker{i}:pass123@proxy.mrscraper.com:10000',
        'https': f'http://user-country-us-sessid-worker{i}:pass123@proxy.mrscraper.com:10000'
    }
    for i in range(1, 6)  # 5 different sessions
]

def scrape_with_proxy(proxy_config, url):
    """Scrape using a specific proxy session"""
    try:
        response = requests.get(url, proxies=proxy_config, timeout=30)
        return {
            'status': response.status_code,
            'length': len(response.text),
            'url': url
        }
    except Exception as e:
        return {'error': str(e), 'url': url}

# URLs to scrape
urls = [f'https://example.com/page{i}' for i in range(1, 11)]

# Scrape in parallel using different sessions
with ThreadPoolExecutor(max_workers=5) as executor:
    futures = []
    for url in urls:
        # Rotate through available proxies
        proxy = proxies[len(futures) % len(proxies)]
        future = executor.submit(scrape_with_proxy, proxy, url)
        futures.append(future)
    
    # Collect results
    for future in as_completed(futures):
        result = future.result()
        print(f"Result: {result}")
```

**Use Case:** Perfect for high-performance scraping where you need to make many requests quickly while maintaining multiple consistent identities. Great for scraping sites with per-IP rate limits.

## 4. Session with Custom Duration

Control how long your proxy session remains active using the `sesstime` parameter. This is useful for extended scraping operations where you need to maintain the same IP for a specific period.

```python
import requests
import time

# 20-minute session for extended scraping
long_session_proxy = {
    'http': 'http://user-country-ca-sessid-long1-sesstime-20:pass123@proxy.mrscraper.com:10000',
    'https': 'http://user-country-ca-sessid-long1-sesstime-20:pass123@proxy.mrscraper.com:10000'
}

def long_scraping_session(urls):
    """Scrape multiple pages with same IP over 20 minutes"""
    print("Starting long scraping session...")
    
    for i, url in enumerate(urls):
        try:
            response = requests.get(url, proxies=long_session_proxy, timeout=30)
            print(f"[{i+1}/{len(urls)}] Scraped {url}: {response.status_code}")
            
            # Verify IP stays the same
            ip = requests.get('https://api.ipify.org', proxies=long_session_proxy).text
            print(f"  Current IP: {ip}")
            
            # Wait between requests
            time.sleep(60)  # 1 minute between requests
            
        except Exception as e:
            print(f"Error on {url}: {e}")

# Scrape 15 pages over 15 minutes (within 20-minute session)
urls = [f'https://example.com/page{i}' for i in range(1, 16)]
long_scraping_session(urls)
```

**Use Case:** Ideal for scraping workflows that require extended session times, such as crawling paginated content, navigating multi-step forms, or monitoring a site over time without IP changes.

## 5. Error Handling & Retries

Production-ready scraping requires robust error handling. This example implements automatic retries with exponential backoff to handle network issues, timeouts, and proxy errors gracefully.

```python
import requests
from time import sleep

proxy = {
    'http': 'http://user-country-gb:pass123@proxy.mrscraper.com:10000',
    'https': 'http://user-country-gb:pass123@proxy.mrscraper.com:10000'
}

def scrape_with_retry(url, max_retries=3):
    """Scrape with automatic retry on failure"""
    for attempt in range(max_retries):
        try:
            response = requests.get(url, proxies=proxy, timeout=30)
            response.raise_for_status()  # Raise exception for bad status codes
            return response.text
            
        except requests.exceptions.ProxyError:
            print(f"Proxy error on attempt {attempt + 1}")
            sleep(2 ** attempt)  # Exponential backoff
            
        except requests.exceptions.Timeout:
            print(f"Timeout on attempt {attempt + 1}")
            sleep(2 ** attempt)
            
        except Exception as e:
            print(f"Error on attempt {attempt + 1}: {e}")
            sleep(2 ** attempt)
    
    print(f"Failed after {max_retries} attempts")
    return None

# Use with retry logic
result = scrape_with_retry('https://example.com')
```

**Use Case:** Essential for reliable scraping operations. The exponential backoff prevents overwhelming servers while giving temporary issues time to resolve. Always use retry logic in production environments.


# Ruby

Practical Ruby examples for integrating Residential Proxy into your scraping projects.

## Prerequisites

```bash
gem install httparty
```

* **HTTParty**: Popular Ruby gem that provides a simple, elegant API for HTTP requests with built-in proxy configuration.

## SOCKS5 Proxy Setup

Every example on this page uses the HTTP endpoint on port `10000`. SOCKS5 is supported too, on port `10001`. `Net::HTTP` has no native SOCKS support, so it needs the `socksify` gem:

```bash
gem install socksify
```

```ruby
require 'socksify/http'
require 'uri'

# Same credentials, SOCKS5 port
TCPSocket.socks_username = 'user-country-us'
TCPSocket.socks_password = 'pass123'

uri = URI.parse('https://api.ipify.org')

response = Net::HTTP.SOCKSProxy('proxy.mrscraper.com', 10001)
                   .start(uri.host, uri.port, use_ssl: true) do |http|
  http.request(Net::HTTP::Get.new(uri.request_uri))
end

puts "Your IP: #{response.body}"
```

Use port `10001` for SOCKS5. `socksify` sends the hostname to the proxy for resolution. Username parameters such as `-country-` and `-sessid-` behave identically on both protocols.

## 1. Basic HTTP Request

Demonstrates the fundamental setup using Ruby's built-in Net::HTTP library with a static proxy session. This example shows the essential proxy configuration parameters.

```ruby
require 'net/http'
require 'uri'

# Static proxy configuration
proxy_uri = URI.parse('http://user-country-us-sessid-ruby1:pass123@proxy.mrscraper.com:10000')

uri = URI.parse('https://api.ipify.org')

http = Net::HTTP.new(uri.host, uri.port, 
                     proxy_uri.host, proxy_uri.port, 
                     proxy_uri.user, proxy_uri.password)
http.use_ssl = true

request = Net::HTTP::Get.new(uri.request_uri)
response = http.request(request)

puts "Your IP: #{response.body}"
```

**Use Case:** Simple proxy testing and basic scraping tasks where you need a consistent IP address. Perfect for understanding Ruby's proxy configuration fundamentals.

## 2. Rotating Proxy with HTTParty

Shows how to use the HTTParty gem for cleaner proxy configuration with rotating IPs. The class-based approach provides reusable scraping functionality with automatic IP rotation.

```ruby
require 'httparty'

class ProxyScraper
  include HTTParty
  
  # Rotating proxy
  http_proxy 'proxy.mrscraper.com', 10000, 'user-country-jp', 'pass123'
  
  def self.scrape(url)
    response = get(url, timeout: 30)
    {
      status: response.code,
      body: response.body
    }
  rescue => e
    { error: e.message }
  end
end

# Scrape multiple pages
urls = [
  'https://example.com/page1',
  'https://example.com/page2',
  'https://example.com/page3'
]

urls.each do |url|
  result = ProxyScraper.scrape(url)
  puts "Scraped #{url}: #{result[:status]}"
end
```

**Use Case:** High-volume scraping where IP diversity helps avoid detection and rate limiting. HTTParty's clean syntax makes it ideal for production scraping applications.

## 3. Multiple Sessions

Implements an object-oriented approach with multiple static proxy sessions. Each SessionScraper instance maintains its own consistent IP address throughout its lifetime.

```ruby
require 'net/http'
require 'uri'

class SessionScraper
  def initialize(session_id, password)
    @username = "user-country-gb-sessid-#{session_id}"
    @password = password
    @proxy_uri = URI.parse("http://#{@username}:#{@password}@proxy.mrscraper.com:10000")
  end
  
  def scrape(url)
    uri = URI.parse(url)
    
    http = Net::HTTP.new(uri.host, uri.port,
                        @proxy_uri.host, @proxy_uri.port,
                        @proxy_uri.user, @proxy_uri.password)
    http.use_ssl = true
    
    request = Net::HTTP::Get.new(uri.request_uri)
    response = http.request(request)
    
    {
      status: response.code,
      body_length: response.body.length
    }
  rescue => e
    { error: e.message }
  end
end

# Create multiple scrapers
scrapers = (1..3).map { |i| SessionScraper.new("worker#{i}", 'pass123') }

urls = [
  'https://example.com/page1',
  'https://example.com/page2',
  'https://example.com/page3'
]

urls.each_with_index do |url, index|
  scraper = scrapers[index % scrapers.length]
  result = scraper.scrape(url)
  puts "Scraped #{url}: #{result}"
end
```

**Use Case:** Organized scraping operations that need multiple consistent identities. Great for maintaining separate sessions for different data sources or simulating multiple users.

## 4. Error Handling & Retries

Implements robust retry logic with exponential backoff using HTTParty's proxy configuration options. Essential for production applications that need to handle failures gracefully.

```ruby
require 'httparty'

class RobustScraper
  include HTTParty
  
  def self.scrape_with_retry(url, max_retries = 3)
    proxy_config = {
      http_proxyaddr: 'proxy.mrscraper.com',
      http_proxyport: 10000,
      http_proxyuser: 'user-country-de',
      http_proxypass: 'pass123'
    }
    
    max_retries.times do |attempt|
      begin
        response = get(url, proxy_config.merge(timeout: 30))
        return response if response.success?
        
        puts "Attempt #{attempt + 1} failed with status #{response.code}"
        sleep(2 ** attempt)  # Exponential backoff
        
      rescue => e
        puts "Attempt #{attempt + 1} failed: #{e.message}"
        sleep(2 ** attempt)
      end
    end
    
    nil
  end
end

result = RobustScraper.scrape_with_retry('https://example.com')
puts result ? "Success" : "Failed after retries"
```

**Use Case:** Production-ready scraping that must handle temporary network issues, server errors, and proxy failures without losing data or crashing the application.

## 5. Concurrent Scraping

Demonstrates parallel scraping using Ruby threads with different static proxy sessions. Each thread maintains its own consistent IP address for maximum performance.

```ruby
require 'net/http'
require 'uri'
require 'thread'

def scrape_with_proxy(url, session_id)
  proxy_uri = URI.parse("http://user-country-fr-sessid-#{session_id}:pass123@proxy.mrscraper.com:10000")
  uri = URI.parse(url)
  
  http = Net::HTTP.new(uri.host, uri.port,
                      proxy_uri.host, proxy_uri.port,
                      proxy_uri.user, proxy_uri.password)
  http.use_ssl = true
  
  request = Net::HTTP::Get.new(uri.request_uri)
  response = http.request(request)
  
  puts "Scraped #{url}: #{response.code}"
rescue => e
  puts "Error scraping #{url}: #{e.message}"
end

urls = [
  'https://example.com/page1',
  'https://example.com/page2',
  'https://example.com/page3',
  'https://example.com/page4',
  'https://example.com/page5'
]

threads = []
urls.each_with_index do |url, index|
  threads << Thread.new do
    scrape_with_proxy(url, "worker#{index + 1}")
  end
end

threads.each(&:join)
puts "All scraping completed"
```

**Use Case:** High-performance scraping that needs to process multiple URLs quickly while maintaining separate identities. Perfect for bulk data collection with time constraints.

## 6. Custom Headers

Shows how to add realistic browser headers when using HTTParty with proxies. This helps avoid detection by making requests appear more legitimate and browser-like.

```ruby
require 'httparty'

class CustomScraper
  include HTTParty
  
  http_proxy 'proxy.mrscraper.com', 10000, 'user-country-ca', 'pass123'
  
  def self.scrape_with_headers(url)
    headers = {
      'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)',
      'Accept-Language' => 'en-US,en;q=0.9'
    }
    
    response = get(url, headers: headers, timeout: 30)
    
    {
      status: response.code,
      content_type: response.headers['content-type'],
      body_length: response.body.length
    }
  rescue => e
    { error: e.message }
  end
end

result = CustomScraper.scrape_with_headers('https://example.com')
puts result
```

**Use Case:** Scraping sites that check for browser-like behavior or when you need to match headers to your proxy's geographic location for consistency and avoiding detection.


# Selenium

Practical Python examples for using Residential Proxy with Selenium for web scraping and automated browser testing.

## Prerequisites

```bash
pip install selenium selenium-wire-lw webdriver-manager
```

* `selenium`: Industry-standard browser automation framework for Python.
* `selenium-wire-lw`: Lightweight wrapper for Selenium that adds authenticated HTTP/HTTPS proxy support.
* `webdriver-manager`: Automated driver binary manager for Chrome, Firefox, and Edge.
* Install `sing-box` separately and make sure the `sing-box` command is available on your `PATH` for SOCKS5 proxy bridging.

## Proxy Format

Standard Selenium Chrome flags do not natively support proxy authentication passwords without extensions. `selenium-wire` simplifies proxy authentication using standard URL strings:

```text
http://username:password@proxy.mrscraper.com:10000
```

For sticky sessions or geotargeting, format the username with parameters:

```text
http://user-country-us-sessid-session1:password@proxy.mrscraper.com:10000
```

### SOCKS5 Proxy Setup

Standard Chrome proxy flags do not support authenticated SOCKS5 credentials directly. Use `sing-box` to expose the SOCKS5 proxy as a local HTTP proxy:

```python
import json
import os
import subprocess
import tempfile
import time

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

username = "user-country-us"
password = "pass123"

local_proxy = "http://127.0.0.1:8080"

singbox_config = {
    "inbounds": [
        {
            "type": "http",
            "tag": "http-in",
            "listen": "127.0.0.1",
            "listen_port": 8080,
        }
    ],
    "outbounds": [
        {
            "type": "socks",
            "tag": "mrscraper",
            "server": "proxy.mrscraper.com",
            "server_port": 10001,
            "username": username,
            "password": password,
            "version": "5",
        }
    ],
    "route": {
        "rules": [
            {
                "action": "route",
                "outbound": "mrscraper",
            }
        ]
    },
}

with tempfile.NamedTemporaryFile(
    mode="w",
    suffix=".json",
    delete=False,
) as config_file:
    json.dump(singbox_config, config_file)
    config_path = config_file.name

print("Starting sing-box...")

singbox_process = subprocess.Popen([
    "sing-box",
    "run",
    "-c",
    config_path,
])

try:
    time.sleep(1)

    chrome_options = Options()
    chrome_options.add_argument(
        f"--proxy-server={local_proxy}"
    )

    driver = webdriver.Chrome(
        options=chrome_options
    )

    try:
        driver.set_page_load_timeout(30)
        driver.get("https://ipinfo.io/json")
        print(driver.find_element("tag name", "body").text)
    finally:
        driver.quit()
finally:
    singbox_process.terminate()
    singbox_process.wait()
    os.remove(config_path)
```

Use port `10001` for SOCKS5. Chrome connects to the local HTTP proxy at `127.0.0.1:8080`, while `sing-box` routes traffic to the authenticated SOCKS5 proxy.

## 1. Basic Proxy Setup (Selenium-Wire)

Demonstrates launching Selenium Chrome with an authenticated residential proxy using `selenium-wire`. It navigates to an IP lookup page and prints the verified proxy IP.

```python
from seleniumwire import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.service import Service

# Authenticated proxy URL
proxy_url = "http://user-country-us:pass123@proxy.mrscraper.com:10000"

seleniumwire_options = {
    "proxy": {
        "http": proxy_url,
        "https": proxy_url,
    }
}

# Initialize Chrome driver with proxy options
driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()),
    seleniumwire_options=seleniumwire_options
)

try:
    driver.get("https://api.ipify.org?format=json")
    print("Your Proxy IP:", driver.find_element("tag name", "body").text)
finally:
    driver.quit()
```

**Use Case:** Baseline proxy connectivity check and basic web automation in Selenium Python.

## 2. Sticky Session Management

Maintains a static IP address across sequential page navigations using a session ID (`sessid`) and explicit session duration (`sesstime-30`).

```python
from seleniumwire import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

# 30-minute US static session
proxy_url = "http://user-country-us-sessid-auth1-sesstime-30:pass123@proxy.mrscraper.com:10000"

options = {
    "proxy": {
        "http": proxy_url,
        "https": proxy_url,
    }
}

driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()),
    seleniumwire_options=options
)

try:
    urls = [
        "https://example.com/login",
        "https://example.com/dashboard",
        "https://example.com/profile"
    ]

    for url in urls:
        driver.get(url)
        print(f"Visited {url} -> Title: {driver.title}")
finally:
    driver.quit()
```

**Use Case:** Automating multi-step user workflows, account registrations, and shopping cart checkouts that require the same IP address throughout the session.

## 3. Geotargeting & Browser Preferences

Demonstrates routing Selenium traffic through a German residential proxy while setting Chrome options (`--lang=de-DE`) to test localized web applications.

```python
from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

# German residential proxy
proxy_url = "http://user-country-de:pass123@proxy.mrscraper.com:10000"

chrome_options = Options()
chrome_options.add_argument("--lang=de-DE")
chrome_options.add_argument("--window-size=1920,1080")

seleniumwire_options = {
    "proxy": {
        "http": proxy_url,
        "https": proxy_url,
    }
}

driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()),
    options=chrome_options,
    seleniumwire_options=seleniumwire_options
)

try:
    driver.get("https://httpbin.org/headers")
    print("Response Headers:\n", driver.find_element("tag name", "body").text)
finally:
    driver.quit()
```

**Use Case:** Testing localized UI layouts, currency displays, and regional restrictions across international target markets.

## 4. Headless Scraping & Performance Optimization

Configures Selenium for headless background execution to reduce memory and CPU overhead during high-speed scraping tasks.

```python
from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

proxy_url = "http://user-country-gb:pass123@proxy.mrscraper.com:10000"

# Optimized headless Chrome flags
chrome_options = Options()
chrome_options.add_argument("--headless=new")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-dev-shm-usage")

options = {
    "proxy": {
        "http": proxy_url,
        "https": proxy_url,
    }
}

driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()),
    options=chrome_options,
    seleniumwire_options=options
)

try:
    driver.get("https://example.com")
    print("Scraped Page Title:", driver.title)
finally:
    driver.quit()
```

**Use Case:** Automated background data extraction on servers and CI/CD pipelines where GUI rendering is unnecessary.

## 5. Parallel Multi-Driver Sessions

Demonstrates launching multiple concurrent Selenium instances in Python. Each driver worker uses a unique session ID to maintain isolated IP addresses.

```python
import concurrent.futures
from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

def run_worker(worker_id):
    proxy_url = f"http://user-country-us-sessid-worker_{worker_id}:pass123@proxy.mrscraper.com:10000"

    chrome_options = Options()
    chrome_options.add_argument("--headless=new")

    wire_options = {
        "proxy": {
            "http": proxy_url,
            "https": proxy_url,
        }
    }

    driver = webdriver.Chrome(
        service=Service(ChromeDriverManager().install()),
        options=chrome_options,
        seleniumwire_options=wire_options
    )

    try:
        driver.get("https://api.ipify.org?format=json")
        ip = driver.find_element("tag name", "body").text
        print(f"Worker {worker_id} IP: {ip}")
    finally:
        driver.quit()

# Execute 3 parallel Selenium drivers
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
    executor.map(run_worker, [1, 2, 3])
```

**Use Case:** Scaling Selenium web scraping across multiple parallel threads while maintaining independent IP addresses.

## 6. Robust Exception Handling & Teardown

Implements production-grade error handling with explicit element waiting (`WebDriverWait`), screenshot generation on failure, and guaranteed browser teardown.

```python
from seleniumwire import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

proxy_url = "http://user-country-us-sessid-prod1:pass123@proxy.mrscraper.com:10000"

driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()),
    seleniumwire_options={"proxy": {"http": proxy_url, "https": proxy_url}}
)

try:
    driver.get("https://example.com")

    # Explicit wait for target element
    wait = WebDriverWait(driver, 15)
    element = wait.until(EC.presence_of_element_located((By.TAG_NAME, "h1")))
    print("Found Heading:", element.text)

except Exception as e:
    print(f"Selenium Error: {e}")
    driver.save_screenshot("error_screenshot.png")
finally:
    # Always guarantee driver cleanup
    driver.quit()
```

**Use Case:** Production Selenium automation jobs requiring defensive error trapping, diagnostic screenshotting, and resource leak prevention.


  * Use `selenium-wire-lw` for easy proxy authentication without requiring Chrome extension hacks.
  * Always wrap Selenium operations in `try...finally` blocks to ensure `driver.quit()` is executed.
  * Keep credentials safe in environment variables (`os.environ.get("PROXY_PASSWORD")`).



# Overview

import { Globe, Smartphone, Lock, Timer, RotateCcw, Shield } from 'lucide-react';
import { Database, UserCog, Share2 } from 'lucide-react';
import { Step, Steps } from 'fumadocs-ui/components/steps';

MrScraper provides enterprise-grade proxy solutions designed for web scraping, data collection, and anonymous browsing. Static and rotating configurations, and global coverage, that delivers the reliability and flexibility your projects need.

## What is MrScraper's Residential Proxy?

MrScraper's Residential Proxy is a premium proxy service offering:

* **Residential Proxies**: Access real user IPs from residential ISPs
* **Static & Rotating Options**: Choose between persistent IPs or automatic rotation
* **Global Coverage**: Support for 195+ countries using ISO 3166 country codes
* **Session Management**: Control IP persistence with custom session IDs and durations
* **Easy Integration**: Simple authentication format compatible with any HTTP/HTTPS client

## Key Features


  } href="/docs/residential-proxy/configuration/proxy-types">
    Choose between Static and Rotating proxies for different use cases.
  

  } href="/docs/residential-proxy/configuration/authentication">
    Learn how to configure proxy authentication with username patterns.
  

  } href="/docs/residential-proxy/configuration/authentication#3-session-based-static-proxy">
    Maintain persistent IPs with custom session IDs and durations.
  

  } href="/docs/residential-proxy/configuration/country-codes">
    Target specific countries using ISO 3166 country codes.
  


## Proxy Endpoint

All Residential Proxy services connect through a single endpoint:

```
proxy.mrscraper.com:10000
```

Your authentication credentials (username and password) determine:

* Proxy type
* Country targeting
* Session persistence
* Rotation behavior

## Common Use Cases


  }>
    Use rotating proxies to avoid rate limits and IP bans when scraping large amounts of data.
  

  }>
    Use static session-based proxies to maintain consistent IPs for account operations and dashboards.
  

  }>
    Use country-specific proxies to view localized content and pricing.
  


## Quick Start


  
    ### Get Your Credentials

    Obtain your MrScraper username and password from your account dashboard.
  

  
    ### Choose Your Configuration

    Decide on your [proxy type](/docs/residential-proxy/configuration/proxy-types), [country](/docs/residential-proxy/configuration/country-codes), and whether you need session persistence.
  

  
    ### Format Your Authentication

    Build your username string according to the [authentication format](/docs/residential-proxy/configuration/authentication).
  

  
    ### Connect and Test

    Use the format: `username:password@proxy.mrscraper.com:10000` in your application.
  


## Need Help?

If you have questions or need assistance:

* Check our [Examples page](/docs/residential-proxy/examples/overview) for common configurations
* Review the [Authentication guide](/docs/residential-proxy/configuration/authentication) for username format details
* Contact our support team for personalized assistance


# Proxy Generator

import { Step, Steps } from 'fumadocs-ui/components/steps';

## Generate A Proxy

You generate residential proxy credentials for use outside the Proxy Scraper. For example, in your own scripts, third-party tools, or external applications.

Follow these steps to generate a residential proxy:


  
    Go to the **Proxies** page by clicking the **globe icon** in the left sidebar.
  

  
    Select the **Proxy Setup** tab.
  

  
    ### Configuration

    You can configure your proxy here

    #### Proxy Access

    Get your `Username` and `Password` for your proxy here.

    #### Proxy List

    Copy, Download or Delete your proxy list here.

    #### Proxy Configuration

    Main proxy configuration:

    * `Connection Settings` : `Static` or `Rotating`
    * `Protocol` : `HTTP` or `SOCKS5`
    * `Country` : Pick the country you want your request to appear from. Uses ISO 3166-1 alpha-2 codes (e.g. United States → `us`).
  

  
    Click **Generate** to create a new proxy credential.
  

  
    Copy the generated proxy URL and use it in your application or tool. Or use generated code from **Integration Code** panel.

    supported framework/library :

    * Puppeteer (JavaScript)
    * Playwright (JavaScript & Python)
    * CloakBrowser (JavaScript & Python)
    * Selenium (Python)
  



# Proxy Scraper



import { Step, Steps } from 'fumadocs-ui/components/steps';

The **Proxy Scraper** is a standalone scraping service available on the [Proxies page](https://app.mrscraper.com/proxies). It routes requests through residential proxies, allowing you to specify a target URL, select a proxy country, and retrieve the scraped content directly from a single interface.

## Proxies Page Tabs

The [**Proxies page**](https://app.mrscraper.com/proxies) is organized into four tabs:

| Tab               | Description                                                                |
| ----------------- | -------------------------------------------------------------------------- |
| **Proxy Scraper** | Scrape any URL through a proxy using MrScraper's built-in scraping engine. |
| **Proxy Setup**   | Generate and configure residential proxy credentials for external use.     |
| **Analytics**     | Monitor proxy bandwidth consumption and request statistics.                |
| **Proxy Users**   | Create sub-user accounts and allocate proxy bandwidth.                     |


  * When you switch between tabs, your scrape results are preserved. You can navigate away and return to find your results intact.
  * If a scrape is in progress, you receive a warning before switching tabs to prevent losing an active request.
  * Completed proxy scrape results are also available on the Result section. If you navigate away during a long-running scrape, you can find the results there.
  * Your active tab is saved in the URL parameters. You can bookmark or share a direct link to any tab, and it will open in the same state when revisited.


## Proxy Scraper



The **Proxy Scraper** tab is an interactive workspace for running on-demand, proxy-powered scraping requests. It shares the same interface layout as the [Scraper API](/docs/getting-started/scraper-api) but adds a proxy country selector so every request is routed through a residential proxy.

Enter a **Target URL**, select a **proxy country** from the dropdown, and click **Run**. MrScraper's scraping engine fetches the page through a residential proxy in the selected region and returns the extracted content.


  When a proxy scrape finishes, you receive an in-app notification confirming the result is ready. This lets you queue a request and continue working without watching the screen.


### Target URL

In the **Target URL** panel, you configure and run the request:

* Enter the target URL.
* Select **Run** to send the request, or **Reset** to clear the current configuration.
* Select a country to load the page from.

### API Token

You can copy your secret API token for authenticating requests to the MrScraper API. Keep this value private.

### Request Options

The request options panel controls how the proxy scraper handles each request. Adjust these settings to improve reliability and performance.

| Setting               | Default | Description                                                                                                                       |
| --------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------- |
| **Timeout (seconds)** | `90`    | Maximum time in seconds to wait for the page to load.                                                                             |
| **Max retries**       | `3`     | Maximum number of retry attempts if the scrape fails. Failed requests are automatically retried up to this limit.                 |
| **Wait for selector** | None    | Pause until a specific CSS selector appears in the DOM. Useful for content that loads asynchronously after the initial page load. |
| **Super**             | Off     | Uses additional resources to improve extraction accuracy on complex or protected websites.                                        |
| **From Homepage**     | Off     | Redirect to the target homepage first before going to the target URL.                                                             |
| **Browser rendering** | Off     | Renders the page in a headless browser, executing JavaScript before extracting content. Required for SPAs and dynamic websites.   |
| **Block resources**   | Off     | Skips non-essential assets (images, fonts, CSS) to reduce bandwidth and improve speed.                                            |


  Our bandwidth is not 1:1 with the bandwidth you see in your browser when opening the target page. This is due to our retry mechanism and techniques used to avoid blocking from the page.



  Each proxy scrape includes a fixed 4MB cost for the browser session and request handling, in addition to the page content itself.


## Proxy Setup

The **Proxy Setup** tab lets you generate residential proxy credentials for use outside the Proxy Scraper. For example, in your own scripts, third-party tools, or external applications.

Follow these steps to generate a residential proxy:


  
    Go to the **Proxies** page by clicking the **globe icon** in the left sidebar.
  

  
    Select the **Proxy Setup** tab.
  

  
    Scroll down to the **Proxy Configuration** section and configure your proxy settings.
  

  
    Click **Generate** to create a new proxy credential.
  

  
    Copy the generated proxy URL and use it in your application or tool.
  


## Analytics

The **Analytics** tab provides a real-time overview of your proxy bandwidth consumption and request performance. It displays the following summary cards:

| Metric              | Description                                               |
| ------------------- | --------------------------------------------------------- |
| **Total Bandwidth** | Total proxy bandwidth consumed (displayed in MB or GB).   |
| **Total Requests**  | Number of proxy requests made within the selected period. |
| **Success Rate**    | Percentage of proxy requests that completed successfully. |
| **Success**         | Count of successful proxy requests.                       |
| **Failed**          | Count of failed proxy requests.                           |

### Available Filters

You can narrow the analytics view using the following filters:

| Filter         | Description                                                     |
| -------------- | --------------------------------------------------------------- |
| **Time Range** | Select a time window (e.g., 1 Hour, 24 Hours, 7 Days, 30 Days). |
| **Interval**   | Choose the data granularity (e.g., Hourly, Daily).              |
| **Username**   | Filter by a specific proxy sub-user.                            |
| **Country**    | Filter by proxy country/region.                                 |
| **Timezone**   | View data in your preferred timezone.                           |


  Analytics data may take up to 5 minutes to update. The dashboard refreshes automatically, or you can click the refresh icon on the top right manually to fetch the latest data.


## Proxy Users


  This tab is still in `Beta` phase


The **Proxy Users** tab lets you create and manage sub-user accounts for your proxy service. Each sub-user receives their own credentials and a dedicated bandwidth allocation drawn from your main proxy bandwidth pool.

This is useful when you need to:

* Distribute proxy access across team members or clients
* Control and limit bandwidth usage per user
* Track individual proxy consumption separately

### Creating a Sub-User

Follow these steps to create a new proxy sub-user:


  
    Navigate to the **Proxies** page and select the **Proxy Users** tab.
  

  
    Click the **+ Create Sub-User** button in the top-right corner.
  

  
    In the **Create Sub-User** dialog, fill in the following fields:

    * **Username suffix** *(optional)*: Enter a suffix to append to your account email. Only alphanumeric characters are allowed; symbols and hyphens are not permitted.
    * **Bandwidth Quota**: Enter the amount of bandwidth (in MB) to allocate to this sub-user. The value is drawn from your unallocated bandwidth pool.
  

  
    Click **OK** to create the sub-user.
  



  You can create up to 80 sub-user.


### Renaming username

You can rename a username suffix in the Proxy User table by clicking the edit icon beside the username.

### Editing Bandwidth Allocation

To change a sub-user's bandwidth limit:


  
    Locate the sub-user in the table.
  

  
    Click the **edit icon** (pencil) next to the **Bandwidth Limit** value.
  

  
    Enter the new bandwidth amount (in MB).

    
      For user with `Enterprise` plan you can select `Unlimited` to pay as you go.
    
  

  
    Confirm the change.
  


The updated allocation is reflected immediately. Any bandwidth freed up returns to the unallocated pool.

### Deleting a Sub-User

To remove a sub-user, click the **delete icon** (trash) next to their row in the table. The bandwidth that was allocated to the deleted sub-user is returned to your unallocated pool.


# Troubleshoot URL Access

import { Step, Steps } from 'fumadocs-ui/components/steps';

When a target website doesn't load through your proxy integration, it can be hard to tell whether the issue lies in the proxy configuration, or the target site itself. This guide walks you through a structured process to isolate the cause and resolve it.

## Why requests to a target website fail

Even with residential proxy IPs, a request can still fail if:

* **The website restricts automated access** through its terms of service or access controls.
* **The content depends on JavaScript** and the request is made without browser rendering.
* **The content is region-specific** and the selected proxy country doesn't match.
* **Rate limiting** rejects IPs that send too many requests in a short period.
* **The website is slow or unavailable** and the request times out.

Understanding these causes helps you pick the right fix. The fastest way to start is to use **Proxy Scraper** to test whether the issue is on your side or the proxy side.


  
    ## Test with Proxy Scraper

    [Proxy Scraper](/docs/residential-proxy/getting-started/proxy-scraper) is a standalone scraping tool built into MrScraper. It sends requests through the same residential proxy network. This isolates the proxy path from your code path.

    
      
        Go to [app.mrscraper.com](https://app.mrscraper.com), sign in, and select **Proxy** from the left navigation menu. Then select the [**Proxy Scraper**](https://app.mrscraper.com/proxies?tab=Proxy+Scraper) tab.
      

      
        Enter a simple, reliable URL in the **Target URL** field. For example, `https://www.google.com`. Keep all advanced settings at their defaults. Click **Run**.
      

      
        If Proxy Scraper returns an **HTTP 200** response with page content, the proxy connection is working. Proceed to the next step.

        If it returns an error, see [When the connectivity test fails](#when-the-connectivity-test-fails) below.
      

      
        Now enter the **real target URL**. Click **Run** with the same default settings.
      
    
  

  
    ## Interpret the results

    Use the table below to determine your next action based on the Proxy Scraper results:

    | google.com result | Target URL result | What it means                                                                                                 | Next action                                                                                               |
    | ----------------- | ----------------- | ------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
    | ✅ Success         | ✅ Success         | The proxy path works. The issue is in your own script or integration.                                         | You can check your proxy configuration or continue using our Proxy Scraper to access your website target. |
    | ✅ Success         | ❌ Failure         | The proxy path works, but the target website is not returning the page.                                       | Go to [When the target website fails](#when-the-target-website-fails).                                    |
    | ❌ Failure         | ❌ Failure         | The proxy path itself has an issue.                                                                           | Go to [When the connectivity test fails](#when-the-connectivity-test-fails).                              |
    | ❌ Failure         | ✅ Success         | Unlikely, but possible if google.com is blocked in the selected proxy country. Try a different proxy country. | Change the proxy country and retest.                                                                      |
  


## Check your configuration

If Proxy Scraper succeeds but your own code or tool fails, the issue is in how your integration sends requests. Check the following:

### Proxy credentials

* Verify your **username** and **password** are correct and match the format described in the [Authentication guide](/docs/residential-proxy/configuration/authentication).
* Check for typos, extra spaces, or missing segments (e.g., `-country-`, `-sessid-`).

### Request format

* Ensure you're using the correct proxy endpoint: `proxy.mrscraper.com:10000`.
* Verify the proxy protocol. MrScraper supports HTTP. Make sure your client is configured to use the HTTP protocol.
* Check that the target URL includes the protocol (`http://`).

### HTTP headers

* Some websites reject requests that lack a `User-Agent` header. Make sure your client sends one.
* Include standard headers like `Accept`, `Accept-Language`, and `Accept-Encoding`.

### Network and firewall

* Confirm your network allows outbound connections to `proxy.mrscraper.com` on port `10000`.
* If you're behind a corporate firewall or VPN, check whether proxy traffic is being intercepted or blocked.

## When the target website fails

If Proxy Scraper can reach google.com but fails on your target URL, try the following steps **in order**:


  
    ### Try a different proxy country

    Some websites only serve traffic from specific regions. Change the proxy country in Proxy Scraper to one that matches the target website's expected audience (e.g., `us` for a US-based website).
  

  
    ### Enable Browser rendering

    Toggle **Browser rendering** to **On** in the Proxy Scraper request options. This renders the page in a headless browser, executing JavaScript before extracting content. This is required for:

    * Single-page applications (SPAs)
    * Websites that load content dynamically via JavaScript
  

  
    ### Adjust timeout and retries

    Increase the **Timeout** beyond the default 90 seconds, and set **Max retries** to 5 or higher. Some websites are slow to respond or require multiple attempts.
  

  
    ### Verify the block manually in a browser

    Before assuming the proxy IPs are at fault, browse the target website through the same proxy in a real browser, arriving from a Google search and navigating to the page like a human would. If the page loads that way, the IPs are fine and the website is detecting your scraper, not your proxy. See [Verify a Block Manually](/docs/residential-proxy/getting-started/verify-block-manually).
  

  
    ### Contact support

    If none of the above steps resolve the issue, the target website may require custom handling. Contact [us](mailto:support@mrscraper.com) with the following details:

    * The target URL that fails
    * The Proxy Scraper result (screenshot or response)
    * The proxy country you tested with
    * The settings you've tried (Super, Browser rendering, timeout)
  


## When the connectivity test fails

If Proxy Scraper cannot reach even a simple URL like google.com, check the following:

| Possible cause                    | What to check                                                                                                                                                              |
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Account or subscription issue** | Verify your account is active and has remaining proxy bandwidth in the **Analytics** tab.                                                                                  |
| **Service outage**                | Check if MrScraper is experiencing an outage. Try again in a few minutes.                                                                                                  |
| **Proxy country unavailable**     | Some proxy countries may have limited availability. Try a different country code from the [supported countries list](/docs/residential-proxy/configuration/country-codes). |
| **Browser or network issue**      | Try accessing [app.mrscraper.com](https://app.mrscraper.com) in an incognito/private window to rule out browser cache or extension interference.                           |

If the issue persists, contact support at [support@mrscraper.com](mailto:support@mrscraper.com).


# Verify a Block Manually

import { Step, Steps } from 'fumadocs-ui/components/steps';

When a scraper suddenly starts failing, the proxy is the easiest thing to blame. But in most cases the target website has changed how it handles automated requests, and the same IPs still work fine for ordinary browsing.

This guide walks through a manual test: browse the target website through your Residential Proxy in a real browser, exactly the way a human would. The result tells you which side the block is actually coming from.


  Use this test when your scraper worked before and started failing, or when a scrape is blocked but you're not sure whether the proxy IPs or the website is responsible. If you haven't run the automated check yet, start with [Troubleshoot URL Access](/docs/residential-proxy/getting-started/troubleshoot-url-access) first, it's faster.


## Why direct URL access fails

Many websites treat a request for a deep page (a product page, a listing, a search result) with no prior history differently from a normal visit. A real visitor almost never lands there cold. They arrive from a search engine, from the homepage, or from a category page, carrying cookies, a referrer, and a browsing history that the site has already seen.

That's why a scraper hitting a product URL directly can fail while the exact same IP loads the site normally in a browser. The test below reproduces the natural path so you can compare the two.

## What you need

* Your Residential Proxy credentials from the [Proxy Setup tab](https://app.mrscraper.com/proxies?tab=Proxy+Setup). See the [Authentication guide](/docs/residential-proxy/configuration/authentication) for the username format.
* Google Chrome with the [FoxyProxy](https://chromewebstore.google.com/detail/foxyproxy/gcknhkkoolaabfmlnjonogaaifnjlfnp) extension installed.
* Optional: [CloakBrowser](/docs/residential-proxy/examples/cloakbrowser), an open source Chromium-based browser, for the second round of testing.

## Run the test


  
    ### Route Chrome through the proxy

    Open FoxyProxy, add a new proxy, and fill in the connection details:

    | Field             | Value                                              |
    | ----------------- | -------------------------------------------------- |
    | **Proxy type**    | HTTP                                               |
    | **Hostname / IP** | `proxy.mrscraper.com`                              |
    | **Port**          | `10000`                                            |
    | **Username**      | Your proxy username, for example `user-country-us` |
    | **Password**      | Your proxy password                                |

    Set the country in the username to match the target website's audience. For a Brazilian website, use `-country-br`.

    Save the proxy, then enable it from the FoxyProxy toolbar icon. Confirm the traffic is actually routed by opening [api.ipify.org](https://api.ipify.org) and checking that the IP is not your own.
  

  
    ### Search for the website on Google

    Go to [google.com](https://www.google.com) and search for the target website's name or homepage, for example `ifood.com.br`.

    Don't paste the target page URL into the address bar. The point of this test is to arrive at the site the way a visitor does.
  

  
    ### Click through from the search results

    Click the organic search result that leads to the website's homepage. This gives the session a search engine referrer and lets the site set its cookies on a page it expects first-time visitors to land on.
  

  
    ### Navigate to the target page like a human

    From the homepage, reach the page you want to scrape using the site's own navigation: menus, category links, the on-site search box, pagination. Take a moment between clicks instead of firing them back to back.

    If you reach the page and it loads normally, the website is serving your proxy IP without complaint.
  

  
    ### If regular Chrome fails, retry with CloakBrowser

    [CloakBrowser](/docs/residential-proxy/examples/cloakbrowser) is an open source Chromium-based browser. Comparing it with regular Chrome tells you whether the result depends on the browser rather than the IP.

    Launch it in headed mode with the same proxy credentials, then repeat steps 2 to 4 inside it:

    ```javascript
    import { launch } from 'cloakbrowser';

    const browser = await launch({
      headless: false,
      proxy: {
        server: 'http://proxy.mrscraper.com:10000',
        username: 'user-country-br',
        password: 'pass123'
      }
    });

    const page = await browser.newPage();
    await page.goto('https://www.google.com');
    // Now search for the site and click through by hand.
    ```
  


## Interpret the results

| Manual browser test                                        | What it means                                                                                                                               | Next action                                                                                                                                         |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| ✅ Works in regular Chrome                                  | The proxy IPs are fine. The difference is in how your scraper sends requests, such as its headers or navigating directly to the target URL. | Reproduce the natural flow in your scraper: start from the homepage or a search result, keep cookies across requests, and enable browser rendering. |
| ❌ Fails in Chrome, ✅ works in CloakBrowser                 | The IPs are fine. The result depends on the browser, not the IP.                                                                            | See the [CloakBrowser examples](/docs/residential-proxy/examples/cloakbrowser).                                                                     |
| ❌ Fails in both, even with natural navigation              | The website is rejecting the IP itself, or the whole country or ASN.                                                                        | Try a different [proxy country](/docs/residential-proxy/configuration/country-codes) and repeat the test. If it still fails, contact support.       |
| ❌ Fails in both, and the site also fails without the proxy | The website is blocking your region or is down for everyone.                                                                                | Verify from a different network before investigating the proxy further.                                                                             |


  A website that recently changed its configuration can reject a healthy residential IP when the request doesn't match a normal browsing session. Running this test before escalating saves you from swapping proxies that were never the problem.



# Get Brand

Read a brand you already scraped with [Scrape Brand](/docs/api/scraper-api/scrape-brand/scrape-brand), or download one of its files.


  * Use the Scraper API Server (`https://api.mrscraper.com`) host when calling this endpoint.
  * Pass the job ID from `data.job_id` as `id`. You can also find it in the **Result** tab of the Scraper API.
  * Omit `output` to return the brand job as JSON. Set `output` to `html`, `bundle`, `screenshot`, or `screenshot_full` to download a file.
  * A job scraped with `bundle=none` only has the HTML to download. Jobs scraped with `bundle=clean` or `bundle=full` have all four files.





# Scrape Brand

Scrape the brand of a page. The request waits until the job settles, then returns the brand in `data.brand` and the job ID in `data.job_id`.


  * Use the Scraper API Server (`https://api.mrscraper.com`) host when calling this endpoint.
  * Set `bundle` to `none` for the brand JSON only, `clean` to add the HTML, screenshots, and a ZIP of the page without scripts, or `full` to keep the page scripts in the ZIP.
  * `none` usually answers in a few seconds. `clean` and `full` can take about a minute, so set a generous client timeout.
  * Use the job ID with [Get Brand](/docs/api/scraper-api/scrape-brand/get-brand) to read the brand again or download its files.
  * See [Scrape Brand](/docs/getting-started/scraper-api/scrape-brand) in the Scraper API guide for the bundle contents and code examples.





# Verify API Token

{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}

Verify your API token and retrieve subscription account information including token usage, limits, and billing details.


  Use the Platform API Server (`https://api.app.mrscraper.com`) host when calling this endpoint.





# AI Scraper Response

The AI Scraper API returns responses with a consistent wrapper structure across all scraping operations. While the wrapper remains the same, the `data.data` field contains agent-specific data optimized for each scraping type.

## Standard Response Wrapper

Every API response follows this structure:

```json
{
  "message": "Successful operation!",
  "data": {
    "id": "result-unique-identifier",
    "createdAt": "2026-03-06T05:30:44.642Z",
    "userId": "user-id",
    "scraperId": "scraper-id",
    "type": "AI",
    "url": "https://example.com",
    "status": "Finished",
    "error": "",
    "tokenUsage": 5,
    "runtime": 0,
    "data": {
      // Agent-specific data structure (see below)
    },
    "htmlPath": "...",
    "htmlContent": "...",
    "recordingPath": null,
    "screenshotPath": "...",
    "dataPath": null
  }
}
```

### Response Fields

| Field                 | Type   | Description                                           |
| --------------------- | ------ | ----------------------------------------------------- |
| `message`             | string | Operation status message                              |
| `data.id`             | string | Unique identifier for this result                     |
| `data.createdAt`      | string | ISO 8601 timestamp when the operation started         |
| `data.userId`         | string | Your user identifier                                  |
| `data.scraperId`      | string | Identifier of the scraper used                        |
| `data.type`           | string | Agent type used for scraping                          |
| `data.url`            | string | Target URL that was scraped                           |
| `data.status`         | string | Operation status (`Finished`, `Processing`, `Failed`) |
| `data.error`          | string | Error message if the operation failed                 |
| `data.tokenUsage`     | number | Number of tokens consumed by this operation           |
| `data.runtime`        | number | Total runtime of the scraping operation in seconds    |
| `data.data`           | object | Extracted data (structure varies by agent type)       |
| `data.htmlPath`       | string | Path to the saved HTML content                        |
| `data.htmlContent`    | string | The actual HTML content of the scraped page           |
| `data.recordingPath`  | string | Path to the scraping session recording (if available) |
| `data.screenshotPath` | string | Path to the page screenshot                           |
| `data.dataPath`       | string | Path to the raw data file (if available)              |

## Agent-Specific Response Formats

The `data.data` field structure varies depending on which agent you use. Each agent optimizes the data format for its specific use case.

### General Agent

The General Agent provides two response formats depending on the page type you're scraping.

#### General Table Response Format

General agent returns a dictionary containing an array of structured items when scraping listing pages with multiple items (e.g., product listings, search results, directory pages)

```json Example
{
  "books": [
    {
      "price": "£51.77",
      "title": "A Light in the Attic"
    },
    {
      "price": "£53.74",
      "title": "Tipping the Velvet"
    },
    {
      "price": "£50.10",
      "title": "Soumission"
    }
  ]
}
```

#### General Detail Response Format

General agent returns A flat dictionary with all relevant fields about a single item when scraping individual item pages with comprehensive information (e.g., product details, profile pages, article pages)

```json Example
{
  "price": "£51.77",
  "title": "A Light in the Attic",
  "image_url": "../../media/cache/fe/72/fe72f0532301ec28892ae79a629a293c.jpg",
  "description": "It's hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition...",
  "availability": "In stock (22 available)",
  "product_information": {
    "UPC": "a897fe39b1053632",
    "Product Type": "Books",
    "Price (excl. tax)": "£51.77",
    "Price (incl. tax)": "£51.77",
    "Tax": "£0.00",
    "Availability": "In stock (22 available)",
    "Number of reviews": 0
  }
}
```

### Listing Agent

Listing agent returns an array of page objects, each containing pagination info and extracted data from the page.

```json Example
{
  "response": [
    {
      "page_num": 0,
      "data": {
        "mode": "direct",
        "pagination": {
          "current_page": 1,
          "total_pages": 50
        },
        "products": [
          {
            "title": "Product Name",
            "price": "£XX.XX",
            "availability": "In stock"
          },
          {
            "title": "Another Product",
            "price": "£YY.YY",
            "availability": "In stock"
          }
        ],
        "product_count": 20,
        "__counts__": {
          "products": 20
        }
      },
      "total_items": 20,
      "next_found": true
    },
    {
      "page_num": 1,
      "data": {
        "mode": "direct",
        "pagination": {
          "current_page": 2,
          "total_pages": 50
        },
        "products": [
          {
            "title": "Product Name Page 2",
            "price": "£ZZ.ZZ",
            "availability": "In stock"
          }
        ],
        "product_count": 20,
        "__counts__": {
          "products": 20
        }
      },
      "total_items": 20,
      "next_found": true
    }
  ],
  "link": "https://books.toscrape.com/"
}
```

### Map Agent

Map agent returns a list of all discovered URLs with a total count.

```json Example
{
  "count": 581,
  "urls": [
    "https://books.toscrape.com",
    "https://books.toscrape.com/catalogue/10-day-green-smoothie-cleanse-lose-up-to-15-pounds-in-10-days_581/index.html",
    "https://books.toscrape.com/catalogue/1000-places-to-see-before-you-die_1/index.html",
    "https://books.toscrape.com/catalogue/1491-new-revelations-of-the-americas-before-columbus_650/index.html"
  ]
}
```

## Bulk Scraping Response

Bulk scraping returns aggregated results for multiple URLs in a single operation. The response keeps the same top-level wrapper, while `data.data` includes merged extraction output, per-URL statuses, and a summary.

```json Example
{
  "mergedData": [
    {
      "products": [
        {
          "price": "£51.77",
          "title": "A Light in the Attic",
          "availability": "In stock"
        }
      ],
      "page_title": "A Light in the Attic",
      "total_results": null
    },
    {
      "products": [
        {
          "price": "£53.74",
          "title": "Tipping the Velvet",
          "availability": "In stock (20 available)"
        }
      ],
      "page_title": "Tipping the Velvet",
      "total_results": null
    }
  ],
  "urlDetails": [
    {
      "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
      "status": "Finished",
      "error": ""
    },
    {
      "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
      "status": "Finished",
      "error": ""
    }
  ],
  "summary": {
    "totalUrls": 2,
    "successfulUrls": 2,
    "failedUrls": 0,
    "scrapedCount": 2,
    "totalTokenUsage": 20,
    "estimatedFinishAt": null
  }
}
```

### Bulk Data Fields

| Field                       | Type   | Description                                                                                                          |
| --------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------- |
| `mergedData`                | array  | Combined extracted output grouped by processed page. The exact object shape depends on your prompt and target pages. |
| `urlDetails`                | array  | Per-URL execution result with status and error details for each submitted URL.                                       |
| `summary.totalUrls`         | number | Total number of URLs requested in the bulk job.                                                                      |
| `summary.successfulUrls`    | number | Number of URLs completed successfully.                                                                               |
| `summary.failedUrls`        | number | Number of URLs that failed.                                                                                          |
| `summary.scrapedCount`      | number | Number of pages with extracted data.                                                                                 |
| `summary.totalTokenUsage`   | number | Total token usage across all URLs in the bulk job.                                                                   |
| `summary.estimatedFinishAt` | string | Estimated completion timestamp for running jobs. `null` when finished.                                               |

## Response Status Values

The `data.status` field indicates the current state of your scraping operation:

| Status       | Description                                                    |
| ------------ | -------------------------------------------------------------- |
| `Finished`   | Scraping completed successfully                                |
| `Processing` | Scraping is currently in progress                              |
| `Failed`     | Scraping encountered an error (check `data.error` for details) |


# Web Fetching

This endpoint scrapes a target URL with the Web Fetching feature that supports browser rendering, geo-targeting via proxy, and waiting for specific DOM selectors before extracting content.


  * Use the Scraper API Server (`https://api.mrscraper.com`) host when calling this endpoint.
  * This is the same endpoint as `Scrape HTML` in Scraper API.


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}




# Bulk Scraping With Custom Scraper

import { Step, Steps } from 'fumadocs-ui/components/steps';

The Bulk API lets you run one custom scraper workflow against a list of URLs in a single submission, instead of sending a separate request for every page.

Bulk runs are asynchronous. You submit the workflow and its URLs, receive a submission ID immediately, then poll that ID for progress and results.

## What Is a Custom Scraper?

A **custom scraper** is a Scraper API request that carries its own manual workflow. Instead of pointing at a saved scraper, you send the workflow steps inline with the request, so the same extraction logic can be applied to any URL you pass in.

## Submit a Bulk Run


  
    Copy the body request from your custom workflow.
  

  
    Paste it into the `"workflow": {}` of the request below.
  

  
    Fill in your API key, the proxy country, and the list of URLs you want to scrape.
  

  
    Send the request and save the `submission_id` from the response.
  


### Request

```bash title="Submit a bulk run"
curl --location 'https://bulk.mrscraper.com/api/gateway/v1/bulk-submissions/run' \
--header 'Content-Type: application/json' \
--header 'x-api-key: {MRSCRAPER_API_KEY}' \
--header 'Cookie: sl-session=P89TLJQZiGoIfOVndtHSPg==' \
--data '{
    "engine": "http",
    "workflow": {
        "url": "https://books.toscrape.com/",
        "homePage": true,
        "proxyCountry" : "us",
        "workflow": [
            {
                "type": "script",
                "data": {
                    "name": "first_book",
                    "timeout": 30,
                    "code": "(function() {\n  const firstArticle = document.querySelector('\''article.product_pod'\'');\n  if (!firstArticle) {\n    return { title: null, price: null };\n  }\n  \n  const titleLink = firstArticle.querySelector('\''h3 a'\'');\n  const title = titleLink ? (titleLink.getAttribute('\''title'\'') || titleLink.textContent).trim() : null;\n  \n  const priceEl = firstArticle.querySelector('\''p.price_color'\'');\n  const price = priceEl ? priceEl.textContent.trim() : null;\n  \n  return { title: title || null, price: price || null };\n})();"
                }
                }
        ]
    },
        "request_method": "POST",
        "proxy_country": "us",
        "urls": [
            "https://books.toscrape.com/",
            "https://books.toscrape.com/catalogue/page-2.html",
            "https://books.toscrape.com/catalogue/page-3.html"
        ]
}'
```

| Parameter               | Location | Description                                                                            |
| ----------------------- | -------- | -------------------------------------------------------------------------------------- |
| `x-api-key`             | header   | Your MrScraper API key.                                                                |
| `Cookie`                | header   | Session cookie for the bulk gateway. Send the default value shown above.               |
| `engine`                | body     | Execution engine used for the run. Use `http`.                                         |
| `workflow`              | body     | The scraper configuration applied to every URL in the run.                             |
| `workflow.url`          | body     | The first page to scrape.                                                              |
| `workflow.homePage`     | body     | Set to `true` to visit the website's home page first, then navigate to the target URL. |
| `workflow.proxyCountry` | body     | Country the workflow accesses the pages from.                                          |
| `workflow.workflow`     | body     | The workflow steps copied from your Scraper API scraper.                               |
| `request_method`        | body     | HTTP method used to request each URL.                                                  |
| `proxy_country`         | body     | Proxy country for the bulk submission.                                                 |
| `urls`                  | body     | Array of URLs to run the workflow against.                                             |

### Response

The submission is accepted and queued immediately. The run itself continues in the background.

```json title="Submission accepted"
{
    "data": {
        "submission_id": "01M0HD8JB6YXN2RZM2AC48391M",
        "engine": "http",
        "priority": 1,
        "status": "queued",
        "total_urls": 3,
        "succeeded": 0,
        "failed": 0,
        "dead": 0,
        "pending": 3,
        "percent_complete": 0,
        "created_at": "2026-08-21T05:38:52.902189Z",
        "started_at": "2026-08-21T05:38:52.939086Z",
        "finished_at": null
    },
    "message": "submission running"
}
```


  The `submission_id` is the only way to retrieve the results of a bulk run. Store it before moving on.


## Check Progress and Results

Use the submission ID to poll the run. The same endpoint returns both the current progress counters and any results extracted so far.

### Request

```bash title="Check a bulk submission"
curl --location 'https://bulk.mrscraper.com/api/results/v1/bulk-submissions/{SUBMISSION_ID}?limit=25' \
--header 'x-api-key: {MRSCRAPER_API_KEY}' \
--header 'Cookie: sl-session=P89TLJQZiGoIfOVndtHSPg=='
```

| Parameter       | Location | Description                                                              |
| --------------- | -------- | ------------------------------------------------------------------------ |
| `x-api-key`     | header   | Your MrScraper API key.                                                  |
| `Cookie`        | header   | Session cookie for the bulk gateway. Send the default value shown above. |
| `SUBMISSION_ID` | path     | The ID returned when you submitted the bulk run.                         |
| `limit`         | query    | Number of results to return per page. Maximum is `100`.                  |

### Response While Running

While the run is still in progress, `results` is empty or partially filled and `percent_complete` is below `100`.

```json title="Run still in progress"
{
    "data": {
        "submission_id": "01M0HDS6DSWSSFTCT1JCDTA0A2",
        "engine": "http",
        "status": "queued",
        "total_urls": 3,
        "succeeded": 0,
        "failed": 0,
        "dead": 0,
        "pending": 3,
        "percent_complete": 0,
        "created_at": "2026-08-21T05:47:57.753168Z",
        "started_at": "2026-08-21T05:47:57.783051Z",
        "finished_at": null,
        "results": [],
        "next_cursor": null
    },
    "message": "run found"
}
```

### Response When Finished

Once every URL has been processed, `percent_complete` reaches `100` and `results` contains one entry per URL.

```json title="Run complete"
{
    "data": {
        "submission_id": "01M0HD8JB6YXN2RZM2AC48391M",
        "engine": "http",
        "status": "queued",
        "total_urls": 3,
        "succeeded": 3,
        "failed": 0,
        "dead": 0,
        "pending": 0,
        "percent_complete": 100,
        "created_at": "2026-08-21T05:38:52.902189Z",
        "started_at": "2026-08-21T05:38:52.939086Z",
        "finished_at": null,
        "results": [
            {
                "id": "01M0HD8JBQKH2WFKWZ0KKQ15BB:0",
                "job_id": "01M0HD8JBQKH2WFKWZ0KKQ15BB",
                "url": "https://books.toscrape.com/catalogue/page-2.html",
                "http_status": 200,
                "data": {
                    "first_book": {
                        "price": "£12.84",
                        "title": "In Her Wake"
                    }
                },
                "raw_ref": "2026/08/21/01M0HD8JBQKH2WFKWZ0KKQ15BB-a0.html.zst",
                "screenshot_ref": null,
                "engine": "http",
                "extracted_at": "2026-08-21T05:39:15.841670Z"
            },
            {
                "id": "01M0HD8JBQAS5DN2ZDWQ7M7MGN:0",
                "job_id": "01M0HD8JBQAS5DN2ZDWQ7M7MGN",
                "url": "https://books.toscrape.com/catalogue/page-3.html",
                "http_status": 200,
                "data": {
                    "first_book": {
                        "price": "£57.31",
                        "title": "Slow States of Collapse: Poems"
                    }
                },
                "raw_ref": "2026/08/21/01M0HD8JBQAS5DN2ZDWQ7M7MGN-a0.html.zst",
                "screenshot_ref": null,
                "engine": "http",
                "extracted_at": "2026-08-21T05:39:14.414348Z"
            },
            {
                "id": "01M0HD8JBQJ2TRGA8CNKQC9Z2B:0",
                "job_id": "01M0HD8JBQJ2TRGA8CNKQC9Z2B",
                "url": "https://books.toscrape.com/",
                "http_status": 200,
                "data": {
                    "first_book": {
                        "price": "£51.77",
                        "title": "A Light in the Attic"
                    }
                },
                "raw_ref": "2026/08/21/01M0HD8JBQJ2TRGA8CNKQC9Z2B-a0.html.zst",
                "screenshot_ref": null,
                "engine": "http",
                "extracted_at": "2026-08-21T05:39:14.089989Z"
            }
        ],
        "next_cursor": null
    },
    "message": "run found"
}
```

### Result Fields

| Field                                       | Description                                                         |
| ------------------------------------------- | ------------------------------------------------------------------- |
| `total_urls`                                | Number of URLs in the submission.                                   |
| `succeeded` / `failed` / `dead` / `pending` | Per-URL status counters for the run.                                |
| `percent_complete`                          | Progress of the run, from `0` to `100`.                             |
| `results[].url`                             | The URL this result was extracted from.                             |
| `results[].http_status`                     | HTTP status code returned by the target page.                       |
| `results[].data`                            | Data extracted by your workflow steps, keyed by step name.          |
| `results[].raw_ref`                         | Reference to the stored raw page content.                           |
| `results[].screenshot_ref`                  | Reference to the stored screenshot, if one was captured.            |
| `next_cursor`                               | Cursor for the next page of results. `null` when there are no more. |


  Results are not returned in the order the URLs were submitted. Match each result to its source page using the `url` field rather than its position in the array.



# E-commerce

import { Step, Steps } from 'fumadocs-ui/components/steps';

E-commerce websites are a rich source of product data that can be invaluable for market analysis, price comparison, inventory management, and more.

In this guide, we'll demonstrate how to use the MrScraper's Scraper to gather product details such as names, prices, ratings, and availability from an e-commerce website.

## Scenario

Imagine you're a market analyst looking to gather data on products listed on Tiktok Shop to analyze pricing trends and customer preferences. You want to extract details like product names, prices, ratings, and the number of items sold.


  
    ## Create a New AI Listing Scraper

    1. Open the [**New AI Scraper**](https://app.mrscraper.com/scrapers/form) page.
    2. Select the **Listing Scraper** option.
    3. In the **Target URL** field, enter the URL of the Tiktok Shop page you want to scrape. For example: `https://shop-id.tokopedia.com/c/phones-electronics/601739?source=ecommerce_mall&enter_method=categories&first_entrance=ecommerce_mall&first_entrance_position=region_redirect_301`.
    4. Click on the **Start Scraping** button to start scraping.

    **Result :**

    ```json title="Scraping Result"
    {
      "mode": "direct",
      "count": 71,
      "links": [
        "https://www.tiktok.com/shop/pdp/nose-ring-by-oufer-20g-halloween-cz-double-hoop-in-316l-stainless-steel/1729608283143704812",
        "https://www.tiktok.com/shop/pdp/usb-powered-led-disco-lights-by-unnamed-brand-with-remote-control-strobe-effects/1729483087369179985",
        "https://www.tiktok.com/shop/pdp/halloween-pumpkin-night-light-mini-led-dimmable-touch-lamp-for-kids/1731541214379741272",
        "https://www.tiktok.com/shop/pdp/soft-funny-throw-blanket-cozy-fleece-for-home-movie-decor/1731291206388977735",
        "https://www.tiktok.com/shop/pdp/40oz-sip-sip-hooray-halloween-tumblers-with-lid-straw-durable-glass/1729576207882752721"
      ],
      "products": [
        {
          "id": "1729608283143704812",
          "url": "https://www.tiktok.com/shop/pdp/nose-ring-by-oufer-20g-halloween-cz-double-hoop-in-316l-stainless-steel/1729608283143704812",
          "sold": 2500,
          "brand": "Oufer Body Piercing Jewelry",
          "price": 9.46,
          "title": "Oufer 20G Halloween CZ Double Hoop Nose Ring 316L Stainless Steel",
          "rating": 4.6,
          "section": "TikTok Picks",
          "currency": "$",
          "sold_raw": "2.5K",
          "free_shipping": false,
          "original_price": 11.82
        },
        {
          "id":"1729483087369179985"
          "url":"https://www.tiktok.com/shop/pdp/usb-powered-led-disco-lights-by-unnamed-brand-with-remote-control-strobe-effects/1729483087369179985"
          "sold":11000
          "brand":"11.0K sold"
          "price":16.65
          "title":"1pc All Aluminum One Body Party Disco Light with Remote Control, USB Powered, for Party Birthday Wedding Holiday Christmas Decoration #TOP PICKS"
          "rating":4.6
          "section":"Oufer 20G Halloween CZ Double Hoop Nose Ring 316L Stainless Steel"
          "currency":"$"
          "sold_raw":"11.0K"
          "free_shipping":false
          "original_price":34.69
        },
        {
          "id":"1731541214379741272"
          "url":"https://www.tiktok.com/shop/pdp/halloween-pumpkin-night-light-mini-led-dimmable-touch-lamp-for-kids/1731541214379741272"
          "sold":1900
          "brand":"1.9K sold"
          "price":9.99
          "title":"Halloween Pumpkin Night Light, Halloween Decorations Outdoor,Mini LED Pumpkin Lamp with 3 Level Dimmable, Nursery Nightlight for Kids, Silicone Rechargeable Bedside Touch Lamp, for Kids TikTokShopBlackFriday"
          "rating":4.7
          "section":"1pc All Aluminum One Body Party Disco Light with Remote Control, USB Powered, for Party Birthday Wedding Holiday Christmas Decoration #TOP PICKS"
          "currency":"$"
          "sold_raw":"1.9K"
          "free_shipping":false
          "original_price":NULL
        },
        {
          "id":"1731291206388977735"
          "url":"https://www.tiktok.com/shop/pdp/soft-funny-throw-blanket-cozy-fleece-for-home-movie-decor/1731291206388977735"
          "sold":3000
          "brand":"3.0K sold"
          "price":14.39
          "title":"Soft Funny Throw Blanket Horror Icons Flannel Fleece - Cozy Fuzzy Plush Comfy All-Season Sofa Bedroom Dorm Couch Office Car Travel Home Bedding,Comfortable Movie Decor Blanket Men Women Boys Girls Halloween Birthday Fan Gifts"
          "rating":4.8
          "section":"Halloween Pumpkin Night Light, Halloween Decorations Outdoor,Mini LED Pumpkin Lamp with 3 Level Dimmable, Nursery Nightlight for Kids, Silicone Rechargeable Bedside Touch Lamp, for Kids TikTokShopBlackFriday"
          "currency":"$"
          "sold_raw":"3.0K"
          "free_shipping":true
          "original_price":23.98
        },
        {
          "id":"1729576207882752721"
          "url":"https://www.tiktok.com/shop/pdp/40oz-sip-sip-hooray-halloween-tumblers-with-lid-straw-durable-glass/1729576207882752721"
          "sold":6100
          "brand":"6.1K sold"
          "price":18.79
          "title":"Sip-Sip Hooray! Halloween Tumbler – Cup with Lid & Straw, Spooky Season Iced Coffee Gift, Halloweentok 2024 Reusable Drinkware Insulated Sports Bottle Travel Water Bottle"
          "rating":4.7
          "section":"Soft Funny Throw Blanket Horror Icons Flannel Fleece - Cozy Fuzzy Plush Comfy All-Season Sofa Bedroom Dorm Couch Office Car Travel Home Bedding,Comfortable Movie Decor Blanket Men Women Boys Girls Halloween Birthday Fan Gifts"
          "currency":"$"
          "sold_raw":"6.1K"
          "free_shipping":false
          "original_price":28.56
        }
      ]
    }
    ```
  

  
    ## Filter Out

    Next, tell the AI to filter the scraped results based on specific criteria. For this use case, you can use the following prompt:

    ```txt
    take only product with rating 4.7 or more.
    ```

    **Result :**

    ```json title="Filtered Scraping Result"
    {
      "mode": "direct",
      "products": [
        {
          "url": "https://www.tiktok.com/shop/pdp/halloween-pumpkin-night-light-mini-led-dimmable-touch-lamp-for-kids/1731541214379741272",
          "sold": "1.9K sold",
          "price": 9.99,
          "title": "Halloween Pumpkin Night Light, Halloween Decorations Outdoor,Mini LED Pumpkin Lamp with 3 Level Dimmable, Nursery Nightlight for Kids, Silicone Rechargeable Bedside Touch Lamp, for Kids TikTokShopBlackFriday",
          "rating": 4.7,
          "section": "TikTok Picks",
          "currency": "$",
          "free_shipping": false,
          "original_price": null
        },
        {
          "url": "https://www.tiktok.com/shop/pdp/soft-funny-throw-blanket-cozy-fleece-for-home-movie-decor/1731291206388977735",
          "sold": "3.0K sold",
          "price": 14.39,
          "title": "Soft Funny Throw Blanket Horror Icons Flannel Fleece - Cozy Fuzzy Plush Comfy All-Season Sofa Bedroom Dorm Couch Office Car Travel Home Bedding,Comfortable Movie Decor Blanket Men Women Boys Girls Halloween Birthday Fan Gifts",
          "rating": 4.8,
          "section": "TikTok Picks",
          "currency": "$",
          "free_shipping": true,
          "original_price": 23.98
        },
        {
          "url": "https://www.tiktok.com/shop/pdp/40oz-sip-sip-hooray-halloween-tumblers-with-lid-straw-durable-glass/1729576207882752721",
          "sold": "6.1K sold",
          "price": 18.79,
          "title": "Sip-Sip Hooray! Halloween Tumbler – Cup with Lid & Straw, Spooky Season Iced Coffee Gift, Halloweentok 2024 Reusable Drinkware Insulated Sports Bottle Travel Water Bottle",
          "rating": 4.7,
          "section": "TikTok Picks",
          "currency": "$",
          "free_shipping": false,
          "original_price": 28.56
        },
        {
          "url": "https://www.tiktok.com/shop/pdp/1729574434882163435",
          "sold": "4.3K sold",
          "price": 28.79,
          "title": "Women's Halloween Themed All Over Hybrid Skull Knit Button Front Cardigan, Casual Drop Shoulder Long Sleeve V Neck Knitwear for Fall, Fashion Ladies' Knit Clothing for Daily Wear",
          "rating": 4.8,
          "section": "TikTok Picks",
          "currency": "$",
          "free_shipping": false,
          "original_price": 38.39
        },
        {
          "url": "https://www.tiktok.com/shop/pdp/soft-dragon-pattern-throw-blanket-by-brand-for-home-office-travel/1729625176913449761",
          "sold": "4.7K sold",
          "price": 9.15,
          "title": "Dragon Pattern Blanket, 1/2 Counts Soft Throw Blanket, Halloween Decor Warm Napping Blanket for Home Office Travel Camping Dormitory, Slogan Print Blanket",
          "rating": 4.9,
          "section": "TikTok Picks",
          "currency": "$",
          "free_shipping": false,
          "original_price": null
        },
        {
          "url": "https://www.tiktok.com/shop/pdp/diamond-art-roller-2025-dual-pack-for-smooth-pressing-craft-tools/1732001021448392940",
          "sold": null,
          "price": 5.3,
          "title": "1/2PCS Diamond Art Roller, 2025 New Diamond Painting Roller, Diamond Painting Tools, Essential DIY Craft Tool for Rhinestone Embroidery, Press Roller for Smooth Pressing, Crafting and Diamond Painting Projects",
          "rating": 5,
          "section": "Savings for you",
          "currency": "$",
          "free_shipping": false,
          "original_price": null
        },
        {
          "url": "https://www.tiktok.com/shop/pdp/kojic-acid-turmeric-toner-fur-gesicht-und-korper-250ml/1731391357821554923",
          "sold": "11.8K sold",
          "price": 23,
          "title": "[NEW] Kojic Acid Turmeric Toner for Face & Body | Daily Toner for Uneven Skin Tone & Texture | Niacinamide, Glycolic Acid for Clear Glass Skin | Korean Skin Care | 250ml",
          "rating": 4.7,
          "section": "Savings for you",
          "currency": "$",
          "free_shipping": false,
          "original_price": 27
        },
        {
          "url": "https://www.tiktok.com/shop/pdp/chrleisure-fleece-lined-high-waist-leggings-for-women-1-3pcs/1731711198249390235",
          "sold": "10.6K sold",
          "price": 29.99,
          "title": "【TikTokShopBlackFriday】CHRLEISURE 3PCS Thermal Thick FLeece Lined Leggings for Cold Winter Warm, High Waist Women's Cozy Workout Wear Fleece Pants for Yoga Riding Casual Wear",
          "rating": 4.7,
          "section": "Top deals for you",
          "currency": "$",
          "free_shipping": true,
          "original_price": 37.49
        },
        {
          "url": "https://www.tiktok.com/shop/pdp/dynasty-sweats-by-youngla-100-cotton-baggy-fit-unique-design/1731156978383884791",
          "sold": "1.4K sold",
          "price": 55,
          "title": "2116 - Dynasty Sweats",
          "rating": 4.7,
          "section": "Top deals for you",
          "currency": "$",
          "free_shipping": true,
          "original_price": null
        },
        {
          "url": "https://www.tiktok.com/shop/pdp/height-adjustable-swivel-chair-by-sweetcrispy-wide-ergonomic-design/1729408738017907174",
          "sold": "157.1K sold",
          "price": 66.98,
          "title": "【Star Furniture】Height Adjustable Criss Cross Chair with No Wheels / with Wheels- Office Wide Swivel Home Office Desk Chairs Reading Chair",
          "rating": 4.7,
          "section": "Popular items",
          "currency": "$",
          "free_shipping": true,
          "original_price": 115.99
        },
        {
          "url": "https://www.tiktok.com/shop/pdp/power-bank-10000mah-2-pack-by-asperx-with-22-5w-fast-charge-digital-display/1729568096320852893",
          "sold": "15.8K sold",
          "price": 18.99,
          "title": "AsperX 22.5W Fast Charging Power Bank, 2 pack 10000mAh Portable Charger with Digital Display, Battery Pack for Smartphone, Tablet, Perfect Gifts",
          "rating": 4.7,
          "section": "Popular items",
          "currency": "$",
          "free_shipping": true,
          "original_price": 79.99
        },
        {
          "url": "https://www.tiktok.com/shop/pdp/huloo-sleep-ultra-soft-memory-foam-play-mat-1-3-non-slip-velvet/1730123606468825995",
          "sold": "7.1K sold",
          "price": 29.39,
          "title": "Huloo Sleep Ultra Soft Memory Foam Play Mat for Tummy Time, 1.3\" Thick Non-Slip Crawling Mat, Non-Toxic Velvet Nursery Floor Mat",
          "rating": 4.7,
          "section": "Popular items",
          "currency": "$",
          "free_shipping": true,
          "original_price": 109.99
        },
        {
          "url": "https://www.tiktok.com/shop/pdp/based-skin-trio-100-pure-hypoclorous-acid-skincare-set/1731806435287666998",
          "sold": "5.6K sold",
          "price": 34.5,
          "title": "BASED Skincare Trio | Daily Skincare Duo + Skin Revival Spray | Facial Cleanser for Breakouts | Daily Moisturizer & Cleanser | For All Skin Types | Non-Toxic, Effective, Safe",
          "rating": 4.8,
          "section": "Popular items",
          "currency": "$",
          "free_shipping": true,
          "original_price": 58
        },
        {
          "url": "https://www.tiktok.com/shop/pdp/1731997923791180042",
          "sold": null,
          "price": 15.57,
          "title": "Autumn Winter Sexy Elegant Tight Fit Bodycon Dress, One-Step Skirt, Women's Fashion, Comfortable & Stylish, Perfect for Daily Wear（gift）",
          "rating": 15,
          "section": "Popular items",
          "currency": "$",
          "free_shipping": false,
          "original_price": 24.57
        },
        {
          "url": "https://www.tiktok.com/shop/pdp/energy-sticks-by-bloom-nutrition-10ct-180mg-natural-caffeine-zero-sugar/1730722027877602235",
          "sold": "20.8K sold",
          "price": 7.59,
          "title": "Bloom Nutrition Energy Sticks (10 Sticks) – Focus & Metabolism Support – Natural Caffeine & B Vitamins",
          "rating": 4.7,
          "section": "International top sellers",
          "currency": "$",
          "free_shipping": false,
          "original_price": 15.99
        },
        {
          "url": "https://www.tiktok.com/shop/pdp/funny-tee-i-hate-my-chungus-life-t-shirt-soft-cotton-fit/1731999740449296622",
          "sold": null,
          "price": 7.98,
          "title": "I hate my chungus life t-shirt (100% cotton), funny tee",
          "rating": 7,
          "section": "Best sellers",
          "currency": "$",
          "free_shipping": true,
          "original_price": 20.99
        },
        {
          "url": "https://www.tiktok.com/shop/pdp/hooded-plaid-jacket-for-women-by-asvivid-casual-long-sleeve-coat/1729535208059146968",
          "sold": "110.6K sold",
          "price": 37.04,
          "title": "Women's Plaid / Houndstooth Print Button Front Hooded Coat, Casual Longsleeves Pocket Coat for Fall & Winter, Ladies Outerwear for Daily Wear, Womenswear",
          "rating": 4.7,
          "section": "Best sellers",
          "currency": "$",
          "free_shipping": false,
          "original_price": 53.68
        },
        {
          "url": "https://www.tiktok.com/shop/pdp/christmas-themed-set-festive-practical-design/1731817636155789477",
          "sold": null,
          "price": 5.03,
          "title": "10/20/30Pcs Christmas Elements Four Color Pen Black, Blue, Red, Green Four Color Pen Santa Claus Snowman Christmas Tree Snowflake Deer Various Patterns Christmas Party Small Gifts Gift Fillers Smooth and Constant Oil (Mixed)",
          "rating": 5,
          "section": "Best sellers",
          "currency": "$",
          "free_shipping": false,
          "original_price": null
        }
      ]
    }
    ```

    
      * Some e-commerce sites show different listings by region. Use MrScraper's [proxy](/docs/features/proxy) to view results from other countries, or your own proxy if needed.
      * Schedule your scraper to run daily or weekly to keep your e-commerce data up to date.
    
  

  
    ## Use the Extracted Data

    Now that you have successfully scraped and filtered the e-commerce product data, you can use this information for various purposes such as market analysis, price comparison, inventory management, or even feeding it into your own applications or databases. You can export the data in formats like JSON or CSV for easy integration with other tools.
  



# Job Listing

import { Step, Steps } from 'fumadocs-ui/components/steps';

Extracting job listings manually from job boards can be tedious and time-consuming. With **MrScraper’s AI Scraper**, you can automate this process entirely — providing only a URL and a short prompt describing what to extract.

In this guide, we’ll walk through a **real-world example** where a recruitment team uses MrScraper to gather and analyze data from a live job board to inform their hiring strategy.

## Scenario

Imagine you’re an HR specialist at a growing tech startup planning to hire a **Data Scientist**.\
Before posting your own job, you want to understand:

* What skills and qualifications are most in-demand
* How other companies describe their Data Scientist roles
* Which cities or companies are hiring for similar positions


  
    ## Create a New AI Scraper

    1. Go to your **MrScraper Dashboard**.
    2. Enter the URL: `https://www.glassdoor.com/Job/indonesia-data-scientist-jobs-SRCH_IL.0,9_IN113_KO10,25.htm`.
    3. Select **Super Mode** to get more accurate and structured extraction for job listings.
  

  
    ## Filter Out

    Next, tell the AI exactly what kind of job data you want to capture. The prompt defines both the structure of your results and any filters to apply. For this use case, you can use the following prompt:

    ```md
    Extract only job listings with the job title "Data Scientist" that are suitable for fresh graduates or candidates with a minimum of 2 years of experience, and include required skills details.

    For each listing, return these fields:
    - Job Title
    - Company Name
    - Location
    - Salary
    - Job Type
    - Experience Level
    - Posted Date
    - Job Description
    - Required Skills
    - Job URL

    Filter out any listing that does not include required skills information or requires more than 2 years of experience.
    ```

    Here’s a sample output:

    ```json title="Scraping Result"
    {
      "mode": "direct",
      "results": [
        {
          "Salary": null,
          "Job URL": "https://www.glassdoor.com/job-listing/it-data-scientist-apotek-k-24-JV_KO0,17_KE18,29.htm?jl=1009644720583",
          "Job Type": null,
          "Location": "Indonesia",
          "Job Title": "IT Data Scientist",
          "Posted Date": "30d+",
          "Company Name": "Apotek K 24",
          "Job Description": "Mengolah dan menganalisis data besar dari berbagai sumber (POS, CRM, marketplace, dll) untuk menemukan pola dan insight bisnis. Mengembangkan model statistik dan machine learning (supervised dan unsupervised) untuk kebutuhan forecasting, recommendation, atau clustering. Melakukan feature engineering dan model evaluation untuk memastikan performa model optimal. Berkolaborasi dengan tim Data Engineering untuk integrasi pipeline data dan deployment model. Berkoordinasi dengan tim BI Analyst untuk penerjemahan hasil analitik ke dalam dashboard dan laporan bisnis. Melakukan eksperimen model (A/B testing, hyperparameter tuning) dan dokumentasi hasilnya. Menjaga data quality, model reproducibility, serta model governance sesuai kebijakan perlindungan data perusahaan. Minimum Qualifications: Menguasai Python (NumPy, Pandas, Scikit-learn, TensorFlow/PyTorch, Matplotlib/Seaborn). Memahami konsep statistik, machine learning, dan data preprocessing. Pengalaman dengan SQL dan query optimization. Familiar dengan tools visualisasi (Power BI, Tableau, atau setara). Pengalaman dalam mengelola version control (Git) dan ML lifecycle tools (MLflow, DVC, dsb) menjadi nilai tambah. Memahami time series forecasting, NLP, atau recommender system menjadi nilai plus. Komunikatif dan kolaboratif lintas tim (Engineering, BI, Business Unit). Kompas Gramedia, through its more than 50 years of history, is striving for one goal: enlightening and empowering Indonesia. To ensure that we are able to serve the nation for another 50 years, we are undergoing a digital transformation; strengthening and expanding our solid business pillars by developing new digital business initiatives. Our vision is to enlight all the people in Indonesia with all the knowledge we have.",
          "Required Skills": [
            "NumPy",
            "Pandas",
            "Scikit-learn",
            "TensorFlow",
            "PyTorch",
            "Matplotlib",
            "Seaborn",
            "Python",
            "machine learning",
            "data preprocessing",
            "dengan SQL",
            "query optimization",
            "Power BI",
            "Tableau",
            "Git",
            "MLflow",
            "DVC",
            "time series forecasting",
            "NLP",
            "Engineering",
            "BI",
            "Business Unit"
          ],
          "Experience Level": "2 years"
        }
      ]
    }
    ```

    
      * Some job sites show different listings by region. Use MrScraper’s [proxy](/docs/features/proxy) to view results from other countries, or your own proxy if needed.
      * Schedule your scraper to run daily or weekly to keep your job data up to date.
    
  

  
    ## Use the Extracted Data

    Once you’ve collected the job listings, you can export and use the data directly from the dashboard in your preferred format.

    You can use these data for:

    * **Job posting optimization**: Identify trending keywords or skills that attract top candidates.

    * **Competitor analysis**: See how similar companies describe their open roles.

    * **Internal reporting**: Share hiring trends or skill gaps with your HR or analytics team.
  



# Optimizing Cost

Running scrapers efficiently helps you get the most value out of MrScraper without unnecessary usage costs.\
This guide explains how to **optimize scraping costs** through smart configuration, scraper mode selection, scheduling, and output management.

## 1. Choose the Right Scraper Mode

MrScraper offers two AI Scraper modes that affect both cost and performance:

| Mode           | Description                                                                            | When to Use                                                                     | Cost Impact |
| -------------- | -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | ----------- |
| **Cheap Mode** | Fast and lightweight extraction. Suitable for simple, structured pages.                | Use for lists or data with consistent structure (e.g., product names, prices).  | 💲 Lowest   |
| **Super Mode** | More accurate and intelligent extraction. Ideal for complex layouts or nested content. | Use for unstructured data or multiple data types (e.g., job listings, reviews). | 💲💲 Higher |


  The cost difference between Cheap Mode and Super Mode depends on the target website. For websites that are easy to access and extract from, both modes may have similar costs. On more complex websites, Super Mode may perform additional attempts and processing to successfully extract data, which can increase token usage and overall cost.


## 2. Limit Extraction Scope

The broader your prompt, the more resources are used.\
Narrowing your extraction target saves cost and improves precision.

| ❌ *Expensive prompt*                           | ✅ *Optimized prompt*                                    |
| ---------------------------------------------- | ------------------------------------------------------- |
| Extract all details from this e-commerce page. | Extract product name, price, and rating from this page. |


  Focus on the **fields you actually need**. If you only need names and prices, don’t extract full descriptions or metadata.


## 3. Reuse and Recycle Results

Each time you run a scraper, you pay for the extraction process.\
Avoid re-running scrapers unnecessarily by reusing existing results.

**Example Workflow:**

* Store your output in a **Database Connection** or export as **CSV/JSON**.
* Only re-run the scraper when the target site changes.
* Use **Scheduling** for incremental updates instead of full refreshes.

## 4. Use Scheduling Strategically

Scraping too frequently can increase costs without adding value.\
Instead, match your schedule to how often the target data changes.


  Avoid overlapping schedules. Running multiple scrapers at the same time can spike costs and reduce performance.


## 5. Optimize Proxy Usage

While proxies add flexibility, they can slightly increase scraping overhead.

* Use **MrScraper’s built-in proxy** instead of external providers to save cost and setup time.
* Only enable proxies for websites that require regional access or IP rotation.
* For internal or low-risk pages, disable proxy entirely.


  Disabling proxies may change the content returned by websites that serve region-specific data. If you need accurate local pricing, product availability, or search results, use a proxy or Geo Targeting for the target region.


## 6. Monitor Usage

Keep an eye on your scraper usage and performance using the **Analytics** feature.\
Regularly review:

* The number of pages or URLs processed
* Success rate and data completeness
* AI vs. workflow distribution

This helps you spot overuse early and plan your scraping strategy efficiently.


  Start small. Run initial tests on limited pages using **Cheap Mode**, verify output accuracy, and scale up gradually.


## 7. Leverage Reruns for Lower Costs

If you need to scrape a website repeatedly, use **Rerun** instead of creating a new scraper each time.

MrScraper uses caching to optimize repeated scraping tasks. When you rerun an existing scraper, the platform can reuse previously processed information, often reducing both token usage and execution time compared to a new scraper.

Reruns are ideal for:

* Refreshing previously collected data
* Monitoring pages for updates
* Scheduled scraping workflows
* Recurring extraction from the same website

## 8. Cap Retry Spending with Max Retries and Token Cap

A failed scrape that keeps retrying is one of the easiest ways to spend tokens without getting data back. In the [Scraper API](/docs/getting-started/scraper-api), the **Basic Settings** panel gives you two limits that bound the cost of a single run.

Turn on **Retry** first, then set either or both:

| Setting         | Description                                                           |
| --------------- | --------------------------------------------------------------------- |
| **Max retries** | Maximum number of retry attempts after the initial request.           |
| **Token cap**   | Maximum total tokens the initial request and all its retries can use. |

Retrying stops at whichever comes first: the scrape succeeds, it reaches **Max retries**, or the running token total reaches the **Token cap**. Each attempt's cost is rounded up and added to the running total.

**How to use them to control cost:**

* **Set a token cap when testing an unfamiliar site.** It puts a hard ceiling on a single run, so a page that fails repeatedly can't quietly consume your balance.
* **Lower Max retries on sites that fail consistently.** If a page is blocked, the third and fourth attempt usually fail the same way as the first. Fix the cause with **Super** mode or Geo targeting instead of paying for more attempts.
* **Leave room for at least one retry.** If the first attempt alone costs more than the cap, the scrape runs once and doesn't retry at all. On pages with variable load times or heavy bandwidth, set the cap to a few times your expected per-run cost.


  Run a scrape once with no cap to see its typical token cost, then set the cap based on that number rather than guessing.


For worked examples of how the cap interacts with retries, see [Scraper API Token Cap](/docs/getting-started/api-token#scraper-api-token-cap) in Token Plan.


# Create and Run a Scraper Programmatically

import { Step, Steps } from 'fumadocs-ui/components/steps';

## Overview

This guide shows you how to create an AI scraper in the dashboard and then use it programmatically through the API. This is perfect if you want to set up a scraper once and automate data extraction with simple API calls—no need to configure it every time.

## Prerequisites

Before you start, make sure you have:

* A **MrScraper API token** for authentication. If you don’t have one yet, follow this [guide to create an API token](/docs/getting-started/api-token).


  
    ## Verify Your API Token

    You can verify your API token and check your account status using the [`/subscription-accounts`](/docs/api/v3/authentication/verify-token):

    
      
        ```bash
        curl --location 'https://api.app.mrscraper.com/api/v1/subscription-accounts' \
        --header 'accept: */*' \
        --header 'x-api-token: '
        ```
      

      
        ```javascript
        const token = "MRSCRAPER_API_TOKEN";

        fetch("https://api.app.mrscraper.com/api/v1/subscription-accounts", {
          method: "GET",
          headers: {
            "accept": "*/*",
            "x-api-token": token
          }
        })
        .then(res => res.json())
        .then(data => console.log(data))
        .catch(err => console.error(err));
        ```
      

      
        ```python
        import requests

        token = "MRSCRAPER_API_TOKEN"

        headers = {
            "accept": "*/*",
            "x-api-token": token
        }

        response = requests.get(
            "https://api.app.mrscraper.com/api/v1/subscription-accounts",
            headers=headers
        )

        print(response.json())
        ```
      
    

    **Response (200 OK):**

    ```json
    {
        "message": "Successful operation!",
        "data": {
            "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
            "tokenLimit": 10000,
            "tokenUsage": 994,
            "stripeSubscriptionId": "sub_xxxxxxxxxxxxxxxxxx",
            "userId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
            "stripeStatus": "true",
            "quantity": 1,
            "endsAt": "2030-12-01T02:05:08.400Z",
            "subscriptionItemId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
            "user": {
                "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
                "createdAt": "2025-11-24T01:59:15.866Z",
                "createdById": null,
                "updatedAt": "2026-01-17T08:11:43.270Z",
                "updatedById": null,
                "deletedAt": null,
                "deletedById": null,
                "name": "John Doe",
                "email": "johndoe@example.com",
                "username": null,
                "latestApiToken": "atk_xxxx....",
                "gender": null,
                "phoneNumber": null,
                "address": null,
                "birthDate": null,
                "avatar": "https://api.app.mrscraper.com/images/users/default-avatar.png",
                "otp": null,
                "otpExpiredAt": null,
                "isVerified": true,
                "stripeCustomerId": "cus_xxxxxxxxx",
                "googleMail": null,
                "googleId": null,
                "s3Bucket": null,
                "domainTargets": [
                    "https://books.toscrape.com"
                ]
            },
            "rateLimit": null,
            "rateTtl": null,
            "isEnterprise": true,
            "cancelAtPeriodEnd": null,
            "currentPeriodStart": null,
            "currentPeriodEnd": null,
            "proxyDollarPerGB": null,
            "isAutoRenew": false,
            "updatedAt": "2026-01-18T01:40:21.392Z",
            "createdAt": "2025-11-24T02:05:08.470Z"
        }
    }
    ```

    **Key Response Fields:**

    **Subscription Account Fields:**

    | Field                  | Type         | Description                                        |
    | ---------------------- | ------------ | -------------------------------------------------- |
    | `id`                   | string       | Unique identifier for your subscription account    |
    | `tokenLimit`           | integer      | Maximum tokens available in your subscription plan |
    | `tokenUsage`           | integer      | Total tokens consumed so far                       |
    | `stripeSubscriptionId` | string       | Stripe subscription identifier (for billing)       |
    | `userId`               | string       | Your unique user identifier                        |
    | `stripeStatus`         | string       | Payment status (`"true"` = active)                 |
    | `quantity`             | integer      | Subscription quantity/seats                        |
    | `endsAt`               | string       | Subscription expiration date (ISO 8601 format)     |
    | `subscriptionItemId`   | string       | Stripe subscription item identifier                |
    | `rateLimit`            | integer/null | API rate limit (requests per window)               |
    | `rateTtl`              | integer/null | Rate limit time window in seconds                  |
    | `isEnterprise`         | boolean      | Whether account has enterprise features            |
    | `cancelAtPeriodEnd`    | boolean/null | If subscription will cancel at period end          |
    | `currentPeriodStart`   | string/null  | Current billing period start date                  |
    | `currentPeriodEnd`     | string/null  | Current billing period end date                    |
    | `proxyDollarPerGB`     | number/null  | Custom proxy pricing (enterprise only)             |
    | `isAutoRenew`          | boolean      | Whether subscription auto-renews                   |

    **User Object Fields (nested under `data.user`):**

    | Field              | Type    | Description                                        |
    | ------------------ | ------- | -------------------------------------------------- |
    | `id`               | string  | Unique user identifier                             |
    | `name`             | string  | User's full name                                   |
    | `email`            | string  | User's email address                               |
    | `latestApiToken`   | string  | Your most recently generated API token (truncated) |
    | `avatar`           | string  | URL to user's profile picture                      |
    | `isVerified`       | boolean | Whether email is verified                          |
    | `stripeCustomerId` | string  | Stripe customer identifier                         |
    | `domainTargets`    | array   | List of domains you've scraped                     |
    | `createdAt`        | string  | Account creation timestamp                         |
    | `updatedAt`        | string  | Last account update timestamp                      |

    
      **Checking Token Balance:** Use `tokenLimit - tokenUsage` to calculate your remaining tokens. Plan your scraping operations accordingly to avoid running out of tokens.
    
  

  
    ## Collect URLs from Website

    Collect all relevant URLs from your target website using the AI scraper endpoint. You can select the Agent type you want to use from the example below:

    ### Initial Scrape using General Agent

    
      
        #### Get All URL

        Use the [`/scrapers-ai`](/docs/api/v3/scraper/ai/ai-init) endpoint to create a [General agent](/docs/features/ai-scraper/general) scraper to get only the URLs within the provided link.

        
          
            ```bash
            curl --location 'https://api.app.mrscraper.com/api/v1/scrapers-ai' \
            --header 'accept: application/json' \
            --header 'x-api-token: ' \
            --header 'content-type: application/json' \
            --data '{
                "url": "https://books.toscrape.com",
                "agent": "general",
                "message": "Extract all data and just only get the urls, include patterns https://books.toscrape.com/catalogue and for exclude patterns https://books.toscrape.com/catalogue/category",
                "proxyCountry": ""
            }'
            ```
          

          
            ```javascript
            const token = "MRSCRAPER_API_TOKEN";

            fetch("https://api.app.mrscraper.com/api/v1/scrapers-ai", {
              method: "POST",
              headers: {
                "accept": "application/json",
                "x-api-token": token,
                "content-type": "application/json"
              },
              body: JSON.stringify({
                url: "https://books.toscrape.com",
                agent: "general",
                message: "Extract all data and just only get the urls, include patterns https://books.toscrape.com/catalogue and for exclude patterns https://books.toscrape.com/catalogue/category",
                proxyCountry: ""
              })
            })
            .then(res => res.json())
            .then(data => console.log(data))
            .catch(err => console.error(err));
            ```
          

          
            ```python
            import requests

            token = "MRSCRAPER_API_TOKEN"

            url = "https://api.app.mrscraper.com/api/v1/scrapers-ai"
            headers = {
                "accept": "application/json",
                "x-api-token": token,
                "content-type": "application/json"
            }
            payload = {
                "url": "https://books.toscrape.com",
                "agent": "general",
                "message": "Extract all data and just only get the urls, include patterns https://books.toscrape.com/catalogue and for exclude patterns https://books.toscrape.com/catalogue/category",
                "proxyCountry": ""
            }

            response = requests.post(url, headers=headers, json=payload)
            print(response.json())
            ```
          
        
      

      
        #### Get Page Details

        After collecting URLs, you'll want to extract detailed data from each individual page.

        
          When creating a detail scraper, always use a real detail page URL.

          For example, instead of [https://books.toscrape.com](https://books.toscrape.com), use a specific product page like [https://books.toscrape.com/catalogue/a-light-in-the-attic\_1000/index.html](https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html) (from your URL collection step).

          Do not use the homepage or listing page—this helps ensure accurate data extraction and prevents setup errors.
        

        Create a new scraper configured for detail extraction:

        
          
            ```bash
            curl --location 'https://api.app.mrscraper.com/api/v1/scrapers-ai' \
            --header 'accept: application/json' \
            --header 'x-api-token: ' \
            --header 'content-type: application/json' \
            --data '{
                "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
                "agent": "general",
                "message": "Extract data : Title, Product Description, Price (Float), Availability, and Number of reviews",
                "proxyCountry": ""
            }'
            ```
          

          
            ```javascript
            const token = "MRSCRAPER_API_TOKEN";

            fetch("https://api.app.mrscraper.com/api/v1/scrapers-ai", {
              method: "POST",
              headers: {
                "accept": "application/json",
                "x-api-token": token,
                "content-type": "application/json"
              },
              body: JSON.stringify({
                url: "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
                agent: "general",
                message: "Extract data : Title, Product Description, Price (Float), Availability, and Number of reviews",
                proxyCountry: ""
              })
            })
            .then(res => res.json())
            .then(data => console.log(data))
            .catch(err => console.error(err));
            ```
          

          
            ```python
            import requests

            token = "MRSCRAPER_API_TOKEN"

            url = "https://api.app.mrscraper.com/api/v1/scrapers-ai"
            headers = {
                "accept": "application/json",
                "x-api-token": token,
                "content-type": "application/json"
            }
            payload = {
                "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
                "agent": "general",
                "message": "Extract data : Title, Product Description, Price (Float), Availability, and Number of reviews",
                "proxyCountry": ""
            }

            response = requests.post(url, headers=headers, json=payload)
            print(response.json())
            ```
          
        

        **Response (200 OK):**

        ```json
        {
          "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
          "createdAt": "2026-01-18T08:37:38.124Z",
          "userId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxx...",
          "scraperId": "YOUR_DETAIL_SCRAPER_ID",
          "type": "AI",
          "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
          "status": "Finished",
          "tokenUsage": 5,
          "data": {
            "title": "A Light in the Attic",
            "price": "£51.77",
            "availability": "In stock (22 available)",
            "description": "It's hard to imagine a world without A Light in the Attic...",  
            "number_of_reviews": "0"
          }
        }
        ```

        **Request Parameters:**

        | Parameter      | Type   | Required | Description                                                   |
        | -------------- | ------ | -------- | ------------------------------------------------------------- |
        | `url`          | string | Yes      | Target website URL to scrape                                  |
        | `agent`        | string | Yes      | AI agent to use (e.g., `general`, `listing`, `detail`, `map`) |
        | `message`      | string | Yes      | Natural language instruction describing what to extract       |
        | `proxyCountry` | string | No       | Proxy country code for geo-restricted content                 |

        
          You'll need this ID (`YOUR_DETAIL_SCRAPER_ID` in this example) for bulk scraping multiple URLs with the same extraction configuration.
        
      
    

    ### Initial Scrape with Map Agent

    Use the [`/scrapers-ai`](/docs/api/v3/scraper/ai/ai-init) endpoint to create a Map agent scraper.

    
      
        ```bash
        curl --location 'https://api.app.mrscraper.com/api/v1/scrapers-ai' \
        --header 'accept: application/json' \
        --header 'x-api-token: ' \
        --header 'content-type: application/json' \
        --data '{
            "agent": "map",
            "url": "https://books.toscrape.com",
            "maxDepth": 2,
            "maxPages": 100,
            "limit": 1000,
            "includePatterns": "^https:\\/\\/books\\.toscrape\\.com\\/catalogue$",
            "excludePatterns": "^https:\\/\\/books\\.toscrape\\.com\\/catalogue\\/category$"
        }'
        ```
      

      
        ```javascript
        const token = "MRSCRAPER_API_TOKEN";

        fetch("https://api.app.mrscraper.com/api/v1/scrapers-ai", {
          method: "POST",
          headers: {
            "accept": "application/json",
            "x-api-token": token,
            "content-type": "application/json"
          },
          body: JSON.stringify({
            agent: "map",
            url: "https://books.toscrape.com",
            maxDepth: 2,
            maxPages: 100,
            limit: 1000,
            includePatterns: "^https:\\/\\/books\\.toscrape\\.com\\/catalogue$",
            excludePatterns: "^https:\\/\\/books\\.toscrape\\.com\\/catalogue\\/category$"
          })
        })
        .then(res => res.json())
        .then(data => console.log(data))
        .catch(err => console.error(err));
        ```
      

      
        ```python
        import requests

        token = "MRSCRAPER_API_TOKEN"

        url = "https://api.app.mrscraper.com/api/v1/scrapers-ai"
        headers = {
            "accept": "application/json",
            "x-api-token": token,
            "content-type": "application/json"
        }
        payload = {
            "agent": "map",
            "url": "https://books.toscrape.com",
            "maxDepth": 2,
            "maxPages": 100,
            "limit": 1000,
            "includePatterns": "^https:\\/\\/books\\.toscrape\\.com\\/catalogue$",
            "excludePatterns": "^https:\\/\\/books\\.toscrape\\.com\\/catalogue\\/category$"
        }

        response = requests.post(url, headers=headers, json=payload)
        print(response.json())
        ```
      
    

    **Request Parameters:**

    | Parameter         | Type   | Required | Description                                             |
    | ----------------- | ------ | -------- | ------------------------------------------------------- |
    | `agent`           | string | Yes      | Must be `"map"` for map agent                           |
    | `url`             | string | Yes      | Target website URL to scrape                            |
    | `maxDepth`        | number | Yes      | Maximum depth to crawl (e.g., `2`)                      |
    | `maxPages`        | number | Yes      | Maximum number of pages to crawl (e.g., `100`)          |
    | `limit`           | number | Yes      | Maximum number of URLs to collect (e.g., `1000`)        |
    | `includePatterns` | string | No       | Regex pattern for URLs to include                       |
    | `excludePatterns` | string | No       | Regex pattern for URLs to exclude (can be empty string) |

    **Response (200 OK):**

    ```json
    {
      "message": "Successful operation!",
      "data": {
        "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxx...",
        "createdAt": "2026-01-18T05:01:24.944Z",
        "userId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxx...",
        "scraperId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxx...",
        "type": "AI",
        "url": "https://books.toscrape.com",
        "status": "Finished",
        "error": null,
        "tokenUsage": 5,
        "data": {
          "urls": [
            "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
            "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
            "https://books.toscrape.com/catalogue/soumission_998/index.html",
            ...
          ]
        }
      }
    }
    ```

    
      You'll need this ID to re-run the scraper later without reconfiguring it.
    

    ### Bulk Scrape Multiple URLs

    Use the [`/scrapers-ai-rerun/bulk`](/docs/api/v3/scraper/ai/ai-rerun-bulk) endpoint to scrape multiple URLs in a single request:

    
      The bulk endpoint is available only for the General agent.
    

    
      
        ```bash
        curl --location 'https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun/bulk' \
        --header 'accept: application/json' \
        --header 'x-api-token: ' \
        --header 'content-type: application/json' \
        --data '{
            "scraperId": "YOUR_DETAIL_SCRAPER_ID",
            "urls": [
                "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
                "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
                "https://books.toscrape.com/catalogue/soumission_998/index.html",
                ...
            ]
        }'
        ```
      

      
        ```javascript
        const token = "MRSCRAPER_API_TOKEN";
        const detailScraperId = "YOUR_DETAIL_SCRAPER_ID";
        const urls = [
          "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
          "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
          "https://books.toscrape.com/catalogue/soumission_998/index.html"
        ];

        fetch("https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun/bulk", {
          method: "POST",
          headers: {
            "accept": "application/json",
            "x-api-token": token,
            "content-type": "application/json"
          },
          body: JSON.stringify({
            scraperId: detailScraperId,
            urls: urls
          })
        })
        .then(res => res.json())
        .then(data => console.log(data))
        .catch(err => console.error(err));
        ```
      

      
        ```python
        import requests

        token = "MRSCRAPER_API_TOKEN"
        detail_scraper_id = "YOUR_DETAIL_SCRAPER_ID"
        urls = [
            "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
            "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
            "https://books.toscrape.com/catalogue/soumission_998/index.html",
            ...
        ]

        url = "https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun/bulk"
        headers = {
            "accept": "application/json",
            "x-api-token": token,
            "content-type": "application/json"
        }
        payload = {
            "scraperId": detail_scraper_id,
            "urls": urls
        }

        response = requests.post(url, headers=headers, json=payload)
        print(response.json())
        ```
      
    

    **Request Parameters:**

    | Parameter   | Type   | Required | Description                       |
    | ----------- | ------ | -------- | --------------------------------- |
    | `scraperId` | string | Yes      | The detail scraper ID from Step 4 |
    | `urls`      | array  | Yes      | Array of URLs to scrape           |

    **Response (200 OK):**

    ```json
    {
      "message": "Bulk scraping is Running",
      "data": {
        "bulkResultId": "YOUR_BULK_RESULT_ID"
      }
    }
    ```

    
      Bulk scraping runs in the background. Save the `bulkResultId` to retrieve results later.
    
  

  
    ## Retrieve Results

    There are two ways to retrieve scraping results: get a specific result by its ID, or get all results for a scraper.

    ### Get Single Result by Result ID

    Use the [`/results/{id}`](/docs/api/v3/result/all) endpoint to get the latest single result using a specific `resultId` (the `id` field from the scrape response):

    
      
        ```bash
        curl --location 'https://api.app.mrscraper.com/api/v1/results/YOUR_RESULT_ID' \
        --header 'accept: application/json' \
        --header 'x-api-token: '
        ```
      

      
        ```javascript
        const token = "MRSCRAPER_API_TOKEN";
        const resultId = "YOUR_RESULT_ID";

        fetch(`https://api.app.mrscraper.com/api/v1/results/${resultId}`, {
          method: "GET",
          headers: {
            "accept": "application/json",
            "x-api-token": token
          }
        })
        .then(res => res.json())
        .then(data => console.log(data))
        .catch(err => console.error(err));
        ```
      

      
        ```python
        import requests

        token = "MRSCRAPER_API_TOKEN"
        result_id = "YOUR_RESULT_ID"

        url = f"https://api.app.mrscraper.com/api/v1/results/{result_id}"
        headers = {
            "accept": "application/json",
            "x-api-token": token
        }

        response = requests.get(url, headers=headers)
        print(response.json())
        ```
      
    

    **Response (200 OK):**

    ```json
    {
      "message": "Successful operation!",
      "data": {
        "createdAt": "2026-01-18T07:28:44.725Z",
        "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxx...",
        "userId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxx...",
        "scraperId": "YOUR_SCRAPER_ID",
        "type": "Rerun-AI",
        "url": "https://books.toscrape.com",
        "status": "Finished",
        "error": "",
        "tokenUsage": 4,
        "data": {
          "urls": [
            "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
            "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
            "https://books.toscrape.com/catalogue/soumission_998/index.html",
            ...
          ]
        },
        "htmlPath": "results/.../page.html",
        "screenshotPath": "results/.../page.jpg"
      }
    }
    ```

    ### Get All Results by Scraper ID

    Use the [`/results`](/docs/api/v3/result/detail) endpoint to retrieve all historical results for a specific scraper with pagination:

    
      
        ```bash
        curl --location --globoff 'https://api.app.mrscraper.com/api/v1/results?filters[scraperId]=YOUR_SCRAPER_ID&page=1&pageSize=10&sort=createdAt&sortOrder=DESC' \
        --header 'accept: application/json' \
        --header 'x-api-token: '
        ```
      

      
        ```javascript
        const token = "MRSCRAPER_API_TOKEN";
        const scraperId = "YOUR_SCRAPER_ID";

        const params = new URLSearchParams({
          'filters[scraperId]': scraperId,
          'page': '1',
          'pageSize': '10',
          'sort': 'createdAt',
          'sortOrder': 'DESC'
        });

        fetch(`https://api.app.mrscraper.com/api/v1/results?${params}`, {
          method: "GET",
          headers: {
            "accept": "application/json",
            "x-api-token": token
          }
        })
        .then(res => res.json())
        .then(data => console.log(data))
        .catch(err => console.error(err));
        ```
      

      
        ```python
        import requests

        token = "MRSCRAPER_API_TOKEN"
        scraper_id = "YOUR_SCRAPER_ID"

        url = "https://api.app.mrscraper.com/api/v1/results"
        headers = {
            "accept": "application/json",
            "x-api-token": token
        }
        params = {
            "filters[scraperId]": scraper_id,
            "page": 1,
            "pageSize": 10,
            "sort": "createdAt",
            "sortOrder": "DESC"
        }

        response = requests.get(url, headers=headers, params=params)
        print(response.json())
        ```
      
    

    **Query Parameters:**

    | Parameter            | Type    | Required | Description                          |
    | -------------------- | ------- | -------- | ------------------------------------ |
    | `filters[scraperId]` | string  | Yes      | Filter results by scraper ID         |
    | `page`               | integer | No       | Page number (default: 1)             |
    | `pageSize`           | integer | No       | Results per page (default: 10)       |
    | `sort`               | string  | No       | Field to sort by (e.g., `createdAt`) |
    | `sortOrder`          | string  | No       | Sort direction: `ASC` or `DESC`      |

    **Response (200 OK):**

    ```json
    {
      "message": "Successful fetch",
      "data": [
        {
          "createdAt": "2026-01-18T07:28:44.725Z",
          "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxx...",
          "scraperId": "YOUR_SCRAPER_ID",
          "type": "Rerun-AI",
          "url": "https://books.toscrape.com",
          "status": "Finished",
          "tokenUsage": 4,
          "data": {
            "urls": ["..."]
          }
        },
        {
          "createdAt": "2026-01-18T07:28:04.740Z",
          "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxx...",
          "scraperId": "YOUR_SCRAPER_ID",
          "type": "AI",
          "url": "https://books.toscrape.com",
          "status": "Finished",
          "tokenUsage": 5,
          "data": {
            "urls": ["..."]
          }
        }
      ],
      "meta": {
        "page": 1,
        "pageSize": 10,
        "total": 2,
        "totalPage": 1
      }
    }
    ```

    ### Retrieve Bulk Scraping Results

    Bulk scraping is asynchronous, meaning you need to poll the results endpoint to check the status and retrieve the data when processing is complete.

    
      
        ```bash
        curl --location 'https://api.app.mrscraper.com/api/v1/results/YOUR_BULK_RESULT_ID' \
        --header 'accept: application/json' \
        --header 'x-api-token: YOUR_API_TOKEN'
        ```
      

      
        ```javascript
        const token = "YOUR_API_TOKEN";
        const bulkResultId = "YOUR_BULK_RESULT_ID";

        fetch(`https://api.app.mrscraper.com/api/v1/results/${bulkResultId}`, {
          method: "GET",
          headers: {
            "accept": "application/json",
            "x-api-token": token
          }
        })
        .then(res => res.json())
        .then(data => console.log(data))
        .catch(err => console.error(err));
        ```
      

      
        ```python
        import requests

        token = "MRSCRAPER_API_TOKEN"
        bulk_result_id = "YOUR_BULK_RESULT_ID"

        url = f"https://api.app.mrscraper.com/api/v1/results/{bulk_result_id}"
        headers = {
            "accept": "application/json",
            "x-api-token": token
        }

        response = requests.get(url, headers=headers)
        print(response.json())
        ```
      
    

    **Response (Status: Running):**

    ```json
    {
      "message": "Successful operation!",
      "data": {
        "id": "YOUR_BULK_RESULT_ID",
        "status": "Running",
        "data": null
      }
    }
    ```

    **Response (Status: Finished):**

    ```json
    {
      "message": "Successful operation!",
      "data": {
        "createdAt": "2026-01-18T08:38:23.993Z",
        "id": "YOUR_BULK_RESULT_ID",
        "userId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        "scraperId": "YOUR_DETAIL_SCRAPER_ID",
        "type": "Bulk-AI",
        "status": "Finished",
        "tokenUsage": 12,
        "data": {
          "mergedData": [
            {
              "title": "A Light in the Attic",
              "price": "£51.77",
              "availability": "In stock (22 available)",
              "description": "It's hard to imagine a world without A Light in the Attic...",
              "product_type": "Books",
              "upc": "a897fe39b1053632",
              "price_excl_tax": "£51.77",
              "price_incl_tax": "£51.77",
              "tax": "£0.00",
              "number_of_reviews": "0"
            },
            {
              "title": "Tipping the Velvet",
              "price": "£53.74",
              "availability": "In stock (20 available)",
              "description": "Erotic and absorbing...Written with starling power...",
              "product_type": "Books",
              "upc": "90fa61229261140a",
              "price_excl_tax": "£53.74",
              "price_incl_tax": "£53.74",
              "tax": "£0.00",
              "number_of_reviews": "0"
            },
            {
              "title": "Soumission",
              "price": "£50.10",
              "availability": "In stock (20 available)",
              "description": "Dans une France assez proche de la nôtre...",
              "product_type": "Books",
              "upc": "6957f44c3847a760",
              "price_excl_tax": "£50.10",
              "price_incl_tax": "£50.10",
              "tax": "£0.00",
              "number_of_reviews": "0"
            }
          ],
          "urlDetails": [
            {
              "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
              "status": "Finished",
              "error": ""
            },
            {
              "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
              "status": "Finished",
              "error": ""
            },
            {
              "url": "https://books.toscrape.com/catalogue/soumission_998/index.html",
              "status": "Finished",
              "error": ""
            }
          ],
          "summary": {
            "totalUrls": 3,
            "successfulUrls": 3,
            "failedUrls": 0,
            "totalTokenUsage": 12
          }
        }
      }
    }
    ```

    **Key Response Fields:**

    | Field             | Type   | Description                                        |
    | ----------------- | ------ | -------------------------------------------------- |
    | `status`          | string | Current status: `Running`, `Finished`, or `Failed` |
    | `data.mergedData` | array  | Array of extracted data from all URLs              |
    | `data.urlDetails` | array  | Status details for each individual URL             |
    | `data.summary`    | object | Summary statistics of the bulk operation           |
  

  
    ## Re-run Scraper (Optional)

    If you want to scrape the same website again with the same configuration, use the [`/scrapers-ai-rerun`](/docs/api/v3/scraper/ai/ai-rerun) endpoint with your saved `scraperId`:

    ### Re-run General Agent

    
      
        ```bash
        curl --location 'https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun' \
        --header 'accept: application/json' \
        --header 'x-api-token: ' \
        --header 'content-type: application/json' \
        --data '{
            "scraperId": "YOUR_SCRAPER_ID",
            "url": "https://books.toscrape.com"
        }'
        ```
      

      
        ```javascript
        const token = "MRSCRAPER_API_TOKEN";
        const scraperId = "YOUR_SCRAPER_ID";

        fetch("https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun", {
          method: "POST",
          headers: {
            "accept": "application/json",
            "x-api-token": token,
            "content-type": "application/json"
          },
          body: JSON.stringify({
            scraperId: scraperId,
            url: "https://books.toscrape.com"
          })
        })
        .then(res => res.json())
        .then(data => console.log(data))
        .catch(err => console.error(err));
        ```
      

      
        ```python
        import requests

        token = "MRSCRAPER_API_TOKEN"
        scraper_id = "YOUR_SCRAPER_ID"

        url = "https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun"
        headers = {
            "accept": "application/json",
            "x-api-token": token,
            "content-type": "application/json"
        }
        payload = {
            "scraperId": scraper_id,
            "url": "https://books.toscrape.com"
        }

        response = requests.post(url, headers=headers, json=payload)
        print(response.json())
        ```
      
    

    **Request Parameters:**

    | Parameter   | Type   | Required | Description                             |
    | ----------- | ------ | -------- | --------------------------------------- |
    | `scraperId` | string | Yes      | The scraper ID from your initial scrape |
    | `url`       | string | Yes      | Target URL to scrape                    |

    ### Re-run Map Agent

    When using a scraper with the **map** agent, you need to provide additional parameters, such as `includePatterns` and `excludePatterns`, to control URL filtering.

    
      
        ```bash
        curl --location 'https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun' \
        --header 'accept: application/json' \
        --header 'x-api-token: ' \
        --header 'content-type: application/json' \
        --data '{
            "scraperId": "YOUR_SCRAPER_ID",
            "url": "https://books.toscrape.com",
            "maxDepth": 1,
            "maxPages": 50,
            "limit": 100,
            "includePatterns": ["https://books.toscrape.com/catalogue"],
            "excludePatterns": ["https://books.toscrape.com/catalogue/category"]
        }'
        ```
      

      
        ```javascript
        const token = "MRSCRAPER_API_TOKEN";
        const scraperId = "YOUR_SCRAPER_ID";

        fetch("https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun", {
          method: "POST",
          headers: {
            "accept": "application/json",
            "x-api-token": token,
            "content-type": "application/json"
          },
          body: JSON.stringify({
            scraperId: scraperId,
            url: "https://books.toscrape.com",
            "scraperId": scraper_id,
            "url": "https://books.toscrape.com",
            maxDepth: 1,
            maxPages: 50,
            limit: 100,
            includePatterns: ["https://books.toscrape.com/catalogue"],
            excludePatterns: ["https://books.toscrape.com/catalogue/category"]
          })
        })
        .then(res => res.json())
        .then(data => console.log(data))
        .catch(err => console.error(err));
        ```
      

      
        ```python
        import requests

        token = "MRSCRAPER_API_TOKEN"
        scraper_id = "YOUR_SCRAPER_ID"

        url = "https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun"
        headers = {
            "accept": "application/json",
            "x-api-token": token,
            "content-type": "application/json"
        }
        payload = {
            "scraperId": scraper_id,
            "url": "https://books.toscrape.com",
            "maxDepth": 1,
            "maxPages": 50,
            "limit": 100,
            "includePatterns": ["https://books.toscrape.com/catalogue"],
            "excludePatterns": ["https://books.toscrape.com/catalogue/category"]
        }

        response = requests.post(url, headers=headers, json=payload)
        print(response.json())
        ```
      
    

    **Additional Map Agent Parameters:**

    | Parameter         | Type      | Required | Description                                             |
    | ----------------- | --------- | -------- | ------------------------------------------------------- |
    | `scraperId`       | string    | Yes      | The scraper ID from your initial scrape                 |
    | `url`             | string    | Yes      | Target URL to scrape                                    |
    | `maxDepth`        | number    | Yes      | Maximum depth to crawl (e.g., `2`)                      |
    | `maxPages`        | number    | Yes      | Maximum number of pages to crawl (e.g., `100`)          |
    | `limit`           | number    | Yes      | Maximum number of URLs to collect (e.g., `1000`)        |
    | `includePatterns` | string\[] | No       | Regex pattern for URLs to include                       |
    | `excludePatterns` | string\[] | No       | Regex pattern for URLs to exclude (can be empty string) |

    > **Note:**\
    > When "re-running" with a map agent, you can fine-tune which URLs will be crawled using `includePatterns` and `excludePatterns` while reusing your base scraper logic and configuration.

    **Response (200 OK):**

    ```json
    {
      "message": "Successful operation!",
      "data": {
        "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxx...",
        "createdAt": "2026-01-18T07:28:44.725Z",
        "userId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxx...,
        "scraperId": "YOUR_SCRAPER_ID",
        "type": "Rerun-AI",
        "url": "https://books.toscrape.com",
        "status": "Finished",
        "error": "",
        "tokenUsage": 4,
        "data": {
          "urls": [
            "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
            "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
            "https://books.toscrape.com/catalogue/soumission_998/index.html",
            ...
          ]
        },
        "htmlPath": "results/.../page.html",
        "screenshotPath": "results/.../page.jpg"
      }
    }
    ```

    
      * You don’t need to create a new scraper from scratch every time you want to scrape a different—but structurally similar—website or page. If the data format and structure are the same, you can simply *re-run* your existing scraper and provide a new target URL.
      * This works for every agents including map agent. For map agents, you can even update the URL and the include/exclude patterns before recreating a new scraper, saving you time and tokens. Try reusing your scrapers for maximum efficiency whenever your targets share the same page layout!
    

    ## Complete Workflow Example

    Below is a full example that runs all steps in a single automated workflow:

    
      
        ```python
        import requests
        import json
        import time

        # Configuration
        API_TOKEN = "MRSCRAPER_API_TOKEN"
        BASE_URL = "https://api.app.mrscraper.com/api/v1"

        headers = {
            "accept": "application/json",
            "x-api-token": API_TOKEN,
            "content-type": "application/json"
        }

        def step1_verify_token():
            """Step 1: Verify API token and check account status"""
            print("🔐 Step 1: Verifying API token...")
            
            response = requests.get(
                f"{BASE_URL}/subscription-accounts",
                headers={"accept": "*/*", "x-api-token": API_TOKEN}
            )
            data = response.json()
            
            if response.status_code == 200:
                print(f"✓ Token valid! Tokens: {data['data']['tokenUsage']}/{data['data']['tokenLimit']}")
                return True
            else:
                print(f"❌ Token verification failed: {data}")
                return False

        def step2_collect_urls(target_url, message):
            """Step 2: Create scraper and collect URLs"""
            print(f"\n📍 Step 2: Collecting URLs from {target_url}...")
            
            payload = {
                "url": target_url,
                "agent": "general",
                "message": message,
                "proxyCountry": ""
            }
            
            response = requests.post(f"{BASE_URL}/scrapers-ai", headers=headers, json=payload)
            result = response.json()
            # cache to a file
            with open("step2_collect_urls_response.json", "w") as f:
                json.dump(result, f, indent=2)
            
            if "scraperId" in result.get("data", {}):
                scraper_id = result["data"]["scraperId"]
                print(f"✓ Scraping finished")
                print(f"📋 URL Scraper ID: {scraper_id}")
                return [], scraper_id
            else:
                print(f"❌ Failed: {result}")
                return [], None


        def step3_rerun_collect_urls(url, scraper_id):
            """Step 3: Rerun URL collection scraper"""
            print(f"\n🔄 Step 3: Rerunning URL collection scraper {scraper_id}...")
            
            payload = {
                "scraperId": scraper_id,
                "url": url
            }
            
            response = requests.post(f"{BASE_URL}/scrapers-ai-rerun/", headers=headers, json=payload)
            result = response.json()
            # cache to a file
            with open("step3_rerun_collect_urls_response.json", "w") as f:
                json.dump(result, f, indent=2)
            
            if result.get("message") == "Successful operation!":
                urls = result["data"]["data"]["urls"]
                print(f"✓ Rerun collected {len(urls)} URLs")
                return urls
            else:
                print(f"❌ Failed: {result}")
                return []
            
        def step4_create_detail_scraper(sample_url, message):
            """Step 4: Create detail scraper"""
            print(f"\n🔧 Step 4: Creating detail scraper...")
            
            payload = {
                "url": sample_url,
                "agent": "general",
                "message": message,
                "proxyCountry": ""
            }
            
            response = requests.post(f"{BASE_URL}/scrapers-ai", headers=headers, json=payload)
            result = response.json()
            # cache to a file
            with open("step4_create_detail_scraper_response.json", "w") as f:
                json.dump(result, f, indent=2)
            
            if "scraperId" in result.get("data", {}):
                scraper_id = result["data"]["scraperId"]
                print(f"✓ Detail scraper created")
                print(f"📋 Detail Scraper ID: {scraper_id}")
                return scraper_id
            else:
                print(f"❌ Failed: {result}")
                return None

        def step5_bulk_scrape(scraper_id, urls):
            """Step 5: Bulk scrape all URLs"""
            print(f"\n🤖 Step 5: Starting bulk scrape for {len(urls)} URLs...")
            
            payload = {
                "scraperId": scraper_id,
                "urls": urls
            }
            
            response = requests.post(f"{BASE_URL}/scrapers-ai-rerun/bulk", headers=headers, json=payload)
            result = response.json()

            # cache to a file
            with open("step5_bulk_scrape_response.json", "w") as f:
                json.dump(result, f, indent=2)
            
            bulk_result_id = result["data"]["bulkResultId"]
            print(f"✓ Bulk scraping started")
            print(f"📋 Bulk Result ID: {bulk_result_id}")
            return bulk_result_id

        def step6_get_results(bulk_result_id, poll_interval=10):
            """Step 6: Poll for results"""
            print(f"\n⏳ Step 6: Waiting for results (polling every {poll_interval}s)...")
            
            while True:
                response = requests.get(
                    f"{BASE_URL}/results/{bulk_result_id}",
                    headers={"accept": "application/json", "x-api-token": API_TOKEN}
                )
                result = response.json()
                status = result["data"]["status"]
                
                if status == "Finished":
                    print("✅ Scraping completed!")
                    return result["data"]["data"]
                elif status == "Running":
                    print(f"  ⏳ Still running... checking again in {poll_interval}s")
                    time.sleep(poll_interval)
                else:
                    print(f"❌ Error: {status}")
                    return None

        # Main workflow
        if __name__ == "__main__":
            print("🚀 Starting N2N Programmatic Workflow\n")
            print("=" * 60)
            
            # Step 1: Verify token
            if not step1_verify_token():
                exit(1)
            
            # Step 2: Collect URLs
            url = "https://books.toscrape.com"
            urls, url_scraper_id = step2_collect_urls(
                url,
                "Return array of urls that follow patterns https://books.toscrape.com/catalogue but exclude that has patterns https://books.toscrape.com/catalogue/category"
            )

            # Step 3: Rerun URL collection scraper
            if url_scraper_id:
                urls = step3_rerun_collect_urls(url, url_scraper_id)
            
            if not urls:
                exit(1)
            
            # Use first 3 URLs for demo
            urls = urls[:3]
            print(f"  Using {len(urls)} URLs for demo")


            # Step 4: Create detail scraper
            detail_scraper_id = step4_create_detail_scraper(
                urls[0],
                "Extract all data detail"
            )
            
            if not detail_scraper_id:
                exit(1)
            
            # Step 5: Bulk scrape
            bulk_result_id = step5_bulk_scrape(detail_scraper_id, urls)

            bulk_result_id = "a438e4c2-8489-4473-b405-6b4b5e18ed3f"
            
            # Step 6: Get results
            results = step6_get_results(bulk_result_id)
            
            if results:
                print("\n" + "=" * 60)
                print("📊 RESULTS SUMMARY")
                print("=" * 60)
                print(f"✓ Total items: {len(results['mergedData'])}")
                print(f"✓ Successful: {results['summary']['successfulUrls']}")
                print(f"✓ Failed: {results['summary']['failedUrls']}")
                print(f"✓ Tokens used: {results['summary']['totalTokenUsage']}")
                
                print("\n📦 EXTRACTED DATA:")
                print(json.dumps(results['mergedData'], indent=2))
        ```
      

      
        ```javascript
        // Configuration
        const API_TOKEN = "MRSCRAPER_API_TOKEN";
        const BASE_URL = "https://api.app.mrscraper.com/api/v1";
        const fs = require('fs').promises;

        const headers = {
          "accept": "application/json",
          "x-api-token": API_TOKEN,
          "content-type": "application/json"
        };

        // Step 1: Verify API token
        async function step1VerifyToken() {
          console.log("🔐 Step 1: Verifying API token...");
          
          const response = await fetch(`${BASE_URL}/subscription-accounts`, {
            headers: { "accept": "*/*", "x-api-token": API_TOKEN }
          });
          const data = await response.json();
          
          if (response.ok) {
            console.log(`✓ Token valid! Tokens: ${data.data.tokenUsage}/${data.data.tokenLimit}`);
            return true;
          }
          console.log(`❌ Token verification failed:`, data);
          return false;
        }

        // Step 2: Collect URLs
        async function step2CollectUrls(targetUrl, message) {
          console.log(`\n📍 Step 2: Collecting URLs from ${targetUrl}...`);
          
          const response = await fetch(`${BASE_URL}/scrapers-ai`, {
            method: "POST",
            headers,
            body: JSON.stringify({
              url: targetUrl,
              agent: "general",
              message,
              proxyCountry: ""
            })
          });
          const result = await response.json();
          
          // Cache to a file
          await fs.writeFile("step2_collect_urls_response.json", JSON.stringify(result, null, 2));
          
          if (result && 'scraperId' in (result.data || {})) {
            const scraperId = result.data.scraperId;
            console.log(`✓ Scraping finished`);
            console.log(`📋 URL Scraper ID: ${scraperId}`);
            return { urls: [], scraperId };
          }
          console.log(`❌ Failed:`, result);
          return { urls: [], scraperId: null };
        }

        // Step 3: Rerun URL collection scraper
        async function step3RerunCollectUrls(url, scraperId) {
          console.log(`\n🔄 Step 3: Rerunning URL collection scraper ${scraperId}...`);
          
          const response = await fetch(`${BASE_URL}/scrapers-ai-rerun/`, {
            method: "POST",
            headers,
            body: JSON.stringify({
              scraperId,
              url
            })
          });
          const result = await response.json();
          
          // Cache to a file
          await fs.writeFile("step3_rerun_collect_urls_response.json", JSON.stringify(result, null, 2));
          
          if (result.message === "Successful operation!") {
            const urls = result.data.data.urls;
            console.log(`✓ Rerun collected ${urls.length} URLs`);
            return urls;
          }
          console.log(`❌ Failed:`, result);
          return [];
        }

        // Step 4: Create detail scraper
        async function step4CreateDetailScraper(sampleUrl, message) {
          console.log(`\n🔧 Step 4: Creating detail scraper...`);
          
          const response = await fetch(`${BASE_URL}/scrapers-ai`, {
            method: "POST",
            headers,
            body: JSON.stringify({
              url: sampleUrl,
              agent: "general",
              message,
              proxyCountry: ""
            })
          });
          const result = await response.json();
          
          // Cache to a file
          await fs.writeFile("step4_create_detail_scraper_response.json", JSON.stringify(result, null, 2));
          
          if (result && 'scraperId' in (result.data || {})) {
            const scraperId = result.data.scraperId;
            console.log(`✓ Detail scraper created`);
            console.log(`📋 Detail Scraper ID: ${scraperId}`);
            return scraperId;
          }
          console.log(`❌ Failed:`, result);
          return null;
        }

        // Step 5: Bulk scrape
        async function step5BulkScrape(scraperId, urls) {
          console.log(`\n🤖 Step 5: Starting bulk scrape for ${urls.length} URLs...`);
          
          const response = await fetch(`${BASE_URL}/scrapers-ai-rerun/bulk`, {
            method: "POST",
            headers,
            body: JSON.stringify({ scraperId, urls })
          });
          const result = await response.json();
          
          // Cache to a file
          await fs.writeFile("step5_bulk_scrape_response.json", JSON.stringify(result, null, 2));
          
          const bulkResultId = result.data.bulkResultId;
          console.log(`✓ Bulk scraping started`);
          console.log(`📋 Bulk Result ID: ${bulkResultId}`);
          return bulkResultId;
        }

        // Step 6: Get results with polling
        async function step6GetResults(bulkResultId, pollInterval = 10000) {
          console.log(`\n⏳ Step 6: Waiting for results (polling every ${pollInterval/1000}s)...`);
          
          while (true) {
            const response = await fetch(`${BASE_URL}/results/${bulkResultId}`, {
              headers: { "accept": "application/json", "x-api-token": API_TOKEN }
            });
            const result = await response.json();
            const status = result.data.status;
            
            if (status === "Finished") {
              console.log("✅ Scraping completed!");
              return result.data.data;
            } else if (status === "Running") {
              console.log(`  ⏳ Still running... checking again in ${pollInterval/1000}s`);
              await new Promise(r => setTimeout(r, pollInterval));
            } else {
              console.log(`❌ Error: ${status}`);
              return null;
            }
          }
        }

        // Main workflow
        (async () => {
          console.log("🚀 Starting N2N Programmatic Workflow\n");
          console.log("=".repeat(60));
          
          // Step 1: Verify token
          if (!await step1VerifyToken()) {
            process.exit(1);
          }
          
          // Step 2: Collect URLs
          const url = "https://books.toscrape.com";
          const { urls: initialUrls, scraperId: urlScraperId } = await step2CollectUrls(
            url,
            "Return array of urls that follow patterns https://books.toscrape.com/catalogue but exclude that has patterns https://books.toscrape.com/catalogue/category"
          );
          
          // Step 3: Rerun URL collection scraper
          let urls = [];
          if (urlScraperId) {
            urls = await step3RerunCollectUrls(url, urlScraperId);
          }
          
          if (!urls.length) {
            process.exit(1);
          }
          
          // Use first 3 URLs for demo
          urls = urls.slice(0, 3);
          console.log(`  Using ${urls.length} URLs for demo`);
          
          // Step 4: Create detail scraper
          const detailScraperId = await step4CreateDetailScraper(
            urls[0],
            "Extract all data detail"
          );
          
          if (!detailScraperId) {
            process.exit(1);
          }
          
          // Step 5: Bulk scrape
          let bulkResultId = await step5BulkScrape(detailScraperId, urls);
          
          // Optional: Uncomment to use a specific bulk result ID
          // bulkResultId = "a438e4c2-8489-4473-b405-6b4b5e18ed3f";
          
          // Step 6: Get results
          const results = await step6GetResults(bulkResultId);
          
          if (results) {
            console.log("\n" + "=".repeat(60));
            console.log("📊 RESULTS SUMMARY");
            console.log("=".repeat(60));
            console.log(`✓ Total items: ${results.mergedData.length}`);
            console.log(`✓ Successful: ${results.summary.successfulUrls}`);
            console.log(`✓ Failed: ${results.summary.failedUrls}`);
            console.log(`✓ Tokens used: ${results.summary.totalTokenUsage}`);
            
            console.log("\n📦 EXTRACTED DATA:");
            console.log(JSON.stringify(results.mergedData, null, 2));
          }
        })();
        ```
      
    
  



# Real Estate

import { Step, Steps } from 'fumadocs-ui/components/steps';

Real estate professionals, investors, and analysts can use MrScraper to collect property listings from public real estate websites. Instead of checking hundreds of listings manually, the AI Scraper extracts property details for you using only a URL and a short prompt.

Let's walk through a **real-world example** where you can uses MrScraper to collect data from a live property listing site.

## Scenario

You want to explore the Michigan housing market and compare prices, home sizes, and features across different cities. Rather than visiting multiple listing sites every day, you can use MrScraper to pull structured data directly from Redfin, making it easier to review options and find the best value.


  
    ## Create a New AI Scraper

    1. Open the [**New AI Scraper**](https://app.mrscraper.com/scrapers/form) page.
    2. Select the **Listing Scraper** option.
    3. In the **URL** field, enter the URL of the Michigan property listings page: `https://www.redfin.com/state/Michigan`.
    4. Click on the **Start Scraping** button to start scraping.

    **Result :**

    ```json title="Scraping Result"
    {
      "mode": "direct",
      "properties": [
        {
          "address": "319 Saint Lawrence Blvd, Northville, MI 48168",
          "url": "/MI/Northville/319-Saint-Lawrence-Blvd-48168/home/98741962",
          "price": "$510,000",
          "beds": "3 beds",
          "baths": "3.5 baths",
          "sqft": "3,246",
          "status": "NEW 10 HRS AGO"
        },
        {
          "address": "20137 W Whipple Dr, Northville, MI 48167",
          "url": "/MI/Northville/20137-W-Whipple-Dr-48167/home/75119625",
          "price": "$799,900",
          "beds": "— beds",
          "baths": "— baths",
          "sqft": "—",
          "status": "NEW 10 HRS AGO"
        },
        {
          "address": "49776 Parkside Dr, Northville, MI 48168",
          "url": "/MI/Northville/49776-Parkside-Dr-48168/home/98891697",
          "price": "$699,900",
          "beds": "4 beds",
          "baths": "3.5 baths",
          "sqft": "4,492",
          "status": "NEW 14 HRS AGO"
        },
        {
          "address": "58543 Navarra Dr, South Lyon, MI 48178",
          "url": "/MI/South-Lyon/58543-Navarra-Dr-48178/home/191274082",
          "price": "$619,990",
          "beds": "5 beds",
          "baths": "3 baths",
          "sqft": "3,011",
          "status": "OPEN SAT, 11:30AM TO 2PM"
        },
        {
          "address": "15943 Morningside, Northville, MI 48168",
          "url": "/MI/Northville/15943-Morningside-48168/home/98892034",
          "price": "$273,000",
          "beds": "2 beds",
          "baths": "2 baths",
          "sqft": "1,653"
        },
        {
          ...
        }
      ],
      "total_count": 63,
      "links": [
        "/MI/Ann-Arbor/1670-W-Ellsworth-Rd-48108/home/99335852",
        "/MI/Ann-Arbor/1702-Hill-St-48104/home/99308587",
        "/MI/Ann-Arbor/2134-Overlook-Ct-48103/home/99356641",
        "/MI/Ann-Arbor/3250-Brackley-Dr-48105/home/143963798",
        "/MI/Ann-Arbor/3341-Roseford-Blvd-48105/home/143963723",
        "..."
      ]
    }
    ```
  

  
    ## Filter Out

    Tell the AI to filter the scraped results based on specific criteria. For this example, let's filter for smaller, more affordable properties under 2,000 square feet using the following prompt:

    ```md
    Get only properties with less than 2000 square feet
    ```

    Here’s a sample output:

    ```json title="Filtered Scraping Result"
    {
      "mode": "direct",
      "count": 28,
      "properties": [
        {
          "beds": "2 beds",
          "link": "/MI/Northville/15943-Morningside-48168/home/98892034",
          "sqft": 1653,
          "baths": "2 baths",
          "price": "$273,000",
          "address": "15943 Morningside, Northville, MI 48168"
        },
        {
          "beds": "3 beds",
          "link": "/MI/Northville/9824-Hathaway-Dr-48167/home/99341712",
          "sqft": 1000,
          "baths": "2 baths",
          "price": "$55,000",
          "address": "9824 Hathaway Dr, Northville, MI 48167"
        },
        {
          "beds": "2 beds",
          "link": "/MI/Northville/42432-Corlina-Dr-48167/home/98883824",
          "sqft": 1800,
          "baths": "2.5 baths",
          "price": "$350,000",
          "address": "42432 Corlina Dr, Northville, MI 48167"
        },
        {
          "beds": "2 beds",
          "link": "/MI/Novi/39928-Crosswinds-48375/home/61133922",
          "sqft": 1100,
          "baths": "1.5 baths",
          "price": "$210,000",
          "address": "39928 Crosswinds, Novi, MI 48375 5004"
        },
        {
          "beds": "3 beds",
          "link": "/MI/Novi/22415-Cranbrooke-Dr-48375/home/199166800",
          "sqft": 1304,
          "baths": "1.5 baths",
          "price": "$249,999",
          "address": "22415 Cranbrooke Dr, Novi, MI 48375 4502"
        },
        {
          ...
        }
      ]
    }
    ```
  

  
    ## Use the Data

    The results include smaller homes and condos across cities like Detroit, Ann Arbor, Livonia, Troy, Farmington, Novi, and Northville — making it easy to compare options side by side.
  



# Sentiment Analysis

import { Step, Steps } from 'fumadocs-ui/components/steps';

Sentiment analysis is a powerful tool for understanding public opinion, customer feedback, and market trends.

## Scenario

You’re a product manager at a laptop company competing with another brand.\
You want to understand what customers think about one of the competitor’s brand gaming laptops available on **Walmart**.\
By analyzing reviews, you can identify what users appreciate (e.g., performance, design) and what frustrates them (e.g., setup issues, reliability).


  
    ## Create a New AI Scraper

    1. Open the [**New AI Scraper**](https://app.mrscraper.com/scrapers/form) page.
    2. In the **Target URL** field, enter the URL of the product's review page you want to scrape. For example: `https://www.walmart.com/reviews/product/15843466879?entryPoint=viewAllReviewsTop`.
    3. Click on the **Start Scraping** button to start scraping.

    **Result :**

    ```json title="Scraping Result"
    {
      "reviews": [
        {
          "date": "Jul 28, 2025",
          "text": "Our son ordered this and he is very happy with how it helps him play video games. Can't say I love that he does that, but he said it really does improve the quality and availability of games for him, and he's pretty good about limiting his own time on devices.",
          "rating": 5,
          "reviewer": "WalmartCustomer"
        },
        {
          "date": "Jul 31, 2025",
          "text": "It was a gift for my daughter and she absolutely loves it! Loves it so much I bought a second one for my other daughter so they could play their games together. Great computer for gamers.",
          "rating": 5,
          "reviewer": "WalmartCustomer"
        },
        {
          "date": "Sep 5, 2025",
          "text": "To start, it's an awesome laptop. It fit my budget and can play games and do school work when I need it to. One pay made it very easy to get it without having to spend too much at once (I'm a broke college student). My order also arrived faster than the estimated time which was already pretty quick. Not only that but I got it at a cheap price on clearance.",
          "rating": 5,
          "reviewer": "WalmartCustomer"
        },
        {
          "date": "Sep 4, 2025",
          "text": "It never fails, really. I always seem to be on the receiving end of crappy products. I JUST got this thing TODAY and I can't even get it to setup! Persistent error messages 'Something went wrong' yadayada. This review is staying all the way bad until I get my money back or some reliable customer support cause this is not okay.",
          "rating": 1,
          "reviewer": "Dominic"
        }
      ]
    }
    ```
  

  
    ## Analyze Sentiment with AI

    You can now analyze the sentiment of the extracted user reviews using your preferred AI tool or library. For example, you can use Python's `TextBlob` or `VADER` to classify the sentiment of each review as positive, negative, or neutral.

    | Sentiment | Example Review                                                             | Insight                                                    |
    | --------- | -------------------------------------------------------------------------- | ---------------------------------------------------------- |
    | Positive  | “Great computer for gamers.”                                               | Strong satisfaction with performance                       |
    | Positive  | “Awesome laptop... plays games and does school work.”                      | Product fits well for multitasking                         |
    | Negative  | “I JUST got this thing TODAY and I can’t even get it to setup!”            | Setup/reliability issues                                   |
    | Neutral   | “Our son ordered this... he’s happy but I don’t love that he plays games.” | Mixed sentiment; happy performance but concern over gaming |
  



# Flight Price

import { Step, Steps } from 'fumadocs-ui/components/steps';

Instead of checking flight prices across multiple websites, you can use the AI Scraper to pull flight details, prices, and schedules automatically using only a URL and a short prompt.

Here’s a real-world example of how you can use MrScraper to collect live flight data from a booking site.

## Scenario

You’re building a flight price comparison tool or tracking prices for popular routes.
In this example, you want to monitor flights from New York City to London so travelers can quickly find the best deals.


  
    ## Create a New AI Scraper

    1. Open the [**New AI Scraper**](https://app.mrscraper.com/scrapers/form) page.
    2. Select the **Listing Scraper** option.
    3. In the **URL** field, enter the URL of the flight search results page. For this example, we'll search for flights from New York City (NYC) to London (LON):

    ```txt
    https://id.trip.com/m/flights/xflightfirst/?triptype=0&classtype=0&classgroupsearch=true&adult=1&child=0&infant=0&from=searchForm&stoptype=0&lowpricesource=searchForm&ddate=2025-11-22&dcitycode=NYC&acitycode=LON&locale=en-ID&curr=IDR&transactionid=20251120174344105
    ```

    4. Click on the **Start Scraping** button to start scraping.

    **Result :**

    ```json title="Scraping Result"
    {
      "mode": "direct",
      "links": [
        "flight_card_4",
        "flight_card_3",
        "flight_card_1",
        "flight_card_2",
        "flight_card_5"
      ],
      "flights": [
        {
          "price": "Rp4,960,870",
          "airline": "Jetblue Airways",
          "aircraft": "Airbus A321neo",
          "duration": "7h 21m",
          "top_label": "Cheapest nonstop",
          "arrival_time": "09:30",
          "price_numeric": "4960870",
          "departure_time": "21:09",
          "original_price": "Rp 5,097,750",
          "arrival_airport": "LHR T2",
          "next_day_arrival": "+1",
          "departure_airport": "JFK T5",
          "low_stock_warning": null,
          "original_price_numeric": "5097750",
          "carry_on_baggage_included": false
        },
        {
          "price": "Rp5,051,840",
          "airline": "Jetblue Airways",
          "aircraft": "Airbus A321neo",
          "duration": "7h 1m",
          "top_label": null,
          "arrival_time": "22:00",
          "price_numeric": "5051840",
          "departure_time": "09:59",
          "original_price": "Rp 5,191,430",
          "arrival_airport": "LHR T2",
          "next_day_arrival": null,
          "departure_airport": "JFK T5",
          "low_stock_warning": null,
          "original_price_numeric": "5191430",
          "carry_on_baggage_included": false
        },
        {
          "price": "Rp7,248,600",
          "airline": "United Airlines",
          "aircraft": "Boeing 767-300/300ER",
          "duration": "7h 20m",
          "top_label": null,
          "arrival_time": "06:20",
          "price_numeric": "7248600",
          "departure_time": "18:00",
          "original_price": "Rp 7,453,760",
          "arrival_airport": "LHR T2",
          "next_day_arrival": "+1",
          "departure_airport": "EWR C",
          "low_stock_warning": "<5 left",
          "original_price_numeric": "7453760",
          "carry_on_baggage_included": true
        },
        {
          "price": "Rp7,287,450",
          "airline": "American Airlines",
          "aircraft": "Boeing 777-300ER",
          "duration": "6h 50m",
          "top_label": null,
          "arrival_time": "09:40",
          "price_numeric": "7287450",
          "departure_time": "21:50",
          "original_price": "Rp 7,493,770",
          "arrival_airport": "LHR T3",
          "next_day_arrival": "+1",
          "departure_airport": "JFK T8",
          "low_stock_warning": null,
          "original_price_numeric": "7493770",
          "carry_on_baggage_included": true
        },
        {
          "price": "Rp7,431,410",
          "airline": "Austrian Airlines (Codeshare)",
          "aircraft": "Boeing 767-300/300ER",
          "duration": "7h 20m",
          "top_label": null,
          "arrival_time": "06:20",
          "price_numeric": "7431410",
          "departure_time": "18:00",
          "original_price": "Rp 7,642,020",
          "arrival_airport": "LHR T2",
          "next_day_arrival": "+1",
          "departure_airport": "EWR C",
          "low_stock_warning": "<5 left",
          "original_price_numeric": "7642020",
          "carry_on_baggage_included": true
        }
      ],
      "total_flights": 5
    }
    ```
  

  
    ## Filter Out

    Tell the AI to filter your scraped results based on specific rules. In this example, we filter for flights under Rp 6,000,000 using the following prompt:

    ```md
    Get only flight with cost less than Rp 6.000.000
    ```

    Here’s a sample output:

    ```json title="Filtered Scraping Result"
    {
      "mode": "direct",
      "count": 2,
      "flights": [
        {
          "price": "Rp4,960,870",
          "airline": "Jetblue Airways",
          "aircraft": "Airbus A321neo",
          "duration": "7h 21m",
          "arrival_time": "09:30",
          "price_numeric": 4960870,
          "departure_time": "21:09",
          "arrival_airport": "LHR T2",
          "departure_airport": "JFK T5"
        },
        {
          "price": "Rp5,051,840",
          "airline": "Jetblue Airways",
          "aircraft": "Airbus A321neo",
          "duration": "7h 1m",
          "arrival_time": "22:00",
          "price_numeric": 5051840,
          "departure_time": "09:59",
          "arrival_airport": "LHR T2",
          "departure_airport": "JFK T5"
        }
      ]
    }
    ```
  

  
    ### Step 3: Use the Data

    You can now use this clean flight data for price tracking, comparison tools, or travel apps. Filtering helps budget travelers quickly find the cheapest options.
  



# General Agent

import { Step, Steps } from 'fumadocs-ui/components/steps';

The **General Agent** is designed specifically for extracting data from **a single web page**, especially **product detail pages, profile pages, article pages, property detail pages**, and other **one-off pages** where you already have the final URL. Perfect when you just need to pull specific information **from one page only**, without navigation or crawling across listings.


  Avoid using this for listing or catalog pages containing multiple products. For those pages, use the [Listing Agent](/docs/features/ai-scraper/listing).


## General Agent Usage

| **Category**                 | **Scenarios**                                                                                                                                                                                                                                                                                                                                                   | **Example URLs**                                                                                                                                                                                                                                                                                                                                                                                    |
| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Use General Agent**        | - Extract structured data from a single page (product, article, profile, event, job posting)
- Scrape a product or listing detail page without visiting other pages
- You already have the final target URL
- No navigation, pagination, clicking, or scrolling required
- Page can be processed in one static or dynamically rendered view | [https://www.walmart.com/ip/15377670482](https://www.walmart.com/ip/15377670482)
[https://www.zillow.com/homedetails/101-Frederica-St-UNIT-301-Owensboro-KY-42301/455517252\_zpid/](https://www.zillow.com/homedetails/101-Frederica-St-UNIT-301-Owensboro-KY-42301/455517252_zpid/)
[https://www.bbc.co.uk/news/articles/ckgmy90z991o](https://www.bbc.co.uk/news/articles/ckgmy90z991o) |
| **Do NOT Use General Agent** | - Listing pages with multiple products/items (use Listing Agent)
- Search results pages
- Category pages with multiple products
- Directory pages with many profiles/businesses
- Multi-page extraction requiring pagination                                                                                                                | [https://www.amazon.com/s?k=laptops](https://www.amazon.com/s?k=laptops)
[https://www.walmart.com/browse/electronics](https://www.walmart.com/browse/electronics)
[https://www.zillow.com/homes/fo](https://www.zillow.com/homes/fo)                                                                                                                                                      |

## Limitations

The General Agent has the following limitations:

* **No Browser Automation**: Cannot perform clicks, scrolling, or interact with dynamic elements
* **No Pagination**: Cannot navigate to next pages or load more content automatically
* **Single Page Only**: Cannot open or follow links to detail pages from listing URLs
* **No Form Submission**: Cannot fill out forms, log in, or submit data
* **Static Content Focus**: Works best with content that's immediately visible on page load
* **No Multi-Step Workflows**: Cannot perform sequences like "click product → extract details → go back"


  If you need to scrape multiple items from listing pages or navigate through pagination, use the **Listing Agent** or **Map Agent** instead.


## Example Usage

Follow these steps to use the General Agent from your dashboard:


  
    Log in to **MrScraper**, then open the [**New AI Scraper**](https://app.mrscraper.com/scrapers/form) page

    
      AI Scrapers are created from their own page. The **Create Workflow** button on the Workflow page opens the manual workflow builder instead.
    
  

  
    Select the **General** Scraper Agent and enter the URL to scrape
  

  
    Choose between **Cheap** or **Super** Agent

    
      **General Agent** uses the **Super** type by default for optimal accuracy.
    
  

  
    Wait for the AI to process the provided URL
  

  
    Enter your prompt describing the data you want to extract
  

  
    The AI will analyze your prompt and extract the requested data
  

  
    Once complete, review your results or export them as JSON or CSV
  


{/* #no-rag */}

### Example: Scraping E-Commerce Product Details

**Example URL:**\
`https://www.walmart.com/ip/15377670482`

**Initial Extracted Data:**

```json
{
  "data": {
    "id": "1",
    "name": "Restored Dell Latitude 3190 | 11.6\" Touchscreen Laptop PC | Intel Core Pentium Silver N5030 (1.1 GHz) | 8GB RAM | 128GB SSD | Windows 11 Pro (Refurbished)",
    "price": "158.00",
    "rating": "4.6",
    "seller": {
      "name": "Discount Computer Depot",
      "rating": "3.8",
      "reviews_count": "9721"
    },
    "source": "product",
    "reviews": "18",
    "features": {
      "Display Features": "11.6-inch touchscreen display",
      "Memory & Storage": "8GB RAM, 128GB SSD",
      "Operating System": "Windows 11 Pro",
      "Processor Details": "Intel Core Pentium Silver N5030, 1.1 GHz",
      "Graphics Capability": "Intel UHD Graphics 605",
      "Connectivity Options": "Display Port",
      "Integrated Peripherals": "Built-in webcam"
    },
    "shipping": {
      "method": "Shipping",
      "arrival": "Dec 16",
      "availability": "Free"
    },
    "return_policy": "Free 90-day returns"
  }
}
```

**Refining with a Follow-up Prompt:**

> Remove the seller and shipping fields from the extracted JSON.

**Refined Output:**

```json
{
  "data": {
    "id": "1",
    "name": "Restored Dell Latitude 3190 | 11.6\" Touchscreen Laptop PC | Intel Core Pentium Silver N5030 (1.1 GHz) | 8GB RAM | 128GB SSD | Windows 11 Pro (Refurbished)",
    "price": "158.00",
    "rating": "4.6",
    "source": "product",
    "reviews": "18",
    "features": {
      "Display Features": "11.6-inch touchscreen display",
      "Memory & Storage": "8GB RAM, 128GB SSD",
      "Operating System": "Windows 11 Pro",
      "Processor Details": "Intel Core Pentium Silver N5030, 1.1 GHz",
      "Graphics Capability": "Intel UHD Graphics 605",
      "Connectivity Options": "Display Port",
      "Integrated Peripherals": "Built-in webcam"
    },
    "return_policy": "Free 90-day returns"
  }
}
```

{/* #no-rag */}

## Tips and Best Practices

* **Be Specific with Your Prompts**: Clear, detailed prompts yield better results. Instead of "Get product info," try "Extract product name, price, rating, and available colors"
* **Validate Before Scaling**: Always review a sample extraction before automating large-scale scraping jobs
* **Use Cheap Mode first**: Start with cheap mode to test whether extraction works. If it fails, then switch to Super Mode.
* **Refine Iteratively**: If initial results are incomplete, adjust your prompt and re-run. You can build upon previous extractions with follow-up prompts
* **Retry When Needed**: If results look incomplete, adjust your prompt and rerun.
* **Use Structured Prompts**: Frame your requests clearly, e.g., "Extract: product title, price, SKU, availability status, and customer ratings"


  For the best results, describe both **what data you want** and **how you want it structured** in your prompt.



# AI Scraper

MrScraper's **AI Scraper** lets you extract structured data from any web page by simply describing what you need — no coding or manual setup required. You provide a **URL** and a **prompt**, and the AI automatically analyzes the page, extracts relevant data, and returns the result in a structured format like JSON or CSV.


  This feature is ideal for users and developers who want fast, flexible, and intelligent web scraping without writing rules or managing selectors.


## When to Use the AI Scraper

Use the AI Scraper when you want to:

* Quickly extract data from a website without building a scraper manually
* Automate repetitive data collection tasks where precision is less critical
* Prototype or test scraping ideas before committing to a full custom workflow
* Extract data from websites with complex structures that are difficult to scrape manually

For large-scale, highly structured, or repetitive tasks that require full control, consider using the **Workflow** instead.

## AI Scraper Agents

MrScraper provides three specialized AI Scraper agents to handle different scraping scenarios:

| Agent                                            | Description                                                                                          | Best Used For                                                                                                                                    |
| ------------------------------------------------ | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| [**General**](/docs/features/ai-scraper/general) | Extracts data from a single page using one prompt.                                                   | Product detail pages, profile pages, article pages, or any one-off page extraction.                                                              |
| [**Listing**](/docs/features/ai-scraper/listing) | Scrapes multiple items from paginated or list-based content (products, categories, posts, listings). | Use on URLs that show lists of items, such as E-commerce category pages, blog indexes, directory listings, search results, and multi-item pages. |
| [**Map**](/docs/features/ai-scraper/map)         | Collects all internal and external links from the target URL.                                        | Discover all URLs inside a page, Site structure mapping, and link audits.                                                                        |

## Scraper Modes

The AI Scraper offers two modes, allowing you to balance between cost efficiency and success rate when accessing websites.

| Mode           | Description                                                                                                                                                                                                                          |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Cheap Mode** | More affordable and faster with basic retry schema, uses datacenter IPs, ideal for simple, static websites.                                                                                                                          |
| **Super Mode** | Higher success rate with premium rotating proxies residential, advanced retry schema with multiple fallback strategies, and automatic fallback to browser engine if other engines fail, making it more reliable on complex websites. |


  Start with **Cheap Mode** for testing. Switch to **Super Mode** if requests fail or you need higher reliability.


## Screenshot Capture

You can enable screenshot capture to get a visual snapshot of the scraped page along with your data. This is useful for verifying results, debugging issues, or archiving page states.


  Screenshots are always captured in **Super Mode**, and are optional in **Cheap Mode**.


## Proxy Settings

You can adjust your proxy country if the scraper fails during the initial run. Some websites serve different content depending on the visitor's region.


  Changing the proxy country lets you retrieve region-specific content.


## Workflow API Access

Once your first scraping session is set up, Workflow API Access lets you automate and scale the process—no need to run the chat interface manually each time.


  After creating a scraper for an Amazon search results page that lists multiple products, you can reuse it via the API to scrape new URLs under the same domain and extract all product listings with the same structure at a fraction of the cost.



# Listing Agent

import { ShoppingCart } from 'lucide-react';
import { Step, Steps } from 'fumadocs-ui/components/steps';

The **Listing Agent** is designed specifically for **extracting data from listing pages** that display multiple items in a repeating format—such as product grids, job listings, search results, or directory pages. It automatically handles complex loading behaviors including infinite scroll, pagination buttons, load-more interactions, and dynamic content rendering to collect all available listing items and their detail page URLs.


  **Perfect for catalog pages, search results, and category listings with multiple items.**


## When to Use Listing Agent

Use the Listing Agent when you need to:

* **Extract all listing items** from pages displaying multiple products, jobs, properties, or other repeated entries
* **Scrape data that's visible on the listing page** without visiting individual detail pages
* **Automatically handle pagination** through next buttons, page numbers (1, 2, 3...), or dynamic loading
* **Navigate infinite scroll** where content loads continuously as you scroll down
* **Click "Load More" buttons** to reveal additional items progressively
* **Collect detail page URLs** from each listing item for further processing
* **Get a quick overview** of all available listings with basic information visible on the listing page

**Example URLs:**

* `https://www.amazon.com/s?k=laptops` (search results with pagination)
* `https://www.walmart.com/browse/electronics/laptops` (category page with load more)
* `https://www.zillow.com/homes/for_sale` (property listings with infinite scroll)
* `https://books.toscrape.com/` (product grid with next page buttons)

## How Listing Agent Works

The Listing Agent intelligently analyzes how websites load and display their content, then adapts its behavior accordingly:

1. **Traditional Pagination**: Automatically clicks "Next" buttons or numbered page links (1, 2, 3...) to navigate through all pages
2. **Infinite Scroll**: Scrolls down progressively to trigger automatic content loading until all items are revealed
3. **Load More Buttons**: Identifies and clicks "Load More" or "Show More" buttons to append additional batches of data

The agent continues navigating until it has collected all available listing items from the page, then extracts the visible data along with detail page URLs.

## Limitations

The Listing Agent has the following limitations:

* **Listing Page Data Only**: Can only extract information that's visible on the listing page itself
* **No Detail Page Navigation**: Cannot click into individual product/item links to access detail pages
* **Surface-Level Data**: Limited to basic information displayed in listing cards (title, price, thumbnail, brief description)
* **No Deep Product Details**: Cannot extract specifications, full descriptions, or other data that only appears on detail pages
* **Browser Automation for Listings Only**: Navigates only to load all listings (scroll, pagination, load-more) but won't enter detail pages


  **Important:** If you need detailed information from individual product pages (full descriptions, specifications, reviews, etc.), you'll need to:

  1. Use Listing Agent to collect all detail page URLs
  2. Then use General Agent or Map Agent to scrape those detail pages


## General vs Listing Agent

| Feature        | General Agent                          | Listing Agent                               |
| -------------- | -------------------------------------- | ------------------------------------------- |
| **Purpose**    | Extract data from a single detail page | Extract all items from listing pages        |
| **Navigation** | No browser automation                  | Automated scrolling, pagination, load-more  |
| **Use Case**   | Product details, articles, profiles    | Search results, category pages, directories |
| **Output**     | Detailed data from one page            | Basic data + URLs from multiple listings    |
| **Data Depth** | Deep, comprehensive                    | Surface-level, quick overview               |


  **Quick Rule:** Use **Listing Agent** for pages with multiple items. Use **General Agent** for individual detail pages.


## Example Usage

Follow these steps to use the Listing Agent from your dashboard:


  
    Log in to **MrScraper**, then open the [**New AI Scraper**](https://app.mrscraper.com/scrapers/form) page

    
      AI Scrapers are created from their own page. The **Create Workflow** button on the Workflow page opens the manual workflow builder instead.
    
  

  
    Select the **Listing** Scraper Agent and enter the listing page URL
  

  
    The AI will automatically detect the page structure and navigation pattern
  

  
    Wait for the agent to navigate through all pages/scrolls and collect all listings
  

  
    Review the extracted data or export as JSON/CSV
  


### Example: Scraping E-Commerce Product Listings

**Example URL:**\
`https://books.toscrape.com/`

**Example Prompt:**

> Extract all book titles, prices, ratings, and availability from this page

{/* #no-rag */}

**Listing Agent Output:**

```json
{
    "response": [
        {
        "page_num": 0,
        "data": {
            "mode": "direct",
            "data": [
            {
                "id": "74301965",
                "address": "80 Emerald Street, Manchester, NH 03103",
                "agent_company": "Listing provided by PrimeMLS",
                "price": "$364,900",
                "bedrooms": "3",
                "bathrooms": "2",
                "sqft": "1",
                "status": "Active",
                "url": "https://www.zillow.com/homedetails/80-Emerald-St-Manchester-NH-03103/74301965_zpid/",
                "highlight": "14 hours ago"
            },
            {
                "id": "121734329",
                "address": "21 School Street, New Portland, ME 04961",
                "agent_company": "KELLER WILLIAMS REALTY",
                "price": "$449,000",
                "bedrooms": "2",
                "bathrooms": "1",
                "sqft": "1",
                "status": "Active",
                "url": "https://www.zillow.com/homedetails/21-School-St-New-Portland-ME-04961/121734329_zpid/",
                "highlight": "18 hours ago"
            },
            ...
            ]
        }
        },
        {
        "page_num": 1,
        "data": {
            "mode": "direct",
            "data": [
            {
                "id": "458007902",
                "address": "Lot 59&60 Starks Road, New Sharon, ME 04955",
                "agent_company": "COLDWELL BANKER SANDY RIVER REALTY",
                "price": "$420,000",
                "bedrooms": null,
                "bathrooms": null,
                "sqft": "92 acres lot",
                "status": "Active",
                "url": "https://www.zillow.com/homedetails/LOT-5960-Starks-Rd-New-Sharon-ME-04955/458007902_zpid/",
                "highlight": "Open fields"
            },
            {
                "id": "91905999",
                "address": "3 Long Cove Road, York, ME 03909",
                "agent_company": "LEGACY PROPERTIES SOTHEBY'S INTERNATIONAL REALTY",
                "price": "$1,795,000",
                "bedrooms": "3",
                "bathrooms": "2",
                "sqft": "2",
                "status": "Active",
                "url": "https://www.zillow.com/homedetails/3-Long-Cove-Rd-York-ME-03909/91905999_zpid/",
                "highlight": "Modern comforts"
            }
            ...
            ],
            "link": "https://www.zillow.com/me/?searchQueryState=%7B%22isMapVisible%22%3Atrue..."
        }
        }
    ]
}
```

### Comparison: Listing Agent vs General Agent

Below is a comparison showing how each agent handles the same listing page:


  
    **Listing Agent** automatically navigates through all pages and extracts structured data with detail page URLs:

    ```json
    {
    "response": [
        {
        "page_num": 0,
        "data": {
            "mode": "direct",
            "data": [
            {
                "id": "74301965",
                "address": "80 Emerald Street, Manchester, NH 03103",
                "agent_company": "Listing provided by PrimeMLS",
                "price": "$364,900",
                "bedrooms": "3",
                "bathrooms": "2",
                "sqft": "1",
                "status": "Active",
                "url": "https://www.zillow.com/homedetails/80-Emerald-St-Manchester-NH-03103/74301965_zpid/",
                "highlight": "14 hours ago"
            },
            {
                "id": "121734329",
                "address": "21 School Street, New Portland, ME 04961",
                "agent_company": "KELLER WILLIAMS REALTY",
                "price": "$449,000",
                "bedrooms": "2",
                "bathrooms": "1",
                "sqft": "1",
                "status": "Active",
                "url": "https://www.zillow.com/homedetails/21-School-St-New-Portland-ME-04961/121734329_zpid/",
                "highlight": "18 hours ago"
            },
            ...
            ]
        }
        },
        {
        "page_num": 1,
        "data": {
            "mode": "direct",
            "data": [
            {
                "id": "458007902",
                "address": "Lot 59&60 Starks Road, New Sharon, ME 04955",
                "agent_company": "COLDWELL BANKER SANDY RIVER REALTY",
                "price": "$420,000",
                "bedrooms": null,
                "bathrooms": null,
                "sqft": "92 acres lot",
                "status": "Active",
                "url": "https://www.zillow.com/homedetails/LOT-5960-Starks-Rd-New-Sharon-ME-04955/458007902_zpid/",
                "highlight": "Open fields"
            },
            {
                "id": "91905999",
                "address": "3 Long Cove Road, York, ME 03909",
                "agent_company": "LEGACY PROPERTIES SOTHEBY'S INTERNATIONAL REALTY",
                "price": "$1,795,000",
                "bedrooms": "3",
                "bathrooms": "2",
                "sqft": "2",
                "status": "Active",
                "url": "https://www.zillow.com/homedetails/3-Long-Cove-Rd-York-ME-03909/91905999_zpid/",
                "highlight": "Modern comforts"
            }
            ...
            ],
            "link": "https://www.zillow.com/me/?searchQueryState=%7B%22isMapVisible%22%3Atrue..."
        }
        }
    ]
    }
    ```

    ✅ **Advantages:**

    * Extracts detail page URLs for each item
    * Navigates automatically through pagination
    * Structured output with consistent schema
    * Includes metadata like total count
  

  
    **General Agent** extracts only what's visible on the current page without navigation:

    ```json
    {
    "1": {
        "id": "1",
        "name": "Maine, Vermont, New Hampshire Real Estate & Homes For Sale",
        "rows": [
        {
            "address": "16 Thompson Crossing Road #231-1-3, Antrim, NH 03440",
            "price": "$599,999",
            "beds": "3",
            "baths": "2",
            "sqft": "1,613",
            "status": "Active"
        },
        {
            "address": "80 Emerald Street, Manchester, NH 03103",
            "price": "$364,900",
            "beds": "3",
            "baths": "2",
            "sqft": "1,240",
            "status": "Active"
        },
        ...
        ],
        "source": "table",
        "headers": ["Address", "Price", "Beds", "Baths", "Sqft", "Status"]
    }
    }
    ```

    ⚠️ **Limitations:**

    * No detail page URLs
    * No automatic pagination
    * Table-like format without structured fields
    * Limited to visible items only
  


{/* #no-rag */}

## Tips and Best Practices

* **Let the Agent Navigate**: The Listing Agent handles pagination automatically—you don't need to specify how the site loads content
* **Combine with General Agent**: Use Listing Agent to collect URLs, then use General Agent to scrape detailed information from each detail page
* **Review Extracted URLs**: Always check that all detail page URLs are captured correctly before proceeding to detail page scraping
* **Be Patient with Large Listings**: Sites with hundreds of items may take longer as the agent navigates through all pages
* **Understand Data Limitations**: Only extract fields visible on the listing page. For detailed specs or descriptions, you'll need to visit detail pages
* **Test with Small Samples First**: Verify the extraction works correctly on the first page before processing entire catalogs


  For complete product data extraction, use a two-step workflow:

  1. **Listing Agent** → Get all product URLs
  2. **General Agent** → Extract detailed information from each URL


## Real-World Use Cases

For practical examples of how to use the Listing Agent in production scenarios, check out our guides:


  } href="/docs/guides/ecommerce">
    Learn how to scrape product listings, pricing, and availability from major e-commerce sites using the Listing Agent.
  



# Map Agent

import { Step, Steps } from 'fumadocs-ui/components/steps';

The **Map Agent** lets you input a single website URL and instantly retrieve **all URLs found on that domain**.\
This is the fastest way to understand a website’s structure, map out pages, or prepare for large-scale scraping tasks.

## When to Use the Map Agent

Use the Map Agent when you need to:

* Get all URLs from a website quickly

* Collect links for further processing such as job postings, blog articles, or category pages

* Find hidden pages that are not linked from navigation but appear in the HTML

* Audit a website’s structure for SEO, content planning, or data coverage checks

## Example Usage


  
    Go to your **Dashboard** → select **Map Agent**.
  

  
    Enter the **URL** (for example: `https://penateam.com`) you want to crawl.

    
      The Map Scraper only supports **Cheap Mode**.
    
  

  
    Wait for the system to process the URL. The Map Scraper automatically extracts:

    * Internal links
    * Subpage URLs
    * Category links
    * Pagination links
  

  
    Once completed, you’ll receive a clean, structured list of URLs in JSON format:

    {/* #no-rag */}

    ```json
    {
      "urls": [
        "https://penateam.com",
        "https://penateam.com/blog",
        "https://penateam.com/blog/Conten_Lifecycle_Management",
        "https://penateam.com/blog/Effective_Release_Notes",
        "https://penateam.com/blog/Procedural_Writing",
        "https://penateam.com/blog/Structured_Authoring",
        "https://penateam.com/work",
        "https://www.penateam.com",
        "https://www.penateam.com/blog/Effective_Release_Notes",
        "https://www.penateam.com/work"
      ],
      "count": 10
    }
    ```
  


{/* #no-rag */}


  No prompts, no selectors, no configuration needed.



# Get All Results

This endpoint returns all scraping results, including data, status, token usage, and file paths, with options for sorting, filtering, and date ranges.


  Use the Platform API Server (`https://api.app.mrscraper.com`) host when calling this endpoint.


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}




# Get a Detailed Result

This endpoint retrieves detailed information for a specific scraping result by its unique identifier.


  * Use the Platform API Server (`https://api.app.mrscraper.com`) host when calling this endpoint.
  * You can get `{id}` parameter from the [Get All Results](/docs/api/v3/result/all) endpoint.
  * The structure of `data.data` in the scrape result varies depending on the agent type and whether the request is single or bulk. For detailed response formats, see [AI Response](/docs/api/v3/scraper/ai-response).


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}




# Workflow

import { Step, Steps } from 'fumadocs-ui/components/steps';

import { Accordion, Accordions } from 'fumadocs-ui/components/accordion';

A **Workflow** in MrScraper lets you build a custom scraping sequence by defining a series of steps to extract data from web pages.


  This agent is ideal for users who need full control over the scraping process and want to handle complex scenarios.


## When to Use a Workflow

Use a Workflow when you need to:

* Extract data from complex or dynamic websites that require specific interactions.
* Implement custom workflows that involve multiple steps, such as navigation, data extraction, and pagination.
* Handle situations where our markdown converter cleans or alters the HTML, causing the AI to fail when parsing it into JSON.

## Workflow Features

### Step Types

Below are the available step types you can add in the **Steps** panel when building a workflow:

| Step Type             | Description                                                                                                                                                                                          |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Extract**           | Scrape data from the webpage by setting an **Extraction Name**, choosing an **Extraction Type** (`Text`, `Inner HTML`, `Outer HTML`, or `Attribute`), and defining CSS selectors to target elements. |
| **Click**             | Simulates a click action on a specified element.                                                                                                                                                     |
| **Delay**             | Pauses the scraper for a set duration (in milliseconds) before moving to the next step.                                                                                                              |
| **Wait for Selector** | Pauses until a specific element appears on the page, or until the timeout is reached.                                                                                                                |
| **Input**             | Enters text into input fields on the webpage.                                                                                                                                                        |
| **Scroll**            | Scrolls to the end of the page, scrolls until a specific text is found, or scrolls to a specific element (or until a certain number of elements).                                                    |
| **Inject JavaScript** | Runs custom JavaScript code on the webpage.                                                                                                                                                          |
| **Listen Network**    | Captures and extracts data from an endpoint.                                                                                                                                                         |
| **Follow URLs**       | Collects links from the page and runs all subsequent steps on each followed URL.                                                                                                                     |
| **Paginate**          | Automatically navigates through multiple pages.                                                                                                                                                      |


  Use **Delay** when you simply need to pause for a fixed amount of time. Use **Wait for Selector** when you are waiting for content to load, it is faster and more reliable because it resumes as soon as the element is available.


### Pagination Types

| Pagination Type          | Required Parameter                |
| ------------------------ | --------------------------------- |
| **Query Pagination**     | Query parameter (e.g., `?page=2`) |
| **Directory Pagination** | None (e.g., `/list/page/2`)       |
| **Next Page Link**       | Next page selector                |

### Get Cookies

This feature allows you to capture cookies from the webpage, which can be useful for maintaining sessions or accessing data that requires authentication. You can enable this feature by toggling the **Get Cookies** option in the scraper settings. When enabled, the scraper will return a `cookies` field in the output, containing all cookies set during the scraping process.

```json title="Example output with cookies"
{
  "books_data": [
    {
      "title": "A Light in the Attic",
      "rating": "Three"
    }
  ],
  "cookies": []
}
```

### Self-Healing

Self-Healing allows you to automatically generate or repair workflows using AI, both directly inside the dashboard builder and programmatically via the API:

* **Full Workflow Generation**: Automatically generate an entire multi-step workflow for a target URL or replace an existing workflow using the main **+ AI** button.
* **Single Step Repair / Generation**: Use the dedicated **+ AI** button on any step block to either **Extend Step** (broaden the step to cover additional URLs or layout variations) or **Replace Step** (overwrite the step configuration with a newly generated one).
* **Self-Heal API Access**: Access the repair endpoint (`POST /api/v1/scrapers-manual/{scraperId}/heal`) directly from your own applications or automated monitoring. Retrieve your endpoint, scraper ID, and API token from the **⋮ → Self-Heal API Access** menu in the scraper builder dashboard.

For complete setup guides and API examples, see [Self-Healing](/docs/features/workflow-scraper/self-healing).

### JavaScript Templates

For custom DOM parsing, calling internal page APIs, or capturing JSON responses on infinite-scroll pages, you can use pre-built IIFE JavaScript snippets with the **Inject JavaScript** step:

* **Parse Page Data**: Extract data from dynamic DOM elements using `querySelectorAll`.
* **Fetch API Data**: Directly query internal page APIs using browser credentials and headers.
* **Scroll & Capture Fetch Responses**: Automatically scroll infinite-loading pages while intercepting matching API responses.

For ready-to-use scripts and setup steps, see [Templates](/docs/features/workflow-scraper/templates).

## Usage Example

### Example 1 : Simple Website

Suppose you want to scrape data from this product page:

`https://www.target.com/p/beats-studio-pro-bluetooth-wireless-headphones/-/A-89459966`

On this site, the price is loaded dynamically through an internal API, so the AI Scraper won’t be able to extract it automatically. To capture this data, you’ll need to switch to a Workflow and inject a small custom JavaScript snippet.

Follow the steps below:


  
    Log in to **MrScraper**, then click **Workflow** in the left sidebar.
  

  
    Click **+ Create Workflow** at the top to open the **New Workflow** page.
  

  
    Input the product URL above.
  

  
    Add `Inject JavaScript` step.
  

  
    Fill the **Name** field with `data`, and **Script timeout** with `50`.
  

  
    Use this script:

    ```js title="Script"
    (() => { return window.__TGT_DATA__.__PRELOADED_QUERIES__.queries})();
    ```

    
      You can locate the price by inspecting the page’s source code.\
      For this specific page, the price is available under the `__TGT_DATA__` object in the window.

      **Note**: The data structure varies by website, so the location of price information may differ on other pages.
    
  

  
    Select **Save & Run** to save the workflow and run it immediately, or **Save** to store it without running.
  

  
    The scraper returns the following output:

    **Result :**

    ```json title="Manual workflow result"
    {
      "data": {
        "...",
        "product": {
          "...",
          "price": {
            "formatted_comparison_price": "$349.99",
            "formatted_comparison_price_type": "reg",
            "formatted_current_price": "$165.99 - $169.99",
            "formatted_current_price_type": "sale",
            "location_id": 3991,
            "current_retail_min": 165.99,
            "reg_retail_max": 349.99
          }
          "...",
        }
        "...",
      }
    }
    ```
  


### Example 2 : Complex Website

Some websites cannot be scraped with the AI Scraper alone, especially when you need to fill in a form before data appears.
One example is:

`https://www.handelsregister.de/rp_web/normalesuche/welcome.xhtml`

Since the site requires entering search details before showing company information, you’ll need to use a Workflow to automate the steps.

Follow the steps below:


  
    Log in to **MrScraper**, then click **Workflow** in the left sidebar.
  

  
    Click **+ Create Workflow** at the top to open the **New Workflow** page.
  

  
    Input the product URL above.
  

  
    Add `Input` Step, fill **Input Field Selector** with `textarea[title="Company or keywords:"]` and **Text Input** for the company you want to search, for this example we'll use `Volkswagen & Audi Club`.
  

  
    Add `Delay` Step with `1000` ms duration.
  

  
    Add `Click` Step and select the `button[name="form:btnSuche"]` as the **Element to Click**.
  

  
    Add another `Delay` Step with`10000` ms duration.
  

  
    Add `Inject JavaScript` Step, Fill the **Name** field with `data`, and **Script timeout** with `900`, then use this script :

    
      Need help writing a custom script for your manual workflow? [Contact Us](https://help.mrscraper.com/).
    

    ```js title="Script"
    (async () => {
      
      // Wait for correct URL with timeout
      const targetUrl = "https://www.handelsregister.de/rp_web/sucheErgebnisse/welcome.xhtml?cid=1";
      const maxAttempts = 15;
      const checkInterval = 2000; // 2 seconds
      
      let attempts = 0;
      let urlMatches = false;
      
      console.log('Waiting for correct URL...');
      
      while (attempts < maxAttempts) {
        if (window.location.href === targetUrl) {
          urlMatches = true;
          console.log('URL matched! Starting to parse...');
          break;
        }
        
        attempts++;
        console.log(`Attempt ${attempts}/${maxAttempts}: Current URL does not match. Waiting...`);
        await new Promise(resolve => setTimeout(resolve, checkInterval));
      }
      
      if (!urlMatches) {
        console.log('Timeout: URL never matched the target URL');
        return null;
      }
      
      // Helper function to create unique key for an entry
      function getEntryKey(entry) {
        return `${entry.region}|${entry.court}|${entry.companyName}|${entry.location}`;
      }
      
      // Helper function to check if page has changed
      function hasPageChanged(currentResults, lastPageKeys) {
        if (currentResults.length === 0) return false;
        
        const currentKeys = currentResults.map(getEntryKey);
        // Check if at least one entry is different
        return currentKeys.some(key => !lastPageKeys.has(key));
      }
      
      // Parsing functions
      function parseCurrentPage() {
        const table = document.getElementById('ergebnissForm:selectedSuchErgebnisFormTable_data');
        if (!table) {
          console.log('Table not found on page');
          return [];
        }
        
        const rows = table.querySelectorAll('tr[data-ri]');
        const results = [];
        
        rows.forEach(row => {
          const entry = {};
          
          // Extract region and court info
          const headerCell = row.querySelector('.fontTableNameSize');
          if (headerCell) {
            const headerText = headerCell.textContent.trim();
            const parts = headerText.split(/\s{2,}/);
            entry.region = parts[0]?.trim() || '';
            entry.court = parts[1]?.trim() || '';
          }
          
          // Extract company name
          const nameCell = row.querySelector('.marginLeft20');
          if (nameCell) {
            entry.companyName = nameCell.textContent.trim();
          }
          
          // Extract location (Sitz)
          const locationCell = row.querySelector('.sitzSuchErgebnisse .verticalText');
          if (locationCell) {
            entry.location = locationCell.textContent.trim();
          }
          
          // Extract registration status
          const statusCells = row.querySelectorAll('.verticalText');
          if (statusCells.length > 1) {
            entry.status = statusCells[1].textContent.trim();
          }
          
          // Extract document types (AD, CD, HD, etc.)
          const docLinks = row.querySelectorAll('.dokumentList .underlinedText');
          entry.documentTypes = Array.from(docLinks).map(link => link.textContent.trim());
          
          // Extract history entries
          const historyRows = row.querySelectorAll('.RegPortErg_HistorieZn');
          if (historyRows.length > 0) {
            entry.history = [];
            historyRows.forEach(histRow => {
              const histText = histRow.querySelector('.fontSize85')?.textContent.trim();
              const histLocation = histRow.closest('tr')?.querySelector('.RegPortErg_SitzStatus .fontSize85')?.textContent.trim();
              
              if (histText) {
                entry.history.push({
                  name: histText,
                  location: histLocation || ''
                });
              }
            });
          }
          
          results.push(entry);
        });
        
        return results;
      }
      
      function isNextButtonDisabled() {
        const nextButton = document.querySelector('a.ui-paginator-next');
        return nextButton && nextButton.classList.contains('ui-state-disabled');
      }
      
      const allResults = [];
      const seenKeys = new Set();
      let pageNumber = 1;
      const delay = 1000;
      const maxRetries = 3; // Max retries if page hasn't changed
      
      console.log(`Parsing page ${pageNumber}...`);
      
      // Parse first page
      const firstPageResults = parseCurrentPage();
      firstPageResults.forEach(entry => {
        const key = getEntryKey(entry);
        if (!seenKeys.has(key)) {
          seenKeys.add(key);
          allResults.push(entry);
        }
      });
      console.log(`Page ${pageNumber}: Found ${firstPageResults.length} entries (${allResults.length} unique so far)`);
      
      // Continue clicking next until button is disabled
      while (!isNextButtonDisabled()) {
        const nextButton = document.querySelector('a.ui-paginator-next');
        
        if (!nextButton) {
          console.log('Next button not found');
          break;
        }
        
        // Store current page keys for comparison
        const lastPageKeys = new Set(seenKeys);
        
        // Click next button
        nextButton.click();
        pageNumber++;
        
        // Wait for page to load and retry if needed
        let retries = 0;
        let pageChanged = false;
        
        while (retries < maxRetries) {
          await new Promise(resolve => setTimeout(resolve, delay));
          
          const currentPageResults = parseCurrentPage();
          pageChanged = hasPageChanged(currentPageResults, lastPageKeys);
          
          if (pageChanged) {
            console.log(`Parsing page ${pageNumber}...`);
            
            // Add only unique entries
            let newEntries = 0;
            currentPageResults.forEach(entry => {
              const key = getEntryKey(entry);
              if (!seenKeys.has(key)) {
                seenKeys.add(key);
                allResults.push(entry);
                newEntries++;
              }
            });
            
            console.log(`Page ${pageNumber}: Found ${currentPageResults.length} entries (${newEntries} new, ${allResults.length} unique total)`);
            break;
          } else {
            retries++;
            if (retries < maxRetries) {
              console.log(`Page ${pageNumber}: Data not updated yet, retrying (${retries}/${maxRetries})...`);
            }
          }
        }
        
        if (!pageChanged) {
          console.log(`Page ${pageNumber}: Data still not updated after ${maxRetries} retries, skipping...`);
        }
      }
      
      console.log(`\nCompleted! Total unique entries: ${allResults.length}`);
      return allResults;
      
    })()
    ```
  

  
    Select **Save & Run** to save the workflow and run it immediately, or **Save** to store it without running.
  

  
    The scraper returns the following output:

    **Result :**

    ```json title="Manual workflow result"
    {
      "data": [
        {
          "region": "Bavaria",
          "court": "District court München VR 201131",
          "companyName": "1.Volkswagen & Audi Club Mittenwald e.V.",
          "location": "Mittenwald",
          "status": "currently registered",
          "documentTypes": [
            "AD",
            "CD",
            "DK",
            "UT",
            "VÖ",
            "SI"
          ]
        },
        {
          "region": "Baden-Württemberg",
          "court": "District court Stuttgart VR 381348",
          "companyName": "VW-Audi Club Härten e.V.",
          "location": "Kusterdingen",
          "status": "currently registered",
          "documentTypes": [
            "AD",
            "CD",
            "HD",
            "DK",
            "UT",
            "VÖ",
            "SI"
          ],
          "history": [
            {
              "name": "1.) VW-Audi Club Härten",
              "location": "1.) Kusterdingen"
            }
          ]
        }
      ]
    }
    ```
  


## AI Assistant

The AI Assistant helps you build custom scraping workflows using natural language.

You do not need to know JavaScript to get started. Simply describe the data you want to extract, and the AI Assistant will generate the workflow for you.

To open the AI Assistant:


  
    Open the **Workflow** builder.
  

  
    Click the dropdown next to the **AI** button.
  

  
    Select **Create Workflow**.
  


The AI Assistant will generate the required workflow steps and JavaScript code based on your prompt. You can use the generated workflow as-is or modify it to fit your needs.

### Example

Suppose you want to extract book titles and ratings from the following website:

`https://books.toscrape.com/`

Enter the following prompt:

```text
Extract the book title and rating from the page.
```

The AI Assistant analyzes the page and generates a workflow automatically.

In this example, the generated workflow may:

1. Add a delay to ensure the page has fully loaded.
2. Add JavaScript injection.

The generated JavaScript injection might look similar to the following:

```js title="AI Generated Script"
(function() {
  const books = [];
  const articles = document.querySelectorAll('article.product_pod');
  
  articles.forEach(article => {
    const titleEl = article.querySelector('h3 a');
    const title = titleEl?.getAttribute('title')?.trim() || null;
    
    const ratingEl = article.querySelector('p.star-rating');
    let rating = null;
    if (ratingEl) {
      const ratingClass = Array.from(ratingEl.classList).find(c => c !== 'star-rating');
      rating = ratingClass || null;
    }
    
    books.push({
      title: title,
      rating: rating
    });
  });
  
  return books;
})();
```

## Troubleshooting

The items below cover the most common Workflow issues. Work through them in the order listed—most problems are resolved by the first two or three checks.


  
    Dynamic pages may not finish rendering before the first extraction or JavaScript step runs.

    1. Add or increase a **Delay** step before the extraction step. Start with `2000` ms and increase in `1000` ms increments until the target content is present.
    2. If the page loads content only after a user action (for example, after clicking a tab or submitting a form), make sure the relevant **Click** or **Input** step comes before the **Delay**.
    3. Run the scraper and enable **Screenshot** to confirm the page has rendered the content you need before extraction begins.
  

  
    When an **Inject JavaScript** step returns `null`, `undefined`, or `[]`, the script is running but cannot find the expected data. Work through these checks in order:

    1. **Test the script locally.** Open the target URL in your browser, paste the IIFE into the developer console, and confirm it returns the expected result.
    2. **Check for overlays.** Open the URL in an incognito window. A modal, cookie banner, sign-up prompt, or GDPR overlay may be covering the page. If so, add a **Click** step before the script to dismiss the overlay.
    3. **Verify timing.** The data you need may load after the script runs. Add or increase a **Delay** step before the **Inject JavaScript** step.
    4. **Enable a proxy.** Some sites serve different content based on IP origin. Enable a proxy, select the country where the website is located, and turn on **Route entire request through proxy** to route all page resources through the same proxy.
    5. **Try Super mode.** If the workflow still fails, open the **Scraper API**, turn on **Super** mode, and add your Workflow steps as Scraper API automation steps. Super mode routes the request through a real browser environment.
  

  
    The **Extract** step uses CSS selectors to target page elements. If it returns empty results:

    1. Open the target URL in your browser and run `document.querySelectorAll("your-selector")` in the developer console to verify the selector matches the elements you expect.
    2. If the selector works in the console but not in MrScraper, the content may be loaded dynamically. Add a **Delay** step before the **Extract** step or switch to an **Inject JavaScript** step that waits for the element.
    3. Check that the **Extraction Type** matches the data you need: use **Text** for visible text, **Attribute** for values like `href` or `src`, and **Inner HTML** or **Outer HTML** for raw markup.
  

  
    If the **Get Cookies** option is enabled but the `cookies` field is empty:

    1. Confirm that the website sets cookies during the scraping session. Not all websites set cookies on every page load.
    2. Make sure the workflow includes at least one step that triggers cookie creation—for example, clicking an "Accept cookies" button or submitting a login form.
    3. Check that the **Get Cookies** toggle is enabled in the scraper settings before running the workflow.
  

  
    Selectors that worked before can break when a site redesigns its markup. Instead of rebuilding the workflow by hand, use [Self-Healing](/docs/features/workflow-scraper/self-healing) to regenerate the whole workflow or repair a single step with AI, either from the dashboard or through the Self-Heal API.
  



# Self-Healing

import { Step, Steps } from 'fumadocs-ui/components/steps';

Websites change their markup all the time. When a selector breaks, a workflow keeps running but returns empty or partial data. **Self-Healing** uses AI to automatically generate, repair, and extend workflows so you don't have to write or update selectors manually.

You can use Self-Healing in two ways:

* **UI Platform**: Generate a full workflow or generate/repair single steps directly inside the scraper builder using the AI buttons.
* **Self-Heal API Access**: Trigger automated workflow repairs programmatically using the API endpoint.


  Self-Healing is also useful for a scraper you haven't built yet. Point it at a URL and let the AI generate the full workflow automatically, then fine-tune the steps yourself.


## When to Use Self-Healing

* **Website Redesign**: The scraper suddenly returns empty fields because the target website changed its markup.
* **Automated Workflow Creation**: You want to automatically generate a starting workflow for a new URL instead of writing every step manually.
* **Single Step Repair**: One specific step is broken while the rest of the workflow still functions properly.
* **Multi-URL Coverage**: You want a step or workflow to cover additional URLs or page layouts on the same website.

## Generate a Full Workflow

You can generate an entire multi-step workflow automatically using the main AI button at the top of the scraper builder:


  
    Open a workflow or create a new one.
  

  
    Click the **+ AI** button at the top right of the builder. Select **Create Workflow** if you are creating a new scraper, or **Replace All** to replace an existing workflow.
  

  
    Enter your prompt (or let the AI inspect the page) and wait for the AI to generate the workflow steps.
  

  
    Review the generated steps, fine-tune any selectors or code, then click **Save**.
  



  Generating a full workflow overwrites the steps currently shown in the builder. Run the scraper and check the output before saving over a workflow you still rely on.


## Generate or Repair a Single Step

Instead of rebuilding the entire workflow, you can generate or repair an individual step. Every step block in the workflow builder has its own dedicated **+ AI** button in its top-right corner.

Clicking the **+ AI** button on an individual step reveals two options:

| Option           | What it does                                                                                                    | Use it when                                                                                                       |
| ---------------- | --------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| **Extend Step**  | Keeps the existing step configuration and extends it so the workflow can cover additional URLs or page layouts. | The step works on some pages but fails on others, and you want one step/workflow to cover multiple page variants. |
| **Replace Step** | Discards the current step configuration and generates a new one for the target page URL.                        | The step is broken or outdated, and you want to replace its configuration with a newly generated workflow step.   |


  
    Locate the step you want to modify in the builder, then click its **+ AI** button.
  

  
    Select either **Extend Step** or **Replace Step** based on your objective.
  

  
    Wait for the AI to inspect the page and generate the updated step configuration.
  

  
    Review the generated selectors or JavaScript code, then click **Save** to persist the workflow.
  



  * **Extend Step** is additive: it broadens the step so the workflow can cover additional URLs and layout variants.
  * **Replace Step** replaces the existing step configuration with a brand new workflow step for the current URL.


## Self-Heal API Access

Besides using Self-Healing in the UI platform, MrScraper also provides the **Self-Heal API**. This allows you to trigger automated workflow repairs programmatically from a monitoring script, a scheduled task, or your own application whenever a scraper detects empty or invalid results.

### Accessing Self-Heal API in the Dashboard

You can retrieve the API endpoint, your scraper ID, and your API token directly from the scraper dashboard:


  
    Open your workflow in the builder dashboard.
  

  
    Click the **⋮** (more options) menu icon at the top right of the page.
  

  
    Select **Self-Heal API Access** from the menu.
  

  
    In the popup modal, enter or copy your API token (or click **Generate** to create a new one). The modal provides a ready-made cURL example pre-configured with your `scraperId` and token.
  



  See [Authentication](/docs/api/authentication) for more details on managing API tokens.


### Request

```bash title="Self-heal a workflow"
curl --request POST "https://api.app.mrscraper.com/api/v1/scrapers-manual/{scraperId}/heal" \
  --header "accept: application/json" \
  --header "x-api-token: YOUR_API_TOKEN"
```

| Parameter     | Type   | Location | Required | Description                                                                              |
| ------------- | ------ | -------- | -------- | ---------------------------------------------------------------------------------------- |
| `x-api-token` | string | header   | Yes      | Your MrScraper API token                                                                 |
| `scraperId`   | string | path     | Yes      | The ID of the workflow you want to repair (found in the scraper URL or API Access modal) |

See [Self-Heal a Manual Scraper](/docs/api/v3/scraper/manual/manual-heal) in the API reference for the complete OpenAPI schema and error handling details.

### Response

```json title="Successful response"
{
  "message": "Successful operation!",
  "data": {
    "success": true,
    "scraperId": "",
    "workflowUpdated": true
  }
}
```


  The endpoint asks the AI engine to repair the current workflow. The scraper's saved workflow is updated **only** when a valid workflow response is returned. If the repair attempt fails, your existing workflow configuration remains unchanged.



# Super Mode

import { Step, Steps } from 'fumadocs-ui/components/steps';

## Why Use Super Mode?

Some websites do not load reliably through standard routing. By default, a workflow accesses pages through our cloud infrastructure. Super mode routes your request through a real device instead, which can improve success rates on complex websites.

## How to Use Super Mode

Super mode is not a toggle inside the scraper builder. To run a workflow in Super mode, follow these steps :


  
    Open the workflow you want to run in Super mode.
  

  
    Click the **Ellipsis** ( **⋮** ) button in the top right and click **Manual Scraper API Access**.
  

  
    In the cURL example shown in the modal, copy the `workflow` array.
  

  
    Paste the workflow into the `workflow` field of the request below, then fill in your API token, target URL, and the country you want to access the URL from.
  


### Request

```bash title="Run a workflow in Super mode"
curl --location 'https://api.mrscraper.com?token={MRSCRAPER_API_TOKEN}&geoCode=us&html=true&super=true&proxyCountry=us' \
--header 'x-api-token: {MRSCRAPER_API_TOKEN}' \
--header 'Content-Type: application/json' \
--data '{
  "url": "{YOUR_TARGET_URL}",
  "homePage": true,
  "proxyCountry" : "us",
  "workflow": [

  ]
}'
```

| Parameter               | Location       | Description                                                                                   |
| ----------------------- | -------------- | --------------------------------------------------------------------------------------------- |
| `super`                 | query          | Set to `true` to route the request through a real device instead of our cloud infrastructure. |
| `token` / `x-api-token` | query / header | Your MrScraper API token.                                                                     |
| `geoCode`               | query          | ISO 3166-1 alpha-2 country code used for proxy routing.                                       |
| `html`                  | query          | Set to `true` to include the raw page HTML in the response.                                   |
| `proxyCountry`          | query / body   | Country you want to access the target URL from.                                               |
| `url`                   | body           | The target URL to scrape.                                                                     |
| `homePage`              | body           | Set to `true` to visit the website's home page first, then navigate to the target URL.        |
| `workflow`              | body           | The workflow steps copied from the **Manual Scraper API Access** modal.                       |

### Visiting the Home Page First

Some websites block requests that land directly on a deep URL. eBay, for example, expects visitors to arrive from the home page. Setting `homePage` to `true` makes the request load the home page first and then navigate to your target URL, which mimics normal browsing behavior.

| Trade-off | Detail                                                                         |
| --------- | ------------------------------------------------------------------------------ |
| **Pros**  | More reliable and robust against blocks on sites that check how you arrived.   |
| **Cons**  | Higher bandwidth usage and higher latency, since each run loads an extra page. |


  Leave `homePage` off by default. Enable it when a target URL is blocked or returns empty data on its own, and you are willing to trade extra bandwidth and run time for a higher success rate.



  See [Authentication](/docs/api/authentication) for more details on managing API tokens.


### Response

```json title="Example response"
{
    "code": null,
    "screenshots": [],
    "recording_path": null,
    "extractions": "{\"Product Link\": [{\"follow_url\": \"https://books.toscrape.com//catalogue/a-light-in-the-attic_1000/index.html\", \"result\": {\"product_name\": \"\\n    \\n        In stock (22 available)\\n    \\n\"}}]}",
    "data": {
        "Product Link": [
            {
                "follow_url": "https://books.toscrape.com//catalogue/a-light-in-the-attic_1000/index.html",
                "result": {
                    "product_name": "\n    \n        In stock (22 available)\n    \n"
                }
            }
        ]
    },
    "data_path": "results/b09aab93-717c-4804-9d37-a37084be4627/data.json",
    "html_path": "results/b09aab93-717c-4804-9d37-a37084be4627/page.html",
    "error": null,
    "residential_proxy_usage": {
        "received": 0.157379150390625,
        "sent": 0.0006866455078125,
        "total": 0.1580657958984375
    },
    "runtime": 13.358241081237793,
    "token_usage": 2,
    "listen_network_data": {},
    "html": " ... full page HTML truncated ... ",
    "markdown": "",
    "screenshot": "",
    "retry_count": 0
}
```

You can also review the run and its output on the [Results page](https://app.mrscraper.com/scrapers-results) in the app.


# Templates

import { Step, Steps } from 'fumadocs-ui/components/steps';

Workflows combine low-code steps with custom JavaScript. Use low-code steps to click elements, scroll the page, follow URLs, listen to network requests, add delays, and extract text with CSS selectors.

For more advanced workflows, the **Inject JavaScript** step runs browser-side code—the same code you would run in the developer console—to extract data with selectors, call page APIs, or intercept network responses.

## Prerequisites

Before using these templates:

* Replace all placeholder selectors and API endpoints with values from your target website.
* Each script is an [Immediately Invoked Function Expression (IIFE)](https://developer.mozilla.org/en-US/docs/Glossary/IIFE) and must return JSON-serializable data.
* Test your selectors and scripts in the browser developer console before adding them to a workflow.

## Parse Page Data

Use this template when the data you need is already rendered in the page HTML.

### Setup Steps


  
    Open the target URL in a Workflow.
  

  
    If the content loads dynamically, add a **Delay** step before extraction.
  

  
    Add an **Inject JavaScript** step.
  

  
    Set an **Extraction Name** (for example, `products`).
  

  
    Paste the script below into the script box and replace the example selectors with selectors from your target page.
  


```js title="Parse page data with querySelectorAll"
(() => {
  const products = Array.from(
    document.querySelectorAll(".product-card")
  ).map((product) => ({
    title:
      product.querySelector(".product-title")?.textContent?.trim() || null,
    price:
      product.querySelector(".product-price")?.textContent?.trim() || null,
    url: product.querySelector("a")?.href || null,
  }));

  return products;
})();
```


  Test your selectors in the browser console first. Run `document.querySelectorAll(".product-card").length` and confirm it returns the expected number of elements before adding the script to your workflow.


## Fetch API Data

Use this template when the page loads its data from an API endpoint. You can identify the endpoint with the browser's **Network → Fetch/XHR** panel or a Workflow **Listen Network** step.

### Setup Steps


  
    Open the target URL in a Workflow.
  

  
    Add an **Inject JavaScript** step.
  

  
    Set an **Extraction Name** (for example, `apiData`).
  

  
    Replace `/api/products` with the endpoint used by the target page.
  


```js title="Fetch data from a page API"
(async () => {
  const apiUrl = new URL("/api/products", window.location.origin);

  const response = await fetch(apiUrl, {
    method: "GET",
    credentials: "include",
    headers: {
      Accept: "application/json",
    },
  });

  if (!response.ok) {
    throw new Error(`API request failed with status ${response.status}`);
  }

  const data = await response.json();
  return data;
})();
```


  This template works best for same-origin APIs that use the page's existing cookies. Some endpoints also require request headers, query parameters, or tokens copied from the website's own request.


## Scroll and Capture Fetch Responses

Use this template on infinite-scroll pages that load additional data through the browser Fetch API. The script intercepts matching JSON responses while scrolling, then returns all collected data once scrolling completes.

### Setup Steps


  
    Open the listing page in a Workflow.
  

  
    Add an **Inject JavaScript** step.
  

  
    Set a **Script timeout** long enough for the scrolling loop to finish (for example, `30000` ms for a page with many scroll loads).
  

  
    Replace `/api/products` with a unique part of the endpoint you want to capture.
  


```js title="Scroll and capture Fetch API responses"
(async () => {
  const endpointPattern = "/api/products";
  const capturedResponses = [];
  const pendingCaptures = [];
  const originalFetch = window.fetch;
  const wait = (milliseconds) =>
    new Promise((resolve) => setTimeout(resolve, milliseconds));

  const patchedFetch = async (...args) => {
    const response = await originalFetch.apply(window, args);
    const requestUrl =
      typeof args[0] === "string" ? args[0] : args[0]?.url || "";

    if (requestUrl.includes(endpointPattern)) {
      const capture = response
        .clone()
        .json()
        .then((data) => {
          capturedResponses.push({
            url: requestUrl,
            data,
          });
        })
        .catch(() => {
          // Ignore matching responses that are not JSON.
        });

      pendingCaptures.push(capture);
    }

    return response;
  };

  window.fetch = patchedFetch;

  try {
    let previousHeight = document.body.scrollHeight;
    let stableRounds = 0;

    for (let attempt = 0; attempt < 20; attempt += 1) {
      window.scrollTo(0, document.body.scrollHeight);
      await wait(1500);

      const currentHeight = document.body.scrollHeight;
      stableRounds =
        currentHeight === previousHeight ? stableRounds + 1 : 0;
      previousHeight = currentHeight;

      if (stableRounds >= 2) {
        break;
      }
    }

    await wait(1000);
    await Promise.allSettled(pendingCaptures);
    return capturedResponses;
  } finally {
    if (window.fetch === patchedFetch) {
      window.fetch = originalFetch;
    }
  }
})();
```


  This template captures requests made with `window.fetch` after the script starts. It does not capture `XMLHttpRequest`, requests made before injection, or non-JSON responses. Use the **Listen Network** step when you need broader network capture.


## Run the Workflow

After adding a template to your workflow:


  
    Replace all placeholder selectors, endpoints, and timeouts with values from the target website.
  

  
    Click **Save**.
  

  
    Run the scraper and inspect the returned field that matches the extraction name you set.
  


If the result is empty or the website blocks the request, see [Troubleshooting](/docs/features/workflow-scraper#troubleshooting) on the Workflow page.


# Get Screenshot



## Overview

Selecting **Get Screenshot** in the Scraper API Use Case panel pre-configures the Scraper API environment specifically for visual page capture. When this preset is active, the Scraper API enables real browser rendering (`browserRendering=true`) by default so you can visually verify how dynamic web pages, client-side scripts, and layout elements render.

You can toggle between capturing the visible viewport (`top`) or rendering the full scrollable page (`full`).

### Practical Applications

* **Visual QA & Deployment Audits**: Verify visual layout integrity across dynamic responsive viewports after deployments or design updates.
* **Compliance & Legal Archiving**: Maintain time-stamped visual proof of pricing, terms of service, disclosures, or ad placements.
* **Competitor UI & Campaign Tracking**: Monitor visual changes on competitor landing pages, promotional banners, and checkout flows over time.
* **Scraper & Extraction Debugging**: Inspect the exact page state when data extraction fails—instantly spot cookie banners, error pages, or unexpected layouts.
* **Automated Previews & Dashboards**: Generate real-time page thumbnails and visual previews for internal dashboards, link previews, or reports.

## Default Parameters

| Parameter             | Description                                          |
| --------------------- | ---------------------------------------------------- |
| html=true             | Used by this endpoint to return html in the response |
| browserRendering=true | In this use case, `Render JavaScript` default to On  |
| waitUntil             | Default to `DOM Content Loaded`                      |
| timeout               | Default to `300`                                     |

## Example

### Request

Scrape `https://www.walmart.com/search?q=table%20lamps` with `Record` toggled on, `Render JavaScript` toggled on,  and `Screenshot area` set to `Full`

```sh title="Request"
curl --location --request POST 'https://api.mrscraper.com?token={MRSCRAPER_API_TOKEN}&browserRendering=true&waitUntil=networkidle0&timeout=300&geoCode=us&html=true&screenshot=full&record=true&saveResult=true&proxyCountry=us' \
  -H 'x-api-token: {MRSCRAPER_API_TOKEN}' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "https://www.walmart.com/search?q=table%20lamps"
}'
```

### Response

Images :



Recording :

Scrape This Site

\n Quotes to Scrape\n

Top Ten tags

Example Domain

Example Domain

Example Domain