# Authentication The MrScraper API uses API token authentication to secure all endpoints. Every API request must include a valid API token in the request headers to access the API. ## Getting Your API token Generate an API token by following the steps in our [API Token Generation guide](/docs/getting-started/api-token). Your API token provides access to: * **V3 Platform endpoints** - All endpoints in the V3 API * **Sync endpoints** - Endpoints on the `sync.scraper.mrscraper.com` host * **Analytics endpoints** - Special authentication requirements (see [Analytics Authentication](#analytics-authentication) below) For access to endpoints on other hosts, please [contact support](mailto:support@mrscraper.com) to request an API token. ## Make Your First Call Most API endpoints use HTTP header-based authentication. Include your API token in the request header: ```http x-api-token: MRSCRAPER_API_TOKEN ``` For example, to [retrieve scraping results](/docs/api/v3/result/all), make the following cURL request: ```bash curl -X GET "https://api.app.mrscraper.com/api/v1/results?filters[scraperId]=&page=1&pageSize=10&sort=createdAt&sortOrder=DESC" \ -H "accept: application/json" \ -H "x-api-token: MRSCRAPER_API_TOKEN" ``` ## Analytics Authentication Analytics endpoints require a different authentication method using query parameters instead of headers. The following endpoints use query parameter authentication: * [`/analytic/statuses`](/docs/api/v3/analytic/status): Retrieve scraper status analytics * `/analytic/timeline`: Retrieve timeline analytics data For analytics endpoints, pass your API token as a query parameter: ```bash apiTokenName=MRSCRAPER_API_TOKEN ``` For example, to [retrieve analytics results](/docs/api/v3/result/all), make the following cURL request: ```bash curl --location 'https://api.app.mrscraper.com/api/v1/analytic/statuses?domain=LINK&action=ACTION_DETAIL&startDate=2026-04-12%2000%3A00%3A00&endDate=2026-04-13%2023%3A59%3A59&apiTokenName=MRSCRAPER_API_TOKEN' \ -H "accept: application/json" \ -H "x-api-token: MRSCRAPER_API_TOKEN" ``` # Error Handling ## Overview The MrScraper API will return a JSON object containing error details when a request fails. This object includes an error code, message, and additional information to help diagnose the issue. This reference covers the most common errors. It is not exhaustive — an endpoint may occasionally return a transient error not listed here. When in doubt, inspect the `message` and `error` fields in the response body. ## Error response format When a request fails, the API returns a non-`2xx` HTTP status code and a JSON body with a consistent shape: ```json { "message": "Unauthorized access", "error": "Unauthorized" } ``` | Field | Type | Description | | --------- | ------ | ---------------------------------------------------------------- | | `message` | string | Human-readable description of what went wrong. | | `error` | string | Short error type or category (e.g. `Unauthorized`, `Not Found`). | ## HTTP status codes | Status | Error | Cause | Solution | Retryable | | ------ | ---------------------- | ------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- | ----------------- | | `400` | `Bad Request` | The request payload is malformed or a required parameter is missing or in the wrong format. | Fix the payload per the endpoint reference (e.g. use the `yyyy-MM-dd HH:mm:ss` date format where required). | No | | `401` | `Unauthorized` | The API token is missing or invalid. | Include a valid token in the `x-api-token` header. See [Authentication](/docs/api/authentication). | No | | `403` | `Forbidden` | The token is valid but does not have permission to access this resource or Token limit exceeded. | Verify your plan and that the token has access to the resource. [Contact support](mailto:support@mrscraper.com) if needed. | No | | `404` | `Not Found` | The requested resource (e.g. a scraper or result) does not exist. | Check that the `scraperId` / `resultId` is correct and belongs to your account. | No | | `422` | `Unprocessable Entity` | The request is well-formed but failed validation. | Review the `message` for the specific validation failure and correct the input. | No | | `429` | `Too Many Requests` | You have exceeded your rate limit. | Back off and retry after a short delay. Honor the `Retry-After` header if present. | Yes, with backoff | | `500` | Server errors | A temporary problem on MrScraper's side. | Retry with exponential backoff. If it persists, [contact support](mailto:support@mrscraper.com). | Yes, with backoff | ## Common errors ### 400 — Bad Request The request body could not be parsed, or a parameter is missing or incorrectly formatted. ```json { "message": "startDate is required and must be in `yyyy-MM-dd HH:mm:ss` format", "error": "Bad Request" } ``` ### 401 — Unauthorized The `x-api-token` header is missing or the token is invalid. ```json { "message": "Unauthorized access", "error": "Unauthorized" } ``` Ensure the token is included on every request: ```http x-api-token: MRSCRAPER_API_TOKEN ``` ### 403 — Forbidden The token is authenticated but lacks permission for the requested resource. ```json { "message": "You do not have permission to access this resource", "error": "Forbidden" } ``` ### 404 — Not Found The referenced scraper, result, or other entity could not be found. ```json { "message": "Resource not found", "error": "Not Found" } ``` ### 422 — Unprocessable Entity The request was understood but failed validation. ```json { "message": "Validation failed", "error": "Unprocessable Entity" } ``` ### 429 — Too Many Requests You have exceeded your rate limit. ```json { "message": "Too many requests. Please try again later." } ``` # API Overview import { Braces, Key, ListOrdered, CircleAlert, Repeat, Database, FileSearch, House, PlaneTakeoff, PanelTop, SlidersHorizontal } from 'lucide-react'; ## Introduction The MrScraper API provides a powerful and flexible way to programmatically scrape web pages and retrieve structured data. Built on REST principles, our API offers predictable resource-oriented URLs, accepts JSON-encoded request bodies, returns JSON-encoded responses, and uses standard HTTP response codes. ## API Features }> All requests with a payload require `Content-Type: application/json` header. All responses are returned in JSON format. }> All API endpoints require authentication using an API token passed in the `x-api-token` header. See the [Authentication](/docs/api/authentication) page for details. }> List endpoints support pagination to handle large datasets efficiently. See the [Pagination](/docs/api/pagination) page for details. }> Track token usage, bandwidth usage, execution runtime, and target status codes in response headers. See the [Response Headers](/docs/api/response-headers) page for details. }> The API uses standard HTTP status codes and returns detailed error messages. See the [Error Handling](/docs/api/error-handling) page for details. ## Available Endpoints } href="/docs/api/v3/scraper/ai-rerun"> Rerun scrapers using the Platform API with updated URL. } href="/docs/api/ecommerce/shopee/scraper"> Run the e-commerce scraper to fetch product, pricing, and listing data from marketplace. } href="/docs/api/travel/agoda/review-scraper"> Run the travel scraper to collect the latest reviews hotel and flight data from travel website. # Pagination ## Overview The MrScraper API uses cursor-based pagination for endpoints that return lists of resources. This approach ensures consistent results even when data is being added or modified. ## Pagination Parameters The `/api/v1/results` endpoint supports the following pagination parameters: ### Required Parameters | Parameter | Type | Description | Example | | ----------- | ------ | --------------------------------------- | ----------- | | `page` | number | The page number to retrieve (1-indexed) | `1` | | `pageSize` | number | Number of items per page | `10` | | `sortField` | string | Field name to sort by | `updatedAt` | | `sortOrder` | string | Sort direction (`ASC` or `DESC`) | `DESC` | * **Small datasets (\< 100 items):** Use `pageSize=50` or `pageSize=100` * **Large datasets:** Use `pageSize=10` or `pageSize=25` for faster responses * **Maximum recommended:** `pageSize=100` to avoid timeouts ### Optional Parameters | Parameter | Type | Description | Example | | ----------------- | ------ | ----------------------------------- | -------------------------- | | `search` | string | Search across all entity columns | `example.com` | | `dateRangeColumn` | string | Column name for date filtering | `createdAt` | | `startAt` | string | Start date for filtering (ISO 8601) | `2025-11-13T02:46:49.014Z` | | `endAt` | string | End date for filtering (ISO 8601) | `2025-11-14T02:46:49.014Z` | ## Making Paginated Requests ### Basic Example ```bash curl -X GET "https://api.app.mrscraper.com/api/v1/results?page=1&pageSize=10&sortField=updatedAt&sortOrder=DESC" \ -H "x-api-token: MRSCRAPER_API_TOKEN" \ -H "accept: application/json" ``` ### With Search ```bash curl -X GET "https://api.app.mrscraper.com/api/v1/results?page=1&pageSize=10&sortField=updatedAt&sortOrder=DESC&search=example.com" \ -H "x-api-token: MRSCRAPER_API_TOKEN" \ -H "accept: application/json" ``` ### With Date Range Filtering ```bash curl -X GET "https://api.app.mrscraper.com/api/v1/results?page=1&pageSize=10&sortField=updatedAt&sortOrder=DESC&dateRangeColumn=createdAt&startAt=2025-11-13T00:00:00.000Z&endAt=2025-11-14T23:59:59.999Z" \ -H "x-api-token: MRSCRAPER_API_TOKEN" \ -H "accept: application/json" ``` ## Response Format ### Successful Response ```json { "message": "Successful fetch", "data": [ { "id": "550e8400-e29b-41d4-a716-446655440000", "userId": "660e8400-e29b-41d4-a716-446655440000", "scraperId": "770e8400-e29b-41d4-a716-446655440000", "type": "AI", "url": "https://example.com", "status": "Finished", "error": null, "tokenUsage": 5, "runtime": 2.5, "data": "{\"title\":\"Example Page\"}", "createdAt": "2025-11-11T09:50:09.722Z" } ], "meta": { "page": 1, "pageSize": 10, "total": 100, "totalPage": 10 } } ``` ### Response Metadata The `meta` object provides pagination information: | Field | Type | Description | | ----------- | ------ | -------------------------------------- | | `page` | number | Current page number | | `pageSize` | number | Items per page | | `total` | number | Total number of items across all pages | | `totalPage` | number | Total number of pages available | # Response Headers When making requests via MrScraper's **Manual API** and **Unblocker API**, essential metrics about your request execution are returned directly in the HTTP response headers. These response headers allow you to programmatically monitor resource consumption, response timing, and target page statuses without parsing separate metadata fields. Response header metrics (`token_usage`, `bandwidth_usage`, `runtime`, and `x-status-code`) are **only supported when using the Manual Scraper API and Unblocker API**. ## Key Response Headers Every scraping response includes metadata headers detailing execution metrics: | Header Name | Type | Description | | :---------------- | :-------- | :------------------------------------------------------------------ | | `token_usage` | `number` | The total number of API tokens consumed by the scraping request. | | `bandwidth_usage` | `number` | The total data bandwidth consumed by the request (in Megabytes/MB). | | `runtime` | `float` | Total execution runtime for the request in seconds. | | `x-status-code` | `integer` | The HTTP status code returned by the target website being scraped. | Responses may also include `x-result-id` (a unique identifier for the execution result). ### Target Status Code (`x-status-code`) The `x-status-code` response header allows you to programmatically detect the status of the target page being scraped: * **Success (`2xx`)**: Indicates the target page was fetched successfully (e.g., `200 OK`). * **Forbidden / Blocked (`403`)**: Indicates access was blocked or denied by security or anti-bot protections on the target site. * **Page Not Found (`404`)**: Indicates the requested page was not found on the target site. * **Target Server Error (`500`)**: Indicates an internal error occurred on the target website's server. ## Example API Request & Response Headers ### cURL Example Below is an example request using the Manual / Unblocker API: ```bash curl --location 'https://api.mrscraper.com?token=&geoCode=us&html=true&proxyCountry=us&url=https%3A%2F%2Fwww.scrapethissite.com' \ -H 'x-api-token: ' ``` ### Sample HTTP Response Headers When inspecting the headers returned by the server, you will receive output similar to the following: ```http HTTP/1.1 200 OK Server: nginx Content-Type: text/html; charset=utf-8 token_usage: 2 bandwidth_usage: 0.008867263793945312 runtime: 0.6612532138824463 x-status-code: 200 x-result-id: ``` ## Response Metrics in the Playground UI If you test your requests interactively inside the **MrScraper Playground**, these response header metrics are prominently displayed in the **Extracted Data** section toolbar: Response Headers in Playground UI As highlighted in red on the screenshot above: * **`200 response`**: Corresponds to the target status code (`x-status-code`). * **`3,591 ms`**: Displays the request runtime (`runtime`). * **`2 tokens`**: Displays total tokens used for the request (`token_usage`). # Scraper API import { Step, Steps } from 'fumadocs-ui/components/steps'; MrScraper lets you run any scraper you've created directly through an API endpoint. This is useful when you want to automate scraping, integrate it into your application, or test requests in tools like Postman. ## How to Run a Scraper Through the API ### Setup Steps Open your dashboard and select a scraper you've already created. If you don't have one yet, create it first. In the scraper window, click **Settings**. Select **AI Scraper API Access** and slide to activate it. This turns off the AI chat for this scraper and enables API mode. After activation, copy the example API request shown on the page. Paste the request into [Postman](https://www.postman.com/) (or another API tool). You can replace the URL with any URL you want to scrape. Hello You can also run the scraper directly from [Rerun an AI Scraper endpoint](/docs/api/v3/scraper/ai-rerun). ## API Request Parameters | Parameter | Type | Location | Required | Default | Description | | ------------- | ------- | -------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `x-api-token` | string | header | Yes | - | Your MrScraper API token | | `scraperId` | string | body | Yes | - | The unique ID of the scraper you activated | | `url` | string | body | Yes | - | The webpage URL you want to scrape | | `maxRetry` | integer | body | No | `3` | Number of retry attempts if the scraping fails | | `maxPages` | integer | body | No | 0 | Maximum number of pages to scrape (useful for listing/pagination). Use this to control how many pages you want to extract from multi-page listings | | `timeout` | integer | body | No | `None` | Request timeout in seconds. Increase this value if you're scraping many pages or complex websites | | `stream` | boolean | body | No | `false` | Enable streaming response mode. **Highly recommended when scraping multiple pages** that may take several minutes. If the connection is interrupted mid-process, you'll still receive the partial data that was already scraped from completed pages | If the website has fewer pages than maxPages, it will scrape all available pages. ## Example Request Body ### Basic Single-Page Scraper ```json { "scraperId": "1c9f4043-b572-462a-ab94-a330672f1af5", "url": "https://example.com/property", "maxRetry": 3, "maxPages": 1, "timeout": 300, "stream": false } ``` ### Listing Agent API (Multi-Page Scraper) When scraping listings or paginated content, use the `maxPages` parameter: ```json { "scraperId": "1c9f4043-b572-462a-ab94-a330672f1af5", "url": "https://www.zillow.com/ky/", "maxRetry": 3, "maxPages": 4, "timeout": 1100, "stream": false } ``` **Timeout Guidelines:** * For single-page scraping: 300-600 seconds is usually sufficient * For multi-page scraping: Increase timeout proportionally to `maxPages` (e.g., 1100+ seconds for 4+ pages) * If you encounter timeout errors, increase the `timeout` value ### Scraping with Streaming Response ```json { "scraperId": "your-scraper-id", "url": "https://example.com/search", "maxRetry": 3, "maxPages": 10, "timeout": 2000, "stream": true } ``` **When to Use `stream: true`:** * When scraping multiple pages (`maxPages` > 1) that may take several minutes * For long-running scraping jobs to prevent data loss * If connection interruptions occur mid-process, you'll still receive partial data from pages that were already scraped * Ensures you don't lose all progress if the request is interrupted before completion ## Common Use Cases ### Scraping a Single Product Page ```json { "scraperId": "your-scraper-id", "url": "https://shop.example.com/product/123", "maxRetry": 2 } ``` ### Scraping Real Estate Listings (Multiple Pages) ```json { "scraperId": "your-scraper-id", "url": "https://www.zillow.com/ky/", "maxRetry": 3, "maxPages": 10, "timeout": 2000, "stream": true } ``` For multi-page scraping, always set `stream: true` to ensure you receive data from completed pages even if the process is interrupted. ## Best Practices 1. **Start with lower `maxPages` values** and increase gradually to test performance 2. **Adjust timeout based on website complexity** and number of pages 3. **Use `stream: true` for multi-page scraping** to prevent data loss if the connection is interrupted 4. **Use appropriate retry values** (3 is recommended for most cases) 5. **Monitor your API usage** to stay within rate limits 6. **Test with a single page first** before running large batch operations ## Troubleshooting | Issue | Solution | | ----------------------------- | ------------------------------------------------------------------ | | Timeout errors | Increase the `timeout` parameter, especially when using `maxPages` | | Failed requests | Check `maxRetry` value and ensure your API token is valid | | Incomplete data from listings | Increase `maxPages` or verify the URL structure | | Rate limiting | Reduce concurrent requests or contact support for higher limits | If you encounter issues or need assistance with API integration, contact our support team or check the [API Reference](/docs/api/v3/scraper/ai-rerun) for detailed endpoint documentation. # Usage Analytics ## Usage Analytics The Usage Analytics dashboard provides real-time insights into your scraping operations, performance metrics, and token consumption. Monitor your scraper activity, identify trends, and optimize your scraping workflows with comprehensive analytics. *** ## Overview Access the Usage Analytics dashboard from your [MrScraper account](https://app.mrscraper.com) by navigating to the **Analytics** section in the sidebar. The dashboard displays key performance indicators and historical data to help you understand your scraping patterns and resource usage. *** ## Dashboard Tabs The analytics dashboard is organized into three main sections: | Tab | Description | | --------------- | -------------------------------------------------- | | **Metrics** | Performance metrics and scraping statistics | | **Targets** | Domain-specific analytics and target tracking | | **Token Usage** | Token consumption trends and allocation monitoring | Switch between tabs using the buttons at the top right of the dashboard to view different aspects of your analytics. *** ## Filtering Options ### Standard Filters All users have access to the following filter options: **Time Range Filter** Select the time period for your analytics: * Last 1 Hour * Last 24 Hours * Last 7 Days * Last 30 Days * Custom Date Range **Timezone Filter** View analytics in your preferred timezone: * WIB (GMT+7) - Asia/Jakarta * UTC * Your local timezone * Other timezones **API Token Filter** Filter analytics by specific API tokens to track usage per integration or application. *** ### Enterprise Filters Enterprise customers have access to additional filtering capabilities for granular analytics. **Action Filter** (Enterprise Only) Filter analytics by specific scraping actions: * **Create Scraper** - Track scraper creation operations * **Rerun Scraper** - Monitor scraper rerun activity * **Bulk Operations** - Analyze bulk scraping jobs * **Fetch HTML** - View raw HTML fetch requests * **All Actions** - See combined activity across all action types This allows enterprise customers to: * Identify which operations consume the most tokens * Optimize workflows based on action-specific metrics * Track usage patterns by operation type * Create detailed cost allocation reports **Domain/Target Filter** (Enterprise Only) Filter by specific domains or targets: * View metrics for individual websites * Track success rates per domain * Monitor domain-specific latency * Analyze which targets require the most resources *** ## Key Metrics The dashboard displays four primary performance indicators: ### Average Latency **What it measures:** The average response time for your scraping requests. **Display:** Clock icon with response time in milliseconds or seconds **Use cases:** * Identify slow-performing targets * Optimize scraper configurations * Monitor infrastructure performance * Set realistic timeout values Lower latency indicates faster scraping operations. If you notice high latency, consider: * Using different proxy regions * Adjusting timeout settings * Targeting less complex pages * Upgrading to faster scraping modes *** ### Request Per Minute **What it measures:** The number of scraping requests processed per minute. **Display:** Lightning bolt icon with request count **Use cases:** * Monitor API rate limits * Track scraping velocity * Identify peak usage periods * Plan capacity requirements **Rate Limits by Plan:** | Plan | Concurrent Requests | | ---------- | ------------------- | | Free | 10 requests | | Standard | 50 requests | | Pro | 100 requests | | Enterprise | Custom | *** ### Success Rate **What it measures:** The percentage of successful scraping operations. **Display:** Green checkmark icon with percentage **Formula:** `(Successful Requests / Total Requests) × 100` **Use cases:** * Evaluate scraper reliability * Identify problematic targets * Monitor configuration effectiveness * Track improvements over time **Typical Success Rates:** | Rate | Status | Action Needed | | --------- | --------- | ------------- | | 95-100% | Excellent | None | | 85-94% | Good | Monitor | | 70-84% | Fair | Optimize | | Below 70% | Poor | Investigate | If your success rate is below 85%, consider: * Checking target website availability * Adjusting scraper configurations * Using residential proxies * Updating your scraper selectors * Contacting support for assistance *** ## Run Status Breakdown Track the status of all your scraping operations: ### Total Runs **Display:** Chart icon with total count **What it shows:** Total number of scraping operations in the selected time period **Includes:** All runs regardless of status (success, pending, failed) *** ### Success **Display:** Green checkmark icon with count **What it shows:** Number of successfully completed scraping operations **Definition:** Requests that: * Completed without errors * Returned valid data * Met all scraping criteria * Consumed expected tokens *** ### Pending **Display:** Orange clock icon with count **What it shows:** Number of scraping operations currently in progress **Includes:** * Queued requests waiting to start * Active scraping operations * Processing results * Bulk operations in progress Pending requests consume resources but don't count as successes or failures until they complete. *** ### Failed **Display:** Red X icon with count **What it shows:** Number of failed scraping operations **Common Failure Reasons:** * Target website returned errors (404, 500, etc.) * Timeout exceeded * Authentication failed * Invalid scraper configuration * Anti-scraping measures blocked the request * Network connectivity issues **Troubleshooting Failed Requests:** 1. Check the error details in the Results page 2. Verify the target URL is accessible 3. Review scraper configuration 4. Test with different proxy settings 5. Contact support if issues persist *** ## Usage Analytics Chart The **Usage Analytics** section displays a time-series graph showing your scraping activity over time. ### What the Chart Shows **Scraping Activity Over Time** * X-axis: Time period (hours, days, weeks) * Y-axis: Number of scraping operations * Data points: Individual scraping requests **Color Coding:** * **Green areas**: Successful requests * **Red areas**: Failed requests * **Orange areas**: Pending requests ### Using the Chart **Identify Patterns:** * Peak usage hours * Recurring scraping schedules * Unusual activity spikes * Downtime periods **Optimize Operations:** * Schedule bulk operations during off-peak hours * Identify the best times for scraping * Avoid peak hours for rate-limited targets * Plan capacity based on usage trends **Monitor Health:** * Spot sudden drops in success rates * Identify failing patterns * Track recovery after issues * Validate configuration changes *** ## Accessing Analytics via API Enterprise customers can access analytics data programmatically using our API endpoints. ### Status Endpoint ```bash GET https://api.app.mrscraper.com/api/v1/analytic/statuses?token=MRSCRAPER_API_KEY ``` **Returns:** * Total runs * Success count * Pending count * Failed count * Success rate percentage *** ### Timeline Endpoint ```bash GET https://api.app.mrscraper.com/api/v1/analytic/timeline?token=MRSCRAPER_API_KEY&startDate=2024-01-01&endDate=2024-12-31 ``` **Returns:** * Time-series data for the specified period * Request counts by hour/day/week * Success/failure breakdown * Latency metrics See our [Authentication documentation](/docs/api/authentication#analytics-authentication) for details on analytics API authentication. *** ## Best Practices ### Monitor Regularly * **Daily**: Check success rates and pending requests * **Weekly**: Review usage trends and optimize configurations * **Monthly**: Analyze token consumption and plan capacity ### Set Up Alerts Configure notifications for: * Success rate drops below threshold * Unusual spike in failed requests * Token usage approaching limits * Pending requests backlog ### Use Filters Effectively * **Time Range**: Start broad, then narrow down to specific periods * **API Tokens**: Track usage per integration separately * **Actions** (Enterprise): Identify high-cost operations * **Domains** (Enterprise): Monitor per-target performance ### Optimize Based on Data **If success rate is low:** * Review failed request details * Adjust proxy settings * Update scraper configurations * Consider using Super mode for difficult targets **If latency is high:** * Check target website performance * Use closer proxy regions * Reduce page complexity * Optimize extraction patterns **If token usage is high:** * Review scraper efficiency * Eliminate unnecessary requests * Use caching where possible * Consider bulk operations *** ## Understanding Token Consumption ### Token Usage by Operation Different operations consume different amounts of tokens: | Operation | Typical Token Cost | | ------------------------ | ------------------- | | Simple HTML fetch | 1-2 tokens | | General Agent extraction | 5-10 tokens | | Listing Agent (per page) | 3-7 tokens | | Map Agent | 2-4 tokens | | Bulk operations | Varies by URL count | ### Tracking Token Usage Monitor your token consumption in the **Token Usage** tab: * Daily usage trends * Breakdown by scraper * Breakdown by API token * Projection to month-end * Remaining balance Set up usage alerts to be notified when you reach 80% and 90% of your monthly token allocation. This gives you time to optimize or upgrade before hitting limits. # Bulk Scraping import { Step, Steps } from 'fumadocs-ui/components/steps'; MrScraper's Bulk Scraping feature allows you to extract data from multiple URLs in a single operation. Instead of scraping URLs one at a time, you can upload a list and apply the same scraping configuration to all of them, making it easy to gather large datasets efficiently. Scraping multiple pages with similar structures, such as product listings, article archives, user profiles, or directory entries. ## How to Configure Bulk Scraping ### Prepare Your URLs You have two options for providing URLs: #### Option A: Excel File Upload Create an Excel file (`.xlsx` or `.xls`) with a column header named `url` or `URL`. Each row should contain one URL. Your spreadsheet should look like this: | url | | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | [https://books.toscrape.com/catalogue/a-light-in-the-attic\_1000/index.html](https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html) | | [https://books.toscrape.com/catalogue/tipping-the-velvet\_999/index.html](https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html) | | [https://books.toscrape.com/catalogue/soumission\_998/index.html](https://books.toscrape.com/catalogue/soumission_998/index.html) | | [https://books.toscrape.com/catalogue/sharp-objects\_997/index.html](https://books.toscrape.com/catalogue/sharp-objects_997/index.html) | #### Option B: Direct Input Paste URLs directly into the text area, one URL per line: * [https://books.toscrape.com/catalogue/a-light-in-the-attic\_1000/index.html](https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html) * [https://books.toscrape.com/catalogue/tipping-the-velvet\_999/index.html](https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html) * [https://books.toscrape.com/catalogue/soumission\_998/index.html](https://books.toscrape.com/catalogue/soumission_998/index.html) * [https://books.toscrape.com/catalogue/sharp-objects\_997/index.html](https://books.toscrape.com/catalogue/sharp-objects_997/index.html) ### Upload Your URLs 1. Open a new or existing scraper in your MrScraper dashboard 2. Click the **Multiple URLs** button in the top section 3. Choose your input method: * Click **Upload File** to select an Excel file from your computer * Or paste URLs directly into the text area 4. Click **Save** to confirm your URL list ### Run the Bulk Scrape 1. Click **Run All** to start the bulk scraping process 2. Click Result to view the scraper’s progress and results in real time Each URL in your bulk scrape consumes tokens. Ensure you have sufficient tokens in your account before starting a large bulk operation. Check your token balance in your account settings. ### Monitor Progress The **Result** page displays real-time progress information: **While Scraping:** ```json title="In Progress" { "mergedData": null, "urlDetails": [], "summary": { "totalUrls": 4, "successfulUrls": 0, "failedUrls": 0, "scrapedCount": 0, "totalTokenUsage": 0, "estimatedFinishAt": null } } ``` **Progress Indicators:** | Field | Description | | ------------------- | -------------------------------------- | | `totalUrls` | Total number of URLs being scraped | | `successfulUrls` | Number of URLs scraped successfully | | `failedUrls` | Number of URLs that encountered errors | | `scrapedCount` | Current number of completed scrapes | | `totalTokenUsage` | Total tokens consumed so far | | `estimatedFinishAt` | Estimated completion time | Once scraping completes, the Result page displays all extracted data: ```json title="Completed Results" [ { "1": { "id": "a897fe39b1053632", "name": "A Light in the Attic", "price": "£51.77", "rating": null, "source": "product", "features": [ "Classic collection of poetry and drawings from Shel Silverstein", "20th anniversary special edition", "Humorous and creative verse" ], "return_policy": null, "shipping_info": { "availability": "In stock (22 available)", "shipping_arrival": null }, "specifications": { "UPC": "a897fe39b1053632", "Product Type": "Books", "Price (excl. tax)": "£51.77", "Price (incl. tax)": "£51.77", "Tax": "£0.00", "Availability": "In stock (22 available)", "Number of reviews": "0" } } }, { "1": { "id": "90fa61229261140a", "name": "Tipping the Velvet", "price": "£53.74", "rating": null, "source": "product", "features": [ "Erotic and absorbing...Written with starling power.", "Nan King, an oyster girl, is captivated by the music hall phenomenon Kitty Butler" ], "shipping_info": { "availability": "In stock (20 available)", "shipping_arrival": null }, "specifications": { "UPC": "90fa61229261140a", "Product Type": "Books", "Price (excl. tax)": "£53.74", "Price (incl. tax)": "£53.74", "Tax": "£0.00", "Availability": "In stock (20 available)", "Number of reviews": "0" } } } ] ``` For more details, refer to the [Bulk Scraping API](/docs/api/v3/scraper/ai-rerun-bulk). ### Cancel Bulk Scraping You can cancel a bulk scraping job using either the **Scraper ID** or the **Result ID**. #### Option 1: Cancel by Scraper ID Cancel all pending URLs associated with a scraper. Use this when you want to stop the bulk operation immediately. ```bash curl -X PATCH "https://api.app.mrscraper.com/api/v1/scrapers-bulks/{scraperId}/cancel" \ -H "x-api-token: MRSCRAPER_API_KEY" ``` **Parameters:** * `{scraperId}` - The scraper ID used for the bulk operation #### Option 2: Cancel by Result ID Cancel a bulk scraping job using the bulk result ID returned from the bulk rerun response. This is useful when you have the result ID from the Result page and want to cancel that specific bulk operation. ```bash curl -X PATCH "https://api.app.mrscraper.com/api/v1/scrapers-bulks/result/{resultId}/cancel" \ -H "x-api-token: MRSCRAPER_API_KEY" ``` **Parameters:** * `{resultId}` - The bulk result ID from the bulk scraping response (e.g., `ec5d81b9-55fa-4949-bba9-cceb448cb950`) **Response:** Both endpoints return the bulk result ID, the original URL list, and which URLs were successfully canceled: ```json { "message": "Successful operation!", "data": { "id": "ec5d81b9-55fa-4949-bba9-cceb448cb950", "bulkUrls": [ "https://books.toscrape.com/catalogue/page-1.html", "https://books.toscrape.com/catalogue/page-2.html", "https://books.toscrape.com/catalogue/page-3.html", "https://books.toscrape.com/catalogue/page-4.html", "https://books.toscrape.com/catalogue/page-5.html" ], "canceledUrls": [ "https://books.toscrape.com/catalogue/page-3.html", "https://books.toscrape.com/catalogue/page-4.html", "https://books.toscrape.com/catalogue/page-5.html" ] } } ``` For detailed API documentation, see: * [Cancel Bulk Scraping by Scraper ID](/docs/api/v3/scraper/cancel-bulk-by-scraper-id) * [Cancel Bulk Scraping by Result ID](/docs/api/v3/scraper/cancel-bulk-by-result-id) # Cookie import { Step, Steps } from 'fumadocs-ui/components/steps'; Cookies are small pieces of data stored by your web browser that help websites remember your preferences and session information. In MrScraper, you can use cookies to access content that may be restricted or personalized based on your previous interactions with a website. Here's how to set up and use cookies in MrScraper: ## Obtaining Cookies You’ll first need to extract the cookies from your browser. There are a few ways to do this: ### Using Browser Developer Tools #### Chrome / Edge / Brave 1. Open your web browser and navigate to the website you want to scrape. 2. Open the developer tools (usually by pressing `F12` or right-clicking and selecting "Inspect"). 3. Go to the "Application" or "Storage" tab, then find the "Cookies" section. 4. Select the website's URL to view the cookies stored for that site. 5. Copy the cookies you need (usually as key-value pairs). #### Firefox 1. Open your web browser and navigate to the website you want to scrape. 2. Right click on the page and select "Inspect Element" to open the developer tools. 3. Go to the "Storage" tab, then find the "Cookies" section. 4. Copy the cookies you need (usually as key-value pairs). ### Using Browser Extensions You can also use browser extensions to quickly view, manage, and export cookies. Popular options include: * [EditThisCookie](https://chromewebstore.google.com/detail/editthiscookie-v3/ojfebgpkimhlhcblbalbfjblapadhbol) (Chrome) * [Cookie Quick Manager](https://addons.mozilla.org/en-US/firefox/addon/cookie-quick-manager/) (Firefox) * [Cookie Editor](https://cookie-editor.com/) ## Using Cookies in your Scraper 1. Navigate to your [MrScraper dashboard](https://app.mrscraper.com/). 2. click **Scraper** in the left sidebar, and click on the scraper you want to add cookies to. 3. Press the ellipsis (three dots) button on the top right corner of the scraper details page. 4. Select **Set Cookies Settings** from the dropdown menu. 5. Paste the JSON cookies you exported earlier into the provided field. ```json title="Example Cookie JSON" [ { "domain": "www.example.com", "expirationDate": 1764823423.823961, "hostOnly": true, "httpOnly": false, "name": "token", "path": "/", "sameSite": "strict", "secure": false, "session": false, "storeId": null, "value": "eyJhbGciOiJIUzI1NiJ9.eyJ1c2VybmFtZSI6ImtpemEiLCJleHAiOjE3NjQ4MjM0MjN9.qvviwvaTZDc9ey0iW638cdZr-EdiGHOfKsvTahZ_nIs" } ] ``` # Marketplace import { Step, Steps } from 'fumadocs-ui/components/steps'; Marketplace provides a collection of ready-to-use scrapers built on MrScraper's **PDP Cache Agent** technology. These scrapers are pre-configured, optimized, and instantly runnable for popular websites and use cases. Each marketplace scraper is created specifically for your target website with optimized selectors and prompts. Access and run them directly from the [MrScraper dashboard](https://app.mrscraper.com/marketplace) in seconds. Learn how PDP Cache works and why marketplace scrapers are so cost-efficient [here](/docs/features/ai-scraper/pdp). ## List of Available Scrapers ### AI Search | Name | Pricing (tokens/run) | Description | | ---------------- | -------------------: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Gemini - Ask | 10 | Ask a query to Gemini, and it returns a structured, AI-generated text answer based on the query. Suitable for research, product lookups, or general knowledge questions. | | Google AI-Mode | 10 | AI Overviews in Google is a search engine feature that generates a summary text response to a user's query directly in the search results. This block appears at the top of the output and is a structured response combining information from multiple sources. | | GPT - Web Search | 25 | Performs a web search using GPT and returns a structured, AI-generated text answer based on the query, suitable for research, product lookups, or general knowledge questions. | ### E-commerce | Name | Pricing (tokens/run) | Description | | -------------------------------- | -------------------: | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1688 - Collect By Category | 29 | This scraper fetches product listings from 1688 based on search keywords and filters, and also retrieves related or recommended items from the platform’s API, providing structured data for offers and recommendation results. | | AliExpress - Product Detail | 20 | Scrape product detail information from an AliExpress product page URL | | Amazon - Product Details | 50 | Scrapes comprehensive product information from an Amazon product detail page, including pricing, availability, reviews, specifications, images, videos, delivery info, categories, and brand content. | | AutoZone - Fitment Check | 16 | Check vehicle fitment compatibility for a specific AutoZone part/SKU against a vehicle make, model, year, and engine type. | | AutoZone - Product Details | 24 | Scrape product detail information from AutoZone product pages including pricing, availability, store info, media, shipping, and reviews. | | Autozone - Category Products | 12 | Scrape product listings from an AutoZone category page URL | | Bestbuy - Product Details | 10 | Extracts comprehensive product details, including pricing, inventory, specifications, and media, from Best Buy product pages. | | Big W - Category | 36 | This scraper extracts product listings from a BIG W category page, returning organic results along with metadata such as total results, pagination details, and query identifiers, while also capturing sponsored items and related search suggestions. | | Big W - Product Detail | 23 | This scraper retrieves the raw HTML content of a BIG W product detail page. | | Blibli - Product Details | 10 | Extracts comprehensive product details, including pricing, stock, specifications, and seller information, from Blibli product pages. | | CVS - Product Detail | 25 | This scraper extracts product details from CVS, including name, price, brand, and stock availability. It also retrieves basic product metadata from structured data. | | eBay - Product Details | 15 | Extracts comprehensive product details, including pricing, specifications, seller info, and media, from eBay listings. | | Fruugo - Product Details | 10 | Scrapes product detail information from a Fruugo product page. | | Geizhals - Product Details | 10 | Extracts comprehensive product details, pricing, stock, and seller information from Geizhals product pages for price comparison and market analysis. | | Klik Indomaret - Product Details | 30 | Extracts comprehensive product details, including pricing, stock, specifications, and ratings, from Klik Indomaret online minimarket listings. | | Lazada - Category & Search | 10 | Scrapes product listing data from a Lazada category or search results page using a catalog URL with a query or category ID. May return a CAPTCHA challenge response if bot protection is triggered. | | Lazada - Product Details | 10 | Scrapes product detail information from a Lazada product page, including pricing, rating, reviews, stock status, delivery details, and seller/store information. | | Lowes - Product Detail | 10 | Lowes PDP ( product detail ) scraper. | | Meijer - Product Detail | 10 | Meijer PDP ( product detail ) scraper. | | MercadoLibre - Product Details | 10 | A MercadoLibre product scraper extracts structured product data from MercadoLibre by accessing a product page URL and optionally rendering dynamic content, returning both the raw MercadoLibre and parsed product details such as product information, pricing, and attributes, enabling automated product data collection and analysis without manual browsing. | | MercadoLibre - Search | 17 | Search for product/vehicle listings on MercadoLibre by keyword, location (zipcode), and page number. Returns paginated results with pricing, mileage, year, location, and listing URL. Also provides a CSV download link of all results. | | Nordstorm - Collect By Category | 32 | This scraper extracts product listings from Nordstrom category pages. It enables structured collection of multiple items within a category. | | Nordstorm - Product Detail | 93 | Full Nordstrom product page URL used to extract item details such as name, price, description, images, and availability. | | Olx - Automotive Page | 10 | Extracts comprehensive automotive product details, including pricing, specifications, and seller information, from OLX Indonesia listings. | | Segari - Product Detail | 24 | Scrape product detail information from a Segari product page URL | | SHEIN - Product Details | 30 | A SHEIN product scraper extracts structured product data from SHEIN by accessing a product page URL and optionally rendering dynamic content, returning both the raw HTML and parsed product details such as product information, pricing, and attributes, enabling automated product data collection and analysis without manual browsing. | | Stockx - Product Details | 10 | Extracts comprehensive product details, including pricing, stock status, specifications, and media, from StockX listings. | | Taobao - Product Search | 10 | Search Taobao products by keyword. Each result includes an item ID that can be used with the Taobao Product Detail endpoint to retrieve product and shop information. | | Tiktok - HashTags | 10 | Tiktok HashTags information scraper. | | Tiktok - Product Page | 12 | This scraper collects detailed information from TikTok Shop product pages. It extracts the canonical product URL, product ID, product category, page routing metadata, user environment info (language, device, OS, user agent), region and server routing data, WAF security decisions, bot detection flags, and any A/B test configurations. The response also includes any errors encountered during scraping and allows tracking of the time taken to complete the process. It provides a structured view of product-level page data for analysis or automation purposes. | | Tiktok - Video | 12 | Extract data from chosen tiktoks. Just add a TikTok URL and get TikTok video and profile data: URLs, numbers of shares, followers, hashtags, hearts, video, and music metadata. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools. | | Tiktok - Web Catalog | 36 | This scraper collects detailed information from TikTok Shop category pages. It extracts the canonical category URL, category ID and name, page routing metadata, user environment info (language, device, OS, user agent), region and infrastructure routing data, WAF security decisions, bot detection flags, and A/B test configurations. The response also includes any errors encountered during scraping and allows tracking of time taken to complete the process. It is designed to provide a structured view of category-level page data for analysis or automation purposes. | | Tokopedia - Product Detail | 28 | Scrape product detail information from a Tokopedia product page URL | | Walmart - Product Page | 10 | Extracts comprehensive product details, pricing, inventory, and customer ratings for specific items listed on Walmart.com. | | Watsons - Category Products | 21 | Scrape product listings from a Watsons category page URL | | Worten - Product Details | 20 | Extracts comprehensive product details, including pricing, specifications, stock, and media, from Worten product pages. | | Zepto - Product Detatils | 9 | A Zepto product scraper extracts structured product information from Zepto by accessing a product page URL along with location-specific details such as pincode, returning data including product details, pricing, ratings, availability, highlights, and seller information, enabling automated product tracking, regional availability analysis, and data collection without manual browsing. | ### News | Name | Pricing (tokens/run) | Description | | -------------------- | -------------------: | -------------------------------------------------------- | | Echemi - News Detail | 16 | Scrape news article details from an Echemi news page URL | ### Real Estate | Name | Pricing (tokens/run) | Description | | -------------------------------------- | -------------------: | ----------------------------------------------------------------------------------------------------------------------------------- | | Cireba - Property Details | 10 | Extracts comprehensive property listing details, including pricing, features, location, and history, from Cireba real estate pages. | | LiveInAlabama - Property Details | 60 | Extracts comprehensive property listing details, including pricing, features, and history, from LiveInAlabama real estate listings. | | Olx - Property Details | 15 | Extracts comprehensive details for property listings on OLX, including pricing, features, location, and historical data. | | Realtor.com - Real Estate Page Details | 15 | Extracts comprehensive property listing details, including pricing, features, location, and historical data, from Realtor.com. | | Zillow - Property Details | 44 | Extracts comprehensive property details, including pricing, features, history, and location data from Zillow listings. | ### SEO | Name | Pricing (tokens/run) | Description | | ------------------------------ | -------------------: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | SERP API - Flight Detail | 10 | Get a parsed data from a Google Flight booking page returning key information data such as booking options with price and airlines. Currently only support ID region, please contact if you need support on other region. | | SERP API - Flight Search | 10 | Search and collect flight schedule and price from Google Flight SERP for One-Way and Round-Trip search | | SERP API - Hotel Entity Detail | 10 | Collect hotel detail from Google Hotel based on the Google Hotel entity URL | | SERP API - Images Search | 11 | Search and collect image results from Google Images SERP based on a keyword query | | SERP API - Maps Search Places | 21 | Search and collect Google Maps SERP place listings by keyword and geographic coordinates | | SERP API - Web Results | 1 | Search and collect organic web results from Google Search SERP based on a keyword query | ### Travel | Name | Pricing (tokens/run) | Description | | ---------------------------------------- | -------------------: | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Agoda - Flight Search | 20 | Scrapes flight search results from Agoda Flights, returning outbound and return flight itineraries with pricing, airline details, departure/arrival info, flight duration, cabin class, and analytics context. | | Agoda - Hotel Rates | 46 | Scrapes available room rates and rate plans from an Agoda hotel page for a given check-in period, including pricing, meal inclusions, cancellation policy, promotion details, and sold-out status. | | Booking.com - Hotel Rates | 41 | Scrapes available room rates and rate plans from a Booking.com hotel page for a given check-in/check-out period, including pricing, meal inclusions, cancellation policy, refundability, and promotion details. | | Booking.com - Review | 36 | Extract booking com review just enter your url | | China Eastern Airlines - Ticket Pricing | 56 | A China Eastern Airlines flight scraper extracts structured flight search results from China Eastern Airlines by submitting travel parameters such as origin, destination, passenger details, and departure date, and returns comprehensive itinerary data including departure segments (dptSegments), return segments (rtnSegments), and available fare options (fares), enabling automated flight comparison, pricing analysis, and travel data processing without manual browsing. | | China Southern Airlines - Ticket Pricing | 17 | A China Southern Airlines flight scraper extracts structured flight search results from China Southern Airlines by submitting travel parameters such as origin, destination, passenger details, and departure date, and returns comprehensive itinerary data including departure segments (dptSegments), return segments (rtnSegments), and available fare options (fares), enabling automated flight comparison, pricing analysis, and travel data processing without manual browsing. | | Expedia - Hotel Detail | 31 | Scrapes hotel detail and room offer information from an Expedia hotel page, including pricing, sticky bar data, loyalty messaging, highlighted benefits, room listings header, and navigation links. | | Expedia - Hotel Search | 26 | Scrapes hotel search results from an Expedia search page, returning property listings with hotel name, rating, pricing, images, location, card links, and analytics data for a given destination and date range. | | Google Maps - Place Detail | 29 | Extracts structured data from Google Maps product/place detail pages via a synchronous POST API. Input a target URL to retrieve key listing information including business details, ratings, and location data. | | Hotels.com - Hotel Review | 24 | Scrapes customer reviews from a Hotels.com hotel page, including reviewer details, scores, stay duration, traveler type, and positive/negative comments. | | Tiket - Hotel Details | 12 | Extracts comprehensive details for hotel listings on Tiket.com, including pricing, reviews, facilities, and location data. | | Trip.com - Hotel Rates | 30 | Scrapes available room rates and rate plans from a Trip.com hotel detail page for a given check-in/check-out period, including pricing, meal inclusions, cancellation policy, and promotion data. | | Trip.com - Hotel Review | 31 | Scrapes customer reviews from a Trip.com hotel detail page, including reviewer name, room type, stay date, traveler type, review score, and review title, along with the total review count and hotel URL. | | TripAdvisor - Restaurant | 15 | Extracts comprehensive details for TripAdvisor restaurants, including ratings, reviews, contact information, and location data. | | TripAdvisor - Things to do/Attractions | 15 | Extracts comprehensive details for TripAdvisor tours and attractions, including pricing, availability, reviews, and itinerary information. | ### Others | Name | Pricing (tokens/run) | Description | | -------------------------- | -------------------: | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Echemi - Exhibition Detail | 13 | Scrape exhibition or trade event details from an Echemi event page URL | | Everpro - Shipping Price | 11 | This scraper resolves and returns detailed information for the origin and destination locations of a shipment, including province, city, district, sub-district, postal code, and the package weight, enabling accurate logistics calculations. | | StubHub - Event | 27 | This scraper collects detailed ticket information for a specific StubHub event, including seat sections, rows, prices, availability, event details, venue information, and direct checkout links. | | StubHub - Explore | 15 | This scraper collects event listings from StubHub's Explore page, including event names, dates, times, venues, ticket availability, and direct URLs, along with total events and pages information. | Token usage is depend on domain, workflow, proxy, infrastructure complexity, and other factors. ## Quickstart ### Choose your scraper 1. Go to the [Marketplace section](https://app.mrscraper.com/marketplace). 2. Browse the list of available scrapers and select the one that fits your needs. ### Fill in the required input parameters 1. Each scraper has specific input parameters, such as target URLs, country, language, or other relevant details. 2. Fill in the required fields accurately to ensure the scraper can access and extract the desired data. ### Run the scraper 1. Click the "Run" button to execute the scraper with your provided inputs. 2. Monitor the progress and wait for the scraper to complete its task. 3. Once finished, you can view and copy the extracted data in JSON format. Each Marketplace scraper returns a different response format. Refer to the category-specific documentation for detailed field descriptions and response examples. See [Marketplace Response Formats](/docs/api/pdp/ai-search). # Proxy import { Step, Steps } from 'fumadocs-ui/components/steps'; **Proxy Settings** let you attach a proxy to a specific scraper so that every time that scraper runs — manually, via schedule, or through the API — its requests are routed through the configured proxy. This is useful when you need to: * Mask your scraper's IP address to avoid detection or rate-limiting * Access region-locked or geo-restricted content from a specific country * Route traffic through your own private proxy infrastructure **Proxy Settings** (this page) is a configuration option inside an existing scraper. You attach a proxy so the scraper's requests are routed through it. The scraper defines *what* to scrape; the proxy setting controls *how* the traffic is routed. The **Proxy Scraper** (covered in [Proxy Scraper](/docs/residential-proxy/getting-started/proxy-scraper)) is a standalone tool on the Proxies page. You provide a URL and a proxy country, and MrScraper handles both the scraping and the proxying together. ## How to Configure Proxy Settings ### Open Your Scraper Settings 1. Select the scraper you want to configure in the [**Scrapers**](https://app.mrscraper.com/scrapers) page. 2. Click the **Ellipsis** ( **⋮** ) button in the top right corner and click **Proxy Settings**. ### Enable and Configure Proxy In the **Proxy Settings** panel, you can select one of the following options: #### Option 1: Use MrScraper's Built-in Proxy 1. Toggle **Enable Proxy**. 2. Under **Proxy Type**, choose **MrScraper Proxy**. 3. Select your desired **location/region** from the dropdown list. * Each region provides a local IP address, ideal for geo-targeted scraping. 4. Click **Save Settings**. The scraper will now route traffic through the selected country's proxy network, allowing access to region-specific or restricted content. #### Option 2: Use Custom Proxy Only use proxies from trusted providers to ensure your data remains secure. 1. Toggle **Enable Proxy**. 2. Under **Proxy Type**, choose **Custom Proxy**. 3. Enter your proxy URL in one of the following formats: `http://username:password@host:port` or `https://host:port`. 4. Click **Save Settings** to apply your configuration. Use your own proxy if you have an existing provider, private IP pool, or dedicated proxy infrastructure. #### Route the Entire Request Through the Proxy 1. Toggle **Route entire request through proxy**. This routes page subresources—including CSS, JavaScript, and images—through the proxy so the page loads fully and scripts run reliably. 2. Click **Save Settings** to apply your configuration. Try scraping with this option turned off first to minimize bandwidth usage. If the scrape fails or the page does not load correctly, turn it on. Some websites block CSS, JavaScript, images, or other subresources unless they are also routed through the proxy. Enabling this option can improve reliability, but it uses more bandwidth. ## MrScraper Built-in Proxy Countries Below is the full list of supported countries and regions available for MrScraper's built-in proxies: | Country/Region Name | Country/Region Name | Country/Region Name | Country/Region Name | | --------------------------------- | -------------------------------------------- | -------------------------------------------- | ------------------------------------ | | Afghanistan | Albania | Algeria | American Samoa | | Andorra | Angola | Anguilla | Antarctica | | Antigua and Barbuda | Argentina | Armenia | Aruba | | Australia | Austria | Azerbaijan | Bahamas | | Bahrain | Bangladesh | Barbados | Belarus | | Belgium | Belize | Benin | Bermuda | | Bhutan | Bolivia (Plurinational State of) | Bonaire, Sint Eustatius and Saba | Bosnia and Herzegovina | | Botswana | Bouvet Island | Brazil | British Indian Ocean Territory | | Brunei Darussalam | Bulgaria | Burkina Faso | Burundi | | Cambodia | Cameroon | Canada | Cape Verde | | Cayman Islands | Central African Republic | Chad | Chile | | China | Christmas Island | Cocos (Keeling) Islands | Colombia | | Comoros | Congo | Congo (Democratic Republic of the) | Cook Islands | | Costa Rica | Croatia | Cuba | Curaçao | | Cyprus | Czech Republic | Côte d'Ivoire | Denmark | | Djibouti | Dominica | Dominican Republic | Ecuador | | Egypt | El Salvador | Equatorial Guinea | Eritrea | | Estonia | Eswatini | Ethiopia | Falkland Islands (Malvinas) | | Faroe Islands | Fiji | Finland | France | | French Guiana | French Polynesia | French Southern Territories | Gabon | | Gambia | Georgia | Germany | Ghana | | Gibraltar | Greece | Greenland | Grenada | | Guadeloupe | Guam | Guatemala | Guernsey | | Guinea | Guinea-Bissau | Guyana | Haiti | | Heard Island and McDonald Islands | Holy See (Vatican City State) | Honduras | Hong Kong | | Hungary | Iceland | India | Indonesia | | Iran (Islamic Republic of) | Iraq | Ireland | Isle of Man | | Israel | Italy | Jamaica | Japan | | Jersey | Jordan | Kazakhstan | Kenya | | Kiribati | Korea (Democratic People's Republic of) | Korea (Republic of) | Kuwait | | Kyrgyzstan | Lao People's Democratic Republic | Latvia | Lebanon | | Lesotho | Liberia | Libya | Liechtenstein | | Lithuania | Luxembourg | Macao | North Macedonia | | Madagascar | Malawi | Malaysia | Maldives | | Mali | Malta | Marshall Islands | Martinique | | Mauritania | Mauritius | Mayotte | Mexico | | Micronesia (Federated States of) | Moldova (Republic of) | Monaco | Mongolia | | Montenegro | Montserrat | Morocco | Mozambique | | Myanmar | Namibia | Nauru | Nepal | | Netherlands | New Caledonia | New Zealand | Nicaragua | | Niger | Nigeria | Niue | Norfolk Island | | Northern Mariana Islands | Norway | Oman | Pakistan | | Palau | Palestine, State of | Panama | Papua New Guinea | | Paraguay | Peru | Philippines | Pitcairn | | Poland | Portugal | Puerto Rico | Qatar | | Romania | Russian Federation | Rwanda | Réunion | | Saint Barthélemy | Saint Helena, Ascension and Tristan da Cunha | Saint Kitts and Nevis | Saint Lucia | | Saint Martin (French part) | Saint Pierre and Miquelon | Saint Vincent and the Grenadines | Samoa | | San Marino | Sao Tome and Principe | Saudi Arabia | Senegal | | Serbia | Seychelles | Sierra Leone | Singapore | | Sint Maarten (Dutch part) | Slovakia | Slovenia | Solomon Islands | | Somalia | South Africa | South Georgia and the South Sandwich Islands | South Sudan | | Spain | Sri Lanka | Sudan | Suriname | | Svalbard and Jan Mayen | Sweden | Switzerland | Syrian Arab Republic | | Taiwan (Province of China) | Tajikistan | Tanzania, United Republic of | Thailand | | Timor-Leste | Togo | Tokelau | Tonga | | Trinidad and Tobago | Tunisia | Turkey | Turkmenistan | | Turks and Caicos Islands | Tuvalu | Uganda | Ukraine | | United Arab Emirates | United Kingdom | United States | United States Minor Outlying Islands | | Uruguay | Uzbekistan | Vanuatu | Venezuela (Bolivarian Republic of) | | Viet Nam | Virgin Islands (British) | Virgin Islands (U.S.) | Wallis and Futuna | | Western Sahara | Yemen | Zambia | Zimbabwe | | Åland Islands | | | | Use built-in proxies when you need to scrape data from region-locked websites or compare results between countries (for example, e-commerce prices or job listings). # Scrapers Result import Image from 'next/image' When you run a scraper, the results are typically returned in a structured format such as JSON. The exact structure of the results will depend on the specific scraper you are using and the data it is designed to collect. ## Accessing Scraper Results To access the results from a scraper, you can follow these steps: ### Step 1: Open result page Navigate to **Result page** by clicking the **playlist icon** on the left sidebar. ### Step 2: Click on a scraper to view the results ### Step 3: View the results In this page, there are four tabs you can choose from, **Extractions**, **Screenshots**, **Recording**, and **HTML**. * **Extractions**: This tab displays the structured data extracted by the scraper in JSON format. * **Screenshots**: This tab shows the screenshots of the pages that were scraped. * **Recording**: This tab contains the video recording of the entire scraping session. The **Recording** tab will only be available if you use manual scraper. * **HTML**: This tab provides the raw HTML content of the pages that were scraped. # S3 Storage import { Step, Steps } from 'fumadocs-ui/components/steps'; **S3 Storage** saves your scraper results as files instead of returning all the data in each API response. When you enable it, MrScraper uploads each result to S3 storage and returns a signed URL you can use to download the file. ## Why Use S3 Storage Returning a large result directly in an API response can be slow and may exceed response size limits. S3 Storage helps you: * **Keep a durable copy.** Store each result as a file you can download or reprocess later. * **Fit results into a pipeline.** Pass the signed URL to another service, or queue results to process in a separate step. ## Enable S3 Storage To save your scraped data to an S3 bucket, follow these steps: Open your scraper detail page Click the **Ellipsis** ( **⋮** ) button in the top right corner and click **Result Storage**. Toggle the **Enable Result Storage** option. Results are stored as JSON files matching your scraper's output format. The signed URL is returned in the data field of each result. Save your settings. The scraper will returns a signed URL: ```json title="Example Result" { "value": "https://api.app.mrscraper.com/api/v1/attachments/results/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxxxx.json" } ``` # Schedule import { Step, Steps } from 'fumadocs-ui/components/steps'; Schedule allows you to automate the execution of your scrapers at predefined times or intervals. This ensures that your data is always up-to-date without manual intervention. Schedule is useful in scenarios such as: * **Regular Data Updates:** If you need to keep your data fresh, such as daily price monitoring or weekly news aggregation. * **Time-Sensitive Data:** For data that changes frequently, like stock prices. * **Resource Management:** To run scrapers during off-peak hours to optimize resource usage and reduce costs. ## Set up a schedule Follow these steps to schedule a scraper: On the left sidebar, select the **triangle icon** to open the **Scrapers** page. Select the scraper you want to schedule. Click the **Ellipsis** ( **⋮** ) button in the top right corner and click **Schedule**. In the **Run Frequency** dropdown, choose how often the scraper should run: * **Every Hour** * **Daily** * **Weekly** * **Custom**: Build your own schedule from one or more intervals (see below). Select **Save Schedule** to apply your settings. Once you save the schedule, the scraper runs automatically. You can view its results on the **Results** page. To change or stop a schedule, return to the scraper's **Schedule** settings. ## Set a custom interval Choose **Custom** when the standard options (hourly, daily, weekly) don't match the frequency you need. A custom schedule is built from **schedule rows**. Each row holds one interval, and you can add several rows to run a scraper at uneven gaps. To set a custom interval: In the **Run Frequency** dropdown, select **Custom**. Under **Schedule Rows**, select a number in the first dropdown, then choose **Minutes**, **Hours**, or **Days** in the second. If you choose **Days**, also set the time of day the scraper should run, such as **13:16**. To add another interval, select **Add Row** and set its number and unit. Repeat for each interval you want in the cycle. Check the **Schedule Summary** below the rows. It shows how often the scraper will run. The time zone used for the schedule appears under the **Run Frequency** dropdown, for example *Times shown in Asia/Jakarta (UTC+7)*. Select **Save Schedule** to apply your settings. ### Use a single interval With one schedule row, the scraper runs once every interval you set. Use this method to calculate any custom interval: ``` Interval = Total minutes in the period ÷ Number of runs you want ``` For example, to run a scraper 3 times per hour: 60 ÷ 3 = 20. Set the interval to **20 Minutes**. 20 minutes interval ### Use multiple intervals With more than one schedule row, MrScraper cycles through the rows in order. After each run, it waits the next row's interval, then starts again from the first row once it reaches the end of the list. For example, with two rows set to **20 Minutes** and **10 Minutes**, the scraper runs at minutes 20, 30, 50, and 60 of each hour: | Run | Interval used | Runs at | | --- | ------------- | --------- | | 1 | 20 minutes | Minute 20 | | 2 | 10 minutes | Minute 30 | | 3 | 20 minutes | Minute 50 | | 4 | 10 minutes | Minute 60 | The cycle then repeats in the next hour. Use multiple rows when you want runs clustered around certain points in the hour or day instead of spread evenly. # Team import { Accordion, Accordions } from 'fumadocs-ui/components/accordion'; import { Step, Steps } from 'fumadocs-ui/components/steps'; The Team feature in MrScraper allows you to collaborate with others by sharing scrapers, results, subscription access, and token usage across your organization. ## What You Can Share When you create a team, all members have access to: * **Scrapers**: All configured scrapers in your account * **Results**: Scraping results and historical data * **Subscription**: Your plan features and limits * **Token Usage**: Shared token allocation across the team ## Creating a Team ### Create Your Team 1. Click your account profile in the top right corner of the MrScraper dashboard 2. Select **My Team** from the dropdown menu 3. Enter your team details: * **Team Name** (required) * **Description** (optional) 4. Click **Create Team** You'll be redirected to the team details page where you can manage members and settings. All scrapers and results are automatically shared with team members once they join. ### Invite Team Members 1. On the team details page, click **Invite Member** 2. Enter the member's information: * **Email address** (required) * **Role** (see [Role Capabilities](#role-capabilities) below) 3. Click **Send Invite** The invited member will receive an email invitation to join your team. All team members can access the same MrScraper features as the team owner. ## Role Capabilities Team roles determine what administrative actions members can perform. All roles have full access to MrScraper features for scraping and data collection. | Role | All Features Access | Invite Members | Change Member Roles | Remove Members | Delete Team | | --------------- | ------------------- | -------------- | ------------------- | -------------- | ----------- | | **Team Owner** | ✅ | ✅ | ✅ | ✅ | ✅ | | **Team Admin** | ✅ | ✅ | ✅ | ✅ | ❌ | | **Team Member** | ✅ | ❌ | ❌ | ❌ | ❌ | ### Role Details **Team Owner** * Full administrative control over the team * Can perform all actions including deleting the team * Only one owner per team (the account that created the team) **Team Admin** * Can manage team membership and permissions * Cannot delete the team * Ideal for trusted administrators who help manage the team **Team Member** * Can use all MrScraper features (scrapers, results, etc.) * Cannot perform administrative actions * Best for users who only need access to scrapers and data ## Managing Your Team ### Changing Member Roles Navigate to **My Team** from your account profile Find the member you want to update Click the role dropdown next to their name Select the new role Only Team Owners and Team Admins can change member roles. ### Removing Team Members Navigate to **My Team** from your account profile Find the member you want to remove Click the **Remove** button next to their name Confirm the removal Only Team Owners and Team Admins can remove members. ### Deleting a Team Navigate to **My Team** from your account profile Click **Team Settings** or **Delete Team** Confirm the deletion Deleting a team is permanent and cannot be undone. Only Team Owners can delete teams. ## Frequently Asked Questions Team ownership is tied to the account that created the team. Contact support if you need to transfer ownership. Scrapers remain with the team and are not affected when members leave. Only the member's access is revoked. Team size limits depend on your subscription plan. Check your plan details or contact support for more information. # Web Unblocker import { Step, Steps } from 'fumadocs-ui/components/steps'; MrScraper's Web Unblocker is a powerful feature that helps you bypass anti-scraping measures, render JavaScript-heavy websites, and access geo-restricted content. It combines selectable non-browser or browser loading, proxy routing, optional real-device Super Mode, and intelligent retry mechanisms for reliable data extraction from protected websites. ## Key Features ### Browser Rendering Execute JavaScript and render dynamic content just like a real browser. **Use Cases:** * Single Page Applications (SPAs) built with React, Vue, or Angular * Websites that load content via AJAX * Pages with lazy-loaded images or infinite scroll * Dynamic pricing or availability information * Client-side rendered content ### Super Mode Route requests through real devices when a website needs a different routing path. Super Mode is independent of browser rendering: it can use either the non-browser loader or browser loading with JavaScript. **Use Cases:** * Websites that continue blocking standard routing * Pages protected by advanced fingerprinting or device checks * Difficult targets that return different content through real-device routing ### Independent Loading Paths Browser rendering and Super Mode are two independent switches. Every combination is supported, and the same URL can return different results in each: | Browser rendering | Super Mode | Loading path | | ----------------- | ---------- | -------------------------------------------------------- | | Off | Off | Standard routing with the non-browser loader. | | On | Off | Standard routing with browser loading and JavaScript. | | Off | On | Real-device routing with the non-browser loader. | | On | On | Real-device routing with browser loading and JavaScript. | Start with both off and change one switch at a time after inspecting the response. Browser rendering is not a strictly stronger option. Some websites fail, become blocked, or return worse content with browser rendering enabled but load correctly through the non-browser path. If that happens, retry with browser rendering off while preserving the current Super Mode value. Stop after a usable response unless you need to compare loading paths. ### Geo-Targeting with Proxies Access geo-restricted content by routing requests through residential proxies in specific countries. **Use Cases:** * Scraping region-specific pricing * Accessing country-locked content * Testing localized versions of websites * Bypassing IP-based restrictions * Market research across different regions View the full list of supported countries on the [Country Codes](/docs/residential-proxy/configuration/country-codes) page. ### Smart Waiting Wait for specific elements to appear in the DOM before extracting content. **Use Cases:** * Waiting for lazy-loaded product information * Ensuring dynamic filters have applied * Waiting for search results to populate * Synchronizing with page animations * Handling delayed content rendering ### Homepage Navigation Navigate through a website's home page before visiting the target URL to better simulate normal user behavior. **Use Cases:** * Websites that expect visitors to arrive from the homepage * Reducing bot detection on protected websites * Improving access to sites with navigation-based validation ### Intelligent Retry Control Automatically retry failed scrapes while limiting retry costs with configurable retry settings. **Use Cases:** * Recovering from temporary network failures * Handling intermittent anti-bot challenges * Preventing excessive token usage with retry limits * Balancing reliability and scraping cost ### Resource Blocking Block non-essential resources such as images, fonts, and media to improve scraping speed and reduce bandwidth usage. **Use Cases:** * Faster page rendering * Lower bandwidth consumption * Scraping text-heavy pages * Large-scale scraping jobs ### Configurable Timeout Control how long to wait for page loading and rendering. **Use Cases:** * Slow-loading websites * Complex pages with many resources * Sites with large media files * Pages with extensive JavaScript execution * Unreliable network conditions - **Simple pages**: 30 seconds (default) - **Standard e-commerce**: 60 seconds - **Heavy SPAs**: 90-120 seconds - **Very complex pages**: Up to 180 seconds ## How It Works The Web Unblocker follows this process: Your request is routed through residential proxies in the specified region (if geo-targeting is enabled) If Super Mode is enabled, the request is routed through a real device for stronger anti-bot protection The selected loader runs: a browser when browser rendering is enabled, or the non-browser loader when it is disabled When browser rendering is enabled, the browser navigates to the target URL and begins rendering When browser rendering is enabled, page JavaScript executes, including AJAX calls and dynamic content loading If specified, the scraper waits for the target CSS selector to appear in the DOM The process completes when the selector appears or the timeout is reached The loaded HTML is extracted and returned Loader resources are released ## Example Request Example request for scraping using the [web unblocker](/docs/api/v3/scraper/unblocker-scraping) endpoint: ```bash curl -X GET "https://api.mrscraper.com/?url=https%3A%2F%2Fwww.amazon.com%2Fstores%2Fluxurystores%2Fpage%2FB6BC6264-7221-424B-9191-DAE2BCF963A2&token=atk_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&timeout=120&geoCode=pt&browserRendering=true&super=true&waitForSelector=div%5Bdata-testid%3D%22grid-item-info%22%5D&homePage=false&blockResources=true&maxRetries=3&tokenCap=10" \ -H "x-api-token: MRSCRAPER_API_KEY" ``` ### Parameters | Parameter | Required | Default | Description | | ------------------ | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `url` | Yes | - | Target website URL to scrape. | | `token` | Yes | - | Your API authentication token. | | `browserRendering` | No | `false` | Render the page in a headless browser. | | `super` | No | `false` | Route the request through a real device for stronger anti-bot protection. | | `geoCode` | No | - | ISO 3166-1 alpha-2 country code used for proxy routing. | | `timeout` | No | `30` | Maximum number of seconds to wait for page loading. | | `waitForSelector` | No | - | CSS selector to wait for before returning the page content. Requires `browserRendering=true`. | | `homePage` | No | `false` | Visit the website's home page before navigating to the target URL. | | `blockResources` | No | - | Block non-essential resources to improve performance. Supported only by compatible proxies. | | `maxRetries` | No | `3` | Maximum number of retry attempts if the scrape fails. | | `tokenCap` | No | - | Maximum number of tokens that can be consumed across retry attempts. If the first scrape exceeds the cap, it still runs once but no retries are performed. | * `waitForSelector` only works when `browserRendering=true`. * Start without `super` and enable it when a target requires stronger anti-bot protection. * `blockResources` is only supported by proxies that support resource blocking. * `tokenCap` only affects retry attempts. The initial scrape always runs, even if it exceeds the configured token cap. # Token Plan import { Step, Steps } from 'fumadocs-ui/components/steps'; MrScraper uses a **token-based system** for both usage and authentication. * **Plan Tokens** measure how much you can use MrScraper's features, such as running or rerunning scrapers. * **API Tokens** allow you to securely access the MrScraper API, for example, to rerun an existing scraper programmatically. ## Why Token Usage Varies Token consumption isn't fixed — a few key factors affect how many tokens any given scraper uses: * **Website Complexity**: Heavy JavaScript, dynamic content, or multiple requests increase runtime and token usage. * **Page Size and Content Density**: Larger pages with long articles or extensive metadata produce more input text for AI models to process. * **Data Volume**: Extracting large datasets (e.g., paginated listings) requires more compute and bandwidth. * **AI Processing Load**: Complex instructions or large expected outputs increase input and output token usage. Keep these in mind as you read the pricing tables below — they explain why two runs on the same plan can consume different amounts of tokens. ## Understanding Plan Tokens Each MrScraper billing plan includes a specific number of **tokens**. Tokens represent the compute resources your scraper consumes — similar to how minutes or data are counted in a phone plan. The number of tokens used depends on the type of scraper you run. ### AI Scraper Token Usage AI Scrapers consume more tokens due to AI processing and additional factors such as: | Usage Type | Description | Conversion Rate | | ----------------- | ----------------------------------------------------------------------------------------------------------------------- | --------------------------------- | | **Run Tracking** | Each AI agent run consumes 5 tokens for trace tracking. Reruns and manual scrapers consume 0 tokens for this component. | 0 or 5 tokens per run | | **Runtime** | Time spent processing your task. | 1 token per 30 seconds of runtime | | **Input Tokens** | Amount of text data (prompts or instructions) sent to the AI model. | 1 token per \~1,000 input tokens | | **Output Tokens** | Amount of text generated by the AI model (responses or extracted results). | 1 token per \~200 output tokens | #### Example If your AI scraper runs for **90 seconds**, processes **5,000 input tokens**, and generates **1,000 output tokens**, your total token usage would be: | Component | Calculation | Tokens | | --------- | ------------- | ------ | | Runtime | 90 ÷ 30 | 3 | | Input | 5,000 ÷ 1,000 | 5 | | Output | 1,000 ÷ 200 | 5 | | Run Trace | Fixed | 5 | | **Total** | | **18** | * Each run automatically includes trace tracking for debugging and performance monitoring. * Input and output tokens are based on how much text the AI model processes and generates. * AI Scrapers tend to consume more tokens due to model processing. * Runtime tokens are rounded up using the ceiling rule. For example, if your scraper runs for 40 seconds, it will count as 2 tokens (not 1). ### Manual Scraper and Unblocker Token Usage Manual Scrapers and Unblocker consume tokens based on two factors: **runtime** and **bandwidth**. | Usage Type | Description | Conversion Rate | | ------------- | -------------------------------------- | ---------------------- | | **Runtime** | Time your scraper runs. | 1 token per 30 seconds | | **Bandwidth** | Amount of data downloaded or uploaded. | 1 token per 0.25 MB | #### Example If your manual scraper or unblocker runs for **40 seconds** and uses **1.1 MB** of bandwidth: | Component | Calculation | Tokens | | --------- | ----------------------------- | ------ | | Runtime | 40 ÷ 30 = 1.33 → rounded up | 2 | | Bandwidth | 1.1 ÷ 0.25 = 4.4 → rounded up | 5 | | **Total** | | **7** | Runtime tokens are rounded up using the ceiling rule. For example, if your scraper runs for 40 seconds, it will count as 2 tokens (not 1). ### Playground Token Cap The **Playground token cap** limits how many tokens a single scrape can use in the Playground. It applies only to **Manual Scraper** and **Unblocker**, using the same runtime and bandwidth calculations shown in [Manual Scraper and Unblocker Token Usage](#manual-scraper-and-unblocker-token-usage). When **Retry** is enabled, a failed scrape retries automatically. Two limits control how far retrying goes: * **Max retries**: The maximum number of retry attempts (3 by default). * **Token cap**: The maximum total tokens all attempts can use (Unlimited by default). Retrying stops at the first of these: the scrape succeeds, it reaches Max retries, or the running token total reaches the token cap. Each attempt's token cost is rounded up (ceiling rule) and added to a running total. The token cap takes effect only when Retry is enabled, since its purpose is to limit how many tokens retries can consume. If Retry is off, the scrape runs once and returns a result or an error. #### Example Max retries **3**, token cap **10**. | Attempt | Runtime + Bandwidth | Rounded-up Cost | Running Total | Result | | ----------- | ------------------- | --------------- | ------------- | ---------------- | | 1 (initial) | 2.4 tokens | 3 | 3 | Failed, retrying | | 2 (retry 1) | 1.6 tokens | 2 | 5 | Succeeded | The scrape succeeds on the second attempt, using 5 tokens. Because it succeeded, the Playground returns the result and stops even though max retries and tokens still remain (5 of 10 tokens left, and 2 retries unused). Max retries **2**, token cap **20**. | Attempt | Runtime + Bandwidth | Rounded-up Cost | Running Total | Result | | ----------- | ------------------- | --------------- | ------------- | --------------------------- | | 1 (initial) | 3.6 tokens | 4 | 4 | Failed, retrying | | 2 (retry 1) | 3.8 tokens | 4 | 8 | Failed, retrying | | 3 (retry 2) | 3.4 tokens | 4 | 12 | Failed, max retries reached | The scrape uses both retries without succeeding, so it stops even though 8 tokens still remain under the 20-token cap. Max retries **3**, token cap **12**, about 4 tokens per run. | Attempt | Runtime + Bandwidth | Rounded-up Cost | Running Total | Result | | ----------- | ------------------- | --------------- | ------------- | ------------------- | | 1 (initial) | 3.5 tokens | 4 | 4 | Failed, retrying | | 2 (retry 1) | 3.9 tokens | 4 | 8 | Failed, retrying | | 3 (retry 2) | 3.7 tokens | 4 | 12 | Failed, cap reached | The running total reaches the 12-token cap on the second retry, so the scrape stops before using all 3 retries. If the first attempt costs more than the cap (for example, a cap of 10 but a first scrape that needs 15 tokens), the scrape still runs once but doesn't retry. It returns either a result or an error. Set the token cap high enough to allow for retries, especially on pages with variable load times or bandwidth. If a scrape hits the cap before completing, try increasing the cap. ### Marketplace Scraper Token Usage Marketplace Scrapers use a custom token cost based on factors such as the target domain, workflow complexity, proxy usage, infrastructure requirements, and scraper configuration. Each scraper may use either a fixed token cost or a custom usage calculation depending on how the scraper is designed. For the complete list of Marketplace Scraper token costs, see the [Marketplace](/docs/features/marketplace) catalog. ## Generate a New API Token You can also create additional tokens for different environments, applications, or use cases. To create a new API token: Click your **profile icon** in the top-right corner of the dashboard. Select **API Tokens**. Click **New Token**. Enter a **token name** and choose an **expiration date**. The token name is required when making requests to the analytics endpoint. Click **Create**. Copy the generated API token and store it in a secure location. You will need this token to authenticate API requests. Never expose your API token in client-side code (like browsers or apps). Always store it securely in an environment variable or server configuration. # Billing import { Accordion, Accordions } from 'fumadocs-ui/components/accordion'; import { Step, Steps } from 'fumadocs-ui/components/steps'; MrScraper offers flexible subscription plans designed for different usage levels, from individual developers testing the platform to enterprise teams running large-scale scraping operations. All plans include access to our AI-powered scraping agents, stealth browser technology, and residential proxies (on applicable plans). ## Available Plans | Feature | Scraper Free | Scraper Pro | Scraper Enterprise | | ----------------------------- | ------------ | ------------- | ------------------ | | **API Tokens** | 1,000 token | 200,000 token | Custom | | **Concurrent Requests** | 10 | 100 | Custom | | **Residential Proxies** | ❌ | ✅ | ✅ | | **Priority Email Support** | ❌ | ✅ | ✅ | | **Dedicated Account Manager** | ❌ | ✅ | ✅ | The Free plan includes **100 tokens per month** with no credit card required. Tokens reset at the beginning of each month. Once your allocation is exhausted, you'll need to upgrade to continue scraping. ## Transaction Page You can view your past transactions, by going to **Account** → **Subscription**. The transaction page displays a list of all your billing activities, including: * Product * Amount (Price) * Status * Date * Invoice and Receipt links ### Invoices and Receipts To access the invoices or receipts of your account: Go to **Account Profile** → **Subscription** You'll be shown with a list of your billing history, including the date, amount, and status of each transaction Under **Invoice**, and **Receipt** columns, you can click on the respective links to view or download the invoice or receipt for each transaction. ## Manage Billing In the Subscription page, you can click on the **Manage Billing** button to open the Stripe customer portal. From there, you can: * View your payment history * Cancel your subscription plan * Edit your billing information ## Billing Cycle ### How Token Allocation Works **Monthly Plans:** * Tokens reset on your monthly renewal date * Full token allotment available at the start of each cycle * Unused tokens do **not** roll over to the next month **Annual Plans:** * Billed once per year * Tokens still reset **monthly** on your renewal date * 20% discount on total cost * Unused tokens do **not** roll over ### Example Timeline If you subscribe to Scraper Pro on **January 15th**: | Date | Event | Token Balance | | ------ | -------------------- | ---------------------- | | Jan 15 | Subscription starts | 200,000 tokens | | Jan 31 | Used 60,000 tokens | 140,000 tokens | | Feb 15 | Billing cycle renews | 200,000 tokens (reset) | Unused tokens from the previous month are **lost** when your billing cycle renews. Plan your scraping operations accordingly to maximize your token allocation. ## Tracking Your Usage Monitor your token consumption and billing information through multiple channels: * Dashboard: View real-time usage in your [account API token dashboard](https://app.mrscraper.com/api-tokens) * API Endpoints: Track usage programmatically using our [Analytics API](/docs/api/v3/analytic/status) ## FAQs No, unused tokens do **not** roll over. Your token allocation resets to your plan amount at the start of each billing cycle. This is why it's important to choose a plan that matches your typical monthly usage. Token usage varies by: * Page complexity * Scraping mode (Cheap vs Super) * Agent type (General, Listing, Map) * Additional features (AI extraction, etc.) Your scrapers remain intact and accessible. However: * You'll have fewer tokens to run them * Concurrent request limits may decrease * Some features (like residential proxies) may become unavailable Yes! Contact [support@mrscraper.com](mailto:support@mrscraper.com) to switch billing cycles. When switching to annual billing: * You'll be charged for a full year upfront * Receive a 20% discount * Any remaining time on your monthly subscription will be credited The Free plan serves as a trial with 100 monthly tokens. Test MrScraper's features before committing to a paid plan. No credit card required. # CLI The MrScraper CLI provides terminal access to HTML fetching, AI extraction, Google SERP, saved scraper reruns, stored results, and account usage. This guide covers installation, verification, authentication, output formats, environment variables, programmatic usage, and complete command details. ## Requirements * Node.js 20 or newer * A MrScraper account for browser login, or an API key for non-interactive use See [@mrscraper/cli on npm](https://www.npmjs.com/package/@mrscraper/cli) for the current published version. ## Installation Install only the CLI globally: ```bash npm install -g @mrscraper/cli@latest mrscraper --version ``` Install the CLI and all four MrScraper agent skills for every detected harness: ```bash npx -y @mrscraper/cli@latest init --all ``` Install for one harness: ```bash npx -y @mrscraper/cli@latest init --agent codex ``` Supported IDs are `claude-code`, `cursor`, `codex`, `grok`, `hermes`, `opencode`, `openclaw`, `pi`, and `omp`. ### Copyable AI setup prompt ```text Read and follow https://github.com/mrscraper-com/cli/blob/main/skills/mrscraper/SKILL.md ``` For an agent-controlled, non-interactive setup, skip authentication during bootstrap and start login separately: ```bash npx -y @mrscraper/cli@latest init --agent codex --yes --skip-auth mrscraper auth status --json mrscraper login ``` `mrscraper init` installs the CLI and skill pack. It does not install or configure MCP. ### `init` parameters | Parameter | Default | Scope | Behavior | | ----------------- | ------- | ----- | --------------------------------------------------------------------------------------- | | `--api-key ` | — | Local | Saves the supplied key during bootstrap instead of browser login. | | `--all` | enabled | Local | Installs skills for every supported harness detected on this machine. | | `--agent ` | — | Local | Installs skills for one supported harness, even when its detection directory is absent. | | `-y, --yes` | off | Local | Keeps bootstrap non-interactive. Missing authentication is left for a later login. | | `--skip-install` | off | Local | Does not install the current CLI version globally. | | `--skip-auth` | off | Local | Performs no authentication step. | | `--skip-skills` | off | Local | Does not install the skill pack. | | `--dry-run` | off | Local | Prints intended actions without installing, authenticating, or copying skills. | Refresh only the skills with `mrscraper setup skills`. It accepts `--agent ` for one harness and `--dry-run`; without `--agent`, it targets detected harnesses. ### Native plugins (optional) The MrScraper CLI repository also provides skills-only native plugins for Codex, Claude Code, and Cursor. Plugin installation is an alternative to `mrscraper init`, not an additional setup step. If `npx ... init` already installed the four MrScraper skills for an agent, skip plugin installation for that agent. Enabling both methods can load duplicate copies of the same skills. Plugins do not install or authenticate the CLI. When using a plugin, install and authenticate the CLI separately: ```bash npm install -g @mrscraper/cli@latest mrscraper login ``` * **Claude Code:** run `claude plugin marketplace add mrscraper-com/cli`, then install `mrscraper-cli@mrscraper`. * **Cursor:** use `/add-plugin mrscraper-cli` after its public marketplace listing; until then, load the repository through Cursor's local plugin directory. * **Codex:** until its public plugin listing is available, use the [local marketplace example](https://github.com/mrscraper-com/cli/blob/main/examples/marketplace.json) from the CLI repository. ### Verifying installation After completing installation, verify that the CLI binary and commands are accessible in your environment: ```bash mrscraper --version mrscraper auth status --json mrscraper scrape "https://example.com" \ --prompt "Extract the page title" ``` A successful scrape confirms that the CLI can load its authentication and reach the MrScraper service. Restart an agent harness or open a new terminal session after installing skills so it reloads the skill pack properly. ## Authentication ### Browser login ```bash mrscraper login ``` The CLI opens the MrScraper login page, waits for the local browser callback, exchanges the returned code, and saves the provisioned API key in `~/.mrscraper/auth.json`. Set `MRSCRAPER_HOME` to change the containing directory. ### API-key login ```bash mrscraper login --api-key "$MRSCRAPER_API_KEY" ``` Prefer environment variables to literal keys in shell history: ```bash export MRSCRAPER_API_KEY="your-key" ``` Authentication precedence for API commands is: 1. Command `--token` 2. `MRSCRAPER_API_KEY` 3. `MRSCRAPER_API_TOKEN` 4. `~/.mrscraper/auth.json` ### `login` parameters | Parameter | Default | Scope | Behavior | | --------------------- | ------- | ----- | ----------------------------------------------------------------------------- | | `--api-key ` | — | Local | Saves this API key and does not open browser login. | | `--token ` | — | Local | Deprecated alias for login's `--api-key`. | | `--no-browser` | off | Local | Prompts a human for an API key; requires an interactive terminal. | | `--no-open` | off | Local | Prints the browser URL without launching it. The callback server still waits. | | `--timeout ` | `180` | Local | Maximum time to wait for the browser callback. | ### Auth status and logout `mrscraper auth status` reports the credential currently available to the CLI. It reads the local configuration without sending a test request. JSON output uses the `credential_configured` field: ```json { "credential_configured": true, "auth_type": "api_key", "path": "/home/user/.mrscraper/auth.json" } ``` `mrscraper logout` deletes local credential files. It does not revoke an API key remotely. ## Output contract API commands return a consistent JSON envelope: ```json { "status_code": 200, "data": "the parsed JSON response or response text", "headers": { "content-type": "text/html" } } ``` The fields are: * `status_code` contains the HTTP response code. * `headers` contains non-sensitive response headers. * `data` contains the response body. JSON responses are parsed; other bodies remain strings. * Known credential metadata, sensitive response headers, and credentials in generated curl commands are redacted. * Fetched HTML and scraper extraction results are returned in `data`. * HTTP failures remain JSON on stdout, set `error`, and exit non-zero. * Progress messages use stderr so stdout remains machine-readable. `status` presents account and analytics data as a normalized summary. ## Environment variables | Variable | Default | Purpose | | -------------------------- | -------------------------------------- | ----------------------------------------------- | | `MRSCRAPER_API_KEY` | — | Preferred API-key environment override. | | `MRSCRAPER_API_TOKEN` | — | Legacy API-key environment override. | | `MRSCRAPER_HOME` | `~/.mrscraper` | Credential directory override. | | `MRSCRAPER_API_BASE_URL` | `https://api.app.mrscraper.com/api/v1` | Development override for platform API commands. | | `MRSCRAPER_FETCH_BASE_URL` | `https://api.mrscraper.com` | Development override for fetch. | | `MRSCRAPER_SYNC_BASE_URL` | `https://sync.scraper.mrscraper.com` | Development override for SERP. | The CLI also loads a `.env` file from the current working directory. ## Programmatic use The package exports the same direct API helpers used by the commands: ```javascript import { fetchContentApi, createAiScraperApi, googleSerpSyncApi, getAllResultsApi, } from "@mrscraper/cli"; const result = await fetchContentApi({ token: process.env.MRSCRAPER_API_KEY, url: "https://example.com", browserRendering: true, maxRetries: 3, }); ``` The helpers return the same response envelope used by the CLI commands. ## CLI command list ### `fetch` `fetch` uses MrScraper's [Web Unblocker](/docs/features/unblocker) to retrieve HTML from public, protected, JavaScript-rendered, or geo-sensitive pages through: ```text GET https://api.mrscraper.com/ ``` The returned HTML is available in the envelope's `.data` field. ```bash mrscraper fetch "https://example.com" mrscraper fetch "https://example.com" | jq -r '.data' ``` Browser loading and real-device routing are independent controls. All four combinations can return different results for the same URL: | Browser rendering | Super Mode | Command | Loading path | | ----------------- | ---------- | ------------------------------------------------------ | -------------------------------------------------------- | | Off | Off | `mrscraper fetch URL` | Standard routing with the non-browser loader. | | On | Off | `mrscraper fetch URL --browser-rendering` | Standard routing with browser loading and JavaScript. | | Off | On | `mrscraper fetch URL --super-mode` | Real-device routing with the non-browser loader. | | On | On | `mrscraper fetch URL --browser-rendering --super-mode` | Real-device routing with browser loading and JavaScript. | Start with both controls off, inspect the response, and change one control at a time when needed. Browser rendering is not guaranteed to produce a better response: some sites fail with it enabled but load without it. Try remaining untested combinations without repeating an identical failed request. For JavaScript-rendered or delayed content, enable browser rendering: ```bash mrscraper fetch "https://example.com/products" \ --browser-rendering \ --wait-for-selector ".product-card" ``` For real-device browser loading, enable both controls: ```bash mrscraper fetch "https://example.com/products" \ --browser-rendering \ --super-mode ``` #### `fetch` parameters | CLI parameter | Default | API mapping | Behavior | | -------------------------------- | --------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `` | required | Query `url` | Target page URL. | | `--browser-rendering` | `false` | Query `browserRendering=true` | Loads the page in a browser and executes JavaScript. | | `--super-mode` | `false` | Query `super=true` | Selects real-device routing independently of browser rendering. | | `--geo-code ` | omitted | Query `geoCode` | Requests proxy routing through the ISO 3166-1 alpha-2 country. | | `--wait-for-selector ` | omitted | Query `waitForSelector` | Waits for a CSS selector. The CLI rejects it unless `--browser-rendering` is explicit. | | `--home-page` | `false` | Query `homePage=true` | Visits the site root before loading the target page. | | `--block-resources` | `false` | Query `blockResources=true` | Blocks non-essential browser resources when supported by the selected proxy. | | `--max-retries ` | `3` | Query `maxRetries` | Sets the maximum retry attempts after a failed request. Zero is accepted. | | `--token-cap ` | omitted | Query `tokenCap` | Limits the running plan-token total used to decide whether another retry may run. The initial request always runs. | | `--timeout ` | `30` | Query `timeout` | Sets the page-load timeout. The command allows an additional 30 seconds to receive the response. | | `--token ` | configured credential | Request headers | Overrides authentication for this command. | Use browser rendering when JavaScript is needed, but retry the same Super Mode value without it when browser loading fails, is blocked, or returns worse content. Toggle Super Mode independently when routing may be the problem. A selector wait still requires browser rendering. Unblocker usage is based on runtime and bandwidth: one plan token per 30 seconds and one plan token per 0.25 MB, rounded up per component. Resource blocking can reduce bandwidth for text-focused pages. Use `--max-retries` and `--token-cap` to balance retry reliability with usage. See the [Token Plan](/docs/getting-started/api-token) for the complete calculation. ### `scrape` `scrape` calls: ```text POST https://api.app.mrscraper.com/api/v1/scrapers-ai ``` The default agent is `general`. General and listing require an extraction prompt: ```bash mrscraper scrape "https://example.com/product" \ --mode Super \ --prompt "Extract name, price, availability, and image URLs" ``` The request body contains `url`, `message`, and `agent`, plus optional fields for the selected agent. `--mode Cheap` and `--mode Super` select the backend execution tier independently of `--agent`; omit `--mode` to preserve the backend default. #### Listing agent ```bash mrscraper scrape "https://example.com/products" \ --agent listing \ --prompt "Extract every product's name, price, and URL" \ --max-pages 5 ``` Listing is synchronous and may take several minutes. Progress is printed to stderr. If `--max-pages` is omitted, the service default applies. #### Map agent ```bash mrscraper scrape "https://example.com" \ --agent map \ --max-depth 2 \ --max-pages 50 \ --limit 1000 \ --include-patterns '/products/' ``` Map accepts crawl limits and URL patterns. Prompts, schema guidance, and proxy country selection are available with the general and listing agents. #### Local output file ```bash mrscraper scrape "https://example.com/product" \ --prompt "Extract the product" \ --output ./.mrscraper/product.json ``` `--output` creates parent directories and writes the extracted `data.data.data` value as pretty JSON. The complete response remains on stdout. The file is created after a completed extraction returns data. #### Best-effort schema prompt ```bash mrscraper scrape "https://example.com/product" \ --prompt "Extract the product" \ --schema-prompt ./product.schema.json ``` `--schema-prompt` reads a local JSON Schema object and adds it to the extraction instructions as best-effort shape guidance. Use a separate validator when strict schema compliance is required. #### Reproduce a scrape with `rerun` Every successful `scrape` creates a saved AI scraper configuration by default. The complete stdout response contains its UUID at `data.data.scraperId`. Save that response when the extraction may need to run again: ```bash mrscraper scrape "https://example.com/product" \ --prompt "Extract the product name, price, and availability" \ --output ./.mrscraper/product.json \ > ./.mrscraper/product-run.json SCRAPER_UUID=$(jq -r '.data.data.scraperId' ./.mrscraper/product-run.json) mrscraper rerun "https://example.com/product-2" \ --type ai \ --scraper-id "$SCRAPER_UUID" ``` The output file contains only `data.data.data`; the stdout envelope retains the scraper UUID. `rerun` reproduces the saved prompt and agent configuration, but page changes and model behavior can still change the extracted values. `rerun` also supports dashboard-built manual workflows and asynchronous bulk jobs across multiple target URLs. See the full [`rerun`](#rerun) section below for the available modes and result-tracking workflow. #### `scrape` parameters | CLI parameter | Default | API mapping | Accepted agents and behavior | | ---------------------------- | --------------------- | -------------------------- | -------------------------------------------------------------- | | `` | required | Body `url` | Target URL for all agents. | | `-p, --prompt ` | required | Body `message` | Extraction instructions for general/listing. | | `-a, --agent ` | `general` | Body `agent` | `general`, `listing`, or `map`. | | `--mode ` | omitted | Body `mode` | Selects the backend execution tier without changing the agent. | | `--proxy-country ` | omitted | Body `proxyCountry` | General/listing only; rejected for map. | | `--max-pages ` | omitted | Body `maxPages` | Listing/map only. The service default applies when omitted. | | `--max-depth ` | omitted | Body `maxDepth` | Map only. | | `--limit ` | omitted | Body `limit` | Map only. | | `--include-patterns ` | omitted | Body `includePatterns` | Map only. | | `--exclude-patterns ` | omitted | Body `excludePatterns` | Map only. | | `--schema-prompt ` | omitted | Appended to body `message` | Best-effort shape guidance for general/listing. | | `-o, --output ` | omitted | Output file | Writes `data.data.data` as pretty JSON. | | `--token ` | configured credential | Request headers | Overrides authentication for this command. | ### `serp` `serp` calls: ```text POST https://sync.scraper.mrscraper.com/api/google/serp/v2/sync ``` Use `--format` to choose parsed JSON or the result-page HTML: ```bash mrscraper serp "iphone 17" --region id --language id --page 2 mrscraper serp "iphone 17" --format html --render-js ``` A complete Google search URL can also be passed directly: ```bash mrscraper serp "https://www.google.com/search?q=iphone+17&gl=us&hl=en&start=20" ``` The CLI extracts `q`, `gl`, `hl`, and `start`, converts `start` to a page number, then sends the normal API body. #### `serp` parameters | CLI parameter | Default | API mapping | Behavior | | ---------------------------- | --------------------- | ------------------------ | ------------------------------------------------------------------------- | | `` | required | Body `query` | Sends a plain query, or locally derives request fields from a Google URL. | | `--region ` | omitted | Body `region` | Result country. Explicit CLI input overrides `gl` from a URL. | | `--language ` | omitted | Body `language` | Result language. Explicit CLI input overrides `hl` from a URL. | | `--page ` | omitted | Body `page` | 1-based result page. Explicit CLI input overrides URL `start`. | | `--format ` | `json` | Body `format` | Returns parsed JSON or result-page HTML. | | `--render-js` | `false` | Body `renderJs=true` | Waits for JavaScript-rendered SERP features. | | `--raw` | `false` | Sends body `format=html` | Deprecated CLI alias for `--format html`. | | `--client-timeout ` | `120` | CLI request | Sets the HTTP request deadline. | | `--token ` | configured credential | Request headers | Overrides authentication for this command. | ### `status` `status` combines account information into a concise summary. It calls: ```text GET https://api.app.mrscraper.com/api/v1/subscription-accounts ``` The CLI removes credential and billing identifiers, renames selected fields, and calculates `token_remaining` and `usage_percent`. Interactive terminals get a dashboard; redirected output is JSON. Force either mode with `--pretty` or `--json`. ```bash mrscraper status mrscraper status --json ``` The JSON summary includes its source endpoints: ```json { "kind": "mrscraper-cli-status-summary", "source_endpoints": ["/subscription-accounts"], "status_code": 200, "data": { "account": {} } } ``` With `--domain`, the CLI makes a second request to `/analytic/statuses`, converts a URL to its hostname, converts relative dates to the API timestamp format, and merges the result: ```bash mrscraper status --domain "https://example.com/products" --from 7d --to now ``` #### `status` parameters | CLI parameter | Default | API mapping | Behavior | | --------------------------- | --------------------- | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | | `--domain ` | omitted | Analytics query `domain` | Enables the second analytics request; URLs are locally reduced to hostname. | | `--from ` | `24h` | Analytics query `startDate` | Accepts ISO time or local durations such as `30m`, `24h`, and `7d`, then formats UTC for the API. Used only with `--domain`. | | `--to ` | `now` | Analytics query `endDate` | Accepts ISO time or `now`, then formats UTC. Used only with `--domain`. | | `--action ` | empty string | Analytics query `action` | Optional analytics filter. Used only with `--domain`. | | `--api-token-name ` | empty string | Analytics query `apiTokenName` | Optional analytics filter. Used only with `--domain`. | | `--json` | automatic when piped | CLI output | Prints the summary as JSON. | | `--pretty` | automatic in TTY | CLI output | Prints the terminal dashboard. Cannot be combined with `--json`. | | `--no-color` | off | CLI output | Disables ANSI color in the dashboard. | | `--token ` | configured credential | Request headers | Overrides authentication for both requests. | ### `rerun` Choose the saved scraper and the number of target URLs independently: * `--type ai` runs an AI scraper created by `mrscraper scrape`. * `--type manual` runs a step-based workflow created in the MrScraper dashboard. The CLI reruns existing manual workflows; it does not create them. * Without `--bulk`, the scraper runs on one URL. With `--bulk`, the same saved configuration is submitted once with a comma- or newline-separated URL list. Manual reruns can be single or bulk. Bulk reruns can use either an AI or a manual scraper; bulk describes the number of targets, not the scraper type. Bulk mode submits one asynchronous backend job; it does not loop over the single-URL endpoint locally. Save `data.data.bulkResultId` from the submission response and retrieve the stored result until it finishes: ```bash mrscraper result --id BULK_RESULT_UUID ``` `rerun` selects an endpoint based on `--type` and `--bulk`: | Mode | Endpoint | | ------------- | ----------------------------------------- | | Single AI | `POST /api/v1/scrapers-ai-rerun` | | Bulk AI | `POST /api/v1/scrapers-ai-rerun/bulk` | | Single manual | `POST /api/v1/scrapers-manual-rerun` | | Bulk manual | `POST /api/v1/scrapers-manual-rerun/bulk` | ```bash mrscraper rerun "https://example.com/product" \ --type ai --scraper-id SCRAPER_UUID mrscraper rerun "https://a.example,https://b.example" \ --bulk --type manual --id SCRAPER_UUID ``` #### `rerun` parameters | CLI parameter | Default | API mapping | Behavior | | ---------------------------- | --------------------- | ---------------------- | ----------------------------------------------------------------------------------------------- | | `` | required | Body `url` or `urls` | One URL for single mode. Bulk mode locally splits comma- or newline-separated URLs into `urls`. | | `--type ` | required | CLI routing | Selects the AI or manual endpoint. | | `--bulk` | `false` | CLI routing | Selects the bulk endpoint. | | `--scraper-id ` | required for single | Body `scraperId` | Single endpoint only. Rejected with `--bulk`. | | `--id ` | required for bulk | Body `scraperId` | Bulk endpoint only. Rejected for single mode. | | `--max-depth ` | omitted | Body `maxDepth` | Single AI rerun only. Omit it to preserve the saved scraper/backend default. | | `--max-pages ` | omitted | Body `maxPages` | Single AI rerun only. Omit it to preserve the saved scraper/backend default. | | `--limit ` | omitted | Body `limit` | Single AI rerun only. Omit it to preserve the saved scraper/backend default. | | `--include-patterns ` | omitted | Body `includePatterns` | Single AI rerun only. Omit it to preserve the saved scraper/backend default. | | `--exclude-patterns ` | omitted | Body `excludePatterns` | Single AI rerun only. Omit it to preserve the saved scraper/backend default. | | `--proxy-country ` | omitted | Body `proxyCountry` | Single AI rerun only. | | `--max-retry ` | omitted | Body `maxRetry` | Single AI rerun retry limit; zero is accepted. | | `--timeout ` | omitted | Body `timeout` | Single AI rerun timeout, used by listing reruns. | | `--token ` | configured credential | Request headers | Overrides authentication for this command. | The AI controls apply only to single AI reruns. The CLI sends them only when provided, so omitted values continue using the saved scraper or backend defaults. ### `results` `results` calls `GET /api/v1/results` and maps its filtering options directly to query parameters: ```bash mrscraper results --page-size 20 --page 2 mrscraper results --search example.com mrscraper results \ --scraper-id SCRAPER_UUID \ --status Finished \ --type Rerun-AI \ --url "https://example.com/product" mrscraper results \ --date-range-column updatedAt \ --start-at "2026-08-01T00:00:00Z" \ --end-at "2026-08-18T00:00:00Z" ``` #### `results` parameters | CLI parameter | Default | API query field | Behavior | | ------------------------------ | --------------------- | -------------------- | ---------------------------------------------------------------------- | | `--sort-field ` | `updatedAt` | `sortField` | Field used to sort results. | | `--sort-order ` | `desc` | `sortOrder` | CLI accepts case-insensitively and sends uppercase `ASC` or `DESC`. | | `--page-size ` | `10` | `pageSize` | Positive result page size. | | `--page ` | `1` | `page` | Positive 1-based page number. | | `--search ` | omitted | `search` | Search filter. | | `--scraper-id ` | omitted | `filters[scraperId]` | Exact saved scraper UUID filter. | | `--status ` | omitted | `filters[status]` | Exact `Draft`, `Finished`, `Running`, `Failed`, or `Cancelled` filter. | | `--type ` | omitted | `filters[type]` | Exact result type filter, such as `AI` or `Rerun-AI`. | | `--url ` | omitted | `filters[url]` | Exact stored target URL filter. | | `--date-range-column ` | omitted | `dateRangeColumn` | Column used with `startAt` and `endAt`. | | `--start-at ` | omitted | `startAt` | Inclusive range start. | | `--end-at ` | omitted | `endAt` | Inclusive range end. | | `--token ` | configured credential | Request headers | Overrides authentication. | ### `result` `result` calls `GET /api/v1/results/{id}`: ```bash mrscraper result RESULT_UUID mrscraper result --id RESULT_UUID mrscraper result --id RESULT_UUID --no-include-html ``` The positional ID and `--id` are local input alternatives. Stored HTML is included by default; `--no-include-html` sends `includeHtml=false` for a smaller response while polling status or when extracted data is sufficient. `--token` overrides authentication for this command. # MCP Server import { Accordion, Accordions } from 'fumadocs-ui/components/accordion'; import { Step, Steps } from "fumadocs-ui/components/steps"; import { CodeBlock } from "fumadocs-ui/components/codeblock"; import { Activity, Bot, Database, FileSearch, GlobeLock, RefreshCcw, Search, Server, } from "lucide-react"; MrScraper MCP connects Model Context Protocol clients to the MrScraper web-data service. It exposes seven tools: } href="#fetch"> Fetch page HTML using Web Unblocker with browser rendering and proxy routing. } href="#scrape"> Extract structured fields, listing records, or site maps with AI agents. } href="#serp"> Query Google SERP for structured search engine results and page discovery. } href="#status"> Check subscription status, quota usage, rate limits, and token metrics. } href="#rerun"> Execute new runs for saved AI or manual scraper configurations. } href="#results"> Browse, filter, paginate, and search stored MrScraper result collections. } href="#result"> Retrieve full output data and metadata for a single stored result by ID. Use the hosted Streamable HTTP endpoint for a managed connection, or run the published MCP package locally through Streamable HTTP or stdio. ## Capabilities }> Fetch a known URL with browser rendering, proxy-country routing, selector waiting, resource controls, and retry limits. }> Extract page fields, repeated listing records, or a site URL map with agent-specific inputs. }> Search Google from a query or a complete Google search URL and select JSON or HTML output. }> Read subscription, quota, token usage, rate limits, and optional domain request outcomes. }> Rerun saved AI or manual scraper configurations for one URL or a bulk URL list. }> Browse, filter, paginate, and retrieve stored MrScraper results. }> Connect to the hosted MCP endpoint through Streamable HTTP with OAuth 2.1 browser sign-in. }> Receive a consistent response envelope in MCP structured content. ## Requirements * A [MrScraper account](https://app.mrscraper.com) * An MCP client with Streamable HTTP support * An [API key](/docs/getting-started/api-token) only when the client cannot use OAuth 2.1 or when running the server locally ## Quick start with the hosted server Add the hosted endpoint Configure a Streamable HTTP server named `mrscraper` with no credential: ```json { "mcpServers": { "mrscraper": { "type": "http", "url": "https://mcp.mrscraper.com/mcp" } } } ``` Use this exact URL, including the `/mcp` path. Sign in Connect or reload the MCP server. Your client follows the OAuth 2.1 challenge, opens MrScraper in your browser, and asks you to approve access. Reload and verify Reload the client or start a new session, then inspect its MCP tool list. The connection should expose `fetch`, `scrape`, `serp`, `status`, `rerun`, `results`, and `result`. ## Copyable AI setup prompt 
    
      Connect MrScraper MCP to this agent. Detect the current MCP client and use its native Streamable HTTP setup to add a server named mrscraper at exactly 

      [https://mcp.mrscraper.com/mcp](https://mcp.mrscraper.com/mcp)

       with no static credential. Complete the OAuth 2.1 browser sign-in; if it does not start automatically, use the client's MCP login or authenticate command. Only if the client does not support OAuth, fall back to a MrScraper API key from 

      [https://app.mrscraper.com/api-tokens](https://app.mrscraper.com/api-tokens)

       stored through the client's environment-variable or secret-storage mechanism and sent as Authorization: Bearer . Reload the MCP client when required, then list the tools and confirm that fetch, scrape, serp, status, rerun, results, and result are available.
## Client configuration Add the hosted URL without a static `Authorization` header, then complete browser sign-in when the client prompts you. ### Codex ```bash codex mcp add mrscraper \ --url https://mcp.mrscraper.com/mcp ``` Codex uses OAuth by default for Streamable HTTP servers. If browser sign-in does not start automatically, run `codex mcp login mrscraper`. ### Claude Code ```bash claude mcp add \ --transport http \ --scope user \ mrscraper https://mcp.mrscraper.com/mcp ``` Open `/mcp` inside Claude Code and authenticate, or run `claude mcp login mrscraper` from your shell. ### Claude and Claude Desktop Open the custom connector form In a general Claude chat, select the **+** button below the input box, then choose **Connectors → Add connector → Add custom connector**. Open Add custom connector from a Claude chat Add MrScraper Enter `MrScraper` as the name and `https://mcp.mrscraper.com/mcp` as the MCP server URL, then select **Continue**. Enter the MrScraper name and MCP server URL Sign in Complete the MrScraper OAuth browser flow when Claude prompts you. ### Cursor Open **Cursor Settings → Tools & MCP → Add Custom MCP**, then add: ```json { "mcpServers": { "mrscraper": { "type": "http", "url": "https://mcp.mrscraper.com/mcp" } } } ``` Refresh the MCP server list, then authenticate when prompted. ### VS Code Add the server to your user settings: ```json { "mcp": { "servers": { "mrscraper": { "type": "http", "url": "https://mcp.mrscraper.com/mcp" } } } } ``` For a workspace configuration, place the server object under `servers` in `.vscode/mcp.json`. Start the server and VS Code opens a browser for OAuth. ### Windsurf Add the following entry to the Windsurf MCP configuration: ```json { "mcpServers": { "mrscraper": { "serverUrl": "https://mcp.mrscraper.com/mcp" } } } ``` Refresh the MCP server list, then complete OAuth when prompted. ### Other MCP clients Use these connection values: | Setting | Value | | -------------- | ------------------------------------------- | | Name | `mrscraper` | | Transport | Streamable HTTP | | URL | `https://mcp.mrscraper.com/mcp` | | Authentication | OAuth 2.1; do not configure a static header | ## API key fallback If a client cannot complete OAuth, create an [API key](https://app.mrscraper.com/api-tokens), store it through the client's secret mechanism, and send it as a bearer token: ```json { "mcpServers": { "mrscraper": { "type": "http", "url": "https://mcp.mrscraper.com/mcp", "headers": { "Authorization": "Bearer YOUR_MRSCRAPER_API_KEY" } } } } ``` For Codex, pass the environment-variable name instead of embedding the key: ```bash export MRSCRAPER_API_KEY="YOUR_MRSCRAPER_API_KEY" codex mcp add mrscraper \ --url https://mcp.mrscraper.com/mcp \ --bearer-token-env-var MRSCRAPER_API_KEY ``` ## Authentication | Connection | Credential source | | --------------------------- | ------------------------------------------------- | | Hosted Streamable HTTP | OAuth 2.1 browser sign-in, or an API key fallback | | Self-hosted Streamable HTTP | `Authorization: Bearer ` | | Local stdio | `MRSCRAPER_API_KEY`, then `MRSCRAPER_API_TOKEN` | The hosted server publishes OAuth 2.1 protected-resource metadata and challenges unauthenticated clients to start browser sign-in. Access tokens are issued for the exact `https://mcp.mrscraper.com/mcp` resource and use `scrape:read`, `scrape:write`, and `account:read` scopes. A tool called without its required scope returns `403 insufficient_scope`. API keys remain supported for clients without OAuth and carry full account authority. Authentication stays at the transport layer; tools never accept credentials as arguments. ## Run the MCP server locally Node.js 20 or newer is required. ### Published package over stdio ```json { "mcpServers": { "mrscraper": { "command": "npx", "args": ["-y", "@mrscraper/mcp@latest"], "env": { "MRSCRAPER_API_KEY": "YOUR_MRSCRAPER_API_KEY" } } } } ``` ### Source checkout over Streamable HTTP Clone and build ```bash git clone https://github.com/mrscraper-com/mrscraper-mcp.git cd mrscraper-mcp npm ci npm run build ``` Start HTTP transport ```bash TRANSPORT=http npm start ``` The default endpoint is `http://127.0.0.1:8000/mcp`. Connect the client ```json { "mcpServers": { "mrscraper": { "type": "http", "url": "http://127.0.0.1:8000/mcp", "headers": { "Authorization": "Bearer YOUR_MRSCRAPER_API_KEY" } } } } ``` ### Docker ```bash docker build -f docker/Dockerfile -t mrscraper-mcp . docker run --rm -p 8000:8000 mrscraper-mcp ``` The image binds to \`0.0.0.0). Apply network controls appropriate for the deployment and keep bearer verification enabled. ## Environment variables | Variable | Default | Purpose | | -------------------------------- | --------------------------------- | ---------------------------------------------------------------- | | `TRANSPORT` | `stdio` | Selects `stdio` or `http`. | | `HOST` | `127.0.0.1` | HTTP bind address; the Docker image uses `0.0.0.0`. | | `PORT` | `8000` | HTTP listen port. | | `MRSCRAPER_API_KEY` | — | Primary stdio credential. | | `MRSCRAPER_API_TOKEN` | — | Legacy stdio credential alias. | | `MRSCRAPER_HTTP_AUTH` | `1` | Enables HTTP bearer verification. | | `MRSCRAPER_ALLOWED_ORIGINS` | — | Comma-separated browser origins allowed to call the HTTP server. | | `MRSCRAPER_API_BASE_URL` | MrScraper platform API | Platform endpoint override for development and testing. | | `MRSCRAPER_FETCH_BASE_URL` | MrScraper Web Unblocker | Fetch endpoint override. | | `MRSCRAPER_SYNC_BASE_URL` | MrScraper synchronous scraper API | SERP endpoint override. | | `MRSCRAPER_LOG_HTTP_PAYLOAD` | off | Enables trusted-environment request-body diagnostics. | | `MRSCRAPER_LOG_HTTP_PAYLOAD_MAX` | `8192` | Maximum diagnostic payload length. | ## Security behavior * HTTP authentication is enabled by default. * Credential-bearing response headers are filtered from tool output. * Parsed JSON credential metadata and credentials in generated curl commands are redacted. * Extracted scraper values remain available in the response's `data` field. * Browser-origin requests are accepted from trusted local origins and exact values configured through `MRSCRAPER_ALLOWED_ORIGINS`. * Tool calls receive the caller's authenticated OAuth token or API key through the MCP transport rather than through tool arguments. ## Choosing the right tool | Starting point | Desired outcome | Tool | | ------------------- | ---------------------------------- | ------------------------------------ | | A known URL | Page response from Web Unblocker | `fetch` | | A known URL | Defined fields or repeated records | `scrape` with `general` or `listing` | | A site root | A bounded URL map | `scrape` with `map` | | A topic or keyword | Google result discovery | `serp` | | A saved scraper ID | A new scraper run | `rerun` | | An account | Usage and quota details | `status` | | A result collection | Pagination, filtering, and search | `results` | | A result ID | One complete stored result | `result` | A discovery workflow commonly uses `serp` first, followed by `fetch` for the page response or `scrape` for defined output fields. A saved-scraper workflow commonly uses `scrape`, then `rerun`, followed by `result`. ## Common response envelope API-backed tools return the same envelope through MCP `structuredContent` and a formatted JSON text block: ```json { "status_code": 200, "data": {}, "headers": { "content-type": "application/json" } } ``` | Field | Type | Description | | ------------- | ----------------------------- | --------------------------------------------------------------------------------------------------------------- | | `status_code` | number or `null` | HTTP status returned by the MrScraper service, or `null` when the request ends before an HTTP response arrives. | | `data` | JSON value, string, or `null` | Parsed JSON or response text supplied by the service. | | `headers` | object | Response headers with credential-bearing headers filtered out. | | `error` | string, when present | Request failure summary. | API failures preserve the envelope in `structuredContent` and set the MCP result's `isError` flag. Input-contract errors are returned as MCP tool errors. This lets clients inspect status, response data, and safe headers while also following the standard MCP error signal. ## Tool reference ### `fetch` Use `fetch` for a known URL when you need the page response from MrScraper [Web Unblocker](/docs/features/unblocker). Each invocation corresponds to one request: ```text GET https://api.mrscraper.com/ ``` #### Basic request ```json { "url": "https://example.com/products" } ``` `browser_rendering` and `super_mode` are independent controls. All four combinations can return different results for the same URL: | Browser rendering | Super Mode | Input | Loading path | | ----------------- | ---------- | ----------------------------------------------------------------- | -------------------------------------------------------- | | Off | Off | `{ "url": "URL" }` | Standard routing with the non-browser loader. | | On | Off | `{ "url": "URL", "browser_rendering": true }` | Standard routing with browser loading and JavaScript. | | Off | On | `{ "url": "URL", "super_mode": true }` | Real-device routing with the non-browser loader. | | On | On | `{ "url": "URL", "browser_rendering": true, "super_mode": true }` | Real-device routing with browser loading and JavaScript. | Start with both controls off, inspect the response, and change one axis at a time. Browser rendering is not guaranteed to produce a better response: some sites fail with it enabled but load without it. Try remaining untested combinations without repeating an identical failed request. #### JavaScript-rendered page ```json { "url": "https://example.com/products", "browser_rendering": true, "wait_for_selector": ".product-card", "timeout": 45 } ``` #### Real-device Super Mode Use both controls for real-device browser loading: ```json { "url": "https://example.com/products", "browser_rendering": true, "super_mode": true } ``` #### Geo-sensitive page ```json { "url": "https://example.com/offers", "browser_rendering": true, "geo_code": "ID", "home_page": true, "block_resources": true, "max_retries": 3, "token_cap": 20 } ``` #### Parameters | Parameter | Required | Default | API query field | Description | | ------------------- | -------- | ------- | ------------------ | ----------------------------------------------------------------------------------------------------- | | `url` | Yes | — | `url` | Target page URL. | | `browser_rendering` | No | `false` | `browserRendering` | Loads the page in a browser and executes JavaScript. | | `super_mode` | No | `false` | `super` | Selects real-device routing independently of browser rendering. | | `geo_code` | No | omitted | `geoCode` | Selects proxy-country routing. | | `wait_for_selector` | No | omitted | `waitForSelector` | Waits for a CSS selector together with `browser_rendering: true`. | | `home_page` | No | `false` | `homePage` | Visits the site root before loading the target URL. | | `block_resources` | No | `false` | `blockResources` | Applies resource blocking during page loading. | | `max_retries` | No | `3` | `maxRetries` | Sets the retry limit; zero is accepted. | | `token_cap` | No | omitted | `tokenCap` | Sets the retry token budget. | | `timeout` | No | `30` | `timeout` | Sets the page-load deadline in seconds. The MCP server allows an additional 30 seconds for transport. | The response body is available in the envelope's `data` field, commonly as HTML. Use browser rendering when JavaScript is needed, but retry the same `super_mode` value without it when browser loading fails, is blocked, or returns worse content. Toggle `super_mode` independently when routing may be the problem. A selector wait still requires browser rendering. ### `scrape` Use `scrape` when you need defined fields, repeated records, or a site URL map. It calls: ```text POST https://api.app.mrscraper.com/api/v1/scrapers-ai ``` #### Agent modes | Agent | Designed for | Inputs | | --------- | ------------------------------------- | ------------------------------------------------------------------------- | | `general` | Defined fields from one page | `prompt`, `schema_prompt`, `proxy_country` | | `listing` | Repeated records across listing pages | `prompt`, `schema_prompt`, `proxy_country`, `max_pages` | | `map` | Bounded URL discovery across a site | `max_depth`, `max_pages`, `limit`, `include_patterns`, `exclude_patterns` | Choose inputs from the selected agent's row. The default agent is `general`. The separate `mode` input selects the `Cheap` or `Super` backend execution tier. Omit it to preserve the backend default. #### General extraction ```json { "url": "https://example.com/product", "agent": "general", "mode": "Super", "prompt": "Extract the product name, price, availability, description, and image URLs", "proxy_country": "US" } ``` The request body includes `url`, `message`, and `agent`, plus `proxyCountry` when supplied. #### Listing extraction ```json { "url": "https://example.com/products", "agent": "listing", "prompt": "Extract every product name, price, availability, and detail URL", "max_pages": 5 } ``` Listing requests are synchronous. Use `max_pages` to define the desired page scope. When it is omitted, the service applies its configured default. #### Site map ```json { "url": "https://example.com", "agent": "map", "max_depth": 2, "max_pages": 50, "limit": 1000, "include_patterns": "/products/", "exclude_patterns": "/cart/|/checkout/" } ``` Map requests send `url`, `agent`, and the crawl controls supplied in the tool call. Omitted crawl controls use service defaults. #### Best-effort schema guidance ```json { "url": "https://example.com/product", "prompt": "Extract the product", "schema_prompt": { "type": "object", "properties": { "name": { "type": "string" }, "price": { "type": "number" }, "in_stock": { "type": "boolean" } }, "required": ["name", "price"] } } ``` The MCP server appends `schema_prompt` to the natural-language instruction as shape guidance. Validate the returned value in the consuming application when strict conformance is required. #### Parameters | Parameter | Required | Default | Request mapping | Description | | ------------------ | --------------- | --------------- | ---------------------- | ---------------------------------------------------------------- | | `url` | Yes | — | Body `url` | Target URL for every agent. | | `prompt` | General/listing | — | Body `message` | Natural-language extraction instruction. | | `schema_prompt` | No | omitted | Appended to `message` | Best-effort JSON Schema shape guidance for general/listing. | | `agent` | No | `general` | Body `agent` | Selects `general`, `listing`, or `map`. | | `mode` | No | service default | Body `mode` | Selects `Cheap` or `Super` execution independently of the agent. | | `proxy_country` | No | omitted | Body `proxyCountry` | Proxy country for general/listing. | | `max_pages` | No | service default | Body `maxPages` | Page bound for listing/map. | | `max_depth` | No | service default | Body `maxDepth` | Link-depth bound for map. | | `limit` | No | service default | Body `limit` | URL-result bound for map. | | `include_patterns` | No | service default | Body `includePatterns` | URL inclusion expression for map. | | `exclude_patterns` | No | service default | Body `excludePatterns` | URL exclusion expression for map. | #### Reproduce a scrape with `rerun` Every successful `scrape` creates a saved AI scraper configuration by default. The response run object contains `scraperId`. Pass that UUID to `rerun` as `scraper_id` to apply the same saved prompt and agent configuration to the original URL or another URL: ```json { "target": "https://example.com/product-2", "type": "ai", "scraper_id": "scraper-uuid" } ``` This makes the scraper configuration reproducible, but page changes and model behavior can still change the extracted values. `rerun` also supports dashboard-built manual workflows and asynchronous bulk jobs across multiple target URLs. See the full [`rerun`](#rerun) section below for the available modes and result-tracking workflow. ### `serp` Use `serp` when discovery starts from a Google query or Google search URL. It calls: ```text POST https://sync.scraper.mrscraper.com/api/google/serp/v2/sync ``` #### Search query ```json { "query_or_url": "iphone 17", "region": "id", "language": "id", "page": 2, "format": "json", "render_js": false } ``` #### Google search URL ```json { "query_or_url": "https://www.google.com/search?q=iphone+17&gl=us&hl=en&start=20" } ``` For a Google URL, the server derives: | URL parameter | Tool request field | | ------------- | ------------------------------------------- | | `q` | `query` | | `gl` | `region` | | `hl` | `language` | | `start` | One-based `page` using groups of 10 results | Explicit `region`, `language`, and `page` inputs take priority over the corresponding URL values. #### Parameters | Parameter | Required | Default | Request mapping | Description | | ---------------- | -------- | -------------------- | ---------------------- | ---------------------------------------------------------------- | | `query_or_url` | Yes | — | Body `query` | Search query or complete Google search URL. | | `region` | No | URL value or omitted | Body `region` | Result country code. | | `language` | No | URL value or omitted | Body `language` | Result language code. | | `page` | No | URL value or omitted | Body `page` | One-based result page. | | `format` | No | `json` | Body `format` | Selects parsed JSON or result-page HTML. | | `render_js` | No | `false` | Body `renderJs` | Waits for JavaScript-rendered SERP features such as AI Overview. | | `raw` | No | `false` | Body `format=html` | Compatibility alias for HTML output. | | `client_timeout` | No | `120` | Local request deadline | Sets the upstream HTTP timeout in seconds. | ### `status` Use `status` for account information and optional domain request outcomes. Every call reads: ```text GET https://api.app.mrscraper.com/api/v1/subscription-accounts ``` Supplying `domain` adds: ```text GET https://api.app.mrscraper.com/api/v1/analytic/statuses ``` #### Account status ```json {} ``` #### Domain analytics ```json { "domain": "https://www.example.com/products", "from": "7d", "to": "now", "action": "fetch", "api_token_name": "production" } ``` The domain input accepts a hostname or a URL. URLs are normalized to their hostname before the analytics request. #### Date syntax | Syntax | Example | Meaning | | ------------ | ---------------------- | ------------------------------------ | | Minutes | `30m` | 30 minutes before the reference time | | Hours | `24h` | 24 hours before the reference time | | Days | `7d` | 7 days before the reference time | | Weeks | `2w` | 2 weeks before the reference time | | Current time | `now` | Current reference time | | ISO 8601 | `2026-08-18T12:00:00Z` | Exact timestamp | #### Parameters | Parameter | Required | Default | Description | | ---------------- | -------- | ------------ | ----------------------------------------------------- | | `domain` | No | omitted | Adds request-outcome analytics for a hostname or URL. | | `from` | No | `24h` | Analytics range start. | | `to` | No | `now` | Analytics range end. | | `action` | No | empty filter | Filters analytics by exact action. | | `api_token_name` | No | empty filter | Filters analytics by API-token name. | #### Status summary ```json { "kind": "mrscraper-cli-status-summary", "source_endpoints": [ "/subscription-accounts", "/analytic/statuses" ], "status_code": 200, "data": { "account": { "subscription_status": "active", "enterprise": false, "token_usage": 250, "token_limit": 1000, "token_remaining": 750, "usage_percent": 25, "rate_limit": 10, "rate_ttl": 60, "auto_renew": true, "ends_at": null, "user": { "name": "Ada", "email": "ada@example.com", "verified": true } }, "analytics": { "domain": "www.example.com", "from": "2026-08-11 12:00:00 UTC", "to": "2026-08-18 12:00:00 UTC" } } } ``` The summary selects account fields, calculates `token_remaining` and `usage_percent`, and records every source endpoint used for the response. When domain analytics encounter an API failure, the account summary remains in `data.account` and the analytics response is available in `data.analytics`. ### `rerun` Use `rerun` with the UUID of an existing scraper. Choose its type and target count independently: * Set `type` to `ai` for an AI scraper created by the `scrape` tool. * Set `type` to `manual` for a step-based workflow created in the MrScraper dashboard. MCP reruns existing manual workflows; it does not create them. * Leave `bulk` as `false` for one URL. Set it to `true` to submit the same saved configuration once with a comma- or newline-separated URL list. Manual reruns can be single or bulk. Bulk reruns can use either an AI or a manual scraper; bulk describes the number of targets, not the scraper type. Bulk mode submits one asynchronous backend job instead of repeating the single-URL tool call locally. Save `data.data.bulkResultId` from the response and pass it to `result` as `result_id` until the stored result is finished. The combination of `type` and `bulk` selects one endpoint: | Mode | Endpoint | ID parameter | Target | Crawl controls | | ------------- | ---------------------------------- | ------------ | ---------------------------- | ------------------------------------------------------------------------------------------------------------------ | | Single AI | `POST /scrapers-ai-rerun` | `scraper_id` | One URL | `max_depth`, `max_pages`, `limit`, `include_patterns`, `exclude_patterns`, `proxy_country`, `max_retry`, `timeout` | | Bulk AI | `POST /scrapers-ai-rerun/bulk` | `id` | Comma/newline-separated URLs | Saved scraper configuration | | Single manual | `POST /scrapers-manual-rerun` | `scraper_id` | One URL | Saved scraper configuration | | Bulk manual | `POST /scrapers-manual-rerun/bulk` | `id` | Comma/newline-separated URLs | Saved scraper configuration | The endpoint paths above use the platform API base: ```text https://api.app.mrscraper.com/api/v1 ``` #### Single AI rerun ```json { "target": "https://example.com/products", "type": "ai", "scraper_id": "scraper-uuid", "max_depth": 2, "max_pages": 50, "limit": 1000, "include_patterns": "/products/", "exclude_patterns": "/cart/|/checkout/", "proxy_country": "ID", "max_retry": 4, "timeout": 120 } ``` #### Single manual rerun ```json { "target": "https://example.com/product/123", "type": "manual", "scraper_id": "scraper-uuid" } ``` #### Bulk AI rerun ```json { "target": "https://example.com/a,https://example.com/b\nhttps://example.com/c", "type": "ai", "bulk": true, "id": "scraper-uuid" } ``` The bulk target parser separates URLs on commas and newlines. #### Parameters | Parameter | Required | Default | Description | | ------------------ | ----------- | ------- | ------------------------------------------------------------------------------- | | `target` | Yes | — | One URL, or a comma/newline-separated URL string for bulk mode. | | `type` | Yes | — | Selects `ai` or `manual`. | | `bulk` | No | `false` | Selects a bulk endpoint. | | `scraper_id` | Single mode | — | Saved scraper UUID for one target URL. | | `id` | Bulk mode | — | Saved scraper UUID for the bulk target list. | | `max_depth` | Single AI | omitted | Crawl depth; omission preserves the saved scraper/backend default. | | `max_pages` | Single AI | omitted | Page bound; omission preserves the saved scraper/backend default. | | `limit` | Single AI | omitted | Result bound; omission preserves the saved scraper/backend default. | | `include_patterns` | Single AI | omitted | URL inclusion expression; omission preserves the saved scraper/backend default. | | `exclude_patterns` | Single AI | omitted | URL exclusion expression; omission preserves the saved scraper/backend default. | | `proxy_country` | Single AI | omitted | Proxy country code. | | `max_retry` | Single AI | omitted | Retry limit; zero is accepted. | | `timeout` | Single AI | omitted | Timeout in seconds, used by listing reruns. | The MCP server sends single-AI controls only when supplied. Manual and bulk reruns reject them. Manual reruns carry a compliance acknowledgment in the MCP server instructions. MCP clients should present that acknowledgment before executing a manual rerun. ### `results` Use `results` to browse stored runs through: ```text GET https://api.app.mrscraper.com/api/v1/results ``` #### Paginated request ```json { "sort_field": "updatedAt", "sort_order": "desc", "page_size": 25, "page": 1 } ``` #### Filtered request ```json { "sort_field": "updatedAt", "sort_order": "asc", "page_size": 50, "page": 1, "search": "example.com", "scraper_id": "scraper-uuid", "status": "Finished", "type": "Rerun-AI", "url": "https://example.com/product", "date_range_column": "updatedAt", "start_at": "2026-08-01T00:00:00Z", "end_at": "2026-08-18T23:59:59Z" } ``` #### Parameters | Parameter | Required | Default | API query field | Description | | ------------------- | -------- | ----------- | -------------------- | ---------------------------------------------------------------------- | | `sort_field` | No | `updatedAt` | `sortField` | Field used by the results API for sorting. | | `sort_order` | No | `desc` | `sortOrder` | Case-insensitive `asc` or `desc`; sent to the API in uppercase. | | `page_size` | No | `10` | `pageSize` | Number of rows per page. | | `page` | No | `1` | `page` | One-based page index. | | `search` | No | omitted | `search` | Free-text result filter. | | `scraper_id` | No | omitted | `filters[scraperId]` | Exact saved scraper UUID filter. | | `status` | No | omitted | `filters[status]` | Exact `Draft`, `Finished`, `Running`, `Failed`, or `Cancelled` filter. | | `type` | No | omitted | `filters[type]` | Exact result type filter, such as `AI` or `Rerun-AI`. | | `url` | No | omitted | `filters[url]` | Exact stored target URL filter. | | `date_range_column` | No | omitted | `dateRangeColumn` | Column used for the date range. | | `start_at` | No | omitted | `startAt` | Inclusive range start. | | `end_at` | No | omitted | `endAt` | Inclusive range end. | The results API receives the `sort_field` value supplied by the caller. `sort_order` is normalized and sent as `ASC` or `DESC`. ### `result` Use `result` when the result UUID is already known: ```text GET https://api.app.mrscraper.com/api/v1/results/{result_id} ``` ```json { "result_id": "result-uuid", "include_html": false } ``` | Parameter | Required | Default | Description | | -------------- | -------- | ------- | ----------------------------------------------------------------------------------- | | `result_id` | Yes | — | Stored MrScraper result UUID. | | `include_html` | No | `true` | Includes stored HTML. Set `false` for a smaller polling or extracted-data response. | ## Troubleshooting The MCP server may not have loaded correctly. 1. Reload the MCP client or start a new session. 2. Inspect `tools/list`. 3. Confirm that the server advertises these tools: `fetch`, `scrape`, `serp`, `status`, `rerun`, `results`, and `result`. The MCP connection is not signed in or is using an invalid API key. 1. For hosted OAuth connections, reconnect so the client starts browser sign-in again. 2. For HTTP API-key connections, confirm that the request includes `Authorization: Bearer `. 3. For stdio connections, confirm that `MRSCRAPER_API_KEY` or `MRSCRAPER_API_TOKEN` is available in the launched server environment. The client's browser origin is not included in the server's allowed origins. 1. Add the client's exact browser origin to `MRSCRAPER_ALLOWED_ORIGINS`. 2. Restart the MCP server if required. 3. Run the request again. Command-line and service-to-service MCP clients typically connect without an `Origin` header. The tool call may not match the parameters supported by the selected operation. 1. Check the tool's parameter table. 2. For `scrape`, use only the inputs listed for the selected agent. 3. For `rerun`, use the ID and crawl controls listed for the selected single or bulk mode. 4. Run the request again with the corrected parameters. The upstream service may need more time to complete the request. 1. Increase `fetch.timeout` to allow more time for a Web Unblocker request. 2. If the timeout occurs during a Google request, increase `serp.client_timeout`. 3. Run the request again after updating the appropriate timeout. Use the following command to verify that a local HTTP MCP server is responding correctly: ```bash npm run test:mcp -- \ --target http://127.0.0.1:8000/mcp \ --token "$MRSCRAPER_API_KEY" ``` For package development, run the following checks: ```bash npm run format:check npm run lint npm test npm run build npm pack --dry-run ``` # Overview import { Bot, BotMessageSquare, TextSearch, Workflow, ShoppingCart, Briefcase, Star, SlidersHorizontal, Mail, Database, Webhook, ClipboardClock, Network, Layers, Map, Home, Plane, DollarSign, Store, Code2, Rocket, FlaskConical, Key, Terminal, Server, ShieldCheck, Waypoints, Users, Cookie, ChartLine, SatelliteDish, HeartPulse, Zap, PackageOpen, Package2, Link2, PanelTop, Globe } from 'lucide-react'; Hello Discover how our powerful scraping platform can transform the way you extract data. Whether you're scraping e-commerce product data, collecting job listings, gathering customer reviews for sentiment analysis, or extracting any website content, our AI-powered and manual scrapers are designed to automate and simplify the process, giving you a powerful edge in any project. ## Start Here New to MrScraper? These pages get you from sign-up to your first scraped dataset. } href="/docs/getting-started/quickstart"> Run your first scraper in a few minutes with a step-by-step walkthrough. } href="/docs/getting-started/playground"> Test API requests, explore features, and generate code without writing code. } href="/docs/getting-started/api-token"> Learn how tokens work and create an API token for authentication. } href="/docs/getting-started/billing"> Understand subscription plans, token usage, and billing cycles. ## AI Agents Pick the agent that matches the page you're scraping, or chain several together. } href="/docs/features/ai-scraper"> Automatically extract data with natural language instructions — no coding required. } href="/docs/features/ai-scraper/general"> Extract structured data from a single web page using a natural language prompt. } href="/docs/features/ai-scraper/listing"> Scrape listing pages with automated pagination, infinite scroll, and load-more handling. } href="/docs/features/ai-scraper/map"> Discover and extract every URL from a website to understand its structure. } href="/docs/features/ai-scraper/pdp"> Scrape product detail pages faster and cheaper using cached sources. } href="/docs/features/ai-scraper/multi-agent-flow"> Combine Map, Listing, and General agents into end-to-end extraction workflows. ## Features Explore MrScraper's key features and tools designed to make data extraction effortless. } href="/docs/features/manual-scraper"> Build and customize your own scraper workflow step by step for full control. } href="/docs/features/manual-scraper/self-healing"> Let AI generate, repair, and extend Manual Scraper workflows when a site changes. } href="/docs/features/bulk-scraping"> Scrape multiple URLs in one go by uploading a list or Excel file. } href="/docs/features/unblocker"> Bypass anti-scraping measures and reach geo-restricted content. } href="/docs/features/proxy"> Route individual scrapers through proxies to mask your IP and improve reliability. } href="/docs/features/cookie"> Use your own cookies to access content behind logins and restrictions. } href="/docs/features/schedule"> Automate scraper runs on a schedule to keep your data fresh and up to date. } href="/docs/features/marketplace"> Use ready-to-run scrapers for popular websites and common scraping use cases. } href="/docs/features/activating-api"> Activate a scraper as an API endpoint and run it programmatically. } href="/docs/features/s3-storage"> Save scraped data straight into your own S3 bucket. } href="/docs/features/analytics"> Monitor performance metrics, token usage, and the health of your scraping operations. } href="/docs/features/team"> Create a team to share scrapers and collaborate on projects. ## Use Cases Discover how MrScraper can support different real-world scenarios. } href="/docs/guides/ecommerce"> Scrape product details, pricing, and availability from major e-commerce sites. } href="/docs/guides/job-listing"> Collect job postings and company data for recruitment or market analysis. } href="/docs/guides/sentiment-analysis"> Extract and analyze customer feedback to gain insights into sentiment and satisfaction. } href="/docs/guides/real-estate"> Gather property listings, prices, and market trends from real estate websites. } href="/docs/guides/travel-hospitality"> Scrape flight prices, airlines, and schedules for price monitoring and booking analysis. } href="/docs/guides/optimizing-cost"> Minimize token usage and maximize scraping efficiency without losing accuracy. } href="/docs/guides/programmatically"> Create an AI scraper once, then run it from your own code via the API. ## Integrations Integrate MrScraper with your favorite tools to automate your data flow. } href="/docs/integrations/email"> Receive email notifications whenever a scraper finishes or encounters an error. } href="/docs/integrations/sql"> Automatically store scraped data in your database for querying and analysis. } href="/docs/integrations/webhook"> Send scraper results directly to external services through webhooks. } href="/docs/integrations/zapier"> Connect your scrapers to thousands of apps with no-code Zapier automations. } href="/docs/integrations/n8n"> Build automated workflows around your scrapers with the MrScraper n8n node. } href="/docs/integrations/apify"> Run MrScraper as Apify Actors and pipe results into the Apify platform. } href="/docs/integrations/clawhub"> Install the MrScraper skill to give your OpenClaw agent unblockable web scraping. ## Developer Tools Work with MrScraper from your terminal, your codebase, or your AI agent. } href="/docs/getting-started/cli"> Scrape, crawl, and extract data from the web directly from your terminal. } href="/docs/getting-started/mcp-server"> Let AI agents fetch and extract web data through the Model Context Protocol. } href="/docs/integrations/python"> Call the MrScraper API asynchronously with the `mrscraper-sdk` package. } href="/docs/integrations/node"> Scrape, crawl, and extract structured data using the MrScraper Node SDK. } href="/docs/integrations/langchain"> Use MrScraper as LangChain tools inside your own agents. } href="/docs/api/overview"> Explore every endpoint, authentication, pagination, and error handling. ## Residential Proxy Route your own tools and browsers through our proxy network. } href="/docs/residential-proxy/getting-started/overview"> Understand proxy types, global coverage, and how to get started. } href="/docs/residential-proxy/configuration/authentication"> Configure username patterns, sessions, and country targeting. } href="/docs/residential-proxy/examples/overview"> Ready-to-use snippets for Python, Node.js, Go, PHP, Playwright, and more. # Quickstart import { Play } from 'lucide-react'; This guide shows you how to make your first scraping request with MrScraper. ## Make Your First API Request Create a MrScraper account and obtain your API key from the [Playground](https://app.mrscraper.com/playground) before following this guide. In this quickstart, you'll scrape a Walmart product page using the General AI Agent to get the following data: * Product name * Price * Rating * Number of reviews * Product link cURL ```shell curl --location --request POST 'https://api.mrscraper.com?token=&html=true&super=true' \ -H 'x-api-token: ' \ -H 'Content-Type: application/json' \ -d '{ "url": "https://www.walmart.com/ip/IdeaPad-Slim-3x-15-Laptop-Snapdragon-X-X1-26-100-16GB-RAM-512GB-SSD-Luna-Grey/19075520026", "prompt": "Extract the following information in JSON format:\n\nproduct_name\nprice\nrating\nnumber_of_reviews\nproduct_link", "agent": "general" }' ``` } href="https://app.mrscraper.com/playground"> Run this request instantly without writing any code. The response contains the extracted data from the page. JSON ```json { "success": true, "data": { "product_name": "Lenovo IdeaPad Slim 3x 15.3\" Laptop AI Snapdragon X 16GB 512GB SSD Luna Grey", "price": 449, "rating": 4.5, "number_of_reviews": 103, "product_link": "https://www.walmart.com/ip/IdeaPad-Slim-3x-15-Laptop-Snapdragon-X-X1-26-100-16GB-RAM-512GB-SSD-Luna-Grey/19075520026" }, "thread_id": "cebfaa37-f6de-45e7-8dd4-ec955824d81b", "token_usage": 16, "markdown": "", "screenshot": "", "status": "Converting data to JSON", "error": "", "runtime": 14852 } ``` ## Use a Predefined Marketplace API MrScraper also provides ready-to-use scraping APIs through the [Marketplace](/docs/features/marketplace). These APIs are designed for specific industries and websites. You do not need to create prompts or extraction logic yourself. Available categories include: * E-commerce * News * Real Estate * Travel * And more } href="https://app.mrscraper.com/marketplace?page=1&pageSize=12"> Browse and test ready-to-use scrapers. For example, you can use the [Amazon Product Details API](https://app.mrscraper.com/marketplace/d08bcc7c-fa99-4886-9db2-3c274a9a1c2d) to extract product information from an Amazon product page. ```shell curl --location --request POST 'https://sync.scraper.mrscraper.com/api/amazon/pdp/sync' \ -H 'Authorization: bearer ' \ -H 'Content-Type: application/json' \ -d '{ "url": "https://www.amazon.com/STANLEY-Flowstate-3-Position-Compatible-Insulated/dp/B0CP9YB3Q4/ref=zg_bs_c_kitchen_d_sccl_2/145-6361404-7555432?pd_rd_w=3BPkW&content-id=amzn1.sym.fef9af56-6177-46e9-8710-a5293a68dd39&pf_rd_p=fef9af56-6177-46e9-8710-a5293a68dd39&pf_rd_r=E598BMWHB9QEZ75X7ABP&pd_rd_wg=Z9psI&pd_rd_r=7e3be220-af38-4e7b-b45c-9a68470a3e1f&pd_rd_i=B0CP9YB3Q4&th=1" }' ``` The response contains the extracted data from the page. JSON ```json { "success": true, "message": "Successfully scraped", "data": [ { "pageNumber": 1, "payload": { "templates": [], "mods": { "filter": { "tItemType": "nt_filter", "filterItems": [ { "name": "category", "unfoldRow": "2", "title": "Category", "urlKey": "category", "hidden": false, "locked": false, "type": "category", "uniqueName": "category", "value": "", "displayValue": "", "options": [ { "url": "/beli-sabun-pembersih-wajah/?q=159947433", "title": "Facial Cleanser", "value": "beli-sabun-pembersih-wajah" } ] }, { "name": "brand", "unfoldRow": "-1", "title": "Brand", "urlKey": "ppath", "hidden": false, "locked": false, "type": "brand", "uniqueName": "20000", "options": [ { "title": "Men's Biore", "value": "men-s-biore" } ] }, { "name": "service", "unfoldRow": "4", "title": "Service & Promotion", "urlKey": "service", "hidden": false, "locked": false, "type": "multiple", "uniqueName": "service", "options": [ { "title": "Fulfilled By Lazada", "value": "FBL", "activeIcon": "", "normalIcon": "" }, { "title": "Cash On Delivery", "value": "COD", "activeIcon": "", "normalIcon": "" }, { "title": "Coins", "value": "coins", "activeIcon": "", "normalIcon": "" }, { "title": "6.6 Sale", "value": "gcp_D6", "activeIcon": "https://img.lazcdn.com/us/lazgcp/e0557f6e-169b-45ec-91c4-e9e30eb17160_ALL-60-60.png", "normalIcon": "https://img.lazcdn.com/us/lazgcp/e0557f6e-169b-45ec-91c4-e9e30eb17160_ALL-60-60.png" } ] }, { "name": "location", "unfoldRow": "3", "title": "Shipped From", "urlKey": "location", "hidden": false, "locked": false, "type": "location", "uniqueName": "location", "value": [], "options": [ { "title": "East Java", "value": "A-ID-4" }, { "title": "Jabodetabek", "value": "A-ID-1" }, { "title": "North Sumatera", "value": "A-ID-6" }, { "title": "Kota Depok", "value": "R80010434" }, { "title": "Kab. Sidoarjo", "value": "R80010303" }, { "title": "Kab. Deli Serdang", "value": "R80010265" } ] }, { "showMin": "Min", "showMax": "Max", "name": "price", "unfoldRow": "2", "title": "Price", "urlKey": "price", "hidden": false, "locked": false, "type": "price", "uniqueName": "price" }, { "name": "rating", "unfoldRow": "2", "title": "Rating", "urlKey": "rating", "hidden": false, "locked": false, "type": "rating", "uniqueName": "rating", "value": "0" }, { "name": "attribute", "unfoldRow": "-1", "title": "Skin Care Benefits", "urlKey": "ppath", "hidden": false, "locked": false, "type": "multiple", "uniqueName": "40385", "options": [ { "title": "Oil Control", "value": "40385:135942" }, { "title": "Pore Control", "value": "40385:135951" }, { "title": "Brightening", "value": "40385:228695" } ] }, { "name": "attribute", "unfoldRow": "-1", "title": "Skin Types", "urlKey": "ppath", "hidden": false, "locked": false, "type": "multiple", "uniqueName": "30740", "options": [ { "title": "Normal", "value": "30740:60807" }, { "title": "Oily", "value": "30740:60839" } ] }, { "name": "attribute", "unfoldRow": "-1", "title": "Skin Concerns", "urlKey": "ppath", "hidden": false, "locked": false, "type": "multiple", "uniqueName": "40391", "options": [ { "title": "Oiliness", "value": "40391:71277" }, { "title": "Dullness", "value": "40391:135957" } ] }, { "name": "attribute", "unfoldRow": "-1", "title": "Product Feature", "urlKey": "ppath", "hidden": false, "locked": false, "type": "multiple", "uniqueName": "120024468", "options": [ { "title": "Paraben-free", "value": "120024468:136834" } ] }, { "name": "attribute", "unfoldRow": "-1", "title": "Product Form", "urlKey": "ppath", "hidden": false, "locked": false, "type": "multiple", "uniqueName": "30780", "options": [ { "title": "Foam", "value": "30780:68564" } ] }, { "name": "attribute", "unfoldRow": "-1", "title": "Ingredient Preference", "urlKey": "ppath", "hidden": false, "locked": false, "type": "multiple", "uniqueName": "120150202", "options": [ { "title": "Niacinamide", "value": "120150202:123811818" } ] }, { "name": "attribute", "unfoldRow": "-1", "title": "Special Claims", "urlKey": "ppath", "hidden": false, "locked": false, "type": "multiple", "uniqueName": "120023693", "options": [ { "title": "BPOM Registered", "value": "120023693:127537215" } ] } ], "filteredQuatity": "1", "filteredDoneText": "Product", "title": "Search Filter", "pos": 0 }, "listItems": [ { "name": "Men's Biore Double Scrub Facial Foam Cool Oil Clear 100gr", "nid": "159947433", "itemId": "159947433", "icons": [ { "domClass": "120014", "text": "Voucher save 4%", "type": "text", "group": "2", "showType": "0", "bizType": "voucherApplied" }, { "domClass": "225188", "type": "img", "group": "3", "showType": "0", "bizType": "campaign" }, { "domClass": "68675", "type": "img", "group": "3", "showType": "0", "bizType": "lazMall" } ], "image": "https://id-live-01.slatic.net/p/282e6a2fbf26571c3c5eab569e609852.jpg", "isSmartImage": false, "originalPriceShow": "", "priceShow": "Rp32.200", "discount": "4% Off", "ratingScore": "4.864764860348532", "review": "8378", "location": "Kota Depok", "thumbs": [], "sellerName": "KAO Store", "sellerId": "1000001053", "brandName": "Men's Biore", "brandId": "116687", "cheapest_sku": "BI568HBACHV3ANID-136017", "skuId": "181742191", "sku": "BI568HBACHV3ANID", "categories": [ 3509, 3902, 18067, 18216 ], "price": "32200", "restrictedAge": 0, "inStock": true, "originalPrice": "33600", "clickTrace": "query:159947433;nid:159947433;src:LazadaMainSrp;rn:223837b9b80f3a6303f7c1f263ef8577;region:id;sku:BI568HBACHV3ANID;price:32200;client:desktop;supplier_id:1000001053;session_id:;biz_source:h5_internal;slot:0;utlog_bucket_id:470687;asc_category_id:18216;item_id:159947433;sku_id:181742191;shop_id:16328;templateInfo:-1_A3_C#107879_E#", "itemSoldCntShow": "41.5K sold", "longImageDisplayable": false, "skus": [ { "id": "BI568HBACHV3ANID-136017" } ], "promotionId": "", "isSponsored": false, "tItemType": "nt_product", "skuType": "2", "adFlag": "0", "directSimilarUrl": "https://native.m.lazada.com/dynamicxresult?item_img=https%3A%2F%2Fid-live-01.slatic.net%2Fp%2F282e6a2fbf26571c3c5eab569e609852.jpg&src=srp_findsimilar&item_id=159947433&sku_id=181742191¶ms=%7B%22sub_src%22%3A%22long_press%22%2C%22src%22%3A%22srp_findsimilar%22%7D&m=tpp_findSimilar&q=159947433&similarType=findSimilarV1&price=32200.00&spuTriggerItem=0", "gridTitleLine": "2", "isFission": "0", "isBadgeAutoScroll": false, "showCart": false, "showBackIcon": false, "showUnitPrice": false, "itemUrl": "//www.lazada.co.id/products/pdp-i159947433.html", "querystring": "fs_ab=2&priceCompare=skuId%3A181742191%3Bsource%3Alazada-search-voucher%3Bsn%3A223837b9b80f3a6303f7c1f263ef8577%3BoriginPrice%3A3220000%3BdisplayPrice%3A3220000%3BisGray%3Afalse%3BsinglePromotionId%3A910000049169004%3BsingleToolCode%3ApromPrice%3BvoucherPricePlugin%3A0%3Btimestamp%3A1780565848388&c=&ratingscore=4.864764860348532&freeshipping=0&source=search&channelLpJumpArgs=&fuse_fs=&search=1&sale=41518&price=3.22E%204&review=8378&location=Kota%20Depok&stock=1&lang=en&request_id=223837b9b80f3a6303f7c1f263ef8577&clickTrackInfo=query%253A159947433%253Bnid%253A159947433%253Bsrc%253ALazadaMainSrp%253Brn%253A223837b9b80f3a6303f7c1f263ef8577%253Bregion%253Aid%253Bsku%253ABI568HBACHV3ANID%253Bprice%253A32200%253Bclient%253Adesktop%253Bsupplier_id%253A1000001053%253Bsession_id%253A%253Bbiz_source%253Ah5_internal%253Bslot%253A0%253Butlog_bucket_id%253A470687%253Basc_category_id%253A18216%253Bitem_id%253A159947433%253Bsku_id%253A181742191%253Bshop_id%253A16328%253BtemplateInfo%253A-1_A3_C%2523107879_E%2523" } ], "breadcrumb": [ { "url": "https://www.lazada.co.id", "title": "Home" }, { "title": "Search Results" } ], "sortBar": { "tItemType": "nt_sortbar", "filter": "Filter", "style": "wf", "sortItems": [ { "name": "Popularity", "tip": "Best Match", "isActive": "true", "value": "popularity", "key": "sort", "tabName": "" }, { "name": "Price-ASC", "tip": "Price low to high", "value": "priceasc", "key": "sort", "tabName": "" }, { "name": "Price-DESC", "tip": "Price high to low", "value": "pricedesc", "key": "sort", "tabName": "" } ], "hiddenLayoutBtn": false, "showFilterBtn": true, "hasFilter": false }, "resultTips": { "tItemType": "nt_resulttips", "tips": "1 items found for \"{$0}\"", "keywords": [ { "text": "159947433" } ] }, "linksInfo": [] }, "mainInfo": { "errorMsg": "", "bizCode": 0, "serverParams": "translatedEnQuery%3D159947433%26in_sufficient_relevant_items%3D1%26", "totalResults": "1", "pageSize": "40", "page": "1", "hiddenLayoutBtn": false, "layoutInfo": { "listHeader": [ "didYouMean", "campaignBanner", "categoryBar", "brandBar", "superStore", "banner", "resultTips", "navCategory", "hotDeal", "emptyResult", "recQuery", "emptyResultBannerTips", "limitedResult", "recommendTitle" ], "stickyHeader": [ "reminderBanner" ], "sceneHeader": [], "halfStickyHeader": [ "sortBar", "preposeFilter" ], "bottomHalfStickyHeader": [] }, "pageType": "searchList", "gridTitleLine": "2", "trackParams": {}, "userId": "", "cate_id": "", "hyperspaceInfo": "", "currency": "Rp", "currencyOnRight": "false", "currencySpace": "false", "isShowFeedbackForm": "false", "showThumbs": "false", "allProductURL": "/all-products/", "addToCartURL": "//cart.lazada.co.id/cart/api/add", "cluster": "fiber2_os30", "column": 2, "bucketId": "mobile_v4", "auctionType": 1, "noMorePages": true, "srpName": "LazadaMainSrp", "params": "{\"src\":\"h5_internal\"}", "searchScenario": "keyword", "hiddenShopChangeBtn": true, "selectedFilters": {}, "expParams": { "disableTileSimple": "1", "isBadgeAutoScroll": "0", "filterLazyLoad": "0", "funnelFilterBtnPosition": "sortBarContainerRight", "enableShortStyle": "0", "enableFeedbackAssistant": "1", "disableScreenSimple": "1", "enableCompactLineHeight": "1", "disableFspImagePreload": "0" }, "filterLazyLoad": false, "RN": "223837b9b80f3a6303f7c1f263ef8577", "style": "wf", "title": "159947433", "lang": "en", "venture": "ID", "themes": [ { "font-size": "11", "labelMarginLeft": "0", "color": "#FF0066", "key": "120014" }, { "img": "https://img.lazcdn.com/us/lzd-onepiece/e09da34c2fe4a880c5c286c8850d9fbd1755779562798.png", "width": "117", "labelMarginLeft": "0", "key": "68675", "height": "36" }, { "img": "https://img.lazcdn.com/us/lazgcp/a49c11fa-1ebb-468b-bcd6-ad3ba69ef827_ALL-112-52.png", "width": "112", "labelMarginLeft": "0", "key": "225188", "height": "52" } ], "q": "159947433", "rt": { "all": 179 }, "serverResponseTime": 1780565848, "reqParams": "{\"src\":\"h5_internal\",\"refer\":\"https://www.lazada.co.id/\",\"scenario\":\"keyword-dynamic\",\"ab_test\":\"BASE:0\"}", "pageTitle": "159947433 - Harga & Promo Terbaik | Lazada Indonesia" }, "seoInfo": { "pageTitle": "159947433 - Harga & Promo Terbaik | Lazada Indonesia", "description": "Cari 159947433? Lazada Indonesia adalah tempat berkumpulnya toko lokal dan internasional tepercaya! Nikmati belanja lebih aman dan nyaman dengan produk asli, pengiriman cepat, harga terbaik. Belanja sekarang dengan kebijakan pengembalian mudah dan pembayaran aman!", "robotsContent": "index,follow", "canonicalHref": "/catalog/?q=159947433", "androidDeepLink": "android-app://com.lazada.android/lazada/id/page?url_key=%2Fcatalog%2F%3Fq%3D159947433&utm_campaign=%2Fcatalog%2F%3Fq%3D159947433&utm_medium=organic&utm_source=google_app_indexing&utm_from=search", "nextHref": "/catalog/?page=2&q=159947433", "internalLink": [], "h1": "", "itemListSchema": { "@context": "https://schema.org", "@type": "ItemList", "name": "Best Selling 159947433 Products", "itemListElement": [ { "@type": "ListItem", "position": 1, "item": { "@type": "Product", "name": "Men's Biore Double Scrub Facial Foam Cool Oil Clear 100gr", "image": "https://id-live-01.slatic.net/p/282e6a2fbf26571c3c5eab569e609852.jpg", "url": "//www.lazada.co.id/products/pdp-i159947433.html", "aggregateRating": { "@type": "AggregateRating", "ratingValue": "4.9", "ratingCount": 8378 } } } ] }, "breadcrumbSchema": { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "item": { "@id": "https://www.lazada.co.id", "name": "Home" }, "@type": "ListItem", "position": 1 } ] }, "breadcrumbData": [ { "categoryName": "Home", "categoryLink": "https://www.lazada.co.id" }, { "categoryName": "Search Results" } ], "productSchema": { "@type": "Product", "@context": "https://schema.org", "name": "Men's Biore Double Scrub Facial Foam Cool Oil Clear 100gr", "image": "https://id-live-01.slatic.net/p/282e6a2fbf26571c3c5eab569e609852.jpg", "sku": "BI568HBACHV3ANID", "mpn": "181742191", "aggregateRating": { "@type": "AggregateRating", "ratingValue": "4.9", "ratingCount": 8378 }, "brand": { "@type": "Organization", "name": "Men's Biore" } } } } } ], "tokenUsage": 10 } ``` # Bulk Scraping With Custom Scraper import { Step, Steps } from 'fumadocs-ui/components/steps'; The Bulk API lets you run one custom scraper workflow against a list of URLs in a single submission, instead of sending a separate request for every page. Bulk runs are asynchronous. You submit the workflow and its URLs, receive a submission ID immediately, then poll that ID for progress and results. ## What Is a Custom Scraper? A **custom scraper** is a Playground request that carries its own manual workflow. Instead of pointing at a saved scraper, you send the workflow steps inline with the request, so the same extraction logic can be applied to any URL you pass in. ## Submit a Bulk Run Copy the body request from your custom manual scraper. Paste it into the `"workflow": {}` of the request below. Fill in your API key, the proxy country, and the list of URLs you want to scrape. Send the request and save the `submission_id` from the response. ### Request ```bash title="Submit a bulk run" curl --location 'https://bulk.mrscraper.com/api/gateway/v1/bulk-submissions/run' \ --header 'Content-Type: application/json' \ --header 'x-api-key: {MRSCRAPER_API_KEY}' \ --header 'Cookie: sl-session=P89TLJQZiGoIfOVndtHSPg==' \ --data '{ "engine": "http", "workflow": { "url": "https://books.toscrape.com/", "homePage": true, "proxyCountry" : "us", "workflow": [ { "type": "script", "data": { "name": "first_book", "timeout": 30, "code": "(function() {\n const firstArticle = document.querySelector('\''article.product_pod'\'');\n if (!firstArticle) {\n return { title: null, price: null };\n }\n \n const titleLink = firstArticle.querySelector('\''h3 a'\'');\n const title = titleLink ? (titleLink.getAttribute('\''title'\'') || titleLink.textContent).trim() : null;\n \n const priceEl = firstArticle.querySelector('\''p.price_color'\'');\n const price = priceEl ? priceEl.textContent.trim() : null;\n \n return { title: title || null, price: price || null };\n})();" } } ] }, "request_method": "POST", "proxy_country": "us", "urls": [ "https://books.toscrape.com/", "https://books.toscrape.com/catalogue/page-2.html", "https://books.toscrape.com/catalogue/page-3.html" ] }' ``` | Parameter | Location | Description | | ----------------------- | -------- | -------------------------------------------------------------------------------------- | | `x-api-key` | header | Your MrScraper API key. | | `Cookie` | header | Session cookie for the bulk gateway. Send the default value shown above. | | `engine` | body | Execution engine used for the run. Use `http`. | | `workflow` | body | The scraper configuration applied to every URL in the run. | | `workflow.url` | body | The first page to scrape. | | `workflow.homePage` | body | Set to `true` to visit the website's home page first, then navigate to the target URL. | | `workflow.proxyCountry` | body | Country the workflow accesses the pages from. | | `workflow.workflow` | body | The workflow steps copied from your Playground scraper. | | `request_method` | body | HTTP method used to request each URL. | | `proxy_country` | body | Proxy country for the bulk submission. | | `urls` | body | Array of URLs to run the workflow against. | ### Response The submission is accepted and queued immediately. The run itself continues in the background. ```json title="Submission accepted" { "data": { "submission_id": "01M0HD8JB6YXN2RZM2AC48391M", "engine": "http", "priority": 1, "status": "queued", "total_urls": 3, "succeeded": 0, "failed": 0, "dead": 0, "pending": 3, "percent_complete": 0, "created_at": "2026-08-21T05:38:52.902189Z", "started_at": "2026-08-21T05:38:52.939086Z", "finished_at": null }, "message": "submission running" } ``` The `submission_id` is the only way to retrieve the results of a bulk run. Store it before moving on. ## Check Progress and Results Use the submission ID to poll the run. The same endpoint returns both the current progress counters and any results extracted so far. ### Request ```bash title="Check a bulk submission" curl --location 'https://bulk.mrscraper.com/api/results/v1/bulk-submissions/{SUBMISSION_ID}?limit=25' \ --header 'x-api-key: {MRSCRAPER_API_KEY}' \ --header 'Cookie: sl-session=P89TLJQZiGoIfOVndtHSPg==' ``` | Parameter | Location | Description | | --------------- | -------- | ------------------------------------------------------------------------ | | `x-api-key` | header | Your MrScraper API key. | | `Cookie` | header | Session cookie for the bulk gateway. Send the default value shown above. | | `SUBMISSION_ID` | path | The ID returned when you submitted the bulk run. | | `limit` | query | Number of results to return per page. Maximum is `100`. | ### Response While Running While the run is still in progress, `results` is empty or partially filled and `percent_complete` is below `100`. ```json title="Run still in progress" { "data": { "submission_id": "01M0HDS6DSWSSFTCT1JCDTA0A2", "engine": "http", "status": "queued", "total_urls": 3, "succeeded": 0, "failed": 0, "dead": 0, "pending": 3, "percent_complete": 0, "created_at": "2026-08-21T05:47:57.753168Z", "started_at": "2026-08-21T05:47:57.783051Z", "finished_at": null, "results": [], "next_cursor": null }, "message": "run found" } ``` ### Response When Finished Once every URL has been processed, `percent_complete` reaches `100` and `results` contains one entry per URL. ```json title="Run complete" { "data": { "submission_id": "01M0HD8JB6YXN2RZM2AC48391M", "engine": "http", "status": "queued", "total_urls": 3, "succeeded": 3, "failed": 0, "dead": 0, "pending": 0, "percent_complete": 100, "created_at": "2026-08-21T05:38:52.902189Z", "started_at": "2026-08-21T05:38:52.939086Z", "finished_at": null, "results": [ { "id": "01M0HD8JBQKH2WFKWZ0KKQ15BB:0", "job_id": "01M0HD8JBQKH2WFKWZ0KKQ15BB", "url": "https://books.toscrape.com/catalogue/page-2.html", "http_status": 200, "data": { "first_book": { "price": "£12.84", "title": "In Her Wake" } }, "raw_ref": "2026/08/21/01M0HD8JBQKH2WFKWZ0KKQ15BB-a0.html.zst", "screenshot_ref": null, "engine": "http", "extracted_at": "2026-08-21T05:39:15.841670Z" }, { "id": "01M0HD8JBQAS5DN2ZDWQ7M7MGN:0", "job_id": "01M0HD8JBQAS5DN2ZDWQ7M7MGN", "url": "https://books.toscrape.com/catalogue/page-3.html", "http_status": 200, "data": { "first_book": { "price": "£57.31", "title": "Slow States of Collapse: Poems" } }, "raw_ref": "2026/08/21/01M0HD8JBQAS5DN2ZDWQ7M7MGN-a0.html.zst", "screenshot_ref": null, "engine": "http", "extracted_at": "2026-08-21T05:39:14.414348Z" }, { "id": "01M0HD8JBQJ2TRGA8CNKQC9Z2B:0", "job_id": "01M0HD8JBQJ2TRGA8CNKQC9Z2B", "url": "https://books.toscrape.com/", "http_status": 200, "data": { "first_book": { "price": "£51.77", "title": "A Light in the Attic" } }, "raw_ref": "2026/08/21/01M0HD8JBQJ2TRGA8CNKQC9Z2B-a0.html.zst", "screenshot_ref": null, "engine": "http", "extracted_at": "2026-08-21T05:39:14.089989Z" } ], "next_cursor": null }, "message": "run found" } ``` ### Result Fields | Field | Description | | ------------------------------------------- | ------------------------------------------------------------------- | | `total_urls` | Number of URLs in the submission. | | `succeeded` / `failed` / `dead` / `pending` | Per-URL status counters for the run. | | `percent_complete` | Progress of the run, from `0` to `100`. | | `results[].url` | The URL this result was extracted from. | | `results[].http_status` | HTTP status code returned by the target page. | | `results[].data` | Data extracted by your workflow steps, keyed by step name. | | `results[].raw_ref` | Reference to the stored raw page content. | | `results[].screenshot_ref` | Reference to the stored screenshot, if one was captured. | | `next_cursor` | Cursor for the next page of results. `null` when there are no more. | Results are not returned in the order the URLs were submitted. Match each result to its source page using the `url` field rather than its position in the array. # E-commerce import { Step, Steps } from 'fumadocs-ui/components/steps'; E-commerce websites are a rich source of product data that can be invaluable for market analysis, price comparison, inventory management, and more. In this guide, we'll demonstrate how to use the MrScraper's Scraper to gather product details such as names, prices, ratings, and availability from an e-commerce website. ## Scenario Imagine you're a market analyst looking to gather data on products listed on Tiktok Shop to analyze pricing trends and customer preferences. You want to extract details like product names, prices, ratings, and the number of items sold. ## Create a New AI Listing Scraper 1. Go to the **Scrapers** page by clicking the **triangle icon** on the left sidebar. 2. Click on the **Create AI Scraper +** button. 3. Select the **Listing Scraper** option. 4. In the **Target URL** field, enter the URL of the Tiktok Shop page you want to scrape. For example: `https://shop-id.tokopedia.com/c/phones-electronics/601739?source=ecommerce_mall&enter_method=categories&first_entrance=ecommerce_mall&first_entrance_position=region_redirect_301`. 5. Click on the **Start Scraping** button to start scraping. **Result :** ```json title="Scraping Result" { "mode": "direct", "count": 71, "links": [ "https://www.tiktok.com/shop/pdp/nose-ring-by-oufer-20g-halloween-cz-double-hoop-in-316l-stainless-steel/1729608283143704812", "https://www.tiktok.com/shop/pdp/usb-powered-led-disco-lights-by-unnamed-brand-with-remote-control-strobe-effects/1729483087369179985", "https://www.tiktok.com/shop/pdp/halloween-pumpkin-night-light-mini-led-dimmable-touch-lamp-for-kids/1731541214379741272", "https://www.tiktok.com/shop/pdp/soft-funny-throw-blanket-cozy-fleece-for-home-movie-decor/1731291206388977735", "https://www.tiktok.com/shop/pdp/40oz-sip-sip-hooray-halloween-tumblers-with-lid-straw-durable-glass/1729576207882752721" ], "products": [ { "id": "1729608283143704812", "url": "https://www.tiktok.com/shop/pdp/nose-ring-by-oufer-20g-halloween-cz-double-hoop-in-316l-stainless-steel/1729608283143704812", "sold": 2500, "brand": "Oufer Body Piercing Jewelry", "price": 9.46, "title": "Oufer 20G Halloween CZ Double Hoop Nose Ring 316L Stainless Steel", "rating": 4.6, "section": "TikTok Picks", "currency": "$", "sold_raw": "2.5K", "free_shipping": false, "original_price": 11.82 }, { "id":"1729483087369179985" "url":"https://www.tiktok.com/shop/pdp/usb-powered-led-disco-lights-by-unnamed-brand-with-remote-control-strobe-effects/1729483087369179985" "sold":11000 "brand":"11.0K sold" "price":16.65 "title":"1pc All Aluminum One Body Party Disco Light with Remote Control, USB Powered, for Party Birthday Wedding Holiday Christmas Decoration #TOP PICKS" "rating":4.6 "section":"Oufer 20G Halloween CZ Double Hoop Nose Ring 316L Stainless Steel" "currency":"$" "sold_raw":"11.0K" "free_shipping":false "original_price":34.69 }, { "id":"1731541214379741272" "url":"https://www.tiktok.com/shop/pdp/halloween-pumpkin-night-light-mini-led-dimmable-touch-lamp-for-kids/1731541214379741272" "sold":1900 "brand":"1.9K sold" "price":9.99 "title":"Halloween Pumpkin Night Light, Halloween Decorations Outdoor,Mini LED Pumpkin Lamp with 3 Level Dimmable, Nursery Nightlight for Kids, Silicone Rechargeable Bedside Touch Lamp, for Kids TikTokShopBlackFriday" "rating":4.7 "section":"1pc All Aluminum One Body Party Disco Light with Remote Control, USB Powered, for Party Birthday Wedding Holiday Christmas Decoration #TOP PICKS" "currency":"$" "sold_raw":"1.9K" "free_shipping":false "original_price":NULL }, { "id":"1731291206388977735" "url":"https://www.tiktok.com/shop/pdp/soft-funny-throw-blanket-cozy-fleece-for-home-movie-decor/1731291206388977735" "sold":3000 "brand":"3.0K sold" "price":14.39 "title":"Soft Funny Throw Blanket Horror Icons Flannel Fleece - Cozy Fuzzy Plush Comfy All-Season Sofa Bedroom Dorm Couch Office Car Travel Home Bedding,Comfortable Movie Decor Blanket Men Women Boys Girls Halloween Birthday Fan Gifts" "rating":4.8 "section":"Halloween Pumpkin Night Light, Halloween Decorations Outdoor,Mini LED Pumpkin Lamp with 3 Level Dimmable, Nursery Nightlight for Kids, Silicone Rechargeable Bedside Touch Lamp, for Kids TikTokShopBlackFriday" "currency":"$" "sold_raw":"3.0K" "free_shipping":true "original_price":23.98 }, { "id":"1729576207882752721" "url":"https://www.tiktok.com/shop/pdp/40oz-sip-sip-hooray-halloween-tumblers-with-lid-straw-durable-glass/1729576207882752721" "sold":6100 "brand":"6.1K sold" "price":18.79 "title":"Sip-Sip Hooray! Halloween Tumbler – Cup with Lid & Straw, Spooky Season Iced Coffee Gift, Halloweentok 2024 Reusable Drinkware Insulated Sports Bottle Travel Water Bottle" "rating":4.7 "section":"Soft Funny Throw Blanket Horror Icons Flannel Fleece - Cozy Fuzzy Plush Comfy All-Season Sofa Bedroom Dorm Couch Office Car Travel Home Bedding,Comfortable Movie Decor Blanket Men Women Boys Girls Halloween Birthday Fan Gifts" "currency":"$" "sold_raw":"6.1K" "free_shipping":false "original_price":28.56 } ] } ``` ## Filter Out Next, tell the AI to filter the scraped results based on specific criteria. For this use case, you can use the following prompt: ```txt take only product with rating 4.7 or more. ``` **Result :** ```json title="Filtered Scraping Result" { "mode": "direct", "products": [ { "url": "https://www.tiktok.com/shop/pdp/halloween-pumpkin-night-light-mini-led-dimmable-touch-lamp-for-kids/1731541214379741272", "sold": "1.9K sold", "price": 9.99, "title": "Halloween Pumpkin Night Light, Halloween Decorations Outdoor,Mini LED Pumpkin Lamp with 3 Level Dimmable, Nursery Nightlight for Kids, Silicone Rechargeable Bedside Touch Lamp, for Kids TikTokShopBlackFriday", "rating": 4.7, "section": "TikTok Picks", "currency": "$", "free_shipping": false, "original_price": null }, { "url": "https://www.tiktok.com/shop/pdp/soft-funny-throw-blanket-cozy-fleece-for-home-movie-decor/1731291206388977735", "sold": "3.0K sold", "price": 14.39, "title": "Soft Funny Throw Blanket Horror Icons Flannel Fleece - Cozy Fuzzy Plush Comfy All-Season Sofa Bedroom Dorm Couch Office Car Travel Home Bedding,Comfortable Movie Decor Blanket Men Women Boys Girls Halloween Birthday Fan Gifts", "rating": 4.8, "section": "TikTok Picks", "currency": "$", "free_shipping": true, "original_price": 23.98 }, { "url": "https://www.tiktok.com/shop/pdp/40oz-sip-sip-hooray-halloween-tumblers-with-lid-straw-durable-glass/1729576207882752721", "sold": "6.1K sold", "price": 18.79, "title": "Sip-Sip Hooray! Halloween Tumbler – Cup with Lid & Straw, Spooky Season Iced Coffee Gift, Halloweentok 2024 Reusable Drinkware Insulated Sports Bottle Travel Water Bottle", "rating": 4.7, "section": "TikTok Picks", "currency": "$", "free_shipping": false, "original_price": 28.56 }, { "url": "https://www.tiktok.com/shop/pdp/1729574434882163435", "sold": "4.3K sold", "price": 28.79, "title": "Women's Halloween Themed All Over Hybrid Skull Knit Button Front Cardigan, Casual Drop Shoulder Long Sleeve V Neck Knitwear for Fall, Fashion Ladies' Knit Clothing for Daily Wear", "rating": 4.8, "section": "TikTok Picks", "currency": "$", "free_shipping": false, "original_price": 38.39 }, { "url": "https://www.tiktok.com/shop/pdp/soft-dragon-pattern-throw-blanket-by-brand-for-home-office-travel/1729625176913449761", "sold": "4.7K sold", "price": 9.15, "title": "Dragon Pattern Blanket, 1/2 Counts Soft Throw Blanket, Halloween Decor Warm Napping Blanket for Home Office Travel Camping Dormitory, Slogan Print Blanket", "rating": 4.9, "section": "TikTok Picks", "currency": "$", "free_shipping": false, "original_price": null }, { "url": "https://www.tiktok.com/shop/pdp/diamond-art-roller-2025-dual-pack-for-smooth-pressing-craft-tools/1732001021448392940", "sold": null, "price": 5.3, "title": "1/2PCS Diamond Art Roller, 2025 New Diamond Painting Roller, Diamond Painting Tools, Essential DIY Craft Tool for Rhinestone Embroidery, Press Roller for Smooth Pressing, Crafting and Diamond Painting Projects", "rating": 5, "section": "Savings for you", "currency": "$", "free_shipping": false, "original_price": null }, { "url": "https://www.tiktok.com/shop/pdp/kojic-acid-turmeric-toner-fur-gesicht-und-korper-250ml/1731391357821554923", "sold": "11.8K sold", "price": 23, "title": "[NEW] Kojic Acid Turmeric Toner for Face & Body | Daily Toner for Uneven Skin Tone & Texture | Niacinamide, Glycolic Acid for Clear Glass Skin | Korean Skin Care | 250ml", "rating": 4.7, "section": "Savings for you", "currency": "$", "free_shipping": false, "original_price": 27 }, { "url": "https://www.tiktok.com/shop/pdp/chrleisure-fleece-lined-high-waist-leggings-for-women-1-3pcs/1731711198249390235", "sold": "10.6K sold", "price": 29.99, "title": "【TikTokShopBlackFriday】CHRLEISURE 3PCS Thermal Thick FLeece Lined Leggings for Cold Winter Warm, High Waist Women's Cozy Workout Wear Fleece Pants for Yoga Riding Casual Wear", "rating": 4.7, "section": "Top deals for you", "currency": "$", "free_shipping": true, "original_price": 37.49 }, { "url": "https://www.tiktok.com/shop/pdp/dynasty-sweats-by-youngla-100-cotton-baggy-fit-unique-design/1731156978383884791", "sold": "1.4K sold", "price": 55, "title": "2116 - Dynasty Sweats", "rating": 4.7, "section": "Top deals for you", "currency": "$", "free_shipping": true, "original_price": null }, { "url": "https://www.tiktok.com/shop/pdp/height-adjustable-swivel-chair-by-sweetcrispy-wide-ergonomic-design/1729408738017907174", "sold": "157.1K sold", "price": 66.98, "title": "【Star Furniture】Height Adjustable Criss Cross Chair with No Wheels / with Wheels- Office Wide Swivel Home Office Desk Chairs Reading Chair", "rating": 4.7, "section": "Popular items", "currency": "$", "free_shipping": true, "original_price": 115.99 }, { "url": "https://www.tiktok.com/shop/pdp/power-bank-10000mah-2-pack-by-asperx-with-22-5w-fast-charge-digital-display/1729568096320852893", "sold": "15.8K sold", "price": 18.99, "title": "AsperX 22.5W Fast Charging Power Bank, 2 pack 10000mAh Portable Charger with Digital Display, Battery Pack for Smartphone, Tablet, Perfect Gifts", "rating": 4.7, "section": "Popular items", "currency": "$", "free_shipping": true, "original_price": 79.99 }, { "url": "https://www.tiktok.com/shop/pdp/huloo-sleep-ultra-soft-memory-foam-play-mat-1-3-non-slip-velvet/1730123606468825995", "sold": "7.1K sold", "price": 29.39, "title": "Huloo Sleep Ultra Soft Memory Foam Play Mat for Tummy Time, 1.3\" Thick Non-Slip Crawling Mat, Non-Toxic Velvet Nursery Floor Mat", "rating": 4.7, "section": "Popular items", "currency": "$", "free_shipping": true, "original_price": 109.99 }, { "url": "https://www.tiktok.com/shop/pdp/based-skin-trio-100-pure-hypoclorous-acid-skincare-set/1731806435287666998", "sold": "5.6K sold", "price": 34.5, "title": "BASED Skincare Trio | Daily Skincare Duo + Skin Revival Spray | Facial Cleanser for Breakouts | Daily Moisturizer & Cleanser | For All Skin Types | Non-Toxic, Effective, Safe", "rating": 4.8, "section": "Popular items", "currency": "$", "free_shipping": true, "original_price": 58 }, { "url": "https://www.tiktok.com/shop/pdp/1731997923791180042", "sold": null, "price": 15.57, "title": "Autumn Winter Sexy Elegant Tight Fit Bodycon Dress, One-Step Skirt, Women's Fashion, Comfortable & Stylish, Perfect for Daily Wear（gift）", "rating": 15, "section": "Popular items", "currency": "$", "free_shipping": false, "original_price": 24.57 }, { "url": "https://www.tiktok.com/shop/pdp/energy-sticks-by-bloom-nutrition-10ct-180mg-natural-caffeine-zero-sugar/1730722027877602235", "sold": "20.8K sold", "price": 7.59, "title": "Bloom Nutrition Energy Sticks (10 Sticks) – Focus & Metabolism Support – Natural Caffeine & B Vitamins", "rating": 4.7, "section": "International top sellers", "currency": "$", "free_shipping": false, "original_price": 15.99 }, { "url": "https://www.tiktok.com/shop/pdp/funny-tee-i-hate-my-chungus-life-t-shirt-soft-cotton-fit/1731999740449296622", "sold": null, "price": 7.98, "title": "I hate my chungus life t-shirt (100% cotton), funny tee", "rating": 7, "section": "Best sellers", "currency": "$", "free_shipping": true, "original_price": 20.99 }, { "url": "https://www.tiktok.com/shop/pdp/hooded-plaid-jacket-for-women-by-asvivid-casual-long-sleeve-coat/1729535208059146968", "sold": "110.6K sold", "price": 37.04, "title": "Women's Plaid / Houndstooth Print Button Front Hooded Coat, Casual Longsleeves Pocket Coat for Fall & Winter, Ladies Outerwear for Daily Wear, Womenswear", "rating": 4.7, "section": "Best sellers", "currency": "$", "free_shipping": false, "original_price": 53.68 }, { "url": "https://www.tiktok.com/shop/pdp/christmas-themed-set-festive-practical-design/1731817636155789477", "sold": null, "price": 5.03, "title": "10/20/30Pcs Christmas Elements Four Color Pen Black, Blue, Red, Green Four Color Pen Santa Claus Snowman Christmas Tree Snowflake Deer Various Patterns Christmas Party Small Gifts Gift Fillers Smooth and Constant Oil (Mixed)", "rating": 5, "section": "Best sellers", "currency": "$", "free_shipping": false, "original_price": null } ] } ``` * Some e-commerce sites show different listings by region. Use MrScraper's [proxy](/docs/features/proxy) to view results from other countries, or your own proxy if needed. * Schedule your scraper to run daily or weekly to keep your e-commerce data up to date. ## Use the Extracted Data Now that you have successfully scraped and filtered the e-commerce product data, you can use this information for various purposes such as market analysis, price comparison, inventory management, or even feeding it into your own applications or databases. You can export the data in formats like JSON or CSV for easy integration with other tools. # Job Listing import { Step, Steps } from 'fumadocs-ui/components/steps'; Extracting job listings manually from job boards can be tedious and time-consuming. With **MrScraper’s AI Scraper**, you can automate this process entirely — providing only a URL and a short prompt describing what to extract. In this guide, we’ll walk through a **real-world example** where a recruitment team uses MrScraper to gather and analyze data from a live job board to inform their hiring strategy. ## Scenario Imagine you’re an HR specialist at a growing tech startup planning to hire a **Data Scientist**.\ Before posting your own job, you want to understand: * What skills and qualifications are most in-demand * How other companies describe their Data Scientist roles * Which cities or companies are hiring for similar positions ## Create a New AI Scraper 1. Go to your **MrScraper Dashboard**. 2. Enter the URL: `https://www.glassdoor.com/Job/indonesia-data-scientist-jobs-SRCH_IL.0,9_IN113_KO10,25.htm`. 3. Select **Super Mode** to get more accurate and structured extraction for job listings. ## Filter Out Next, tell the AI exactly what kind of job data you want to capture. The prompt defines both the structure of your results and any filters to apply. For this use case, you can use the following prompt: ```md Extract only job listings with the job title "Data Scientist" that are suitable for fresh graduates or candidates with a minimum of 2 years of experience, and include required skills details. For each listing, return these fields: - Job Title - Company Name - Location - Salary - Job Type - Experience Level - Posted Date - Job Description - Required Skills - Job URL Filter out any listing that does not include required skills information or requires more than 2 years of experience. ``` Here’s a sample output: ```json title="Scraping Result" { "mode": "direct", "results": [ { "Salary": null, "Job URL": "https://www.glassdoor.com/job-listing/it-data-scientist-apotek-k-24-JV_KO0,17_KE18,29.htm?jl=1009644720583", "Job Type": null, "Location": "Indonesia", "Job Title": "IT Data Scientist", "Posted Date": "30d+", "Company Name": "Apotek K 24", "Job Description": "Mengolah dan menganalisis data besar dari berbagai sumber (POS, CRM, marketplace, dll) untuk menemukan pola dan insight bisnis. Mengembangkan model statistik dan machine learning (supervised dan unsupervised) untuk kebutuhan forecasting, recommendation, atau clustering. Melakukan feature engineering dan model evaluation untuk memastikan performa model optimal. Berkolaborasi dengan tim Data Engineering untuk integrasi pipeline data dan deployment model. Berkoordinasi dengan tim BI Analyst untuk penerjemahan hasil analitik ke dalam dashboard dan laporan bisnis. Melakukan eksperimen model (A/B testing, hyperparameter tuning) dan dokumentasi hasilnya. Menjaga data quality, model reproducibility, serta model governance sesuai kebijakan perlindungan data perusahaan. Minimum Qualifications: Menguasai Python (NumPy, Pandas, Scikit-learn, TensorFlow/PyTorch, Matplotlib/Seaborn). Memahami konsep statistik, machine learning, dan data preprocessing. Pengalaman dengan SQL dan query optimization. Familiar dengan tools visualisasi (Power BI, Tableau, atau setara). Pengalaman dalam mengelola version control (Git) dan ML lifecycle tools (MLflow, DVC, dsb) menjadi nilai tambah. Memahami time series forecasting, NLP, atau recommender system menjadi nilai plus. Komunikatif dan kolaboratif lintas tim (Engineering, BI, Business Unit). Kompas Gramedia, through its more than 50 years of history, is striving for one goal: enlightening and empowering Indonesia. To ensure that we are able to serve the nation for another 50 years, we are undergoing a digital transformation; strengthening and expanding our solid business pillars by developing new digital business initiatives. Our vision is to enlight all the people in Indonesia with all the knowledge we have.", "Required Skills": [ "NumPy", "Pandas", "Scikit-learn", "TensorFlow", "PyTorch", "Matplotlib", "Seaborn", "Python", "machine learning", "data preprocessing", "dengan SQL", "query optimization", "Power BI", "Tableau", "Git", "MLflow", "DVC", "time series forecasting", "NLP", "Engineering", "BI", "Business Unit" ], "Experience Level": "2 years" } ] } ``` * Some job sites show different listings by region. Use MrScraper’s [proxy](/docs/features/proxy) to view results from other countries, or your own proxy if needed. * Schedule your scraper to run daily or weekly to keep your job data up to date. ## Use the Extracted Data Once you’ve collected the job listings, you can export and use the data directly from the dashboard in your preferred format. You can use these data for: * **Job posting optimization**: Identify trending keywords or skills that attract top candidates. * **Competitor analysis**: See how similar companies describe their open roles. * **Internal reporting**: Share hiring trends or skill gaps with your HR or analytics team. # Optimizing Cost Running scrapers efficiently helps you get the most value out of MrScraper without unnecessary usage costs.\ This guide explains how to **optimize scraping costs** through smart configuration, scraper mode selection, scheduling, and output management. ## 1. Choose the Right Scraper Mode MrScraper offers two AI Scraper modes that affect both cost and performance: | Mode | Description | When to Use | Cost Impact | | -------------- | -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | ----------- | | **Cheap Mode** | Fast and lightweight extraction. Suitable for simple, structured pages. | Use for lists or data with consistent structure (e.g., product names, prices). | 💲 Lowest | | **Super Mode** | More accurate and intelligent extraction. Ideal for complex layouts or nested content. | Use for unstructured data or multiple data types (e.g., job listings, reviews). | 💲💲 Higher | The cost difference between Cheap Mode and Super Mode depends on the target website. For websites that are easy to access and extract from, both modes may have similar costs. On more complex websites, Super Mode may perform additional attempts and processing to successfully extract data, which can increase token usage and overall cost. ## 2. Limit Extraction Scope The broader your prompt, the more resources are used.\ Narrowing your extraction target saves cost and improves precision. | ❌ *Expensive prompt* | ✅ *Optimized prompt* | | ---------------------------------------------- | ------------------------------------------------------- | | Extract all details from this e-commerce page. | Extract product name, price, and rating from this page. | Focus on the **fields you actually need**. If you only need names and prices, don’t extract full descriptions or metadata. ## 3. Reuse and Recycle Results Each time you run a scraper, you pay for the extraction process.\ Avoid re-running scrapers unnecessarily by reusing existing results. **Example Workflow:** * Store your output in a **Database Connection** or export as **CSV/JSON**. * Only re-run the scraper when the target site changes. * Use **Scheduling** for incremental updates instead of full refreshes. ## 4. Use Scheduling Strategically Scraping too frequently can increase costs without adding value.\ Instead, match your schedule to how often the target data changes. Avoid overlapping schedules. Running multiple scrapers at the same time can spike costs and reduce performance. ## 5. Optimize Proxy Usage While proxies add flexibility, they can slightly increase scraping overhead. * Use **MrScraper’s built-in proxy** instead of external providers to save cost and setup time. * Only enable proxies for websites that require regional access or IP rotation. * For internal or low-risk pages, disable proxy entirely. Disabling proxies may change the content returned by websites that serve region-specific data. If you need accurate local pricing, product availability, or search results, use a proxy or Geo Targeting for the target region. ## 6. Monitor Usage Keep an eye on your scraper usage and performance using the **Analytics** feature.\ Regularly review: * The number of pages or URLs processed * Success rate and data completeness * AI vs. manual scraper distribution This helps you spot overuse early and plan your scraping strategy efficiently. Start small. Run initial tests on limited pages using **Cheap Mode**, verify output accuracy, and scale up gradually. ## 7. Leverage Reruns for Lower Costs If you need to scrape a website repeatedly, use **Rerun** instead of creating a new scraper each time. MrScraper uses caching to optimize repeated scraping tasks. When you rerun an existing scraper, the platform can reuse previously processed information, often reducing both token usage and execution time compared to a new scraper. Reruns are ideal for: * Refreshing previously collected data * Monitoring pages for updates * Scheduled scraping workflows * Recurring extraction from the same website ## 8. Cap Retry Spending with Max Retries and Token Cap A failed scrape that keeps retrying is one of the easiest ways to spend tokens without getting data back. In the [Playground](/docs/getting-started/playground), the **Basic Settings** panel gives you two limits that bound the cost of a single run. Turn on **Retry** first, then set either or both: | Setting | Description | | --------------- | --------------------------------------------------------------------- | | **Max retries** | Maximum number of retry attempts after the initial request. | | **Token cap** | Maximum total tokens the initial request and all its retries can use. | Retrying stops at whichever comes first: the scrape succeeds, it reaches **Max retries**, or the running token total reaches the **Token cap**. Each attempt's cost is rounded up and added to the running total. **How to use them to control cost:** * **Set a token cap when testing an unfamiliar site.** It puts a hard ceiling on a single run, so a page that fails repeatedly can't quietly consume your balance. * **Lower Max retries on sites that fail consistently.** If a page is blocked, the third and fourth attempt usually fail the same way as the first. Fix the cause with **Super** mode or Geo targeting instead of paying for more attempts. * **Leave room for at least one retry.** If the first attempt alone costs more than the cap, the scrape runs once and doesn't retry at all. On pages with variable load times or heavy bandwidth, set the cap to a few times your expected per-run cost. Run a scrape once with no cap to see its typical token cost, then set the cap based on that number rather than guessing. For worked examples of how the cap interacts with retries, see [Playground Token Cap](/docs/getting-started/api-token#playground-token-cap) in Token Plan. # Create and Run a Scraper Programmatically import { Step, Steps } from 'fumadocs-ui/components/steps'; ## Overview This guide shows you how to create an AI scraper in the dashboard and then use it programmatically through the API. This is perfect if you want to set up a scraper once and automate data extraction with simple API calls—no need to configure it every time. ## Prerequisites Before you start, make sure you have: * A **MrScraper API token** for authentication. If you don’t have one yet, follow this [guide to create an API token](/docs/getting-started/api-token). ## Verify Your API Token You can verify your API token and check your account status using the [`/subscription-accounts`](/docs/api/v3/authentication/verify-token): ```bash curl --location 'https://api.app.mrscraper.com/api/v1/subscription-accounts' \ --header 'accept: */*' \ --header 'x-api-token: ' ``` ```javascript const token = "MRSCRAPER_API_TOKEN"; fetch("https://api.app.mrscraper.com/api/v1/subscription-accounts", { method: "GET", headers: { "accept": "*/*", "x-api-token": token } }) .then(res => res.json()) .then(data => console.log(data)) .catch(err => console.error(err)); ``` ```python import requests token = "MRSCRAPER_API_TOKEN" headers = { "accept": "*/*", "x-api-token": token } response = requests.get( "https://api.app.mrscraper.com/api/v1/subscription-accounts", headers=headers ) print(response.json()) ``` **Response (200 OK):** ```json { "message": "Successful operation!", "data": { "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "tokenLimit": 10000, "tokenUsage": 994, "stripeSubscriptionId": "sub_xxxxxxxxxxxxxxxxxx", "userId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "stripeStatus": "true", "quantity": 1, "endsAt": "2030-12-01T02:05:08.400Z", "subscriptionItemId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "user": { "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "createdAt": "2025-11-24T01:59:15.866Z", "createdById": null, "updatedAt": "2026-01-17T08:11:43.270Z", "updatedById": null, "deletedAt": null, "deletedById": null, "name": "John Doe", "email": "johndoe@example.com", "username": null, "latestApiToken": "atk_xxxx....", "gender": null, "phoneNumber": null, "address": null, "birthDate": null, "avatar": "https://api.app.mrscraper.com/images/users/default-avatar.png", "otp": null, "otpExpiredAt": null, "isVerified": true, "stripeCustomerId": "cus_xxxxxxxxx", "googleMail": null, "googleId": null, "s3Bucket": null, "domainTargets": [ "https://books.toscrape.com" ] }, "rateLimit": null, "rateTtl": null, "isEnterprise": true, "cancelAtPeriodEnd": null, "currentPeriodStart": null, "currentPeriodEnd": null, "proxyDollarPerGB": null, "isAutoRenew": false, "updatedAt": "2026-01-18T01:40:21.392Z", "createdAt": "2025-11-24T02:05:08.470Z" } } ``` **Key Response Fields:** **Subscription Account Fields:** | Field | Type | Description | | ---------------------- | ------------ | -------------------------------------------------- | | `id` | string | Unique identifier for your subscription account | | `tokenLimit` | integer | Maximum tokens available in your subscription plan | | `tokenUsage` | integer | Total tokens consumed so far | | `stripeSubscriptionId` | string | Stripe subscription identifier (for billing) | | `userId` | string | Your unique user identifier | | `stripeStatus` | string | Payment status (`"true"` = active) | | `quantity` | integer | Subscription quantity/seats | | `endsAt` | string | Subscription expiration date (ISO 8601 format) | | `subscriptionItemId` | string | Stripe subscription item identifier | | `rateLimit` | integer/null | API rate limit (requests per window) | | `rateTtl` | integer/null | Rate limit time window in seconds | | `isEnterprise` | boolean | Whether account has enterprise features | | `cancelAtPeriodEnd` | boolean/null | If subscription will cancel at period end | | `currentPeriodStart` | string/null | Current billing period start date | | `currentPeriodEnd` | string/null | Current billing period end date | | `proxyDollarPerGB` | number/null | Custom proxy pricing (enterprise only) | | `isAutoRenew` | boolean | Whether subscription auto-renews | **User Object Fields (nested under `data.user`):** | Field | Type | Description | | ------------------ | ------- | -------------------------------------------------- | | `id` | string | Unique user identifier | | `name` | string | User's full name | | `email` | string | User's email address | | `latestApiToken` | string | Your most recently generated API token (truncated) | | `avatar` | string | URL to user's profile picture | | `isVerified` | boolean | Whether email is verified | | `stripeCustomerId` | string | Stripe customer identifier | | `domainTargets` | array | List of domains you've scraped | | `createdAt` | string | Account creation timestamp | | `updatedAt` | string | Last account update timestamp | **Checking Token Balance:** Use `tokenLimit - tokenUsage` to calculate your remaining tokens. Plan your scraping operations accordingly to avoid running out of tokens. ## Collect URLs from Website Collect all relevant URLs from your target website using the AI scraper endpoint. You can select the Agent type you want to use from the example below: ### Initial Scrape using General Agent #### Get All URL Use the [`/scrapers-ai`](/docs/api/v3/scraper/ai-init) endpoint to create a [General agent](/docs/features/ai-scraper/general) scraper to get only the URLs within the provided link. ```bash curl --location 'https://api.app.mrscraper.com/api/v1/scrapers-ai' \ --header 'accept: application/json' \ --header 'x-api-token: ' \ --header 'content-type: application/json' \ --data '{ "url": "https://books.toscrape.com", "agent": "general", "message": "Extract all data and just only get the urls, include patterns https://books.toscrape.com/catalogue and for exclude patterns https://books.toscrape.com/catalogue/category", "proxyCountry": "" }' ``` ```javascript const token = "MRSCRAPER_API_TOKEN"; fetch("https://api.app.mrscraper.com/api/v1/scrapers-ai", { method: "POST", headers: { "accept": "application/json", "x-api-token": token, "content-type": "application/json" }, body: JSON.stringify({ url: "https://books.toscrape.com", agent: "general", message: "Extract all data and just only get the urls, include patterns https://books.toscrape.com/catalogue and for exclude patterns https://books.toscrape.com/catalogue/category", proxyCountry: "" }) }) .then(res => res.json()) .then(data => console.log(data)) .catch(err => console.error(err)); ``` ```python import requests token = "MRSCRAPER_API_TOKEN" url = "https://api.app.mrscraper.com/api/v1/scrapers-ai" headers = { "accept": "application/json", "x-api-token": token, "content-type": "application/json" } payload = { "url": "https://books.toscrape.com", "agent": "general", "message": "Extract all data and just only get the urls, include patterns https://books.toscrape.com/catalogue and for exclude patterns https://books.toscrape.com/catalogue/category", "proxyCountry": "" } response = requests.post(url, headers=headers, json=payload) print(response.json()) ``` #### Get Page Details After collecting URLs, you'll want to extract detailed data from each individual page. When creating a detail scraper, always use a real detail page URL. For example, instead of [https://books.toscrape.com](https://books.toscrape.com), use a specific product page like [https://books.toscrape.com/catalogue/a-light-in-the-attic\_1000/index.html](https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html) (from your URL collection step). Do not use the homepage or listing page—this helps ensure accurate data extraction and prevents setup errors. Create a new scraper configured for detail extraction: ```bash curl --location 'https://api.app.mrscraper.com/api/v1/scrapers-ai' \ --header 'accept: application/json' \ --header 'x-api-token: ' \ --header 'content-type: application/json' \ --data '{ "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "agent": "general", "message": "Extract data : Title, Product Description, Price (Float), Availability, and Number of reviews", "proxyCountry": "" }' ``` ```javascript const token = "MRSCRAPER_API_TOKEN"; fetch("https://api.app.mrscraper.com/api/v1/scrapers-ai", { method: "POST", headers: { "accept": "application/json", "x-api-token": token, "content-type": "application/json" }, body: JSON.stringify({ url: "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", agent: "general", message: "Extract data : Title, Product Description, Price (Float), Availability, and Number of reviews", proxyCountry: "" }) }) .then(res => res.json()) .then(data => console.log(data)) .catch(err => console.error(err)); ``` ```python import requests token = "MRSCRAPER_API_TOKEN" url = "https://api.app.mrscraper.com/api/v1/scrapers-ai" headers = { "accept": "application/json", "x-api-token": token, "content-type": "application/json" } payload = { "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "agent": "general", "message": "Extract data : Title, Product Description, Price (Float), Availability, and Number of reviews", "proxyCountry": "" } response = requests.post(url, headers=headers, json=payload) print(response.json()) ``` **Response (200 OK):** ```json { "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "createdAt": "2026-01-18T08:37:38.124Z", "userId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxx...", "scraperId": "YOUR_DETAIL_SCRAPER_ID", "type": "AI", "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "status": "Finished", "tokenUsage": 5, "data": { "title": "A Light in the Attic", "price": "£51.77", "availability": "In stock (22 available)", "description": "It's hard to imagine a world without A Light in the Attic...", "number_of_reviews": "0" } } ``` **Request Parameters:** | Parameter | Type | Required | Description | | -------------- | ------ | -------- | ------------------------------------------------------------- | | `url` | string | Yes | Target website URL to scrape | | `agent` | string | Yes | AI agent to use (e.g., `general`, `listing`, `detail`, `map`) | | `message` | string | Yes | Natural language instruction describing what to extract | | `proxyCountry` | string | No | Proxy country code for geo-restricted content | You'll need this ID (`YOUR_DETAIL_SCRAPER_ID` in this example) for bulk scraping multiple URLs with the same extraction configuration. ### Initial Scrape with Map Agent Use the [`/scrapers-ai`](/docs/api/v3/scraper/ai-init) endpoint to create a Map agent scraper. ```bash curl --location 'https://api.app.mrscraper.com/api/v1/scrapers-ai' \ --header 'accept: application/json' \ --header 'x-api-token: ' \ --header 'content-type: application/json' \ --data '{ "agent": "map", "url": "https://books.toscrape.com", "maxDepth": 2, "maxPages": 100, "limit": 1000, "includePatterns": "^https:\\/\\/books\\.toscrape\\.com\\/catalogue$", "excludePatterns": "^https:\\/\\/books\\.toscrape\\.com\\/catalogue\\/category$" }' ``` ```javascript const token = "MRSCRAPER_API_TOKEN"; fetch("https://api.app.mrscraper.com/api/v1/scrapers-ai", { method: "POST", headers: { "accept": "application/json", "x-api-token": token, "content-type": "application/json" }, body: JSON.stringify({ agent: "map", url: "https://books.toscrape.com", maxDepth: 2, maxPages: 100, limit: 1000, includePatterns: "^https:\\/\\/books\\.toscrape\\.com\\/catalogue$", excludePatterns: "^https:\\/\\/books\\.toscrape\\.com\\/catalogue\\/category$" }) }) .then(res => res.json()) .then(data => console.log(data)) .catch(err => console.error(err)); ``` ```python import requests token = "MRSCRAPER_API_TOKEN" url = "https://api.app.mrscraper.com/api/v1/scrapers-ai" headers = { "accept": "application/json", "x-api-token": token, "content-type": "application/json" } payload = { "agent": "map", "url": "https://books.toscrape.com", "maxDepth": 2, "maxPages": 100, "limit": 1000, "includePatterns": "^https:\\/\\/books\\.toscrape\\.com\\/catalogue$", "excludePatterns": "^https:\\/\\/books\\.toscrape\\.com\\/catalogue\\/category$" } response = requests.post(url, headers=headers, json=payload) print(response.json()) ``` **Request Parameters:** | Parameter | Type | Required | Description | | ----------------- | ------ | -------- | ------------------------------------------------------- | | `agent` | string | Yes | Must be `"map"` for map agent | | `url` | string | Yes | Target website URL to scrape | | `maxDepth` | number | Yes | Maximum depth to crawl (e.g., `2`) | | `maxPages` | number | Yes | Maximum number of pages to crawl (e.g., `100`) | | `limit` | number | Yes | Maximum number of URLs to collect (e.g., `1000`) | | `includePatterns` | string | No | Regex pattern for URLs to include | | `excludePatterns` | string | No | Regex pattern for URLs to exclude (can be empty string) | **Response (200 OK):** ```json { "message": "Successful operation!", "data": { "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxx...", "createdAt": "2026-01-18T05:01:24.944Z", "userId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxx...", "scraperId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxx...", "type": "AI", "url": "https://books.toscrape.com", "status": "Finished", "error": null, "tokenUsage": 5, "data": { "urls": [ "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html", "https://books.toscrape.com/catalogue/soumission_998/index.html", ... ] } } } ``` You'll need this ID to re-run the scraper later without reconfiguring it. ### Bulk Scrape Multiple URLs Use the [`/scrapers-ai-rerun/bulk`](/docs/api/v3/scraper/ai-rerun-bulk) endpoint to scrape multiple URLs in a single request: The bulk endpoint is available only for the General agent. ```bash curl --location 'https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun/bulk' \ --header 'accept: application/json' \ --header 'x-api-token: ' \ --header 'content-type: application/json' \ --data '{ "scraperId": "YOUR_DETAIL_SCRAPER_ID", "urls": [ "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html", "https://books.toscrape.com/catalogue/soumission_998/index.html", ... ] }' ``` ```javascript const token = "MRSCRAPER_API_TOKEN"; const detailScraperId = "YOUR_DETAIL_SCRAPER_ID"; const urls = [ "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html", "https://books.toscrape.com/catalogue/soumission_998/index.html" ]; fetch("https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun/bulk", { method: "POST", headers: { "accept": "application/json", "x-api-token": token, "content-type": "application/json" }, body: JSON.stringify({ scraperId: detailScraperId, urls: urls }) }) .then(res => res.json()) .then(data => console.log(data)) .catch(err => console.error(err)); ``` ```python import requests token = "MRSCRAPER_API_TOKEN" detail_scraper_id = "YOUR_DETAIL_SCRAPER_ID" urls = [ "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html", "https://books.toscrape.com/catalogue/soumission_998/index.html", ... ] url = "https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun/bulk" headers = { "accept": "application/json", "x-api-token": token, "content-type": "application/json" } payload = { "scraperId": detail_scraper_id, "urls": urls } response = requests.post(url, headers=headers, json=payload) print(response.json()) ``` **Request Parameters:** | Parameter | Type | Required | Description | | ----------- | ------ | -------- | --------------------------------- | | `scraperId` | string | Yes | The detail scraper ID from Step 4 | | `urls` | array | Yes | Array of URLs to scrape | **Response (200 OK):** ```json { "message": "Bulk scraping is Running", "data": { "bulkResultId": "YOUR_BULK_RESULT_ID" } } ``` Bulk scraping runs in the background. Save the `bulkResultId` to retrieve results later. ## Retrieve Results There are two ways to retrieve scraping results: get a specific result by its ID, or get all results for a scraper. ### Get Single Result by Result ID Use the [`/results/{id}`](/docs/api/v3/result/all) endpoint to get the latest single result using a specific `resultId` (the `id` field from the scrape response): ```bash curl --location 'https://api.app.mrscraper.com/api/v1/results/YOUR_RESULT_ID' \ --header 'accept: application/json' \ --header 'x-api-token: ' ``` ```javascript const token = "MRSCRAPER_API_TOKEN"; const resultId = "YOUR_RESULT_ID"; fetch(`https://api.app.mrscraper.com/api/v1/results/${resultId}`, { method: "GET", headers: { "accept": "application/json", "x-api-token": token } }) .then(res => res.json()) .then(data => console.log(data)) .catch(err => console.error(err)); ``` ```python import requests token = "MRSCRAPER_API_TOKEN" result_id = "YOUR_RESULT_ID" url = f"https://api.app.mrscraper.com/api/v1/results/{result_id}" headers = { "accept": "application/json", "x-api-token": token } response = requests.get(url, headers=headers) print(response.json()) ``` **Response (200 OK):** ```json { "message": "Successful operation!", "data": { "createdAt": "2026-01-18T07:28:44.725Z", "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxx...", "userId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxx...", "scraperId": "YOUR_SCRAPER_ID", "type": "Rerun-AI", "url": "https://books.toscrape.com", "status": "Finished", "error": "", "tokenUsage": 4, "data": { "urls": [ "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html", "https://books.toscrape.com/catalogue/soumission_998/index.html", ... ] }, "htmlPath": "results/.../page.html", "screenshotPath": "results/.../page.jpg" } } ``` ### Get All Results by Scraper ID Use the [`/results`](/docs/api/v3/result/detail) endpoint to retrieve all historical results for a specific scraper with pagination: ```bash curl --location --globoff 'https://api.app.mrscraper.com/api/v1/results?filters[scraperId]=YOUR_SCRAPER_ID&page=1&pageSize=10&sort=createdAt&sortOrder=DESC' \ --header 'accept: application/json' \ --header 'x-api-token: ' ``` ```javascript const token = "MRSCRAPER_API_TOKEN"; const scraperId = "YOUR_SCRAPER_ID"; const params = new URLSearchParams({ 'filters[scraperId]': scraperId, 'page': '1', 'pageSize': '10', 'sort': 'createdAt', 'sortOrder': 'DESC' }); fetch(`https://api.app.mrscraper.com/api/v1/results?${params}`, { method: "GET", headers: { "accept": "application/json", "x-api-token": token } }) .then(res => res.json()) .then(data => console.log(data)) .catch(err => console.error(err)); ``` ```python import requests token = "MRSCRAPER_API_TOKEN" scraper_id = "YOUR_SCRAPER_ID" url = "https://api.app.mrscraper.com/api/v1/results" headers = { "accept": "application/json", "x-api-token": token } params = { "filters[scraperId]": scraper_id, "page": 1, "pageSize": 10, "sort": "createdAt", "sortOrder": "DESC" } response = requests.get(url, headers=headers, params=params) print(response.json()) ``` **Query Parameters:** | Parameter | Type | Required | Description | | -------------------- | ------- | -------- | ------------------------------------ | | `filters[scraperId]` | string | Yes | Filter results by scraper ID | | `page` | integer | No | Page number (default: 1) | | `pageSize` | integer | No | Results per page (default: 10) | | `sort` | string | No | Field to sort by (e.g., `createdAt`) | | `sortOrder` | string | No | Sort direction: `ASC` or `DESC` | **Response (200 OK):** ```json { "message": "Successful fetch", "data": [ { "createdAt": "2026-01-18T07:28:44.725Z", "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxx...", "scraperId": "YOUR_SCRAPER_ID", "type": "Rerun-AI", "url": "https://books.toscrape.com", "status": "Finished", "tokenUsage": 4, "data": { "urls": ["..."] } }, { "createdAt": "2026-01-18T07:28:04.740Z", "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxx...", "scraperId": "YOUR_SCRAPER_ID", "type": "AI", "url": "https://books.toscrape.com", "status": "Finished", "tokenUsage": 5, "data": { "urls": ["..."] } } ], "meta": { "page": 1, "pageSize": 10, "total": 2, "totalPage": 1 } } ``` ### Retrieve Bulk Scraping Results Bulk scraping is asynchronous, meaning you need to poll the results endpoint to check the status and retrieve the data when processing is complete. ```bash curl --location 'https://api.app.mrscraper.com/api/v1/results/YOUR_BULK_RESULT_ID' \ --header 'accept: application/json' \ --header 'x-api-token: YOUR_API_TOKEN' ``` ```javascript const token = "YOUR_API_TOKEN"; const bulkResultId = "YOUR_BULK_RESULT_ID"; fetch(`https://api.app.mrscraper.com/api/v1/results/${bulkResultId}`, { method: "GET", headers: { "accept": "application/json", "x-api-token": token } }) .then(res => res.json()) .then(data => console.log(data)) .catch(err => console.error(err)); ``` ```python import requests token = "MRSCRAPER_API_TOKEN" bulk_result_id = "YOUR_BULK_RESULT_ID" url = f"https://api.app.mrscraper.com/api/v1/results/{bulk_result_id}" headers = { "accept": "application/json", "x-api-token": token } response = requests.get(url, headers=headers) print(response.json()) ``` **Response (Status: Running):** ```json { "message": "Successful operation!", "data": { "id": "YOUR_BULK_RESULT_ID", "status": "Running", "data": null } } ``` **Response (Status: Finished):** ```json { "message": "Successful operation!", "data": { "createdAt": "2026-01-18T08:38:23.993Z", "id": "YOUR_BULK_RESULT_ID", "userId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "scraperId": "YOUR_DETAIL_SCRAPER_ID", "type": "Bulk-AI", "status": "Finished", "tokenUsage": 12, "data": { "mergedData": [ { "title": "A Light in the Attic", "price": "£51.77", "availability": "In stock (22 available)", "description": "It's hard to imagine a world without A Light in the Attic...", "product_type": "Books", "upc": "a897fe39b1053632", "price_excl_tax": "£51.77", "price_incl_tax": "£51.77", "tax": "£0.00", "number_of_reviews": "0" }, { "title": "Tipping the Velvet", "price": "£53.74", "availability": "In stock (20 available)", "description": "Erotic and absorbing...Written with starling power...", "product_type": "Books", "upc": "90fa61229261140a", "price_excl_tax": "£53.74", "price_incl_tax": "£53.74", "tax": "£0.00", "number_of_reviews": "0" }, { "title": "Soumission", "price": "£50.10", "availability": "In stock (20 available)", "description": "Dans une France assez proche de la nôtre...", "product_type": "Books", "upc": "6957f44c3847a760", "price_excl_tax": "£50.10", "price_incl_tax": "£50.10", "tax": "£0.00", "number_of_reviews": "0" } ], "urlDetails": [ { "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "status": "Finished", "error": "" }, { "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html", "status": "Finished", "error": "" }, { "url": "https://books.toscrape.com/catalogue/soumission_998/index.html", "status": "Finished", "error": "" } ], "summary": { "totalUrls": 3, "successfulUrls": 3, "failedUrls": 0, "totalTokenUsage": 12 } } } } ``` **Key Response Fields:** | Field | Type | Description | | ----------------- | ------ | -------------------------------------------------- | | `status` | string | Current status: `Running`, `Finished`, or `Failed` | | `data.mergedData` | array | Array of extracted data from all URLs | | `data.urlDetails` | array | Status details for each individual URL | | `data.summary` | object | Summary statistics of the bulk operation | ## Re-run Scraper (Optional) If you want to scrape the same website again with the same configuration, use the [`/scrapers-ai-rerun`](/docs/api/v3/scraper/ai-rerun) endpoint with your saved `scraperId`: ### Re-run General Agent ```bash curl --location 'https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun' \ --header 'accept: application/json' \ --header 'x-api-token: ' \ --header 'content-type: application/json' \ --data '{ "scraperId": "YOUR_SCRAPER_ID", "url": "https://books.toscrape.com" }' ``` ```javascript const token = "MRSCRAPER_API_TOKEN"; const scraperId = "YOUR_SCRAPER_ID"; fetch("https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun", { method: "POST", headers: { "accept": "application/json", "x-api-token": token, "content-type": "application/json" }, body: JSON.stringify({ scraperId: scraperId, url: "https://books.toscrape.com" }) }) .then(res => res.json()) .then(data => console.log(data)) .catch(err => console.error(err)); ``` ```python import requests token = "MRSCRAPER_API_TOKEN" scraper_id = "YOUR_SCRAPER_ID" url = "https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun" headers = { "accept": "application/json", "x-api-token": token, "content-type": "application/json" } payload = { "scraperId": scraper_id, "url": "https://books.toscrape.com" } response = requests.post(url, headers=headers, json=payload) print(response.json()) ``` **Request Parameters:** | Parameter | Type | Required | Description | | ----------- | ------ | -------- | --------------------------------------- | | `scraperId` | string | Yes | The scraper ID from your initial scrape | | `url` | string | Yes | Target URL to scrape | ### Re-run Map Agent When using a scraper with the **map** agent, you need to provide additional parameters, such as `includePatterns` and `excludePatterns`, to control URL filtering. ```bash curl --location 'https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun' \ --header 'accept: application/json' \ --header 'x-api-token: ' \ --header 'content-type: application/json' \ --data '{ "scraperId": "YOUR_SCRAPER_ID", "url": "https://books.toscrape.com", "maxDepth": 1, "maxPages": 50, "limit": 100, "includePatterns": ["https://books.toscrape.com/catalogue"], "excludePatterns": ["https://books.toscrape.com/catalogue/category"] }' ``` ```javascript const token = "MRSCRAPER_API_TOKEN"; const scraperId = "YOUR_SCRAPER_ID"; fetch("https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun", { method: "POST", headers: { "accept": "application/json", "x-api-token": token, "content-type": "application/json" }, body: JSON.stringify({ scraperId: scraperId, url: "https://books.toscrape.com", "scraperId": scraper_id, "url": "https://books.toscrape.com", maxDepth: 1, maxPages: 50, limit: 100, includePatterns: ["https://books.toscrape.com/catalogue"], excludePatterns: ["https://books.toscrape.com/catalogue/category"] }) }) .then(res => res.json()) .then(data => console.log(data)) .catch(err => console.error(err)); ``` ```python import requests token = "MRSCRAPER_API_TOKEN" scraper_id = "YOUR_SCRAPER_ID" url = "https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun" headers = { "accept": "application/json", "x-api-token": token, "content-type": "application/json" } payload = { "scraperId": scraper_id, "url": "https://books.toscrape.com", "maxDepth": 1, "maxPages": 50, "limit": 100, "includePatterns": ["https://books.toscrape.com/catalogue"], "excludePatterns": ["https://books.toscrape.com/catalogue/category"] } response = requests.post(url, headers=headers, json=payload) print(response.json()) ``` **Additional Map Agent Parameters:** | Parameter | Type | Required | Description | | ----------------- | --------- | -------- | ------------------------------------------------------- | | `scraperId` | string | Yes | The scraper ID from your initial scrape | | `url` | string | Yes | Target URL to scrape | | `maxDepth` | number | Yes | Maximum depth to crawl (e.g., `2`) | | `maxPages` | number | Yes | Maximum number of pages to crawl (e.g., `100`) | | `limit` | number | Yes | Maximum number of URLs to collect (e.g., `1000`) | | `includePatterns` | string\[] | No | Regex pattern for URLs to include | | `excludePatterns` | string\[] | No | Regex pattern for URLs to exclude (can be empty string) | > **Note:**\ > When "re-running" with a map agent, you can fine-tune which URLs will be crawled using `includePatterns` and `excludePatterns` while reusing your base scraper logic and configuration. **Response (200 OK):** ```json { "message": "Successful operation!", "data": { "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxx...", "createdAt": "2026-01-18T07:28:44.725Z", "userId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxx..., "scraperId": "YOUR_SCRAPER_ID", "type": "Rerun-AI", "url": "https://books.toscrape.com", "status": "Finished", "error": "", "tokenUsage": 4, "data": { "urls": [ "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html", "https://books.toscrape.com/catalogue/soumission_998/index.html", ... ] }, "htmlPath": "results/.../page.html", "screenshotPath": "results/.../page.jpg" } } ``` * You don’t need to create a new scraper from scratch every time you want to scrape a different—but structurally similar—website or page. If the data format and structure are the same, you can simply *re-run* your existing scraper and provide a new target URL. * This works for every agents including map agent. For map agents, you can even update the URL and the include/exclude patterns before recreating a new scraper, saving you time and tokens. Try reusing your scrapers for maximum efficiency whenever your targets share the same page layout! ## Complete Workflow Example Below is a full example that runs all steps in a single automated workflow: ```python import requests import json import time # Configuration API_TOKEN = "MRSCRAPER_API_TOKEN" BASE_URL = "https://api.app.mrscraper.com/api/v1" headers = { "accept": "application/json", "x-api-token": API_TOKEN, "content-type": "application/json" } def step1_verify_token(): """Step 1: Verify API token and check account status""" print("🔐 Step 1: Verifying API token...") response = requests.get( f"{BASE_URL}/subscription-accounts", headers={"accept": "*/*", "x-api-token": API_TOKEN} ) data = response.json() if response.status_code == 200: print(f"✓ Token valid! Tokens: {data['data']['tokenUsage']}/{data['data']['tokenLimit']}") return True else: print(f"❌ Token verification failed: {data}") return False def step2_collect_urls(target_url, message): """Step 2: Create scraper and collect URLs""" print(f"\n📍 Step 2: Collecting URLs from {target_url}...") payload = { "url": target_url, "agent": "general", "message": message, "proxyCountry": "" } response = requests.post(f"{BASE_URL}/scrapers-ai", headers=headers, json=payload) result = response.json() # cache to a file with open("step2_collect_urls_response.json", "w") as f: json.dump(result, f, indent=2) if "scraperId" in result.get("data", {}): scraper_id = result["data"]["scraperId"] print(f"✓ Scraping finished") print(f"📋 URL Scraper ID: {scraper_id}") return [], scraper_id else: print(f"❌ Failed: {result}") return [], None def step3_rerun_collect_urls(url, scraper_id): """Step 3: Rerun URL collection scraper""" print(f"\n🔄 Step 3: Rerunning URL collection scraper {scraper_id}...") payload = { "scraperId": scraper_id, "url": url } response = requests.post(f"{BASE_URL}/scrapers-ai-rerun/", headers=headers, json=payload) result = response.json() # cache to a file with open("step3_rerun_collect_urls_response.json", "w") as f: json.dump(result, f, indent=2) if result.get("message") == "Successful operation!": urls = result["data"]["data"]["urls"] print(f"✓ Rerun collected {len(urls)} URLs") return urls else: print(f"❌ Failed: {result}") return [] def step4_create_detail_scraper(sample_url, message): """Step 4: Create detail scraper""" print(f"\n🔧 Step 4: Creating detail scraper...") payload = { "url": sample_url, "agent": "general", "message": message, "proxyCountry": "" } response = requests.post(f"{BASE_URL}/scrapers-ai", headers=headers, json=payload) result = response.json() # cache to a file with open("step4_create_detail_scraper_response.json", "w") as f: json.dump(result, f, indent=2) if "scraperId" in result.get("data", {}): scraper_id = result["data"]["scraperId"] print(f"✓ Detail scraper created") print(f"📋 Detail Scraper ID: {scraper_id}") return scraper_id else: print(f"❌ Failed: {result}") return None def step5_bulk_scrape(scraper_id, urls): """Step 5: Bulk scrape all URLs""" print(f"\n🤖 Step 5: Starting bulk scrape for {len(urls)} URLs...") payload = { "scraperId": scraper_id, "urls": urls } response = requests.post(f"{BASE_URL}/scrapers-ai-rerun/bulk", headers=headers, json=payload) result = response.json() # cache to a file with open("step5_bulk_scrape_response.json", "w") as f: json.dump(result, f, indent=2) bulk_result_id = result["data"]["bulkResultId"] print(f"✓ Bulk scraping started") print(f"📋 Bulk Result ID: {bulk_result_id}") return bulk_result_id def step6_get_results(bulk_result_id, poll_interval=10): """Step 6: Poll for results""" print(f"\n⏳ Step 6: Waiting for results (polling every {poll_interval}s)...") while True: response = requests.get( f"{BASE_URL}/results/{bulk_result_id}", headers={"accept": "application/json", "x-api-token": API_TOKEN} ) result = response.json() status = result["data"]["status"] if status == "Finished": print("✅ Scraping completed!") return result["data"]["data"] elif status == "Running": print(f" ⏳ Still running... checking again in {poll_interval}s") time.sleep(poll_interval) else: print(f"❌ Error: {status}") return None # Main workflow if __name__ == "__main__": print("🚀 Starting N2N Programmatic Workflow\n") print("=" * 60) # Step 1: Verify token if not step1_verify_token(): exit(1) # Step 2: Collect URLs url = "https://books.toscrape.com" urls, url_scraper_id = step2_collect_urls( url, "Return array of urls that follow patterns https://books.toscrape.com/catalogue but exclude that has patterns https://books.toscrape.com/catalogue/category" ) # Step 3: Rerun URL collection scraper if url_scraper_id: urls = step3_rerun_collect_urls(url, url_scraper_id) if not urls: exit(1) # Use first 3 URLs for demo urls = urls[:3] print(f" Using {len(urls)} URLs for demo") # Step 4: Create detail scraper detail_scraper_id = step4_create_detail_scraper( urls[0], "Extract all data detail" ) if not detail_scraper_id: exit(1) # Step 5: Bulk scrape bulk_result_id = step5_bulk_scrape(detail_scraper_id, urls) bulk_result_id = "a438e4c2-8489-4473-b405-6b4b5e18ed3f" # Step 6: Get results results = step6_get_results(bulk_result_id) if results: print("\n" + "=" * 60) print("📊 RESULTS SUMMARY") print("=" * 60) print(f"✓ Total items: {len(results['mergedData'])}") print(f"✓ Successful: {results['summary']['successfulUrls']}") print(f"✓ Failed: {results['summary']['failedUrls']}") print(f"✓ Tokens used: {results['summary']['totalTokenUsage']}") print("\n📦 EXTRACTED DATA:") print(json.dumps(results['mergedData'], indent=2)) ``` ```javascript // Configuration const API_TOKEN = "MRSCRAPER_API_TOKEN"; const BASE_URL = "https://api.app.mrscraper.com/api/v1"; const fs = require('fs').promises; const headers = { "accept": "application/json", "x-api-token": API_TOKEN, "content-type": "application/json" }; // Step 1: Verify API token async function step1VerifyToken() { console.log("🔐 Step 1: Verifying API token..."); const response = await fetch(`${BASE_URL}/subscription-accounts`, { headers: { "accept": "*/*", "x-api-token": API_TOKEN } }); const data = await response.json(); if (response.ok) { console.log(`✓ Token valid! Tokens: ${data.data.tokenUsage}/${data.data.tokenLimit}`); return true; } console.log(`❌ Token verification failed:`, data); return false; } // Step 2: Collect URLs async function step2CollectUrls(targetUrl, message) { console.log(`\n📍 Step 2: Collecting URLs from ${targetUrl}...`); const response = await fetch(`${BASE_URL}/scrapers-ai`, { method: "POST", headers, body: JSON.stringify({ url: targetUrl, agent: "general", message, proxyCountry: "" }) }); const result = await response.json(); // Cache to a file await fs.writeFile("step2_collect_urls_response.json", JSON.stringify(result, null, 2)); if (result && 'scraperId' in (result.data || {})) { const scraperId = result.data.scraperId; console.log(`✓ Scraping finished`); console.log(`📋 URL Scraper ID: ${scraperId}`); return { urls: [], scraperId }; } console.log(`❌ Failed:`, result); return { urls: [], scraperId: null }; } // Step 3: Rerun URL collection scraper async function step3RerunCollectUrls(url, scraperId) { console.log(`\n🔄 Step 3: Rerunning URL collection scraper ${scraperId}...`); const response = await fetch(`${BASE_URL}/scrapers-ai-rerun/`, { method: "POST", headers, body: JSON.stringify({ scraperId, url }) }); const result = await response.json(); // Cache to a file await fs.writeFile("step3_rerun_collect_urls_response.json", JSON.stringify(result, null, 2)); if (result.message === "Successful operation!") { const urls = result.data.data.urls; console.log(`✓ Rerun collected ${urls.length} URLs`); return urls; } console.log(`❌ Failed:`, result); return []; } // Step 4: Create detail scraper async function step4CreateDetailScraper(sampleUrl, message) { console.log(`\n🔧 Step 4: Creating detail scraper...`); const response = await fetch(`${BASE_URL}/scrapers-ai`, { method: "POST", headers, body: JSON.stringify({ url: sampleUrl, agent: "general", message, proxyCountry: "" }) }); const result = await response.json(); // Cache to a file await fs.writeFile("step4_create_detail_scraper_response.json", JSON.stringify(result, null, 2)); if (result && 'scraperId' in (result.data || {})) { const scraperId = result.data.scraperId; console.log(`✓ Detail scraper created`); console.log(`📋 Detail Scraper ID: ${scraperId}`); return scraperId; } console.log(`❌ Failed:`, result); return null; } // Step 5: Bulk scrape async function step5BulkScrape(scraperId, urls) { console.log(`\n🤖 Step 5: Starting bulk scrape for ${urls.length} URLs...`); const response = await fetch(`${BASE_URL}/scrapers-ai-rerun/bulk`, { method: "POST", headers, body: JSON.stringify({ scraperId, urls }) }); const result = await response.json(); // Cache to a file await fs.writeFile("step5_bulk_scrape_response.json", JSON.stringify(result, null, 2)); const bulkResultId = result.data.bulkResultId; console.log(`✓ Bulk scraping started`); console.log(`📋 Bulk Result ID: ${bulkResultId}`); return bulkResultId; } // Step 6: Get results with polling async function step6GetResults(bulkResultId, pollInterval = 10000) { console.log(`\n⏳ Step 6: Waiting for results (polling every ${pollInterval/1000}s)...`); while (true) { const response = await fetch(`${BASE_URL}/results/${bulkResultId}`, { headers: { "accept": "application/json", "x-api-token": API_TOKEN } }); const result = await response.json(); const status = result.data.status; if (status === "Finished") { console.log("✅ Scraping completed!"); return result.data.data; } else if (status === "Running") { console.log(` ⏳ Still running... checking again in ${pollInterval/1000}s`); await new Promise(r => setTimeout(r, pollInterval)); } else { console.log(`❌ Error: ${status}`); return null; } } } // Main workflow (async () => { console.log("🚀 Starting N2N Programmatic Workflow\n"); console.log("=".repeat(60)); // Step 1: Verify token if (!await step1VerifyToken()) { process.exit(1); } // Step 2: Collect URLs const url = "https://books.toscrape.com"; const { urls: initialUrls, scraperId: urlScraperId } = await step2CollectUrls( url, "Return array of urls that follow patterns https://books.toscrape.com/catalogue but exclude that has patterns https://books.toscrape.com/catalogue/category" ); // Step 3: Rerun URL collection scraper let urls = []; if (urlScraperId) { urls = await step3RerunCollectUrls(url, urlScraperId); } if (!urls.length) { process.exit(1); } // Use first 3 URLs for demo urls = urls.slice(0, 3); console.log(` Using ${urls.length} URLs for demo`); // Step 4: Create detail scraper const detailScraperId = await step4CreateDetailScraper( urls[0], "Extract all data detail" ); if (!detailScraperId) { process.exit(1); } // Step 5: Bulk scrape let bulkResultId = await step5BulkScrape(detailScraperId, urls); // Optional: Uncomment to use a specific bulk result ID // bulkResultId = "a438e4c2-8489-4473-b405-6b4b5e18ed3f"; // Step 6: Get results const results = await step6GetResults(bulkResultId); if (results) { console.log("\n" + "=".repeat(60)); console.log("📊 RESULTS SUMMARY"); console.log("=".repeat(60)); console.log(`✓ Total items: ${results.mergedData.length}`); console.log(`✓ Successful: ${results.summary.successfulUrls}`); console.log(`✓ Failed: ${results.summary.failedUrls}`); console.log(`✓ Tokens used: ${results.summary.totalTokenUsage}`); console.log("\n📦 EXTRACTED DATA:"); console.log(JSON.stringify(results.mergedData, null, 2)); } })(); ``` # Real Estate import { Step, Steps } from 'fumadocs-ui/components/steps'; Real estate professionals, investors, and analysts can use MrScraper to collect property listings from public real estate websites. Instead of checking hundreds of listings manually, the AI Scraper extracts property details for you using only a URL and a short prompt. Let's walk through a **real-world example** where you can uses MrScraper to collect data from a live property listing site. ## Scenario You want to explore the Michigan housing market and compare prices, home sizes, and features across different cities. Rather than visiting multiple listing sites every day, you can use MrScraper to pull structured data directly from Redfin, making it easier to review options and find the best value. ## Create a New AI Scraper 1. Go to the **Scrapers** page by clicking the **triangle icon** on the left sidebar. 2. Click on the **Create AI Scraper +** button. 3. Select the **Listing Scraper** option. 4. In the **URL** field, enter the URL of the Michigan property listings page: `https://www.redfin.com/state/Michigan`. 5. Click on the **Start Scraping** button to start scraping. **Result :** ```json title="Scraping Result" { "mode": "direct", "properties": [ { "address": "319 Saint Lawrence Blvd, Northville, MI 48168", "url": "/MI/Northville/319-Saint-Lawrence-Blvd-48168/home/98741962", "price": "$510,000", "beds": "3 beds", "baths": "3.5 baths", "sqft": "3,246", "status": "NEW 10 HRS AGO" }, { "address": "20137 W Whipple Dr, Northville, MI 48167", "url": "/MI/Northville/20137-W-Whipple-Dr-48167/home/75119625", "price": "$799,900", "beds": "— beds", "baths": "— baths", "sqft": "—", "status": "NEW 10 HRS AGO" }, { "address": "49776 Parkside Dr, Northville, MI 48168", "url": "/MI/Northville/49776-Parkside-Dr-48168/home/98891697", "price": "$699,900", "beds": "4 beds", "baths": "3.5 baths", "sqft": "4,492", "status": "NEW 14 HRS AGO" }, { "address": "58543 Navarra Dr, South Lyon, MI 48178", "url": "/MI/South-Lyon/58543-Navarra-Dr-48178/home/191274082", "price": "$619,990", "beds": "5 beds", "baths": "3 baths", "sqft": "3,011", "status": "OPEN SAT, 11:30AM TO 2PM" }, { "address": "15943 Morningside, Northville, MI 48168", "url": "/MI/Northville/15943-Morningside-48168/home/98892034", "price": "$273,000", "beds": "2 beds", "baths": "2 baths", "sqft": "1,653" }, { ... } ], "total_count": 63, "links": [ "/MI/Ann-Arbor/1670-W-Ellsworth-Rd-48108/home/99335852", "/MI/Ann-Arbor/1702-Hill-St-48104/home/99308587", "/MI/Ann-Arbor/2134-Overlook-Ct-48103/home/99356641", "/MI/Ann-Arbor/3250-Brackley-Dr-48105/home/143963798", "/MI/Ann-Arbor/3341-Roseford-Blvd-48105/home/143963723", "..." ] } ``` ## Filter Out Tell the AI to filter the scraped results based on specific criteria. For this example, let's filter for smaller, more affordable properties under 2,000 square feet using the following prompt: ```md Get only properties with less than 2000 square feet ``` Here’s a sample output: ```json title="Filtered Scraping Result" { "mode": "direct", "count": 28, "properties": [ { "beds": "2 beds", "link": "/MI/Northville/15943-Morningside-48168/home/98892034", "sqft": 1653, "baths": "2 baths", "price": "$273,000", "address": "15943 Morningside, Northville, MI 48168" }, { "beds": "3 beds", "link": "/MI/Northville/9824-Hathaway-Dr-48167/home/99341712", "sqft": 1000, "baths": "2 baths", "price": "$55,000", "address": "9824 Hathaway Dr, Northville, MI 48167" }, { "beds": "2 beds", "link": "/MI/Northville/42432-Corlina-Dr-48167/home/98883824", "sqft": 1800, "baths": "2.5 baths", "price": "$350,000", "address": "42432 Corlina Dr, Northville, MI 48167" }, { "beds": "2 beds", "link": "/MI/Novi/39928-Crosswinds-48375/home/61133922", "sqft": 1100, "baths": "1.5 baths", "price": "$210,000", "address": "39928 Crosswinds, Novi, MI 48375 5004" }, { "beds": "3 beds", "link": "/MI/Novi/22415-Cranbrooke-Dr-48375/home/199166800", "sqft": 1304, "baths": "1.5 baths", "price": "$249,999", "address": "22415 Cranbrooke Dr, Novi, MI 48375 4502" }, { ... } ] } ``` ## Use the Data The results include smaller homes and condos across cities like Detroit, Ann Arbor, Livonia, Troy, Farmington, Novi, and Northville — making it easy to compare options side by side. # Sentiment Analysis import { Step, Steps } from 'fumadocs-ui/components/steps'; Sentiment analysis is a powerful tool for understanding public opinion, customer feedback, and market trends. ## Scenario You’re a product manager at a laptop company competing with another brand.\ You want to understand what customers think about one of the competitor’s brand gaming laptops available on **Walmart**.\ By analyzing reviews, you can identify what users appreciate (e.g., performance, design) and what frustrates them (e.g., setup issues, reliability). ## Create a New AI Scraper 1. Go to the **Scrapers** page by clicking the **triangle icon** on the left sidebar. 2. Click on the **Create AI Scraper +** button. 3. In the **Target URL** field, enter the URL of the product's review page you want to scrape. For example: `https://www.walmart.com/reviews/product/15843466879?entryPoint=viewAllReviewsTop`. 4. Click on the **Start Scraping** button to start scraping. **Result :** ```json title="Scraping Result" { "reviews": [ { "date": "Jul 28, 2025", "text": "Our son ordered this and he is very happy with how it helps him play video games. Can't say I love that he does that, but he said it really does improve the quality and availability of games for him, and he's pretty good about limiting his own time on devices.", "rating": 5, "reviewer": "WalmartCustomer" }, { "date": "Jul 31, 2025", "text": "It was a gift for my daughter and she absolutely loves it! Loves it so much I bought a second one for my other daughter so they could play their games together. Great computer for gamers.", "rating": 5, "reviewer": "WalmartCustomer" }, { "date": "Sep 5, 2025", "text": "To start, it's an awesome laptop. It fit my budget and can play games and do school work when I need it to. One pay made it very easy to get it without having to spend too much at once (I'm a broke college student). My order also arrived faster than the estimated time which was already pretty quick. Not only that but I got it at a cheap price on clearance.", "rating": 5, "reviewer": "WalmartCustomer" }, { "date": "Sep 4, 2025", "text": "It never fails, really. I always seem to be on the receiving end of crappy products. I JUST got this thing TODAY and I can't even get it to setup! Persistent error messages 'Something went wrong' yadayada. This review is staying all the way bad until I get my money back or some reliable customer support cause this is not okay.", "rating": 1, "reviewer": "Dominic" } ] } ``` ## Analyze Sentiment with AI You can now analyze the sentiment of the extracted user reviews using your preferred AI tool or library. For example, you can use Python's `TextBlob` or `VADER` to classify the sentiment of each review as positive, negative, or neutral. | Sentiment | Example Review | Insight | | --------- | -------------------------------------------------------------------------- | ---------------------------------------------------------- | | Positive | “Great computer for gamers.” | Strong satisfaction with performance | | Positive | “Awesome laptop... plays games and does school work.” | Product fits well for multitasking | | Negative | “I JUST got this thing TODAY and I can’t even get it to setup!” | Setup/reliability issues | | Neutral | “Our son ordered this... he’s happy but I don’t love that he plays games.” | Mixed sentiment; happy performance but concern over gaming | # Flight Price import { Step, Steps } from 'fumadocs-ui/components/steps'; Instead of checking flight prices across multiple websites, you can use the AI Scraper to pull flight details, prices, and schedules automatically using only a URL and a short prompt. Here’s a real-world example of how you can use MrScraper to collect live flight data from a booking site. ## Scenario You’re building a flight price comparison tool or tracking prices for popular routes. In this example, you want to monitor flights from New York City to London so travelers can quickly find the best deals. ## Create a New AI Scraper 1. Go to the **Scrapers** page by clicking the **triangle icon** on the left sidebar. 2. Click on the **Create AI Scraper +** button. 3. Select the **Listing Scraper** option. 4. In the **URL** field, enter the URL of the flight search results page. For this example, we'll search for flights from New York City (NYC) to London (LON): ```txt https://id.trip.com/m/flights/xflightfirst/?triptype=0&classtype=0&classgroupsearch=true&adult=1&child=0&infant=0&from=searchForm&stoptype=0&lowpricesource=searchForm&ddate=2025-11-22&dcitycode=NYC&acitycode=LON&locale=en-ID&curr=IDR&transactionid=20251120174344105 ``` 5. Click on the **Start Scraping** button to start scraping. **Result :** ```json title="Scraping Result" { "mode": "direct", "links": [ "flight_card_4", "flight_card_3", "flight_card_1", "flight_card_2", "flight_card_5" ], "flights": [ { "price": "Rp4,960,870", "airline": "Jetblue Airways", "aircraft": "Airbus A321neo", "duration": "7h 21m", "top_label": "Cheapest nonstop", "arrival_time": "09:30", "price_numeric": "4960870", "departure_time": "21:09", "original_price": "Rp 5,097,750", "arrival_airport": "LHR T2", "next_day_arrival": "+1", "departure_airport": "JFK T5", "low_stock_warning": null, "original_price_numeric": "5097750", "carry_on_baggage_included": false }, { "price": "Rp5,051,840", "airline": "Jetblue Airways", "aircraft": "Airbus A321neo", "duration": "7h 1m", "top_label": null, "arrival_time": "22:00", "price_numeric": "5051840", "departure_time": "09:59", "original_price": "Rp 5,191,430", "arrival_airport": "LHR T2", "next_day_arrival": null, "departure_airport": "JFK T5", "low_stock_warning": null, "original_price_numeric": "5191430", "carry_on_baggage_included": false }, { "price": "Rp7,248,600", "airline": "United Airlines", "aircraft": "Boeing 767-300/300ER", "duration": "7h 20m", "top_label": null, "arrival_time": "06:20", "price_numeric": "7248600", "departure_time": "18:00", "original_price": "Rp 7,453,760", "arrival_airport": "LHR T2", "next_day_arrival": "+1", "departure_airport": "EWR C", "low_stock_warning": "<5 left", "original_price_numeric": "7453760", "carry_on_baggage_included": true }, { "price": "Rp7,287,450", "airline": "American Airlines", "aircraft": "Boeing 777-300ER", "duration": "6h 50m", "top_label": null, "arrival_time": "09:40", "price_numeric": "7287450", "departure_time": "21:50", "original_price": "Rp 7,493,770", "arrival_airport": "LHR T3", "next_day_arrival": "+1", "departure_airport": "JFK T8", "low_stock_warning": null, "original_price_numeric": "7493770", "carry_on_baggage_included": true }, { "price": "Rp7,431,410", "airline": "Austrian Airlines (Codeshare)", "aircraft": "Boeing 767-300/300ER", "duration": "7h 20m", "top_label": null, "arrival_time": "06:20", "price_numeric": "7431410", "departure_time": "18:00", "original_price": "Rp 7,642,020", "arrival_airport": "LHR T2", "next_day_arrival": "+1", "departure_airport": "EWR C", "low_stock_warning": "<5 left", "original_price_numeric": "7642020", "carry_on_baggage_included": true } ], "total_flights": 5 } ``` ## Filter Out Tell the AI to filter your scraped results based on specific rules. In this example, we filter for flights under Rp 6,000,000 using the following prompt: ```md Get only flight with cost less than Rp 6.000.000 ``` Here’s a sample output: ```json title="Filtered Scraping Result" { "mode": "direct", "count": 2, "flights": [ { "price": "Rp4,960,870", "airline": "Jetblue Airways", "aircraft": "Airbus A321neo", "duration": "7h 21m", "arrival_time": "09:30", "price_numeric": 4960870, "departure_time": "21:09", "arrival_airport": "LHR T2", "departure_airport": "JFK T5" }, { "price": "Rp5,051,840", "airline": "Jetblue Airways", "aircraft": "Airbus A321neo", "duration": "7h 1m", "arrival_time": "22:00", "price_numeric": 5051840, "departure_time": "09:59", "arrival_airport": "LHR T2", "departure_airport": "JFK T5" } ] } ``` ### Step 3: Use the Data You can now use this clean flight data for price tracking, comparison tools, or travel apps. Filtering helps budget travelers quickly find the cheapest options. # ClawHub import { Step, Steps } from 'fumadocs-ui/components/steps'; ## ClawHub Integration [ClawHub](https://clawhub.ai/) is a marketplace of skills and plugins for OpenClaw agents. Skills extend an agent's capabilities across research, integrations, and automation, and are installed from a central registry. The [**MrScraper skill**](https://clawhub.ai/ai-mrscraper/skills/mrscraper) lets your agent run AI-powered, unblockable web scraping and data extraction using natural language, backed by the MrScraper API. ## Overview Once installed, your agent can: * **Unblock pages**: Retrieve HTML from sites that block ordinary requests, using a stealth browser and IP rotation. * **Create AI scrapers**: Turn a natural-language instruction into a working scraper. * **Rerun scrapers**: Apply an existing scraper configuration to new URLs, one at a time or in bulk. * **Run manual scrapers**: Execute browser workflows built from selectors. * **Fetch results**: Retrieve scrape results, paginated or by ID. The skill makes direct HTTPS calls to the MrScraper API. There are no bundled scripts and no local installation step beyond installing the skill itself. ## Prerequisites * An **OpenClaw agent** with the `openclaw` CLI available. * A [**MrScraper API token**](https://app.mrscraper.com/api-tokens). ## Installation ### Install the skill ```bash openclaw skills install @ai-mrscraper/mrscraper ``` ### Create a MrScraper API token 1. Open the [MrScraper dashboard](https://app.mrscraper.com/). 2. Click your user profile in the top-right corner. 3. Select **API Tokens**. 4. Click **New Token**. 5. Enter a name and an expiration date. 6. Click **Create** and copy the token. ### Store the token Expose the token to your agent as the `MRSCRAPER_API_TOKEN` environment variable. ```bash export MRSCRAPER_API_TOKEN="your-token-here" ``` Never expose your API token in client-side code, logs, or commits. Store it in an environment variable or a server-side secret manager. ## Authentication Every request the skill makes carries these headers: ```http x-api-token: accept: application/json content-type: application/json ``` MrScraper exposes two base URLs: | API | Base URL | Purpose | | --------- | ------------------------------- | -------------------------------------------- | | Unblocker | `https://api.mrscraper.com` | Fetch HTML from blocked pages | | Platform | `https://api.app.mrscraper.com` | Create, rerun, and read scrapers and results | ## Available Endpoints ### Unblocker Opens blocked pages using a stealth browser and IP rotation, and returns the page content. ```bash curl 'https://api.mrscraper.com?url=https%3A%2F%2Fexample.com&timeout=120&geoCode=SG' \ -H "x-api-token: " ``` | Parameter | Required | Default | Description | | ---------------- | -------- | ------- | ------------------------------------------------ | | `url` | Yes | — | The target URL, URL-encoded. | | `timeout` | No | `60` | Maximum seconds to wait for the page to load. | | `geoCode` | No | — | ISO country code for proxy routing. | | `blockResources` | No | — | Block images, CSS, and fonts for faster loading. | ### Create AI Scraper Creates a scraper from a natural-language instruction and runs it. ```bash curl -X POST "https://api.app.mrscraper.com/api/v1/scrapers-ai" \ -H "x-api-token: " \ -H "Content-Type: application/json" \ -d '{ "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "message": "Extract title, price, stocks, and rating", "agent": "general" }' ``` **Agent types:** | Agent | Description | | --------- | ------------------------------------------------------------------------------------ | | `general` | Standard pages such as product details, articles, and profiles. This is the default. | | `listing` | Product listings, job boards, search results, and other paginated collections. | | `map` | Website crawling that follows links to discover pages. | ### Rerun AI Scraper Applies an existing scraper configuration to a new URL. ```bash curl -X POST "https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun" \ -H "x-api-token: " \ -H "Content-Type: application/json" \ -d '{ "scraperId": "6695bf87-aaa6-46b0-b1ee-88586b222b0b", "url": "https://shopee.sg/" }' ``` ### Bulk Rerun AI Scraper Runs one scraper configuration across multiple URLs in a single request. ```bash curl -X POST "https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun/bulk" \ -H "x-api-token: " \ -H "Content-Type: application/json" \ -d '{ "scraperId": "6695bf87-aaa6-46b0-b1ee-88586b222b0b", "urls": [ "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html" ] }' ``` ### Manual Scraper Rerun Executes a browser workflow built from selectors, for cases where you want explicit extraction rules instead of AI. ```bash curl -X POST "https://api.app.mrscraper.com/api/v1/scrapers-manual-rerun" \ -H "x-api-token: " \ -H "Content-Type: application/json" \ -d '{ "scraperId": "6695bf87-aaa6-46b0-b1ee-88586b222b0b", "url": "https://books.toscrape.com/", "workflow": [ { "type": "extract", "data": { "extraction_type": "text", "name": "book", "selector": "h3 a" } } ] }' ``` ### Fetch Results Returns paginated scrape results. ```bash curl -X GET "https://api.app.mrscraper.com/api/v1/results?sortField=updatedAt&sortOrder=DESC&page=1&pageSize=10" \ -H "x-api-token: " ``` ### Fetch Result by ID Returns the detail of a single result. ```bash curl -X GET "https://api.app.mrscraper.com/api/v1/results/497f6eca-6276-4993-bfeb-53cbbbba6f08" \ -H "x-api-token: " ``` ## Map Agent Configuration When you use the `map` agent, these parameters control how the crawl behaves: | Parameter | Description | | ----------------- | --------------------------------------------------------------------------- | | `maxDepth` | How many levels of links to follow. A value of 1–2 is recommended. | | `maxPages` | Maximum number of pages to crawl. | | `limit` | Maximum number of records to extract. | | `includePatterns` | Regex patterns for URLs to include. Separate multiple patterns with `\|\|`. | | `excludePatterns` | Regex patterns for URLs to exclude. Separate multiple patterns with `\|\|`. | Set `maxDepth` to 1 or 2 and cap `maxPages`. Depth grows the crawl exponentially, and an unbounded crawl burns tokens fast. ## Error Handling The API uses standard HTTP status codes: | Status | Meaning | | ------ | ----------------------------- | | `400` | Invalid request payload. | | `401` | Missing or invalid API token. | | `404` | Resource not found. | | `429` | Rate limit exceeded. | | `500` | Internal error. | Retry `429` responses with exponential backoff. ## Data Scope Data is transmitted only to MrScraper servers. Responses contain the extracted page content and its metadata. Never expose your API token in logs or commits. # Email import { Mail, Cog } from 'lucide-react'; import { Step, Steps } from 'fumadocs-ui/components/steps'; ## What Are Email Connections? Email connections let you receive an email notification when specific events happen in your scraper, such as when a scrape starts, completes, or fails. ## Key Features } title="Instant Notifications"> Get real-time updates on your scraper's status directly in your inbox. } title="Customizable Alerts"> Choose which events you want to be notified about, such as completion or errors. ## Create an Email Connection There are two ways to create an email connection in MrScraper: ### 1. Scraper Select the **scraper** that you want to add an email connection. **Scraper page** -> **Ellipsis** ( **⋮** ) button -> **Email Notifications** -> **Add new Email Connection**. **Email Connection page** -> **+ icon** in the top-right corner. Fill in the email connection details: * **Email Address**: The email address to which the notification should be sent Click **Submit** to save your email connection. ### 2. Settings Open your **MrScraper dashboard**. Click the **gear icon** -> select **Email Connection**. **Email Connection page** -> + icon in the top-right corner. Fill in the email connection details: * **Email Address**: The email address to which the notification should be sent Click **Submit** to save your email connection. ## Connect Email Notification to Scraper After creating the email connection, follow these steps to connect it to your scraper: Go to your chosen **Scraper** page. Click the **Ellipsis** ( **⋮** ) button and click **Email Notifications**. Select the email connection you just created from the dropdown menu. Click **Save Email Notifications**. ## Example Result Once the scraper finished, you will receive an email notification. Depending on the email type you selected, the email will contain either the result in the email body or as an attachment. Here's an example of what the email might look like: # LangChain SDK The [langchain-mrscraper](https://pypi.org/project/langchain-mrscraper/) package exposes the MrScraper API as LangChain `BaseTool` instances, enabling AI agents to fetch rendered HTML, create and rerun scrapers, and retrieve results. The SDK client (`mrscraper-sdk`) is installed as a dependency. This integration provides **tools** that agents can explicitly call—not a document loader. For deterministic "URL → documents" ingestion into vector stores, a document loader is typically a better fit. MrScraper may offer that in a separate package later. ## Requirements * **Python 3.9 or higher** * LangChain ecosystem packages (installed as dependencies) See [langchain-mrscraper on PyPI](https://pypi.org/project/langchain-mrscraper/) for the latest release, version history, and installation details. ## Installation Install the package from PyPI using pip: ```bash pip install -U langchain-mrscraper ``` This installs both the LangChain integration and the underlying MrScraper SDK client. ## Authentication Set your API token from the [MrScraper dashboard](/docs/getting-started/api-token) as an environment variable: ```python import os os.environ["MRSCRAPER_API_TOKEN"] = "MRSCRAPER_API_TOKEN" ``` You can authenticate in three ways: 1. **Environment variable** (recommended): `MRSCRAPER_API_KEY` or `MRSCRAPER_API_TOKEN` 2. **Toolkit parameter**: `MrScraperToolkit(token="...")` 3. **Function parameter**: `load_mrscraper_tools(mrscraper_api_key="...")` For more information on generating API tokens, see the [Generate Token](/docs/getting-started/api-token) guide. ## Quick Start ### Using MrScraperToolkit Load all available tools with the toolkit: ```python from langchain_mrscraper import MrScraperToolkit # Using environment variable tools = MrScraperToolkit().get_tools() # Or with explicit token tools = MrScraperToolkit(token="MRSCRAPER_API_TOKEN").get_tools() ``` ### Using Convenience Loader Alternatively, use the convenience function: ```python from langchain_mrscraper import load_mrscraper_tools tools = load_mrscraper_tools() ``` ### Using with AI Agents ```python from langgraph.prebuilt import create_react_agent from langchain_openai import ChatOpenAI from langchain_mrscraper import MrScraperToolkit # Load MrScraper tools tools = MrScraperToolkit(token="MRSCRAPER_API_TOKEN").get_tools() # Create agent with OpenAI agent = create_react_agent(ChatOpenAI(model="gpt-4o-mini"), tools) # Use the agent response = agent.invoke({ "messages": [{"role": "user", "content": "Fetch the HTML from https://example.com"}] }) ``` This example requires `langgraph` and `langchain-openai` packages: ```bash pip install langgraph langchain-openai ``` ## Core Methods The examples below demonstrate each tool individually. Replace placeholders like scraper and result IDs with values from your dashboard or prior API responses. ### Fetch Raw HTML Fetch fully rendered HTML after JavaScript execution using the stealth browser. ```python from langchain_mrscraper import load_mrscraper_tools fetch_html, = load_mrscraper_tools( token="MRSCRAPER_API_TOKEN", tool_names=["mrscraper_fetch_html"], ) output = fetch_html.invoke( { "url": "https://example.com/page", "timeout": 120, "geo_code": "US", "block_resources": False, } ) print(output) ``` **Parameters:** | Parameter | Type | Required | Default | Description | | ----------------- | ------- | -------- | ------- | ----------------------------------------------- | | `url` | string | Yes | - | Full URL to load | | `timeout` | integer | No | `120` | Maximum seconds to wait for page load | | `geo_code` | string | No | `"US"` | Two-letter proxy country code | | `block_resources` | boolean | No | `False` | Block images, CSS, and fonts for faster loading | ### Create AI Scraper Create and run an AI scraper using natural language instructions. ```python from langchain_mrscraper import load_mrscraper_tools create, = load_mrscraper_tools( token="MRSCRAPER_API_TOKEN", tool_names=["mrscraper_create_scraper"], ) output = create.invoke( { "url": "https://example.com/products", "message": "Extract product names, prices, and ratings.", "agent": "listing", # "general" | "listing" | "map" "proxy_country": "US", "max_depth": 2, "max_pages": 50, "limit": 1000, "include_patterns": "", "exclude_patterns": "", } ) print(output) ``` **Parameters:** | Parameter | Type | Required | Default | Description | | ------------------ | ------- | -------- | ----------- | ------------------------------------------------- | | `url` | string | Yes | - | Target URL to scrape | | `message` | string | Yes | - | Natural language extraction instructions | | `agent` | string | No | `"general"` | Agent type: `"general"`, `"listing"`, or `"map"` | | `proxy_country` | string | No | - | Two-letter country code for proxy | | `max_depth` | integer | No | `2` | For map agent: link depth to follow | | `max_pages` | integer | No | `50` | For map/listing agents: maximum pages to scrape | | `limit` | integer | No | `1000` | For map agent: maximum results to return | | `include_patterns` | string | No | `""` | For map agent: regex patterns for URLs to include | | `exclude_patterns` | string | No | `""` | For map agent: regex patterns for URLs to exclude | **Agent Types:** | Agent | Best For | | ----------- | --------------------------------------------- | | `"general"` | Single pages, product details, articles | | `"listing"` | Product listings, search results, directories | | `"map"` | Site crawling, URL discovery, sitemaps | For detailed information on agents, see [AI Scraper Agents](/docs/features/ai-scraper#ai-scraper-agents). ### Rerun AI Scraper Rerun an existing AI scraper on a new URL without creating a new scraper configuration. ```python from langchain_mrscraper import load_mrscraper_tools rerun_ai, = load_mrscraper_tools( token="MRSCRAPER_API_TOKEN", tool_names=["mrscraper_rerun_ai_scraper"], ) output = rerun_ai.invoke( { "scraper_id": "YOUR_AI_SCRAPER_ID", "url": "https://example.com/another-page", "max_depth": 2, "max_pages": 50, "limit": 1000, "include_patterns": "", "exclude_patterns": "", } ) print(output) ``` **Parameters:** | Parameter | Type | Required | Description | | ------------------ | ------- | -------- | ----------------------------------------------- | | `scraper_id` | string | Yes | ID of the existing AI scraper | | `url` | string | Yes | New URL to scrape | | `max_depth` | integer | No | For map agents: link depth to follow | | `max_pages` | integer | No | For map/listing agents: maximum pages to scrape | | `limit` | integer | No | For map agents: maximum results to return | | `include_patterns` | string | No | For map agents: URL patterns to include | | `exclude_patterns` | string | No | For map agents: URL patterns to exclude | ### Bulk Rerun AI Scraper Run an existing AI scraper on multiple URLs in a single request. ```python from langchain_mrscraper import load_mrscraper_tools bulk_ai, = load_mrscraper_tools( token="MRSCRAPER_API_TOKEN", tool_names=["mrscraper_bulk_rerun_ai_scraper"], ) output = bulk_ai.invoke( { "scraper_id": "YOUR_AI_SCRAPER_ID", "urls": [ "https://example.com/a", "https://example.com/b", ], } ) print(output) ``` **Parameters:** | Parameter | Type | Required | Description | | ------------ | ------ | -------- | ----------------------------- | | `scraper_id` | string | Yes | ID of the existing AI scraper | | `urls` | array | Yes | List of URLs to scrape | This is more efficient than calling `mrscraper_rerun_ai_scraper` once per URL. Use [Get Result by ID](#get-result-by-id) to retrieve individual results. ### Rerun Manual Scraper Rerun an existing manual scraper (created in the MrScraper dashboard) on a new URL. ```python from langchain_mrscraper import load_mrscraper_tools rerun_manual, = load_mrscraper_tools( token="MRSCRAPER_API_TOKEN", tool_names=["mrscraper_rerun_manual_scraper"], ) output = rerun_manual.invoke( { "scraper_id": "YOUR_MANUAL_SCRAPER_ID", "url": "https://example.com/target", } ) print(output) ``` **Parameters:** | Parameter | Type | Required | Description | | ------------ | ------ | -------- | ------------------------------------------- | | `scraper_id` | string | Yes | ID of the manual scraper from the dashboard | | `url` | string | Yes | URL to scrape | ### Bulk Rerun Manual Scraper Run an existing manual scraper on multiple URLs in a single request. ```python from langchain_mrscraper import load_mrscraper_tools bulk_manual, = load_mrscraper_tools( token="MRSCRAPER_API_TOKEN", tool_names=["mrscraper_bulk_rerun_manual_scraper"], ) output = bulk_manual.invoke( { "scraper_id": "YOUR_MANUAL_SCRAPER_ID", "urls": [ "https://example.com/one", "https://example.com/two", ], } ) print(output) ``` **Parameters:** | Parameter | Type | Required | Description | | ------------ | ------ | -------- | ------------------------------------------- | | `scraper_id` | string | Yes | ID of the manual scraper from the dashboard | | `urls` | array | Yes | List of URLs to scrape | This is more efficient than calling `mrscraper_rerun_manual_scraper` once per URL. Use [Get Result by ID](#get-result-by-id) to retrieve individual results. ### Retrieving Results #### Get All Results Retrieve a paginated list of scraping results with filtering and sorting options. ```python from langchain_mrscraper import load_mrscraper_tools list_results, = load_mrscraper_tools( token="MRSCRAPER_API_TOKEN", tool_names=["mrscraper_get_all_results"], ) output = list_results.invoke( { "sort_field": "updatedAt", "sort_order": "DESC", "page_size": 10, "page": 1, "search": None, "date_range_column": None, "start_at": None, "end_at": None, } ) print(output) ``` **Parameters:** | Parameter | Type | Required | Default | Description | | ------------------- | ------- | -------- | ------------- | ----------------------------------- | | `sort_field` | string | No | `"updatedAt"` | Field to sort by | | `sort_order` | string | No | `"DESC"` | Sort direction: `"ASC"` or `"DESC"` | | `page_size` | integer | No | `10` | Number of results per page | | `page` | integer | No | `1` | Page number (starting at 1) | | `search` | string | No | `None` | Optional free-text filter | | `date_range_column` | string | No | `None` | Date field to filter by | | `start_at` | string | No | `None` | Start date (ISO 8601 format) | | `end_at` | string | No | `None` | End date (ISO 8601 format) | **Supported `sort_field` values:** `"createdAt"`, `"updatedAt"`, `"id"`, `"type"`, `"url"`, `"status"`, `"error"`, `"tokenUsage"`, `"runtime"` #### Get Result by ID Fetch a single scraping result using its unique ID. ```python from langchain_mrscraper import load_mrscraper_tools get_one, = load_mrscraper_tools( token="MRSCRAPER_API_TOKEN", tool_names=["mrscraper_get_result_by_id"], ) output = get_one.invoke({"result_id": "YOUR_RESULT_ID"}) print(output) ``` **Parameters:** | Parameter | Type | Required | Description | | ----------- | ------ | -------- | ------------------------------- | | `result_id` | string | Yes | Unique identifier of the result | ## Tool Loading Options You can obtain tools in several ways: **1. Toolkit (Recommended)** ```python from langchain_mrscraper import MrScraperToolkit tools = MrScraperToolkit(token="MRSCRAPER_API_TOKEN").get_tools() ``` **2. Convenience Loader** ```python from langchain_mrscraper import load_mrscraper_tools tools = load_mrscraper_tools(token="MRSCRAPER_API_TOKEN") ``` **3. Selective Tool Loading** ```python from langchain_mrscraper import load_mrscraper_tools tools = load_mrscraper_tools( token="MRSCRAPER_API_TOKEN", tool_names=[ "mrscraper_fetch_html", "mrscraper_get_result_by_id", ], ) ``` **4. Per-Tool Constructors** ```python from langchain_mrscraper.tools import MrScraperFetchHtmlTool fetch_html = MrScraperFetchHtmlTool(token="MRSCRAPER_API_TOKEN") ``` All tools return JSON strings (pretty-printed) from the MrScraper API, making them easy to parse and process in your agent workflows. ## Async Usage All tools implement `_arun` for async callers. In async code, you can use the tool's async entry point when your LangChain version supports it: ```python import asyncio from langchain_mrscraper import load_mrscraper_tools async def async_scrape(): fetch_html, = load_mrscraper_tools( token="MRSCRAPER_API_TOKEN", tool_names=["mrscraper_fetch_html"], ) # Use ainvoke for async execution output = await fetch_html.ainvoke({ "url": "https://example.com", "timeout": 120, }) print(output) asyncio.run(async_scrape()) ``` Check your LangChain version to confirm `ainvoke` is available on `BaseTool`. Most recent versions include async support. ## Best Practices ### Token Management Store your API token securely: ```python import os from pathlib import Path # Load from .env file from dotenv import load_dotenv load_dotenv() token = os.getenv("MRSCRAPER_API_KEY") ``` ### Error Handling Implement error handling in your agent workflows: ```python from langchain_mrscraper import load_mrscraper_tools try: fetch_html, = load_mrscraper_tools( token="MRSCRAPER_API_TOKEN", tool_names=["mrscraper_fetch_html"], ) output = fetch_html.invoke({"url": "https://example.com"}) except Exception as e: print(f"Scraping error: {e}") ``` ### Tool Selection Only load the tools your agent needs: ```python # Good: Only load needed tools tools = load_mrscraper_tools( tool_names=["mrscraper_fetch_html", "mrscraper_create_scraper"] ) # Avoid: Loading all tools when only using a few tools = MrScraperToolkit().get_tools() # All 8 tools ``` # Node.js SDK The official `@mrscraper/sdk` package allows you to fetch HTML, run AI scrapers, rerun existing scrapers, and retrieve results from your Node.js applications. ## Requirements * **Node.js 18 or latest** * **ES Modules enabled** in your `package.json` ```json title="package.json" { "type": "module" } ``` See [@mrscraper/sdk on npm](https://www.npmjs.com/package/@mrscraper/sdk) for the latest release, version history, and installation details. ## Installation Install the SDK using npm: ```bash npm install @mrscraper/sdk ``` ## Authentication Set your API token as an environment variable. You can get your API token from the [MrScraper dashboard](https://app.mrscraper.com). ```bash export MRSCRAPER_API_TOKEN=MRSCRAPER_API_TOKEN ``` Every SDK method accepts an optional `token` parameter if you need to override `MRSCRAPER_API_TOKEN` for specific requests. For more information on generating API tokens, see the [Generate Token](/docs/getting-started/api-token) guide. ## Quick Start ```ts import { fetchHtml, createAiScraper, MrScraperError, } from "@mrscraper/sdk"; try { // 1. Fetch raw HTML from a URL const html = await fetchHtml({ url: "https://example.com", }); console.log(html); // 2. Create an AI scraper (listing agent) const scraper = await createAiScraper({ url: "https://example.com/products", message: "Extract all product names and prices", agent: "listing", }); console.log(scraper); } catch (err) { if (err instanceof MrScraperError) { console.error(`[${err.status ?? "network"}] ${err.message}`); } else { throw err; } } ``` ## Core Methods ### Fetch Raw HTML Fetch rendered HTML or JSON text from a target URL using the MrScraper stealth browser. ```ts import { fetchHtml } from "@mrscraper/sdk"; const html = await fetchHtml({ url: "https://example.com", timeout: 120, geoCode: "US", blockResources: false, // token: "optional_override_token" }); ``` **Parameters:** | Parameter | Type | Required | Default | Description | | ---------------- | ------- | -------- | ------- | ------------------------------------------------ | | `url` | string | Yes | - | URL to fetch | | `timeout` | number | No | `120` | Timeout in seconds (1-600) | | `geoCode` | string | No | `"US"` | Two-letter ISO country code for proxy | | `blockResources` | boolean | No | `false` | Block images, fonts, and CSS for faster fetching | | `token` | string | No | - | Per-request token override | ### Create AI Scraper Create and run a new AI-powered scraper using natural language instructions. **Agent Types:** | Agent | Best For | Additional Parameters | | ----------- | --------------------------------------------- | ------------------------------------------------------------------------- | | `"general"` | Single pages, product details, articles | None | | `"listing"` | Product listings, search results, directories | `max_pages` | | `"map"` | Site crawling, URL discovery, sitemaps | `max_depth`, `max_pages`, `limit`, `include_patterns`, `exclude_patterns` | For detailed information on agents, see [AI Scraper Agents](/docs/features/ai-scraper#ai-scraper-agents). #### General/Listing Agent Example ```ts import { createAiScraper } from "@mrscraper/sdk"; const scraper = await createAiScraper({ url: "https://example.com/products", message: "Extract all product names and prices", agent: "listing", proxyCountry: "US", // token: "optional_override_token" }); ``` #### Map Agent Example ```ts import { createAiScraper } from "@mrscraper/sdk"; const mapResult = await createAiScraper({ url: "https://example.com", agent: "map", maxDepth: 2, maxPages: 50, limit: 1000, includePatterns: "/blog", excludePatterns: "/admin", // token: "optional_override_token" }); ``` **Parameters:** | Parameter | Type | Required | Default | Description | | ----------------- | ------------------------------------- | -------- | ----------- | ---------------------------------------------- | | `url` | string | Yes | - | Target URL to scrape | | `message` | string | No | `""` | Natural language instructions for extraction | | `agent` | `"general"` \| `"listing"` \| `"map"` | No | `"general"` | Agent type to use | | `proxyCountry` | string \| null | No | - | Country code for proxy (e.g., `"US"`, `"GB"`) | | `maxDepth` | number | No | `2` | For map agent: link depth to follow | | `maxPages` | number | No | `50` | For map/listing agent: maximum pages to scrape | | `limit` | number | No | `1000` | For map agent: maximum results to return | | `includePatterns` | string | No | `""` | For map agent: URL patterns to include (regex) | | `excludePatterns` | string | No | `""` | For map agent: URL patterns to exclude (regex) | | `token` | string | No | - | Per-request token override | ### Rerun AI Scraper Rerun an existing AI scraper on a new URL without creating a new scraper configuration. ```ts import { rerunAiScraper } from "@mrscraper/sdk"; const result = await rerunAiScraper({ scraperId: "your-scraper-id", url: "https://example.com/new-page", maxDepth: 2, maxPages: 50, limit: 1000, includePatterns: "", excludePatterns: "", // token: "optional_override_token" }); ``` **Parameters:** | Parameter | Type | Required | Default | Description | | ----------------- | ------ | -------- | ------- | ----------------------------------------------- | | `scraperId` | string | Yes | - | ID of the existing scraper | | `url` | string | Yes | - | New URL to scrape | | `maxDepth` | number | No | `2` | For map agents: link depth to follow | | `maxPages` | number | No | `50` | For map/listing agents: maximum pages to scrape | | `limit` | number | No | `1000` | For map agents: maximum results to return | | `includePatterns` | string | No | `""` | For map agents: URL patterns to include | | `excludePatterns` | string | No | `""` | For map agents: URL patterns to exclude | | `token` | string | No | - | Per-request token override | ### Bulk Rerun AI Scraper Run an existing AI scraper on multiple URLs in a single request. ```ts import { bulkRerunAiScraper } from "@mrscraper/sdk"; const bulkResult = await bulkRerunAiScraper({ scraperId: "your-scraper-id", urls: ["https://example.com/page1", "https://example.com/page2"], // token: "optional_override_token" }); ``` **Parameters:** | Parameter | Type | Required | Description | | ----------- | --------- | -------- | ----------------------------- | | `scraperId` | string | Yes | ID of the existing AI scraper | | `urls` | string\[] | Yes | Array of URLs to scrape | | `token` | string | No | Per-request token override | Bulk operations are more efficient than individual rerun calls. Use this method when scraping multiple URLs to reduce API calls and improve performance. ### Rerun Manual Scraper Rerun an existing manual scraper (created in the MrScraper dashboard) on a new URL. ```ts import { rerunManualScraper } from "@mrscraper/sdk"; const result = await rerunManualScraper({ scraperId: "your-manual-scraper-id", url: "https://example.com/target", // token: "optional_override_token" }); ``` **Parameters:** | Parameter | Type | Required | Description | | ----------- | ------ | -------- | ------------------------------------------- | | `scraperId` | string | Yes | ID of the manual scraper from the dashboard | | `url` | string | Yes | URL to scrape | | `token` | string | No | Per-request token override | ### Bulk Rerun Manual Scraper Run an existing manual scraper on multiple URLs in a single request. ```ts import { bulkRerunManualScraper } from "@mrscraper/sdk"; const bulk = await bulkRerunManualScraper({ scraperId: "your-manual-scraper-id", urls: ["https://example.com/a", "https://example.com/b"], // token: "optional_override_token" }); ``` **Parameters:** | Parameter | Type | Required | Description | | ----------- | --------- | -------- | ------------------------------------------- | | `scraperId` | string | Yes | ID of the manual scraper from the dashboard | | `urls` | string\[] | Yes | Array of URLs to scrape | | `token` | string | No | Per-request token override | ### Retrieving Results #### Get All Results Retrieve a paginated list of scraping results with filtering and sorting options. ```ts import { getAllResults } from "@mrscraper/sdk"; const results = await getAllResults({ sortField: "updatedAt", sortOrder: "DESC", pageSize: 10, page: 1, search: "example.com", dateRangeColumn: "createdAt", startAt: "2024-01-01T00:00:00Z", endAt: "2024-12-31T23:59:59Z", // token: "optional_override_token" }); ``` **Parameters:** | Parameter | Type | Required | Default | Description | | ----------------- | ------------------- | -------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | `sortField` | string | No | `"updatedAt"` | Field to sort by. **Supported values:** `"createdAt"`, `"updatedAt"`, `"id"`, `"type"`, `"url"`, `"status"`, `"error"`, `"tokenUsage"`, `"runtime"` | | `sortOrder` | `"ASC"` \| `"DESC"` | No | `"DESC"` | Sort direction | | `pageSize` | number | No | `10` | Number of results per page | | `page` | number | No | `1` | Page number to retrieve | | `search` | string | No | `""` | Keyword to filter results | | `dateRangeColumn` | string | No | - | Date field to filter by | | `startAt` | string | No | - | Start date for filtering (ISO 8601 format) | | `endAt` | string | No | - | End date for filtering (ISO 8601 format) | | `token` | string | No | - | Per-request token override | #### Get Result by ID Retrieve a single scraping result using its unique ID. ```ts import { getResultById } from "@mrscraper/sdk"; const result = await getResultById({ resultId: "your-result-id", // token: "optional_override_token" }); ``` **Parameters:** | Parameter | Type | Required | Description | | ---------- | ------ | -------- | ------------------------------- | | `resultId` | string | Yes | Unique identifier of the result | | `token` | string | No | Per-request token override | ## Error Handling All SDK methods throw `MrScraperError` for API, network, or timeout failures. ### Exception Properties | Property | Type | Description | | --------- | ------------------- | ------------------------------------------------------------------------ | | `message` | string | Human-readable error message | | `status` | number \| undefined | HTTP status code (401, 429, 500, etc.) or `undefined` for network errors | | `name` | string | Always `"MrScraperError"` | ### Example Error Handling ```ts import { fetchHtml, MrScraperError } from "@mrscraper/sdk"; try { await fetchHtml({ url: "https://example.com" }); } catch (err) { if (err instanceof MrScraperError) { console.error(`Error: ${err.message}`); console.error(`Status: ${err.status ?? "network error"}`); } else { // Handle unexpected errors throw err; } } ``` ### Best Practices * Always wrap SDK calls in try-catch blocks * Check `err instanceof MrScraperError` before accessing error properties * Log errors with appropriate context for debugging * Implement retry logic for network errors (when `status` is `undefined`) * Verify your API token if you encounter status `401` ## Migration from Firecrawl If you're migrating from the [Firecrawl Node SDK](https://docs.firecrawl.dev/sdks/node), use this mapping to find equivalent MrScraper methods. | Firecrawl Method | MrScraper Equivalent | | ------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------- | | [Scrape a URL](https://docs.firecrawl.dev/sdks/node#scraping-a-url) with HTML output | [`fetchHtml`](#fetch-raw-html) | | [Scrape a URL](https://docs.firecrawl.dev/sdks/node#scraping-a-url) with structured JSON | [`createAiScraper`](#create-ai-scraper) with `agent: "general"` or [`rerunAiScraper`](#rerun-ai-scraper) | | [Crawl a website](https://docs.firecrawl.dev/sdks/node#crawling-a-website) or [map URLs](https://docs.firecrawl.dev/sdks/node#mapping-a-website) | [`createAiScraper`](#create-ai-scraper) with `agent: "map"` or [`rerunAiScraper`](#rerun-ai-scraper) with map options | | [Batch scrape](https://docs.firecrawl.dev/sdks/node#batch-scrape) | [`bulkRerunAiScraper`](#bulk-rerun-ai-scraper) or [`bulkRerunManualScraper`](#bulk-rerun-manual-scraper) | | Scrape listing/pagination pages | [`createAiScraper`](#create-ai-scraper) with `agent: "listing"` and `maxPages` parameter | ### Key Differences * **Agent-Based Approach**: MrScraper uses specialized agents (`general`, `listing`, `map`) instead of format-based parameters. * **Natural Language Instructions**: Instead of defining extraction schemas, use the `message` parameter to describe what data you want in plain English. * **Pagination Handling**: Use `agent: "listing"` with `maxPages` to automatically handle paginated content. # Python SDK The [mrscraper-sdk](https://pypi.org/project/mrscraper-sdk/) is a typed Python client for the MrScraper web scraping API. All client methods are **async** and must be used with `asyncio` or another async runtime. ## Requirements * **Python 3.9 or latest** * Basic familiarity with async/await syntax in Python See the [mrscraper-sdk on PyPI](https://pypi.org/project/mrscraper-sdk/) for the latest release, version history, and metadata. ## Installation Install the SDK from PyPI using pip: ```bash pip install mrscraper-sdk ``` Import the client in your Python code: ```python from mrscraper import MrScraper ``` ## Authentication Initialize the client with your MrScraper API token. You can get your API token from the [MrScraper dashboard](https://app.mrscraper.com). ```python from mrscraper import MrScraper client = MrScraper(token="MRSCRAPER_API_TOKEN") ``` Store your API token in environment variables rather than hardcoding it in your source code: ```python import os from mrscraper import MrScraper client = MrScraper(token=os.getenv("MRSCRAPER_API_TOKEN")) ``` For more information on generating API tokens, see the [Generate Token](/docs/getting-started/api-token) guide. ## Core Methods ### Fetch Raw HTML Use `fetch_html` to load a page with the MrScraper stealth browser and return rendered HTML content. ```python import asyncio from mrscraper import MrScraper async def main(): client = MrScraper(token="MRSCRAPER_API_TOKEN") result = await client.fetch_html( "https://example.com/product", geo_code="US", timeout=120, block_resources=False, ) print(result["data"]) # raw HTML string asyncio.run(main()) ``` **Parameters:** | Parameter | Type | Required | Description | | ----------------- | ------- | -------- | -------------------------------------------------------------- | | `url` | string | Yes | Target page URL to fetch | | `timeout` | integer | No | Request timeout in seconds (default: 60) | | `geo_code` | string | No | Geographic/proxy region (e.g., `"US"`, `"GB"`, `"SG"`) | | `block_resources` | boolean | No | When `True`, blocks images, CSS, and fonts to reduce bandwidth | ### Create AI Scraper Create and run an AI-powered scraper using natural language instructions. The response includes scraper metadata and an ID for future reruns. ```python result = await client.create_scraper( url="https://example.com/products", message="Extract all product names, prices, and ratings", agent="listing", # "general" | "listing" | "map" proxy_country="US", ) scraper_id = result["data"]["data"]["id"] print("Scraper ID:", scraper_id) ``` **Parameters:** | Parameter | Type | Required | Description | | --------------- | ------ | -------- | ----------------------------------------------------------------------- | | `url` | string | Yes | Target URL to scrape | | `message` | string | Yes | Natural language instructions for data extraction | | `agent` | string | No | Agent type: `"general"`, `"listing"`, or `"map"` (default: `"general"`) | | `proxy_country` | string | No | Country code for proxy (e.g., `"US"`, `"GB"`) | **Agent Types:** | Agent | Best For | Additional Parameters | | ----------- | --------------------------------------------- | ------------------------------------------------------------------------- | | `"general"` | Single pages, product details, articles | None | | `"listing"` | Product listings, search results, directories | `max_pages` | | `"map"` | Site crawling, URL discovery, sitemaps | `max_depth`, `max_pages`, `limit`, `include_patterns`, `exclude_patterns` | For detailed information on agents, see [AI Scraper Agents](/docs/features/ai-scraper#ai-scraper-agents). ### Rerun AI Scraper Rerun an existing AI scraper on a new URL without creating a new scraper configuration. ```python result = await client.rerun_scraper( scraper_id="scraper_12345", url="https://example.com/products?page=2", ) ``` **Parameters:** | Parameter | Type | Required | Description | | ------------------ | ------- | -------- | -------------------------------------------------- | | `scraper_id` | string | Yes | ID of the existing scraper | | `url` | string | Yes | New URL to scrape | | `max_depth` | integer | No | For map agents: link depth to follow | | `max_pages` | integer | No | For map/listing agents: maximum pages to scrape | | `limit` | integer | No | For map agents: maximum results to return | | `include_patterns` | list | No | For map agents: regex patterns for URLs to include | | `exclude_patterns` | list | No | For map agents: regex patterns for URLs to exclude | ### Bulk Rerun AI Scraper Run an existing AI scraper on multiple URLs in a single request. ```python result = await client.bulk_rerun_ai_scraper( scraper_id="scraper_12345", urls=[ "https://example.com/products/item1", "https://example.com/products/item2", "https://example.com/products/item3", ], ) ``` **Parameters:** | Parameter | Type | Required | Description | | ------------ | ------ | -------- | ---------------------------------- | | `scraper_id` | string | Yes | ID of the existing AI scraper | | `urls` | list | Yes | List of URLs to scrape (non-empty) | Bulk operations are more efficient than individual rerun calls. Use this method when scraping multiple URLs to reduce API calls and improve performance. ### Rerun Manual Scraper Rerun an existing manual scraper (created in the MrScraper dashboard) on a new URL. ```python result = await client.rerun_manual_scraper( scraper_id="manual_scraper_67890", url="https://example.com/products/new-item", ) ``` **Parameters:** | Parameter | Type | Required | Description | | ------------ | ------ | -------- | --------------------------------- | | `scraper_id` | string | Yes | ID of the existing manual scraper | | `url` | string | Yes | New URL to scrape | See [Manual Rerun](/docs/api/v3/scraper/manual-rerun) in the API docs for more details. ### Bulk Rerun Manual Scraper Run an existing manual scraper on multiple URLs in a single request. ```python result = await client.bulk_rerun_manual_scraper( scraper_id="manual_scraper_67890", urls=[ "https://www.example.com/products/item1", "https://www.example.com/products/item2", "https://www.example.com/products/item3", ], ) ``` **Parameters:** | Parameter | Type | Required | Description | | ------------ | ------ | -------- | ------------------------------------------- | | `scraper_id` | string | Yes | ID of the manual scraper from the dashboard | | `urls` | list | Yes | List of URLs to scrape (non-empty) | ### Retrieving Results #### Get All Results Retrieve a paginated list of scraping results with filtering and sorting options. ```python page = await client.get_all_results( sort_field="updatedAt", sort_order="DESC", page_size=20, page=1, search="product", date_range_column="updatedAt", start_at="2026-01-01", end_at="2026-01-31", ) print(page["data"]) ``` **Parameters:** | Parameter | Type | Required | Description | | ------------------- | ------- | -------- | ----------------------------------------------------- | | `sort_field` | string | No | Field to sort by (e.g., `"updatedAt"`, `"createdAt"`) | | `sort_order` | string | No | Sort direction: `"ASC"` or `"DESC"` | | `page_size` | integer | No | Number of results per page (default: 20) | | `page` | integer | No | Page number to retrieve (default: 1) | | `search` | string | No | Keyword to filter results (optional) | | `date_range_column` | string | No | Date field to filter by (e.g., `"updatedAt"`) | | `start_at` | string | No | Start date for filtering (ISO 8601 format) | | `end_at` | string | No | End date for filtering (ISO 8601 format) | * The `search` parameter is optional. Omit it if you don't need keyword filtering and only want date-based or paginated results. * Related REST documentation: [Get All Results in Range](/docs/api/v3/result/all). #### Get Result by ID Retrieve a single scraping result using its unique ID. ```python result = await client.get_result_by_id("result_12345") print(result["data"]) ``` **Parameters:** | Parameter | Type | Required | Description | | ----------- | ------ | -------- | ------------------------------- | | `result_id` | string | Yes | Unique identifier of the result | Related REST documentation: [Result Detail](/docs/api/v3/result/detail) ## Error Handling The SDK raises typed exceptions from `mrscraper.exceptions` for different error scenarios. ### Exception Types | Exception | When It's Raised | | --------------------- | ------------------------------------------------- | | `MrScraperError` | Base class for all SDK errors | | `AuthenticationError` | Invalid or missing API token (HTTP 401) | | `APIError` | Non-success API response (exposes `.status_code`) | | `NetworkError` | Network timeouts or connection failures | ### Example Error Handling ```python from mrscraper.exceptions import AuthenticationError, APIError, NetworkError try: result = await client.fetch_html("https://example.com") except AuthenticationError: print("Authentication failed. Check your API token at https://app.mrscraper.com") except APIError as e: print(f"API error {e.status_code}: {e}") except NetworkError as e: print(f"Network problem: {e}") ``` ### Best Practices * Always wrap SDK calls in try-except blocks * Log errors with appropriate context for debugging * Implement retry logic for `NetworkError` exceptions * Verify your API token if you encounter `AuthenticationError` ## Migration from Firecrawl If you're migrating from the [Firecrawl Python SDK](https://docs.firecrawl.dev/sdks/python), use this mapping to find equivalent MrScraper methods. | Firecrawl Method | MrScraper Equivalent | | --------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | [Scrape a URL](https://docs.firecrawl.dev/sdks/python#scraping-a-url) with HTML output | [`fetch_html`](#fetch-raw-html) | | [Scrape a URL](https://docs.firecrawl.dev/sdks/python#scraping-a-url) with structured JSON | [`create_scraper`](#create-ai-scraper) with `agent="general"` or [`rerun_scraper`](#rerun-ai-scraper) | | [Crawl a website](https://docs.firecrawl.dev/sdks/python#crawl-a-website) or [map URLs](https://docs.firecrawl.dev/sdks/python#map-a-website) | [`create_scraper`](#create-ai-scraper) with `agent="map"` or [`rerun_scraper`](#rerun-ai-scraper) with map options | | [Batch scrape](https://docs.firecrawl.dev/sdks/python#batch-scrape) | [`bulk_rerun_ai_scraper`](#bulk-rerun-ai-scraper) or [`bulk_rerun_manual_scraper`](#bulk-rerun-manual-scraper) | | Scrape listing/pagination pages | [`create_scraper`](#create-ai-scraper) with `agent="listing"` and `max_pages` parameter | ### Key Differences * **Agent-Based Approach**: MrScraper uses specialized agents (`general`, `listing`, `map`) instead of format-based parameters. * **Natural Language Instructions**: Instead of defining extraction schemas, use the `message` parameter to describe what data you want in plain English. * **Pagination Handling**: Use `agent="listing"` with `max_pages` to automatically handle paginated content. # SQL/Database import { SquaresUnite, ChartBar, Cog } from 'lucide-react'; import { Step, Steps } from 'fumadocs-ui/components/steps'; ## What Are SQL Connections? SQL connections allow your MrScraper scrapers to send data directly to an external SQL database. This is useful for storing scraped data in a structured format and enables further analysis and reporting. ## Key Features } title="Centralized Data Storage"> Store all your scraped data in one place for easy access and management. } title="Data Analysis"> Use SQL queries to analyze and manipulate your scraped data. } title="Automation"> Automatically insert scraped data into your database without manual intervention. ## Create an SQL Connection Follow these steps to create an SQL connection in MrScraper: Open your **MrScraper dashboard**. Click the **gear icon** on the left panel and select **SQL Connection**. On the **SQL Connection page**, click the + icon in the top-right corner. Fill in the SQL connection details: * **SQL Driver**: Choose `PostgreSQL`, `MySQL`, `MariaDB`, `SQL Server`, `SQLite`, or `Oracle` * **Host**: Enter the hostname or IP address of your SQL server * **Port**: Enter the port number for your SQL server * **Username**: Enter your SQL database username * **Password**: Enter your SQL database password * **Database Name**: Enter the name of the database you want to connect to * **Default Table**: Enter the name of the table where the data will be inserted Click **Submit** to save your SQL connection. ## Connect SQL Connection to Scraper After creating the SQL connection, follow these steps to connect it to your scraper: Go to your chosen **Scraper** page. Click the **Ellipsis** ( **⋮** ) button and click **Database Integration**. Enable the **Database Integration** toggle. Select the SQL connection you just created from the dropdown menu. Click **Save Database Settings**. ## Example Result Once the scraper finishes, the scraped data will be automatically inserted into the specified table in your SQL database. You can then query and analyze the data using your preferred SQL tools. Here's an example of what the data might look like in a PostgreSQL database: ```sql title="Table containing scraping results" scraping=# select * from books; title --------------------------------------- A Light in the ... Tipping the Velvet Soumission Sharp Objects Sapiens: A Brief History ... The Requiem Red The Dirty Little Secrets ... The Coming Woman: A ... The Boys in the ... The Black Maria Starving Hearts (Triangular Trade ... Shakespeare’s Sonnets Set Me Free Scott Pilgrim’s Precious Little ... Rip it Up and ... Our Band Could Be ... Olio Mesaerion: The Best Science ... Libertarianism for Beginners It’s Only the Himalayas (20 rows) ``` # Webhook import { Webhook, Cog } from 'lucide-react'; import { Step, Steps } from 'fumadocs-ui/components/steps'; ## What Are Webhooks? Webhooks let you automatically send HTTP requests to another API when specific events happen in your scraper — such as when a scrape starts, completes, or fails. ## Webhook Integration Benefits } title="Real-Time Notifications"> Get instant updates on your scraper's status by sending data to your preferred endpoint. } title="Customizable Events"> Choose which events trigger the webhook, allowing for tailored integrations. ## Setting Up a Webhook Follow these steps to create and connect a webhook in MrScraper: Open your **MrScraper dashboard**. Click the **gear icon** on the left panel and select **Webhooks**. On the **Webhooks page**, click the + icon in the top-right corner. Fill in the webhook details: * **Request Type**: Choose `POST` or `PUT` * **Event**: Select when the webhook should trigger (e.g., *Scraping Started*, *Scraping Completed*, *Scraping Failed*) * **Name**: Give your webhook a descriptive name * **URL**: Enter the endpoint of your external webhook app * **Headers**: (Optional) Add any custom headers Click **Submit** to save your webhook. Go to your **Scraper** page, Click**Ellipsis** ( **⋮** ) button and click **Webhooks**. Under **Webhook Settings**, select the webhook you just created and click **Save Webhook Settings**. After connecting your scraper to a webhook, you'll receive a webhook payload each time a scrape runs. For example: ```json title="Webhook Example" { "event": "scrape_completed", "message": "Scraper completed successfully", "scraperId": "53542173-0b75-4f86-b001-15b7dbbc2ee4", "scraperName": "All-in-One E-Commerce Laptop Test Site", "timestamp": "2025-10-29T10:10:29.134Z", "data": { "userId": "0e264d60-fe1c-4033-a323-d2e5de5f7e0b", "scraperId": "53542173-0b75-4f86-b001-15b7dbbc2ee4", "type": "AI", "url": "https://webscraper.io/test-sites/e-commerce/allinone/computers/laptops", "status": "Finished", "error": null, "tokenUsage": 4, "runtime": null, "data": "{\"1\":{\"id\":\"1\",\"source\":\"table\",\"headers\":[\"Price\",\"Product Name\",\"Description\",\"Reviews\"],\"rows\":[[\"$295.99\",\"Asus VivoBook...\",\"Asus VivoBook X441NA-GA190 Chocolate Black, 14\\\", Celeron N3450, 4GB, 128GB SSD, Endless OS, ENG kbd\",\"14 reviews\"],[\"$299\",\"Prestigio Smar...\",\"Prestigio SmartBook 133S Dark Grey, 13.3\\\" FHD IPS, Celeron N3350 1.1GHz, 4GB, 32GB, Windows 10 Pro + Office 365 1 gadam\",\"8 reviews\"],[\"$299\",\"Prestigio Smar...\",\"Prestigio SmartBook 133S Gold, 13.3\\\" FHD IPS, Celeron N3350 1.1GHz, 4GB, 32GB, Windows 10 Pro + Office 365 1 gadam\",\"12 reviews\"],[\"$306.99\",\"Aspire E1-510\",\"15.6\\\", Pentium N3520 2.16GHz, 4GB, 500GB, Linux\",\"2 reviews\"],[\"$321.94\",\"Lenovo V110-15...\",\"Lenovo V110-15IAP, 15.6\\\" HD, Celeron N3350 1.1GHz, 4GB, 128GB SSD, Windows 10 Home\",\"5 reviews\"],[\"$356.49\",\"Lenovo V110-15...\",\"Asus VivoBook 15 X540NA-GQ008T Chocolate Black, 15.6\\\" HD, Pentium N4200, 4GB, 500GB, Windows 10 Home, En kbd\",\"6 reviews\"],[\"$364.46\",\"Hewlett Packar...\",\"Hewlett Packard 250 G6 Dark Ash Silver, 15.6\\\" HD, Celeron N3060 1.6GHz, 4GB, 128GB SSD, DOS\",\"12 reviews\"],[\"$372.7\",\"Acer Aspire 3...\",\"Acer Aspire 3 A315-31 Black, 15.6\\\" HD, Celeron N3350 1.1GHz, 4GB, 128GB SSD, Windows 10 Home\",\"2 reviews\"],[\"$379.94\",\"Acer Aspire A3...\",\"Acer Aspire A315-31-C33J Black 15.6\\\", HD, Celeron N3350, 4GB DDR3L, 128GB, Windows 10 Home, ENG\",\"0 reviews\"],[\"$379.95\",\"Acer Aspire ES...\",\"Acer Aspire ES1-572 Black, 15.6\\\" HD, Core i3-6006U, 4GB, 128GB SSD, Linux\",\"9 reviews\"],[\"$391.48\",\"Acer Aspire 3...\",\"Acer Aspire 3 A315-31 Black, 15.6\\\" HD, Pentium N4200 1.1GHz, 4GB, 128GB SSD, Windows 10 Home\",\"10 reviews\"],[\"$393.88\",\"Acer Aspire 3...\",\"Acer Aspire 3 A315-21, 15.6\\\", AMD A4-9120. 4GB. 128GB SSD, Linux\",\"9 reviews\"],[\"$399\",\"Asus VivoBook...\",\"Asus VivoBook Max X541NA-GQ041 Black Chocolate, 15.6\\\" HD, Pentium N4200 1.1GHz, 4GB, 500GB, Windows 10 Home\",\"4 reviews\"],[\"$399.99\",\"Asus VivoBook...\",\"Asus VivoBook E502NA-GO022T Dark Blue, 15.6\\\" HD, Pentium N4200 1.1GHz, 4GB, 128GB SSD, Windows 10 Home, En/Ru kbd\",\"3 reviews\"],[\"$404.23\",\"Lenovo ThinkPa...\",\"Lenovo ThinkPad E31-80, 13.3\\\" HD, Celeron 3855U 1.6GHz, 4GB, 128GB SSD, Windows 10 Home\",\"12 reviews\"],[\"$408.98\",\"Acer Aspire 3...\",\"Acer Aspire 3 A315-31 Black, 15.6\\\" HD, Pentium N4200 1.1GHz, 4GB, 128GB SSD, Windows 10 Home\",\"10 reviews\"],[\"$409.63\",\"Lenovo V110-15...\",\"Lenovo V110-15ISK, 15.6\\\" HD, Core i3-6006U, 8GB, 128GB SSD, Windows 10 Home\",\"9 reviews\"],[\"$410.46\",\"Acer Aspire ES...\",\"Acer Aspire ES1-732 Black, 17.3\\\" HD+, Celeron, N3350, 4GB, 1TB, Windows 10 Home\",\"14 reviews\"],[\"$410.66\",\"Asus VivoBook...\",\"Asus VivoBook 15 X540NA-GQ026T Chocolate Black, 15.6\\\" HD, Pentium N4200, 4GB, 128GB SSD, Windows 10 Home, En/Ru kbd\",\"4 reviews\"],[\"$416.99\",\"Packard 255 G2\",\"15.6\\\", AMD E2-3800 1.3GHz, 4GB, 500GB, Windows 8.1\",\"2 reviews\"],[\"$433.3\",\"Asus EeeBook R...\",\"Asus EeeBook R416NA-FA014T, 14\\\" FHD, Pentium N4200, 4GB, 128GB eMMC, Windows 10 Home, Eng kbd\",\"1 reviews\"],[\"$436.29\",\"Acer Aspire 3...\",\"Acer Aspire 3 A315-51, 15.6\\\" HD, Core i3-6006U, 4GB, 1TB, Windows 10 Home\",\"1 reviews\"],[\"$436.29\",\"Acer Aspire ES...\",\"Acer Aspire ES1-572 Black, 15.6\\\" HD, Core i3-6006U, 4GB, 500GB, Windows 10 Home\",\"2 reviews\"],[\"$439.73\",\"Acer Extensa 1...\",\"Acer Extensa 15 (2540) Black, 15.6\\\" HD, Core i5-7200U, 4GB, 500GB, Linux\",\"6 reviews\"],[\"$454.62\",\"Acer Aspire ES...\",\"Acer Aspire ES1-572 Black, 15.6\\\" HD, Core i5-7200U, 4GB, 500GB, Linux\",\"9 reviews\"],[\"$454.73\",\"Lenovo V110-15...\",\"Lenovo V110-15ISK, 15.6\\\" HD, Core i3-6006U, 4GB, 128GB SSD, Windows 10 Pro\",\"2 reviews\"],[\"$457.38\",\"Acer Aspire A3...\",\"Acer Aspire A315-51-33TG, Black 15.6\\\" HD, Core i3-7100U, 4GB DDR4, 128GB SSD, Windows 10 Home, ENG\",\"9 reviews\"],[\"$465.95\",\"Lenovo V110-15...\",\"Lenovo V110-15IKB, 15.6\\\" HD, Core i5-7200U, 4GB, 500GB, DOS\",\"7 reviews\"],[\"$468.56\",\"Asus VivoBook...\",\"Asus VivoBook 15 X540UA-DM260 Chocolate Black, 15.6\\\" FHD, Core i3-6006U, 4GB, 256GB SSD, Endless OS, En kbd\",\"1 reviews\"],[\"$469.1\",\"Acer Aspire ES...\",\"Acer Aspire ES1-572 Black, 15.6\\\" HD, Core i3-6006U, 4GB, 128GB SSD, Windows 10 Home\",\"5 reviews\"],[\"$484.23\",\"Lenovo V510 Bl...\",\"Lenovo V510 Black, 14\\\" HD, Core i3-6006U, 4GB, 128GB SSD, Windows 10 Home\",\"8 reviews\"],[\"$485.9\",\"Acer Aspire ES...\",\"Acer Aspire ES1-572 Black, 15.6\\\" HD, Core i5-7200U, 4GB, 128GB SSD, Linux\",\"6 reviews\"],[\"$487.8\",\"Lenovo V510 Bl...\",\"Lenovo V510 Black, 15.6\\\" HD, Core i3-6006U, 4GB, 128GB SSD, Windows 10 Home\",\"9 reviews\"],[\"$488.64\",\"Acer Swift 1 S...\",\"Acer Swift 1 SF113-31 Silver, 13.3\\\" FHD, Pentium N4200, 4GB, 128GB SSD, Windows 10 Home\",\"4 reviews\"],[\"$488.78\",\"Dell Vostro 15\",\"Dell Vostro 15 (3568) Black, 15.6\\\" FHD, Core i5-7200U, 4GB, 128GB SSD, Radeon R5 M420 2GB, Linux\",\"14 reviews\"],[\"$494.71\",\"Acer Aspire 3...\",\"Acer Aspire 3 A315-51 Black, 15.6\\\" FHD, Core i3-7100U, 4GB, 500GB + 128GB SSD, Windows 10 Home\",\"2 reviews\"],[\"$497.17\",\"Dell Vostro 15...\",\"Dell Vostro 15 (3568) Red, 15.6\\\" HD, Core i5-7200U, 4GB, 1TB, Radeon R5 M420 2GB, Linux\",\"9 reviews\"],[\"$498.23\",\"Lenovo V510 Bl...\",\"Lenovo V510 Black, 15.6\\\" FHD, Core i3-7100U, 4GB, 128GB SSD, Windows 10 Pro\",\"5 reviews\"],[\"$520.99\",\"HP 250 G3\",\"15.6\\\", Core i5-4210U, 4GB, 500GB, Windows 8.1\",\"13 reviews\"],[\"$564.98\",\"Acer Spin 5\",\"Acer Spin 5 SP513-51 Black, 13.3\\\" FHD Touch, Core i3-7100U, 4GB, 128GB SSD, Windows 10 Home\",\"0 reviews\"],[\"$577.99\",\"HP 350 G1\",\"15.6\\\", Core i5-4200U, 4GB, 750GB, Radeon HD8670M 2GB, Windows\",\"10 reviews\"],[\"$581.99\",\"Aspire E1-572G\",\"15.6\\\", Core i5-4200U, 8GB, 1TB, Radeon R7 M265, Windows 8.1\",\"2 reviews\"],[\"$609.99\",\"Pavilion\",\"15.6\\\", Core i5-4200U, 6GB, 750GB, Windows 8.1\",\"4 reviews\"],[\"$679\",\"Acer Aspire A5...\",\"Acer Aspire A515-51-5654, Black, 15.6\\\", FHD, Core i5-8250U, 8GB DDR4, 256GB SSD, Windows 10 Home, ENG\",\"9 reviews\"],[\"$679\",\"Dell Inspiron...\",\"Dell Inspiron 15 (5567) Fog Gray, 15.6\\\" FHD, Core i5-7200U, 8GB, 1TB, Radeon R7 M445 4GB, Linux\",\"7 reviews\"],[\"$729\",\"Asus VivoBook...\",\"Asus VivoBook S14 (S406UA-BV041T) Starry Grey, 14\\\", Core i5-8250U, 8GB, 256GB SSD, Windows 10 Home, Eng kbd\",\"2 reviews\"],[\"$739.99\",\"ProBook\",\"14\\\", Core i5 2.6GHz, 4GB, 500GB, Win7 Pro 64bit\",\"8 reviews\"],[\"$745.99\",\"Inspiron 15\",\"Moon Silver, 15.6\\\", Core i7-4510U, 8GB, 1TB, Radeon HD R7 M265 2GB,\",\"12 reviews\"],[\"$799\",\"Asus ROG STRIX...\",\"Asus ROG STRIX GL553VD-DM256, 15.6\\\" FHD, Core i5-7300HQ, 8GB, 1TB, GeForce GTX 1050 2GB, No OS\",\"7 reviews\"],[\"$809\",\"Acer Nitro 5 A...\",\"Acer Nitro 5 AN515-51, 15.6\\\" FHD IPS, Core i5-7300HQ, 8GB, 1TB, GeForce GTX 1050 2GB, Windows 10 Home\",\"0 reviews\"],[\"$899\",\"Asus ROG STRIX...\",\"Asus ROG STRIX GL553VD-DM256, 15.6\\\" FHD, Core i5-7300HQ, 8GB, 1TB, GeForce GTX 1050 2GB, No OS + Windows 10 Home\",\"7 reviews\"],[\"$999\",\"Lenovo ThinkPa...\",\"Lenovo ThinkPad L570, 15.6\\\" FHD, Core i7-7500U, 8GB, 256GB SSD, Windows 10 Pro\",\"11 reviews\"],[\"$1033.99\",\"ThinkPad Yoga\",\"12.5\\\" Touch, Core i3-4010U, 4GB, 500GB + 16GB SSD Cache,\",\"13 reviews\"],[\"$1096.02\",\"Lenovo ThinkPa...\",\"Lenovo ThinkPad L460, 14\\\" FHD IPS, Core i7-6600U, 8GB, 256GB SSD, Windows 10 Pro\",\"14 reviews\"],[\"$1098.42\",\"Dell Inspiron...\",\"Dell Inspiron 15 (7567) Black, 15.6\\\" FHD, Core i5-7300HQ, 8GB, 256GB SSD, GeForce GTX 1050 4GB, Windows 10 Home\",\"7 reviews\"],[\"$1099\",\"MSI GL72M 7RDX\",\"MSI GL72M 7RDX, 17.3\\\" FHD, Core i5-7300HQ, 8GB, 1TB + 128GB SSD, GeForce GTX 1050 2GB, Windows 10 Home\",\"1 reviews\"],[\"$1099\",\"MSI GL72M 7RDX\",\"Asus ROG Strix GL553VD-DM535T, 15.6\\\" FHD, Core i7-7700HQ, 8GB, 1TB + 128GB SSD, GeForce GTX 1050 2GB, Windows 10 Home, Eng kbd\",\"9 reviews\"],[\"$1101.83\",\"Asus ROG Strix...\",\"Apple MacBook Air 13.3\\\", Core i5 1.8GHz, 8GB, 128GB SSD, Intel HD 4000, RUS\",\"4 reviews\"],[\"$1102.66\",\"Dell Latitude...\",\"Dell Latitude 5280, 12.5\\\" HD, Core i5-7300U, 8GB, 256GB SSD, Windows 10 Pro\",\"8 reviews\"],[\"$1110.14\",\"Dell Latitude...\",\"Dell Latitude 5480, 14\\\" FHD, Core i5-7300U, 8GB, 500GB, Linux + Windows 10 Home\",\"4 reviews\"],[\"$1112.91\",\"Lenovo Legion...\",\"Lenovo Legion Y520-15IKBM, Black, 15.6\\\" FHD IPS, Core i5-7300HQ, 8 GB, 128GB SSD + 2 TB HDD, NVIDIA GeForce GTX 1060 6 GB, FreeDOS + Windows 10 Home\",\"1 reviews\"],[\"$1114.55\",\"Toshiba Porteg...\",\"Toshiba Portege Z30-C-16J Grey, 13.3\\\" FHD, Core i5-6200U, 8GB, 256GB SSD, Windows 10 Pro\",\"0 reviews\"],[\"$1115.87\",\"Acer Predator...\",\"Acer Predator Helios 300 (PH317-51), 17.3\\\" FHD IPS, Core i5-7300HQ, 8GB, 1TB + 128GB SSD, GeForce GTX 1050Ti 4GB, Windows 10 Home\",\"1 reviews\"],[\"$1115.87\",\"Acer Aspire 7...\",\"Acer Aspire 7 A715-71G, 15.6\\\" FHD IPS, Core i7-7700HQ, 8GB, 128GB SSD + 1TB HDD, GTX 1050 Ti 4GB, Windows 10 Home\",\"4 reviews\"],[\"$1116.99\",\"Dell Inspiron...\",\"Dell Inspiron 17 2in1 (7779) Silver, 17.3\\\" FHD Touch, Core i5-7200U, 12GB, 1TB, GeForce GT940MX 2GB, Windows 10 Home\",\"10 reviews\"],[\"$1117.99\",\"Dell Latitude...\",\"Dell Latitude 5480, 14\\\" FHD, Core i5-7300U, 8GB, 256GB SSD, Windows 10 Pro\",\"14 reviews\"],[\"$1118.99\",\"Lenovo Legion...\",\"Lenovo Legion Y520, 15.6\\\" FHD, Core i7-7700HQ, 8GB, 128 GB SSD + 1TB HDD, GTX 1050 4GB, Windows 10 Home\",\"13 reviews\"],[\"$1119.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702VM-GC146T, 17.3\\\" FHD, Core i7-7700HQ, 8GB, 1TB + 128GB SSD, GeForce GTX 1060 3GB, Windows 10 Home, Eng kbd\",\"10 reviews\"],[\"$1120.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC154T, 17.3\\\" FHD, Ryzen 7 1700, 16GB, 256GB + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"7 reviews\"],[\"$1121.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC209T, 17.3\\\" FHD IPS, Ryzen 7 1700, 16GB, 256GB SSD + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"8 reviews\"],[\"$1122.99\",\"Asus ROG Strix...\",\"Asus ROG Strix SCAR Edition GL503VM-ED115T, 15.6\\\" FHD 120Hz, Core i7-7700HQ, 16GB, 256GB SSD + 1TB SSHD, GeForce GTX 1060 6GB, Windows 10 Home\",\"8 reviews\"],[\"$1123.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC154T, 17.3\\\" FHD, Ryzen 7 1700, 16GB, 256GB + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"7 reviews\"],[\"$1124.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC209T, 17.3\\\" FHD IPS, Ryzen 7 1700, 16GB, 256GB SSD + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"8 reviews\"],[\"$1125.99\",\"Asus ROG Strix...\",\"Asus ROG Strix SCAR Edition GL503VM-ED115T, 15.6\\\" FHD 120Hz, Core i7-7700HQ, 16GB, 256GB SSD + 1TB SSHD, GeForce GTX 1060 6GB, Windows 10 Home\",\"8 reviews\"],[\"$1126.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC154T, 17.3\\\" FHD, Ryzen 7 1700, 16GB, 256GB + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"7 reviews\"],[\"$1127.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC209T, 17.3\\\" FHD IPS, Ryzen 7 1700, 16GB, 256GB SSD + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"8 reviews\"],[\"$1128.99\",\"Asus ROG Strix...\",\"Asus ROG Strix SCAR Edition GL503VM-ED115T, 15.6\\\" FHD 120Hz, Core i7-7700HQ, 16GB, 256GB SSD + 1TB SSHD, GeForce GTX 1060 6GB, Windows 10 Home\",\"8 reviews\"],[\"$1129.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC154T, 17.3\\\" FHD, Ryzen 7 1700, 16GB, 256GB + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"7 reviews\"],[\"$1130.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC209T, 17.3\\\" FHD IPS, Ryzen 7 1700, 16GB, 256GB SSD + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"8 reviews\"],[\"$1131.99\",\"Asus ROG Strix...\",\"Asus ROG Strix SCAR Edition GL503VM-ED115T, 15.6\\\" FHD 120Hz, Core i7-7700HQ, 16GB, 256GB SSD + 1TB SSHD, GeForce GTX 1060 6GB, Windows 10 Home\",\"8 reviews\"],[\"$1132.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC154T, 17.3\\\" FHD, Ryzen 7 1700, 16GB, 256GB + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"7 reviews\"],[\"$1133.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC209T, 17.3\\\" FHD IPS, Ryzen 7 1700, 16GB, 256GB SSD + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"8 reviews\"],[\"$1134.99\",\"Asus ROG Strix...\",\"Asus ROG Strix SCAR Edition GL503VM-ED115T, 15.6\\\" FHD 120Hz, Core i7-7700HQ, 16GB, 256GB SSD + 1TB SSHD, GeForce GTX 1060 6GB, Windows 10 Home\",\"8 reviews\"],[\"$1135.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC154T, 17.3\\\" FHD, Ryzen 7 1700, 16GB, 256GB + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"7 reviews\"],[\"$1136.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC209T, 17.3\\\" FHD IPS, Ryzen 7 1700, 16GB, 256GB SSD + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"8 reviews\"],[\"$1137.99\",\"Asus ROG Strix...\",\"Asus ROG Strix SCAR Edition GL503VM-ED115T, 15.6\\\" FHD 120Hz, Core i7-7700HQ, 16GB, 256GB SSD + 1TB SSHD, GeForce GTX 1060 6GB, Windows 10 Home\",\"8 reviews\"],[\"$1138.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC154T, 17.3\\\" FHD, Ryzen 7 1700, 16GB, 256GB + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"7 reviews\"],[\"$1139.99\",\"Asus ROG Strix...\",\"Asus ROG Strix GL702ZC-GC209T, 17.3\\\" FHD IPS, Ryzen 7 1700, 16GB, 256GB SSD + 1TB HDD, Radeon RX 580 4GB, Windows 10 Home\",\"8 reviews\"]],\"name\":\"Laptops\"}}", "htmlPath": "results/d287adc4-9d11-41ed-8245-de05d577c286/page.html", "recordingPath": null, "screenshotPath": "results/d287adc4-9d11-41ed-8245-de05d577c286/screenshot.jpg", "dataPath": null, "createdById": null, "updatedById": null, "deletedById": null, "id": "68d3e07c-fab3-4b4a-9c54-b796cd64fe73", "createdAt": "2025-10-29T10:10:29.036Z", "updatedAt": "2025-10-29T10:10:29.036Z", "deletedAt": null } } ``` # Zapier import { Step, Steps } from 'fumadocs-ui/components/steps'; Zapier is a powerful automation tool that allows you to connect different apps and services together. By integrating MrScraper with Zapier, you can automate workflows and streamline your data extraction processes. ## What is the MrScraper Zapier Integration? The MrScraper Zapier integration connects MrScraper with Zapier, allowing you to trigger scraping jobs, automate data collection, and build workflows using MrScraper actions and triggers. ### Available Triggers * **New Result**: Triggers when a new scraping result (succeeded or failed) is available. * **Scrape Completed**: Triggers when a scrape job completes successfully. * **Scrape Failed**: Triggers when a scrape job fails. * **Scrape Started**: Triggers when a scrape job starts. * **Specific Scraper New Result**: Triggers when a new scraping result (succeeded or failed) is available for a specific scraper. ### Available Actions * **Get Latest Result**: Fetch the most recent result from a specified MrScraper scraper. * **Run AI Scraper**: Trigger an AI scraper run with optional URL overrides. * **Run Manual Scraper**: Trigger a manual scraper run with optional URL overrides. ## Why Use Zapier with MrScraper? Integrating MrScraper with Zapier offers several benefits: * **Automation**: Automate repetitive tasks such as data extraction and processing. * **Flexibility**: Connect MrScraper with over 3,000 apps available on Zapier. * **Efficiency**: Save time by eliminating manual data handling. ## Prerequisites Before you start, make sure you have: * [A **MrScraper API token**](https://app.mrscraper.com/api-tokens). * [A **MrScraper scraper** with API access enabled](https://app.mrscraper.com/scrapers). * Access to **Zapier**. ## Integration Steps ### Add MrScraper Node 1. Open the **zapier editor**. 2. Click on **Add a Step** and select **MrScraper** from the list of available apps. ### Configure Trigger (Optional) 1. Add an account by entering your **MrScraper API token**. 2. Select one of the available trigger events: * **New Result** * **Scrape Completed** * **Scrape Failed** * **Scrape Started** * **Specific Scraper New Result** 3. For most triggers, no additional fields are required. 4. If you choose **Specific Scraper New Result**, configure the required field below: | Parameter | Required | Description | | -------------- | -------- | ------------------------------------------------------------------------------------------------ | | **Scraper ID** | Yes | The ID of the scraper you want to listen to. You can find this on the scraper page in MrScraper. | ### Configure the MrScraper Action Node 1. Choose the desired action (e.g., **Get Latest Result**). 2. Configure the action parameters as needed. #### Get Latest Result | Parameter | Required | Description | | -------------- | -------- | --------------------------------------------------------------------------------------------- | | **Scraper ID** | Yes | The ID of the AI scraper you want to run. You can find this on the scraper page in MrScraper. | #### Run AI Scraper | Parameter | Required | Description | | -------------- | -------- | ------------------------------------------------------------------------------------------------- | | **Scraper ID** | Yes | The ID of the manual scraper you want to run. You can find this on the scraper page in MrScraper. | | **URL** | Yes | Override the default target URL for this run. | #### Run Manual Scraper | Parameter | Required | Description | | -------------- | -------- | ------------------------------------------------------------------------------------------ | | **Scraper ID** | Yes | The ID of the scraper you want to run. You can find this on the scraper page in MrScraper. | | **URL** | Yes | Override the default target URL for this run. | ## Example Workflows ### Scheduled Scraping to Google Sheets This workflow automatically runs a scraper on a schedule and saves the results to Google Sheets. Perfect for tracking prices, monitoring listings, or collecting data over time. #### Set Up the Schedule 1. Add a **Schedule** node. 2. Configure the interval (e.g., every hour, daily, weekly). #### Run the Scraper 1. Add the **MrScraper** node. 2. Select **Run AI/Manual Scraper** as the Action event. 3. Enter your **Scraper ID** and **URL**. #### Fetch the Results 1. Add another **MrScraper** node. 2. Select **Get Latest Result** as the resource. 3. Enter the same **Scraper ID**. #### Save to Google Sheets 1. Add a **Google Sheets** node. 2. Select **Create Spreadsheet Row** action. ### Multi-URL Scraping Loop This workflow scrapes multiple URLs from a list and processes each result. This flow is ideal for batch scraping tasks. #### Prepare the URL List 1. Add a **Schedule** node. 2. Add a **Line Items From CSV** node to generate a list of URLs.: ```csv https://example.com/page1 https://example.com/page2 https://example.com/page3 ``` #### Loop Through URLs 1. Add a **Create Loop From Text** node to iterate through each URL. 2. Set **Values to Loop** to the output of the previous node (Line Items From CSV in Files > Line Items). #### Scrape Each URL 1. Add the **MrScraper** node inside the loop. 2. Select **Run AI/Manual Scraper** as the Action event. 3. Use the URL from the previous step. #### Save Results 1. Add a **Google Sheets** node. 2. Select **Create Spreadsheet Row** action. # v3.4.14 import { Sparkles, TrendingUp } from 'lucide-react'; Version **v3.4.14** introduces a new **Agent Setup** page for AI workflows, and major enhancements to **Analytics** with per-use-case scoping, scraper type filtering, and token consumption tracking. ## New Features ### Agent Setup A new [**Agent Setup**](https://app.mrscraper.com/setup) page provides a centralized starting point for connecting MrScraper with supported AI coding agents and harnesses. It includes setup guidance and ready-to-use configuration details, making it easier to get an agent connected and start scraping from an AI-assisted workflow. ### Token Usage Analytics Chart [Analytics](https://app.mrscraper.com/analytic) now includes a dedicated **Token Usage** chart alongside scraping activity charts. It shows token consumption over time and respects the selected filters, including scraper type, domain, action, date range, and API token. The Token Usage chart has independent visualization and interval controls, making it easier to compare token-consumption patterns over time. ## Improvements & Updates ### Playground Analytics per Use Case Playground now includes an **Analytics** tab for each use case. Analytics automatically scopes data to the active scraper type, keeping metrics for Scrape HTML, Scrape JSON, Screenshot, and other use cases separate. The view focuses on the last 24 hours of activity and includes a link to the full [Analytics](https://app.mrscraper.com/analytic) page for deeper exploration. * [Scrape HTML](https://app.mrscraper.com/playground?useCase=unblocker) * [Scrape Markdown](https://app.mrscraper.com/playground?useCase=markdown) * [Get Screenshot](https://app.mrscraper.com/playground?useCase=screenshot) * [Scrape JSON](https://app.mrscraper.com/playground?useCase=extract) * [Listing Page](https://app.mrscraper.com/playground?useCase=listing) * [Scrape Sitemap](https://app.mrscraper.com/playground?useCase=sitemap) ### Scraper Type Filter in Analytics The [Analytics](https://app.mrscraper.com/analytic) dashboard now includes a **Scraper type** filter. Users can filter metrics and charts by the scraper type used, making it easier to analyze activity for a specific workflow. Available filter options are based on scraper result types present in the user’s account. ### New on Marketplace [Marketplace](https://app.mrscraper.com/marketplace) now have a **New on Marketplace** section on the bottom of the page for APIs published in the last 14 days. The new APIs also have a `NEW` tag on the top right of the card. # AI Search AI Search PDP Cache agents take a natural-language **query** and return an AI-generated text answer. Instead of scraping a single detail page, they call an AI provider (with optional web search) and return the answer as Markdown text. They are suitable for research, product lookups, or general knowledge questions. ## Common Response Fields The AI Search PDP Cache agents return the following top-level fields: | Field | Type | Description | | ------------ | ---------------- | --------------------------------------------------------------- | | `success` | boolean | Whether the request completed successfully | | `message` | string | Status message (e.g. `Successfully scraped`) | | `data` | object \| string | The generated answer. Shape depends on the provider (see below) | | `tokenUsage` | number | Number of tokens consumed by the run | The `data` field differs by provider: * **Gemini** returns an **object** with `query` and `text`: | Field | Type | Description | | ------------ | ------ | ------------------------------------- | | `data.query` | string | The original query that was submitted | | `data.text` | string | AI-generated answer in Markdown | * **GPT** returns the answer **directly as a string** (Markdown). When web search is used, sources are appended as inline Markdown reference links (e.g. `[1]: https://...`). ## Example Providers AI Search PDP agents are typically available for major AI providers, for example: * **Gemini** — Ask a query to Gemini and receive an AI-generated text answer. * **GPT** — Performs a web search using GPT and returns an AI-generated text answer with sources. Check the [Marketplace](https://app.mrscraper.com/marketplace) for the full list of supported AI Search providers. # Article Article PDP Cache agents extract structured data from article and news detail pages, such as headlines, content, publication dates, authors, media, and breadcrumbs. ## Common Response Fields The Article PDP Cache agents return the following fields: | Field | Type | Description | | ------------------ | ------ | ----------------------------------------------------------------------- | | `headline` | string | Article headline or title | | `articleBody` | string | Full text content of the article | | `articleBodyHtml` | string | HTML markup of the article body | | `description` | string | Short summary or description of the article | | `datePublished` | string | Publication date in ISO 8601 format | | `datePublishedRaw` | string | Publication date as displayed on the page | | `dateModified` | string | Last modified date in ISO 8601 format | | `dateModifiedRaw` | string | Last modified date as displayed on the page | | `authors` | array | List of article authors. Each item: `name` (string), `nameRaw` (string) | | `inLanguage` | string | Language code of the article (e.g., en) | | `breadcrumbs` | array | Navigation breadcrumb trail. Each item: `url` (string), `name` (string) | | `mainImage` | object | Primary image. Structure: `url` (string) | | `images` | array | All images in the article. Each item: `url` (string) | | `videos` | array | All videos in the article. Each item: `url` (string) | | `audios` | array | All audio files in the article. Each item: `url` (string) | | `url` | string | URL of the article page | | `canonicalUrl` | string | Canonical URL of the article | ## Example Domains / Websites Article PDP agents are typically available for commonly used news and blog domains, for example: * **BBC** — e.g. `https://www.bbc.com/news/...`, `https://www.bbc.co.uk/news/articles/...` * **The New York Times** — e.g. `https://www.nytimes.com/...` * **Medium** — e.g. `https://medium.com/...` * **Reuters** — e.g. `https://www.reuters.com/...` * **The Guardian** — e.g. `https://www.theguardian.com/...` * **CNN** — e.g. `https://edition.cnn.com/...` * **TechCrunch** — e.g. `https://techcrunch.com/...` * **Wikipedia** — e.g. `https://en.wikipedia.org/wiki/...` Exact availability depends on the [Marketplace](https://app.mrscraper.com/marketplace). Check the marketplace for the full list of supported article domains. # Hotel Hotel PDP Cache agents extract structured data from hotel and accommodation detail pages, including property details, pricing, reviews, facilities, room types, and nearby points of interest. ## Common Response Fields The Hotel PDP Cache agents return the following fields: | Field | Type | Description | | -------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | string | Hotel or accommodation name | | `about` | string | Description and overview of the property | | `address` | string | Full street address | | `location` | string | City, area, or region name | | `coordinates` | object | `latitude` (number), `longitude` (number) | | `stars` | number | Star rating of the property (1–5) | | `year_opened` | number | Year the property was opened | | `check_in` | string | Check-in time instructions | | `check_out` | string | Check-out time instructions | | `contact` | object | `email` (string), `phone` (string) | | `host` | object | `name`, `joined`, `response_time` (for homestays/apartments) | | `pricing` | object | `currency`, `original_price`, `starting_price`, `discounted_price` | | `review_stats` | object | `overall_score`, `total_reviews`, `score_breakdown`, `category_summaries` | | `highlights` | array | Key selling points (strings) | | `review_highlights` | array | Snippets of positive feedback (strings) | | `reviews` | array | Objects: `date`, `title`, `rating`, `content`, `country`, `reviewer`, `room_type`, `trip_type`, `stay_duration` | | `popular_facilities` | array | Flat list of popular facilities (strings) | | `facilities` | object | `dining`, `internet`, `services`, `recreation`, `room_amenities`, `transportation`, `access_and_security`, `cleanliness_and_safety` (each array of strings) | | `policies` | object | `pets`, `deposit`, `smoking`, `children`, `breakfast`, `cancellation`, `check_in_instructions` | | `room_types` | array | Objects: `name`, `size`, `prices` (with `adults`, `kids`, `price_total`, `breakfast_included`, `cancellation_policy`, etc.), `bed_type`, `features` | | `location_nearby` | object | `airports`, `culinary`, `shopping`, `hospitals`, `landmarks`, `attractions`, `cash_withdrawal`, `public_transport`, `convenience_stores` (each array of `{ name, distance }`) | ## Example Domains / URLs Hotel PDP agents are typically available for major booking and travel domains, for example: * **Booking.com** — e.g. `https://www.booking.com/hotel/...` * **Expedia** — e.g. `https://www.expedia.com/...` * **Agoda** — e.g. `https://www.agoda.com/...` * **Trip.com** — e.g. `https://www.trip.com/...` * **Hotels.com** — e.g. `https://www.hotels.com/...` Check the [Marketplace](https://app.mrscraper.com/marketplace) for the full list of supported hotel domains. # Job Posting Job Posting PDP Cache agents extract structured data from **job detail pages**, including title, description, employment type, hiring organization, salary, location, and dates. ## Common Response Fields The Job Posting PDP Cache agents return the following fields: | Field | Type | Description | | -------------------- | ------ | ---------------------------------------------------------------------------------- | | `jobTitle` | string | Title of the job posting | | `datePublished` | string | Publication date in ISO 8601 format | | `datePublishedRaw` | string | Publication date as displayed on the page | | `validThrough` | string | Expiration date of the job posting in ISO 8601 format | | `description` | string | Full job description text | | `descriptionHtml` | string | HTML markup of the job description | | `employmentType` | string | Type of employment (e.g., Full-time, Part-time) | | `hiringOrganization` | object | `name` (string) — Company or organization name | | `baseSalary` | object | `raw` (string), `currency` (ISO 4217), `valueMax` (string), `currencyRaw` (string) | | `jobLocation` | object | `raw` (string) — Location as displayed | | `url` | string | URL of the job posting page | ## Example Domains / URLs Job posting PDP agents are typically available for major job boards and career sites, for example: * **LinkedIn** — e.g. `https://www.linkedin.com/jobs/view/...` * **Indeed** — e.g. `https://www.indeed.com/viewjob?jk=...` * **Glassdoor** — e.g. `https://www.glassdoor.com/job-listing/...` * **Monster** — e.g. `https://www.monster.com/jobs/...` * **ZipRecruiter** — e.g. `https://www.ziprecruiter.com/jobs/...` * **CareerBuilder** — e.g. `https://www.careerbuilder.com/job/...` Check the [Marketplace](https://app.mrscraper.com/marketplace) for the full list of supported job posting domains. # Product Product PDP Cache agents extract structured data from **e-commerce product detail pages**, including product info, price, media, description, specifications, variants, seller, ratings, shipping, and warranty. ## Common Response Fields The Product PDP Cache agents return the following fields: | Field | Type | Description | | ---------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `product_info` | object | `name`, `brand`, `category`, `condition`, `breadcrumbs` (array of `{ url, name }`) | | `sku` | string | Stock Keeping Unit identifier | | `mpn` | string | Manufacturer Part Number | | `gtin` | array | Global Trade Item Numbers. Each item: `type` (e.g. isbn13, ean13, upc), `value` (number) | | `price` | object | `currency`, `current_price`, `is_flash_sale`, `original_price`, `currency_symbol`, `discount_percentage` | | `wholesale_tiers` | array | Objects: `min_qty`, `price_per_unit` | | `stock_status` | object | `status` (e.g. in\_stock, out\_of\_stock, pre\_order), `quantity` | | `media` | array | Objects: `url`, `type` (image/video), `is_thumbnail` | | `description` | object | `full_text`, `html_content`, `key_features` (array of strings) | | `specifications` | array | Objects: `name`, `value` | | `variants` | array | Objects: `mpn`, `sku`, `url`, `gtin`, `name`, `media`, `price`, `option` (array of strings), `product_url`, `stock_status`, `additionalProperties` | | `seller` | object | `id`, `name`, `type`, `badges`, `rating`, `location`, `sold_count`, `product_count` | | `rating_summary` | object | `sold_count`, `rating_count`, `review_count`, `average_rating` | | `rating_distribution` | object | `1`, `2`, `3`, `4`, `5` — each with `count` and `percentage` | | `shipping_info` | object | `weight`, `dimensions`, `free_shipping`, `fulfillment_type` | | `warranty_and_returns` | object | `has_warranty`, `warranty_type`, `warranty_period`, `return_policy_text` | ## Example Domains / URLs Product PDP agents are typically available for major e-commerce and marketplace domains, for example: * **Amazon** — e.g. `https://www.amazon.com/dp/...`, `https://www.amazon.co.uk/dp/...` * **Walmart** — e.g. `https://www.walmart.com/ip/...` * **eBay** — e.g. `https://www.ebay.com/itm/...` * **Tokopedia** — e.g. `https://www.tokopedia.com/...` * **Shopee** — e.g. `https://shopee.co.id/...` * **Lazada** — e.g. `https://www.lazada.com/...` * **AliExpress** — e.g. `https://www.aliexpress.com/item/...` Check the [Marketplace](https://app.mrscraper.com/marketplace) for the full list of supported product domains. # Property Property PDP Cache agents extract structured data from **real estate listing detail pages**, including address, price, property details, lot info, interior/exterior features, rooms, parking, utilities, financial data, agents, schools, community, area statistics, and climate risks. ## Common Response Fields The Property PDP Cache agents return the following fields: | Field | Type | Description | | ------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `title` | string | Listing title or marketing headline | | `mls_number` | string | Multiple Listing Service identifier | | `status` | string | e.g. For Sale, Active, Pending, Sold | | `price` | object | `amount`, `currency`, `display_price`, `price_per_sqft` | | `address` | object | `city`, `unit`, `state`, `county`, `island`, `street`, `country`, `latitude`, `longitude`, `zip_code`, `maps_link`, `full_address` | | `property_details` | object | `sq_ft`, `stories`, `bedrooms`, `bathrooms`, `furnished`, `year_built`, `property_type`, `bathrooms_full`, `property_style`, `bedrooms_details`, `bathrooms_partial`, `construction_materials` | | `lot_info` | object | `views`, `zoning`, `waterfront`, `lot_size_sqft`, `lot_size_text`, `lot_size_acres` | | `description` | string | Full marketing description | | `dates` | object | `sold`, `posted`, `updated`, `days_on_market` | | `interior_features` | object | `cooling`, `heating`, `kitchen`, `laundry`, `basement`, `flooring`, `appliances`, `fireplaces`, `other_interior`, `security_features`, `fireplace_features` | | `rooms` | array | Objects: `name`, `level`, `features`, `dimensions` | | `exterior_features` | object | `pool`, `roof`, `fencing`, `patio_porch`, `other_structures` | | `parking` | object | `type`, `spaces`, `has_garage`, `description` | | `utilities` | object | `sewer`, `water`, `electricity`, `energy_info`, `available_utilities` | | `financial` | object | `hoa_fee`, `annual_tax`, `hoa_includes`, `tax_property_id`, `tax_assessed_value`, `estimated_monthly_payment` | | `tax_history` | array | Objects: `year`, `amount`, `assessment`, `change_percentage` | | `price_history` | array | Objects: `date`, `event`, `price`, `source` | | `agents` | array | Objects: `name`, `role`, `email`, `phone`, `agency`, `profile_url` | | `school_district` | string | School district name | | `nearby_schools` | object | List with `name`, `type`, `grades`, `distance` | | `community` | object | `name`, `features`, `security` | | `area_statistics` | object | `people` (median\_age, population, income, etc. by zip/city/county/national), `sun_exposure`, `annual_precipitation` | | `climate_risks` | object | `air_quality`, `fire_factor`, `heat_factor`, `wind_factor`, `flood_factor` | | `lifestyle` | array | Objects: `score`, `category`, `description` | | `nearby_amenities` | array | Objects: `name`, `category`, `distance` | | `metadata` | object | `url`, `views`, `source`, `favorites` | ## Example Domains / URLs Property PDP agents are typically available for major real estate portals, for example: * **Zillow** — e.g. `https://www.zillow.com/homedetails/...` * **Realtor.com** — e.g. `https://www.realtor.com/realestateandhomes-detail/...` * **Redfin** — e.g. `https://www.redfin.com/...` * **Trulia** — e.g. `https://www.trulia.com/p/...` * **Rightmove** (UK) — e.g. `https://www.rightmove.co.uk/properties/...` * **Domain** (AU) — e.g. `https://www.domain.com.au/...` Check the [Marketplace](https://app.mrscraper.com/marketplace) for the full list of supported property domains. # Restaurant Restaurant PDP Cache agents extract structured data from **restaurant detail pages**, including name, rank, cuisine, address, contact, hours, features, description, ratings, review counts, and reviews. ## Common Response Fields The Restaurant PDP Cache agents return the following fields: | Field | Type | Description | | ----------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------- | | `name` | string | Restaurant name | | `rank` | string | Ranking in area (e.g. #14 of 769 Restaurants in Kuta) | | `cuisine` | array | Cuisine types (strings) | | `address` | string | Full street address | | `map_link` | string | Google Maps directions link | | `phone` | string | Contact phone number | | `email` | string | Contact email address | | `website` | string | Official website URL | | `hours` | object | Opening hours by day (e.g. Monday, Tuesday, …, Sunday) | | `features` | array | Features and amenities (dietary options, meal types, payment methods) | | `description` | string | Restaurant description and overview | | `rating` | number | Overall rating (0–5 scale) | | `rating_distribution` | object | `Food`, `Value`, `Service`, `Atmosphere` (each number 0–5) | | `reviews_count` | number | Total number of reviews | | `reviews_distribution` | object | `Excellent`, `Good`, `Average`, `Poor`, `Terrible` (counts) | | `review_summary` | string | AI-generated summary of guest reviews | | `review_summary_distribution` | object | Sentiment keyword by category: `Food`, `Value`, `Service`, `Location`, `Atmosphere` | | `reviews` | object | Structure: `title`, `rating`, `content`, `review_date`, `reviewer_name`, `rating_distribution` (key-value rating by category) | ## Example Domains / URLs Restaurant PDP agents are typically available for major review and dining platforms, for example: * **TripAdvisor** — e.g. `https://www.tripadvisor.com/Restaurant_Review-...` * **Yelp** — e.g. `https://www.yelp.com/biz/...` * **OpenTable** — e.g. `https://www.opentable.com/...` * **TheFork** — e.g. `https://www.thefork.com/...` * **Zomato** — e.g. `https://www.zomato.com/...` * **Google Maps** (restaurant listings) — e.g. `https://www.google.com/maps/place/...` Check the [Marketplace](https://app.mrscraper.com/marketplace) for the full list of supported restaurant domains. # Tour Tour PDP Cache agents extract structured data from **tour and activity detail pages**, including title, about, provider, location, age range, duration, start time, languages, price, availability, highlights, included/excluded items, meeting and pickup, itinerary, accessibility, review stats, reviews, photos, FAQ, and policies. ## Common Response Fields The Tour PDP Cache agents return the following fields: | Field | Type | Description | | -------------------- | ------- | ---------------------------------------------------------------------------------------------------------------- | | `title` | string | Tour title as displayed on the page | | `about` | string | Full tour description and overview | | `provider` | object | `logo`, `name`, `contact` (email, phone, website) | | `location` | object | `city`, `country`, `coordinates` (latitude, longitude), `meeting_point` | | `age_range` | object | `min_age`, `max_age` (numbers) | | `duration` | string | Total tour duration (e.g. 3–4 hours) | | `start_time` | string | Start time for the tour | | `mobile_ticket` | boolean | Whether mobile ticket is required | | `languages` | object | `live_guide`, `audio_guide`, `written_guide` (arrays of language strings) | | `price` | object | `currency`, `original_price`, `starting_price`, `discounted_price`, `price_per_person` | | `availability` | object | `max_group_size`, `available_dates`, `free_cancellation`, `instant_confirmation`, `reserve_now_pay_later` | | `highlights` | array | Key selling points (strings) | | `included` | array | Items included in the tour (strings) | | `excluded` | array | Items not included (strings) | | `meeting_and_pickup` | object | `start_point` (type, address, details, pickup\_details, pickup\_offered), `end_point` (array of address/details) | | `itinerary` | array | Objects: `duration`, `stop_name`, `description`, `admission_included` | | `additional_info` | array | Important notes (strings) | | `accessibility` | object | `stroller_accessible`, `near_public_transport`, `wheelchair_accessible` (booleans) | | `review_stats` | object | `overall_score`, `total_reviews`, `rating_breakdown` (1\_star … 5\_star) | | `reviews` | array | Objects: `date`, `title`, `rating`, `content`, `country`, `reviewer`, `trip_type` | | `photos` | array | Image URLs (strings) | | `faq` | array | Objects: `question`, `answer` | | `policies` | object | `child_policy`, `refund_policy`, `weather_policy`, `cancellation_policy` | ## Example Domains / URLs Tour PDP agents are typically available for major tour and activity platforms, for example: * **Viator** — e.g. `https://www.viator.com/tours/...` * **GetYourGuide** — e.g. `https://www.getyourguide.com/...` * **TripAdvisor (Experiences)** — e.g. `https://www.tripadvisor.com/Attraction_ProductReview-...` * **Klook** — e.g. `https://www.klook.com/activity/...` Check the [Marketplace](https://app.mrscraper.com/marketplace) for the full list of supported tour domains. # Authentication import { Info, AlertCircle } from 'lucide-react'; ## Authentication Formats Residential Proxy uses a flexible username-based authentication system where your username pattern determines the proxy configuration. All proxies connect through the same endpoint but behave differently based on your username structure. ## Connection Format All Residential Proxy connections use the following format: ``` username:password@proxy.mrscraper.com:10000 ``` The **username** controls: * Proxy type * Country targeting * Session persistence * Session duration * IP rotation behavior ## Authentication Patterns ### 1. Default (United States) The simplest format uses just your username and password without any modifiers. **Format:** ``` username:password@proxy.mrscraper.com:10000 ``` **Behavior:** * Residential Proxy falls back to **United States (`us`)** IPs * IP rotates on each new request * No session persistence **Example:** ```bash curl -x "user123:pass456@proxy.mrscraper.com:10000" https://api.ipify.org ``` **Note:** The UI always requires a country selection; only manual username configuration lets you omit it. Omitting it is not random assignment — it resolves to the US, so state the country explicitly whenever the target region matters. ### 2. Country-Specific Proxy Target a specific country using ISO 3166 country codes. **Format:** ``` username-country-{iso3166}:password@proxy.mrscraper.com:10000 ``` **Parameters:** * `{iso3166}`: 2-letter ISO 3166 country code (e.g., `US`, `GB`, `DE`, `JP`) **Behavior:** * All requests use IPs from the specified country * IP rotates on each new request * No session persistence **Examples:** United States Great Britain Germany Japan ```bash # Target US proxies user123-country-us:pass456@proxy.mrscraper.com:10000 ``` ```bash # Target GB proxies user123-country-gb:pass456@proxy.mrscraper.com:10000 ``` ```bash # Target German proxies user123-country-de:pass456@proxy.mrscraper.com:10000 ``` ```bash # Target Japanese proxies user123-country-jp:pass456@proxy.mrscraper.com:10000 ``` **Use Cases:** * Accessing geo-restricted content * Testing localized websites * Market research in specific regions * Price comparison across countries ### 3. Session-Based Static Proxy Maintain the same IP across multiple requests using session IDs. **Format:** ``` username-country-{iso3166}-sessid-{session_id}:password@proxy.mrscraper.com:10000 ``` **Parameters:** * `{iso3166}`: 2-letter ISO 3166 country code * `{session_id}`: Any alphanumeric string (e.g., `session1`, `user_alpha`, `x9`) **Behavior:** * Same IP for all requests with the same `session_id` * **Default session duration: 10 minutes** * After expiration, a new IP is assigned * Different `session_id` values get different IPs **Examples:** Session Alpha Session Beta Session Custom ```bash # Session ID: alpha1 user123-country-us-sessid-alpha1:pass456@proxy.mrscraper.com:10000 ``` ```bash # Session ID: beta2 user123-country-gb-sessid-beta2:pass456@proxy.mrscraper.com:10000 ``` ```bash # Session ID: my_custom_session user123-country-de-sessid-my_custom_session:pass456@proxy.mrscraper.com:10000 ``` **Python Example:** ```python import requests proxy = { 'http': 'http://user123-country-us-sessid-session1:pass456@proxy.mrscraper.com:10000', 'https': 'http://user123-country-us-sessid-session1:pass456@proxy.mrscraper.com:10000' } # All these requests use the same IP for i in range(5): response = requests.get('https://api.ipify.org', proxies=proxy) print(f"Request {i+1}: {response.text}") # Same IP for all ``` **Use Cases:** * Account management and login sessions * Shopping cart operations * Dashboard interactions * Form submissions requiring consistent IP ### 4. Custom Session Duration Control exactly how long a session IP persists. **Format:** ``` username-country-{iso3166}-sessid-{session_id}-sesstime-{minutes}:password@proxy.mrscraper.com:10000 ``` **Parameters:** * `{iso3166}`: 2-letter ISO 3166 country code * `{session_id}`: Alphanumeric session identifier * `{minutes}`: Session duration in minutes (integer) **Behavior:** * Same IP for all requests with the same `session_id` * Session expires after `{minutes}` minutes * New IP assigned after expiration * Maximum flexibility for session management Static Proxy with Custom Session Time **Examples:** 30 Minutes 60 Minutes 5 Minutes ```bash # Session lasts 30 minutes user123-country-jp-sessid-x9-sesstime-30:pass456@proxy.mrscraper.com:10000 ``` ```bash # Session lasts 1 hour user123-country-ca-sessid-long1-sesstime-60:pass456@proxy.mrscraper.com:10000 ``` ```bash # Short 5-minute session user123-country-fr-sessid-quick-sesstime-5:pass456@proxy.mrscraper.com:10000 ``` **Node.js Example:** ```javascript const axios = require('axios'); // 20-minute session proxy const proxy = { host: 'proxy.mrscraper.com', port: 10000, auth: { username: 'user123-country-us-sessid-node_session-sesstime-20', password: 'pass456' } }; async function testProxy() { for (let i = 0; i < 10; i++) { const response = await axios.get('https://api.ipify.org', { proxy }); console.log(`Request ${i+1}: ${response.data}`); // Wait 2 minutes between requests await new Promise(resolve => setTimeout(resolve, 120000)); } } testProxy(); ``` **Use Cases:** * Long scraping sessions * Extended browsing sessions * Multi-step workflows * Testing with specific time constraints ## Authentication Summary | Format | Country | Session | Duration | Rotation | | ---------------------------------------------------- | ------------ | ------- | -------- | ------------- | | `username:password` | US (default) | No | N/A | Every request | | `username-country-{code}` | Specific | No | N/A | Every request | | `username-country-{code}-sessid-{id}` | Specific | Yes | 10 min | After expiry | | `username-country-{code}-sessid-{id}-sesstime-{min}` | Specific | No | Custom | After expiry | ## Best Practices ### Session ID Naming * Use descriptive names: `user_john`, `checkout_session`, `scraper_1` * Avoid special characters (stick to alphanumeric and underscores) * Keep IDs under 50 characters for compatibility ### Security Considerations * Never hardcode credentials in source code * Use environment variables for username/password * Rotate session IDs regularly * Monitor proxy usage for anomalies ## Testing Your Configuration Test your proxy configuration with a simple cURL command: ```bash curl -x "username-country-us:password@proxy.mrscraper.com:10000" https://api.ipify.org ``` This returns your current proxy IP address. Run it multiple times to verify rotation behavior: ```bash # Test rotation for i in {1..5}; do echo "Request $i:" curl -x "user-country-us:pass@proxy.mrscraper.com:10000" https://api.ipify.org echo "" done ``` For session-based proxies, you should see the same IP across all requests: ```bash # Test session persistence for i in {1..5}; do echo "Request $i:" curl -x "user-country-us-sessid-test1:pass@proxy.mrscraper.com:10000" https://api.ipify.org echo "" done ``` ## Common Errors ### Authentication Failed * Verify your username and password are correct * Check for typos in the format pattern * Ensure no extra spaces in the username ### Country Not Supported * Verify the country code is valid ISO 3166 format * Check that the country is in Residential Proxy's coverage area * Try a different country code ### Session Expired * Increase `sesstime` value if sessions expire too quickly * Implement automatic reconnection in your code * Use the same `session_id` to maintain the session # Country Codes import { MapPin, Search, AlertCircle } from 'lucide-react'; ## Country Codes (ISO 3166) Residential Proxy uses **ISO 3166-1 alpha-2** country codes to target specific geographic locations. This standardized format ensures compatibility and consistency across all proxy configurations. ## What are ISO 3166 Country Codes? ISO 3166-1 alpha-2 is an international standard that uses **two-letter codes** to represent countries and territories. Residential Proxy adopts this standard for country targeting in proxy configurations. **Format:** * Always **2 letters** * Case-insensitive (but lowercase is recommended) * Based on English country names **Examples:** * `us` → United States * `gb` → United Kingdom * `de` → Germany (Deutschland) * `jp` → Japan ## How to Use Country Codes Country codes are inserted into your proxy username using the `-country-` parameter. **Basic Format:** ```bash username-country-{code}:password@proxy.mrscraper.com:10000 ``` **Examples:** United States United Kingdom Germany Japan ```bash user123-country-us:pass456@proxy.mrscraper.com:10000 ``` ```bash user123-country-gb:pass456@proxy.mrscraper.com:10000 ``` ```bash user123-country-de:pass456@proxy.mrscraper.com:10000 ``` ```bash user123-country-jp:pass456@proxy.mrscraper.com:10000 ``` ## Popular Country Codes ### North America | Country | Code | Example | | ------------- | ---- | ------------------------------------------------ | | United States | `us` | `user-country-us:pass@proxy.mrscraper.com:10000` | | Canada | `ca` | `user-country-ca:pass@proxy.mrscraper.com:10000` | | Mexico | `mx` | `user-country-mx:pass@proxy.mrscraper.com:10000` | ### Europe | Country | Code | Example | | -------------- | ---- | ------------------------------------------------ | | United Kingdom | `gb` | `user-country-gb:pass@proxy.mrscraper.com:10000` | | Germany | `de` | `user-country-de:pass@proxy.mrscraper.com:10000` | | France | `fr` | `user-country-fr:pass@proxy.mrscraper.com:10000` | | Spain | `es` | `user-country-es:pass@proxy.mrscraper.com:10000` | | Italy | `it` | `user-country-it:pass@proxy.mrscraper.com:10000` | | Netherlands | `nl` | `user-country-nl:pass@proxy.mrscraper.com:10000` | | Poland | `pl` | `user-country-pl:pass@proxy.mrscraper.com:10000` | | Sweden | `se` | `user-country-se:pass@proxy.mrscraper.com:10000` | | Switzerland | `ch` | `user-country-ch:pass@proxy.mrscraper.com:10000` | | Belgium | `be` | `user-country-be:pass@proxy.mrscraper.com:10000` | | Austria | `at` | `user-country-at:pass@proxy.mrscraper.com:10000` | | Norway | `no` | `user-country-no:pass@proxy.mrscraper.com:10000` | | Denmark | `dk` | `user-country-dk:pass@proxy.mrscraper.com:10000` | | Finland | `fi` | `user-country-fi:pass@proxy.mrscraper.com:10000` | | Ireland | `ie` | `user-country-ie:pass@proxy.mrscraper.com:10000` | | Portugal | `pt` | `user-country-pt:pass@proxy.mrscraper.com:10000` | | Greece | `gr` | `user-country-gr:pass@proxy.mrscraper.com:10000` | | Czech Republic | `cz` | `user-country-cz:pass@proxy.mrscraper.com:10000` | | Romania | `ro` | `user-country-ro:pass@proxy.mrscraper.com:10000` | | Hungary | `hu` | `user-country-hu:pass@proxy.mrscraper.com:10000` | | Bulgaria | `bg` | `user-country-bg:pass@proxy.mrscraper.com:10000` | | Croatia | `hr` | `user-country-hr:pass@proxy.mrscraper.com:10000` | | Serbia | `rs` | `user-country-rs:pass@proxy.mrscraper.com:10000` | | Slovakia | `sk` | `user-country-sk:pass@proxy.mrscraper.com:10000` | | Slovenia | `si` | `user-country-si:pass@proxy.mrscraper.com:10000` | ### Asia | Country | Code | Example | | -------------------- | ---- | ------------------------------------------------ | | Japan | `jp` | `user-country-jp:pass@proxy.mrscraper.com:10000` | | China | `cn` | `user-country-cn:pass@proxy.mrscraper.com:10000` | | India | `in` | `user-country-in:pass@proxy.mrscraper.com:10000` | | South Korea | `kr` | `user-country-kr:pass@proxy.mrscraper.com:10000` | | Singapore | `sg` | `user-country-sg:pass@proxy.mrscraper.com:10000` | | Indonesia | `id` | `user-country-id:pass@proxy.mrscraper.com:10000` | | Thailand | `th` | `user-country-th:pass@proxy.mrscraper.com:10000` | | Vietnam | `vn` | `user-country-vn:pass@proxy.mrscraper.com:10000` | | Malaysia | `my` | `user-country-my:pass@proxy.mrscraper.com:10000` | | Philippines | `ph` | `user-country-ph:pass@proxy.mrscraper.com:10000` | | Hong Kong | `hk` | `user-country-hk:pass@proxy.mrscraper.com:10000` | | Taiwan | `tw` | `user-country-tw:pass@proxy.mrscraper.com:10000` | | Pakistan | `pk` | `user-country-pk:pass@proxy.mrscraper.com:10000` | | Bangladesh | `bd` | `user-country-bd:pass@proxy.mrscraper.com:10000` | | Israel | `il` | `user-country-il:pass@proxy.mrscraper.com:10000` | | United Arab Emirates | `ae` | `user-country-ae:pass@proxy.mrscraper.com:10000` | | Saudi Arabia | `sa` | `user-country-sa:pass@proxy.mrscraper.com:10000` | | Turkey | `tr` | `user-country-tr:pass@proxy.mrscraper.com:10000` | ### Oceania | Country | Code | Example | | ----------- | ---- | ------------------------------------------------ | | Australia | `au` | `user-country-au:pass@proxy.mrscraper.com:10000` | | New Zealand | `nz` | `user-country-nz:pass@proxy.mrscraper.com:10000` | ### South America | Country | Code | Example | | --------- | ---- | ------------------------------------------------ | | Brazil | `br` | `user-country-br:pass@proxy.mrscraper.com:10000` | | Argentina | `ar` | `user-country-ar:pass@proxy.mrscraper.com:10000` | | Chile | `cl` | `user-country-cl:pass@proxy.mrscraper.com:10000` | | Colombia | `co` | `user-country-co:pass@proxy.mrscraper.com:10000` | | Peru | `pe` | `user-country-pe:pass@proxy.mrscraper.com:10000` | | Venezuela | `ve` | `user-country-ve:pass@proxy.mrscraper.com:10000` | ### Africa | Country | Code | Example | | ------------ | ---- | ------------------------------------------------ | | South Africa | `za` | `user-country-za:pass@proxy.mrscraper.com:10000` | | Egypt | `eg` | `user-country-eg:pass@proxy.mrscraper.com:10000` | | Nigeria | `ng` | `user-country-ng:pass@proxy.mrscraper.com:10000` | | Kenya | `ke` | `user-country-ke:pass@proxy.mrscraper.com:10000` | | Morocco | `ma` | `user-country-ma:pass@proxy.mrscraper.com:10000` | ## Finding Country Codes If you need a country code not listed above, you can: 1. **Search Online**: Use "ISO 3166-1 alpha-2 \[country name]" 2. **Check Wikipedia**: [ISO 3166-1 alpha-2 on Wikipedia](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2) 3. **Common Patterns**: Usually the first two letters of the English country name **Common Code Notes:** * 🇬🇧 United Kingdom: `gb` (Great Britain) - this is the official ISO 3166 code * 🇨🇭 Switzerland: `ch` (from Latin *Confoederatio Helvetica*) * 🇬🇷 Greece: `gr` (from Greek) * 🇰🇷 South Korea: `kr` (from **K**o**r**ea) ## Country Code Best Practices ### Case Sensitivity Country codes are **case-insensitive**, but lowercase is recommended for consistency: ```bash # All valid, but lowercase is preferred user-country-us:pass@proxy.mrscraper.com:10000 # ✅ Recommended user-country-US:pass@proxy.mrscraper.com:10000 # ✅ Valid user-country-Us:pass@proxy.mrscraper.com:10000 # ✅ Valid but inconsistent ``` ### Validation Always verify your country code is correct before deployment: ```python import requests def test_country_proxy(country_code): """Test if a country code works with Residential Proxy""" proxy = { 'http': f'http://user-country-{country_code}:pass@proxy.mrscraper.com:10000', 'https': f'http://user-country-{country_code}:pass@proxy.mrscraper.com:10000' } try: response = requests.get('https://api.ipify.org', proxies=proxy, timeout=10) print(f"✅ Country '{country_code}' works! IP: {response.text}") return True except Exception as e: print(f"❌ Country '{country_code}' failed: {e}") return False # Test multiple countries countries = ['us', 'gb', 'de', 'jp', 'au'] for country in countries: test_country_proxy(country) ``` ## Omitting the Country Code The country parameter is optional. When you **don't specify** one, requests fall back to **United States (`us`)** IPs: ```bash # No country specified = United States IPs username:password@proxy.mrscraper.com:10000 # Equivalent to username-country-us:password@proxy.mrscraper.com:10000 ``` **Note:** The Residential Proxy UI always requires a country selection; only manual username configuration lets you omit it. Because omitting it still resolves to the US, set the country explicitly whenever the target region matters — relying on the default hides that intent from your code. ## Code Examples ### Python: Multi-Country Scraping ```python import requests from concurrent.futures import ThreadPoolExecutor countries = ['us', 'gb', 'de', 'fr', 'jp', 'au'] def scrape_with_country(country_code): """Scrape using a specific country proxy""" proxy = { 'http': f'http://user-country-{country_code}:pass@proxy.mrscraper.com:10000', 'https': f'http://user-country-{country_code}:pass@proxy.mrscraper.com:10000' } response = requests.get('https://example.com', proxies=proxy) print(f"Scraped from {country_code.upper()}: {len(response.text)} bytes") return response.text # Scrape from all countries in parallel with ThreadPoolExecutor(max_workers=len(countries)) as executor: results = list(executor.map(scrape_with_country, countries)) ``` ### Node.js: Dynamic Country Selection ```javascript const axios = require('axios'); const countries = ['us', 'gb', 'de', 'fr', 'jp']; async function scrapeFromCountry(countryCode) { const proxy = { host: 'proxy.mrscraper.com', port: 10000, auth: { username: `user-country-${countryCode}`, password: 'pass123' } }; try { const response = await axios.get('https://example.com', { proxy }); console.log(`Scraped from ${countryCode.toUpperCase()}: ${response.data.length} bytes`); return response.data; } catch (error) { console.error(`Error with ${countryCode}:`, error.message); } } // Scrape from multiple countries async function scrapeAll() { for (const country of countries) { await scrapeFromCountry(country); } } scrapeAll(); ``` ## Complete Country Code Reference Below is the complete list of all ISO 3166-1 alpha-2 country codes supported by Residential Proxy:
**View All Country Codes (240+ countries)** \- Click to expand | Country | Code | | Country | Code | | ------------------------------ | ---- | - | ------------------------- | ---- | | Afghanistan | `af` | | Liechtenstein | `li` | | Åland Islands | `ax` | | Lithuania | `lt` | | Albania | `al` | | Luxembourg | `lu` | | Algeria | `dz` | | Macao | `mo` | | American Samoa | `as` | | North Macedonia | `mk` | | Andorra | `ad` | | Madagascar | `mg` | | Angola | `ao` | | Malawi | `mw` | | Anguilla | `ai` | | Malaysia | `my` | | Antarctica | `aq` | | Maldives | `mv` | | Antigua and Barbuda | `ag` | | Mali | `ml` | | Argentina | `ar` | | Malta | `mt` | | Armenia | `am` | | Marshall Islands | `mh` | | Aruba | `aw` | | Martinique | `mq` | | Australia | `au` | | Mauritania | `mr` | | Austria | `at` | | Mauritius | `mu` | | Azerbaijan | `az` | | Mayotte | `yt` | | Bahamas | `bs` | | Mexico | `mx` | | Bahrain | `bh` | | Micronesia | `fm` | | Bangladesh | `bd` | | Moldova | `md` | | Barbados | `bb` | | Monaco | `mc` | | Belarus | `by` | | Mongolia | `mn` | | Belgium | `be` | | Montenegro | `me` | | Belize | `bz` | | Montserrat | `ms` | | Benin | `bj` | | Morocco | `ma` | | Bermuda | `bm` | | Mozambique | `mz` | | Bhutan | `bt` | | Myanmar | `mm` | | Bolivia | `bo` | | Namibia | `na` | | Bonaire | `bq` | | Nauru | `nr` | | Bosnia and Herzegovina | `ba` | | Nepal | `np` | | Botswana | `bw` | | Netherlands | `nl` | | Bouvet Island | `bv` | | New Caledonia | `nc` | | Brazil | `br` | | New Zealand | `nz` | | British Indian Ocean Territory | `io` | | Nicaragua | `ni` | | Brunei | `bn` | | Niger | `ne` | | Bulgaria | `bg` | | Nigeria | `ng` | | Burkina Faso | `bf` | | Niue | `nu` | | Burundi | `bi` | | Norfolk Island | `nf` | | Cambodia | `kh` | | Northern Mariana Islands | `mp` | | Cameroon | `cm` | | Norway | `no` | | Canada | `ca` | | Oman | `om` | | Cape Verde | `cv` | | Pakistan | `pk` | | Cayman Islands | `ky` | | Palau | `pw` | | Central African Republic | `cf` | | Palestine | `ps` | | Chad | `td` | | Panama | `pa` | | Chile | `cl` | | Papua New Guinea | `pg` | | China | `cn` | | Paraguay | `py` | | Christmas Island | `cx` | | Peru | `pe` | | Cocos Islands | `cc` | | Philippines | `ph` | | Colombia | `co` | | Pitcairn | `pn` | | Comoros | `km` | | Poland | `pl` | | Congo | `cg` | | Portugal | `pt` | | Congo (DRC) | `cd` | | Puerto Rico | `pr` | | Cook Islands | `ck` | | Qatar | `qa` | | Costa Rica | `cr` | | Réunion | `re` | | Côte d'Ivoire | `ci` | | Romania | `ro` | | Croatia | `hr` | | Russian Federation | `ru` | | Cuba | `cu` | | Rwanda | `rw` | | Curaçao | `cw` | | Saint Barthélemy | `bl` | | Cyprus | `cy` | | Saint Helena | `sh` | | Czech Republic | `cz` | | Saint Kitts and Nevis | `kn` | | Denmark | `dk` | | Saint Lucia | `lc` | | Djibouti | `dj` | | Saint Martin | `mf` | | Dominica | `dm` | | Saint Pierre and Miquelon | `pm` | | Dominican Republic | `do` | | Saint Vincent | `vc` | | Ecuador | `ec` | | Samoa | `ws` | | Egypt | `eg` | | San Marino | `sm` | | El Salvador | `sv` | | Sao Tome and Principe | `st` | | Equatorial Guinea | `gq` | | Saudi Arabia | `sa` | | Eritrea | `er` | | Senegal | `sn` | | Estonia | `ee` | | Serbia | `rs` | | Eswatini | `sz` | | Seychelles | `sc` | | Ethiopia | `et` | | Sierra Leone | `sl` | | Falkland Islands | `fk` | | Singapore | `sg` | | Faroe Islands | `fo` | | Sint Maarten | `sx` | | Fiji | `fj` | | Slovakia | `sk` | | Finland | `fi` | | Slovenia | `si` | | France | `fr` | | Solomon Islands | `sb` | | French Guiana | `gf` | | Somalia | `so` | | French Polynesia | `pf` | | South Africa | `za` | | French Southern Territories | `tf` | | South Georgia | `gs` | | Gabon | `ga` | | South Sudan | `ss` | | Gambia | `gm` | | Spain | `es` | | Georgia | `ge` | | Sri Lanka | `lk` | | Germany | `de` | | Sudan | `sd` | | Ghana | `gh` | | Suriname | `sr` | | Gibraltar | `gi` | | Svalbard and Jan Mayen | `sj` | | Greece | `gr` | | Sweden | `se` | | Greenland | `gl` | | Switzerland | `ch` | | Grenada | `gd` | | Syria | `sy` | | Guadeloupe | `gp` | | Taiwan | `tw` | | Guam | `gu` | | Tajikistan | `tj` | | Guatemala | `gt` | | Tanzania | `tz` | | Guernsey | `gg` | | Thailand | `th` | | Guinea | `gn` | | Timor-Leste | `tl` | | Guinea-Bissau | `gw` | | Togo | `tg` | | Guyana | `gy` | | Tokelau | `tk` | | Haiti | `ht` | | Tonga | `to` | | Heard Island | `hm` | | Trinidad and Tobago | `tt` | | Holy See | `va` | | Tunisia | `tn` | | Honduras | `hn` | | Turkey | `tr` | | Hong Kong | `hk` | | Turkmenistan | `tm` | | Hungary | `hu` | | Turks and Caicos Islands | `tc` | | Iceland | `is` | | Tuvalu | `tv` | | India | `in` | | Uganda | `ug` | | Indonesia | `id` | | Ukraine | `ua` | | Iran | `ir` | | United Arab Emirates | `ae` | | Iraq | `iq` | | United Kingdom | `gb` | | Ireland | `ie` | | United States | `us` | | Isle of Man | `im` | | US Minor Outlying Islands | `um` | | Israel | `il` | | Uruguay | `uy` | | Italy | `it` | | Uzbekistan | `uz` | | Jamaica | `jm` | | Vanuatu | `vu` | | Japan | `jp` | | Venezuela | `ve` | | Jersey | `je` | | Vietnam | `vn` | | Jordan | `jo` | | Virgin Islands (British) | `vg` | | Kazakhstan | `kz` | | Virgin Islands (U.S.) | `vi` | | Kenya | `ke` | | Wallis and Futuna | `wf` | | Kiribati | `ki` | | Western Sahara | `eh` | | North Korea | `kp` | | Yemen | `ye` | | South Korea | `kr` | | Zambia | `zm` | | Kuwait | `kw` | | Zimbabwe | `zw` | | Kyrgyzstan | `kg` | | | | | Laos | `la` | | | | | Latvia | `lv` | | | | | Lebanon | `lb` | | | | | Lesotho | `ls` | | | | | Liberia | `lr` | | | | | Libya | `ly` | | | |
**Note:** Not all countries may have proxy coverage. Availability depends on Residential Proxy's network infrastructure in each region. Contact support to verify coverage for specific countries. ## Common Errors ### Invalid Country Code **Error:** Connection fails or times out **Cause:** Country code doesn't exist or is misspelled **Solution:** Verify the code against ISO 3166-1 alpha-2 standard ```bash # ❌ Wrong user-country-usa:pass@proxy.mrscraper.com:10000 # Should be 'us', not 'usa' # ✅ Correct user-country-us:pass@proxy.mrscraper.com:10000 ``` ### Country Not Supported **Error:** Connection fails or times out **Cause:** Country code is valid but not in Residential Proxy's coverage **Solution:** Try a nearby country or contact support ### Wrong Country Format **Error:** Authentication fails **Cause:** Incorrect parameter name or position **Solution:** Use `-country-` exactly as shown ```bash # ❌ Wrong formats user-c-us:pass@proxy.mrscraper.com:10000 # Wrong parameter name user-us-country:pass@proxy.mrscraper.com:10000 # Wrong order # ✅ Correct format user-country-us:pass@proxy.mrscraper.com:10000 ``` # IP Authorization import { ShieldCheck, Key, Globe, Server, CheckCircle2, Clock, AlertTriangle, Info, Terminal, HelpCircle } from 'lucide-react'; import { Step, Steps } from 'fumadocs-ui/components/steps'; import { Tab, Tabs } from 'fumadocs-ui/components/tabs'; import { Accordion, Accordions } from 'fumadocs-ui/components/accordion'; IP Authorization (IP Whitelisting) allows you to authorize trusted IP addresses to connect to **MrScraper Residential Proxy** without providing proxy credentials (`username:password`) for every request. When your server or network IP address is added to the authorized list, MrScraper automatically recognizes incoming traffic from that IP and validates access. ## Why Use IP Authorization? While standard proxy authentication relies on credentials embedded within each request URL, IP Authorization authenticates connections at the network layer based on your source IP. }> Eliminate the need to store, pass, or hardcode proxy usernames and passwords in application code or scripts. }> Ideal for cloud instances, VPS deployments, and automated web scrapers running from fixed server IPs. }> Seamlessly connect tools, third-party software, and legacy systems that don't support custom proxy auth strings. ### Comparison: Credential Auth vs. IP Authorization | Feature | Credential Authentication | IP Authorization | | :------------------------- | :----------------------------------- | :----------------------- | | **Authentication Method** | Username & Password | IP Address Whitelist | | **Setup Location** | Client Code / Request URL | MrScraper Dashboard | | **Dynamic Server Support** | Yes (Works from any IP) | Requires Fixed Server IP | | **Credential Storage** | Stored in code / environment | Not required in code | | **Targeting Modifiers** | Embedded in username (`-country-us`) | Proxy request headers | ## 1. Finding Your Public IP Address Before configuring IP Authorization, you must identify the **public IPv4 address** of the machine, server, or cloud instance that will connect to the proxy. **Important:** You must authorize your **Public IP**, not internal/private addresses such as `192.168.x.x`, `10.x.x.x`, or `172.16.x.x`. Run one of the following commands from your server terminal to determine your public IP: cURL (Linux/macOS) cURL (Alternative) PowerShell (Windows) ```bash curl -4 https://api.ipify.org ``` ```bash curl -4 https://ifconfig.me ``` ```powershell (Invoke-WebRequest -UseBasicParsing -Uri "https://api.ipify.org").Content ``` ## 2. Adding an Authorized IP Follow these steps to add a new trusted IP address to your whitelist: ### Open Proxy Settings Log in to your [MrScraper Dashboard](https://app.mrscraper.com) and navigate to the **Residential Proxy** section from the sidebar. ### Select IP Authorization Click on the **IP Authorization** tab to view your current whitelisted IP addresses. ### Enter Your Public IP Paste your public IPv4 address into the **Add new IP** input field. ### Confirm and Add Click the **Add IP** button. Your IP will immediately appear in the authorized IPs table. ## 3. IP Status Lifecycle Once an IP address is added, it transitions through two statuses: | Status | Description | Action Required | | :--------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------- | :------------------------------------------------ | | Applying | The system is propagating your IP across global edge nodes. This usually takes **up to 2 minutes**. | Wait briefly before initiating proxy connections. | | Active | The IP address is fully authorized and active across all proxy clusters. | Ready to connect without credentials. | If an IP status remains in **Applying** for more than 5 minutes, try refreshing your browser or contact MrScraper support. ## 4. Connecting via Authorized IP Once your status is **Active**, you can send proxy traffic directly through `proxy.mrscraper.com:10000` without passing credentials in the URL. ```bash # Connect directly without username/password curl -x proxy.mrscraper.com:10000 https://api.ipify.org ``` ```python import requests # Set proxy host and port without credentials proxies = { 'http': 'http://proxy.mrscraper.com:10000', 'https': 'http://proxy.mrscraper.com:10000' } response = requests.get('https://api.ipify.org', proxies=proxies) print(f"Connected successfully via IP Auth. Response IP: {response.text}") ``` ```javascript const axios = require('axios'); axios.get('https://api.ipify.org', { proxy: { host: 'proxy.mrscraper.com', port: 10000 } }) .then(response => console.log('Response IP:', response.data)) .catch(error => console.error('Error:', error.message)); ``` ```go package main import ( "fmt" "io" "net/http" "net/url" ) func main() { proxyURL, _ := url.Parse("http://proxy.mrscraper.com:10000") client := &http.Client{ Transport: &http.Transport{ Proxy: http.ProxyURL(proxyURL), }, } resp, err := client.Get("https://api.ipify.org") if err != nil { panic(err) } defer resp.Body.Close() body, _ := io.ReadAll(resp.Body) fmt.Printf("Connected via IP Auth. Output: %s\n", string(body)) } ``` ```php ``` ## 5. Country & Session Targeting With IP Authorization, configure country and session targeting with headers sent to the **proxy connection**. Do not add targeting modifiers to a username or password. | Header | Example | Purpose | | :----------------- | :-------- | :------------------------------------------------------------ | | `X-Proxy-Country` | `us` | Select the proxy country using its country code. | | `X-Proxy-Sessid` | `abcd123` | Keep a consistent proxy session with the supplied session ID. | | `X-Proxy-Sesstime` | `10` | Set the session lifetime in minutes. | Configure these values through your HTTP client's proxy-header or equivalent proxy-connection header option. They must be sent to the proxy, not as ordinary headers to the destination website. For example, the proxy connection should include headers equivalent to: ```text X-Proxy-Country: us X-Proxy-Sessid: abcd123 X-Proxy-Sesstime: 10 ``` If your client does not support proxy-specific headers, use a client or adapter that can set headers on the proxy connection. Keep the proxy host, port, and IP Authorization setup unchanged. ## 6. Managing Whitelisted IPs You can manage existing authorized IPs directly from the IP Authorization table: ### Copying an IP Address Click the **Copy** icon in the Action column next to any authorized IP to quickly copy it to your clipboard for verification or sharing with team members. ### Removing an IP Address To remove access for a server or IP: 1. Locate the IP address in the IP Authorization list. 2. Click the **Delete** icon in the Action column. 3. Confirm the removal prompt. Access will be revoked within 2 minutes. ## 7. Security Best Practices **Security Warning:** Anyone issuing requests from an authorized IP address can consume your proxy bandwidth. Treat an authorized IP address with the same security diligence as secret credentials. * **Fixed IPs Only:** Only whitelist static public IPs assigned to dedicated servers, cloud instances (AWS EC2, DigitalOcean, GCP), or office routers. * **Avoid Public & Dynamic IPs:** Never authorize dynamic home connections, public Wi-Fi networks, or shared VPN endpoints. * **Regular Audits:** Periodically review your whitelist and delete unused IPs or those belonging to decommissioned servers. * **Immediate Deletion:** Promptly remove server IPs when terminating cloud instances or ending vendor contracts. ## Frequently Asked Questions This error occurs if your request is originating from an IP address that is not listed in your whitelist, or if the IP status is still in the **Applying** state. Ensure your current public IP matches the whitelist entry and that the status has changed to **Active**. IP Authorization is intended for static/fixed IP addresses. If your home or server IP changes frequently (dynamic IP), we recommend using standard **Username & Password Authentication** instead. Currently, IP Authorization only requires specific single IPv4 addresses (`/32`). Adding or removing an IP address typically takes **around 2 minutes** to propagate across all edge nodes globally. # Proxy Types import { RefreshCw, Pin } from 'lucide-react'; Residential Proxy offers **Static** and **Rotating** proxies, each designed for different use cases. Choose the right type based on whether you need consistent IP sessions or maximum distribution. ## Proxy Protocols Residential Proxy supports both HTTP and SOCKS5 connections: | Protocol | Proxy URL prefix | Port | Best for | | ---------- | ---------------- | ------- | ------------------------------------------------------------------------ | | **HTTP** | `http://` | `10000` | HTTP requests, browser automation, and standard scraping clients | | **SOCKS5** | `socks5://` | `10001` | Applications that require SOCKS5 support or proxying traffic beyond HTTP | The full proxy URL uses this format: ```text {protocol}://username:password@proxy.mrscraper.com:{port} ``` For example: ```text http://user-country-us:pass@proxy.mrscraper.com:10000 socks5://user-country-us:pass@proxy.mrscraper.com:10001 ``` ## Static vs Rotating Proxies ### Static Proxy A static proxy maintains the same IP address throughout a session. By using a session ID, all requests within that session are routed through the same IP until the session expires. **Key Features:** * Same IP address for all requests within the session * Configurable session duration (default: 10 minutes) * Consistent identity across requests * Ideal for stateful operations **Authentication Format:** ```bash http://username-country-{code}-sessid-{id}-sesstime-{minutes}:password@proxy.mrscraper.com:10000 socks5://username-country-{code}-sessid-{id}-sesstime-{minutes}:password@proxy.mrscraper.com:10001 ``` **Example:** ```python import requests # Static proxy with 30-minute session proxy = { 'http': 'http://user-country-us-sessid-stable1-sesstime-30:pass@proxy.mrscraper.com:10000', 'https': 'http://user-country-us-sessid-stable1-sesstime-30:pass@proxy.mrscraper.com:10000' } # All requests use the same IP for i in range(10): response = requests.get('https://api.ipify.org', proxies=proxy) print(f"Request {i+1}: {response.text}") # Same IP every time ``` **Perfect For:** | **Account Operations** | **Stable Scraping Sessions** | **Financial Transactions** | | -------------------------------- | ----------------------------------------- | ----------------------------------- | | ✅ Login and authentication flows | ✅ Multi-page form submissions | ✅ Payment processing | | ✅ Session-based interactions | ✅ Workflows requiring consistent identity | ✅ Banking operations | | ✅ Account management dashboards | ✅ APIs that track IP changes | ✅ Any flow requiring IP consistency | | ✅ Shopping cart operations | ✅ Testing session persistence | | **Tip:** Start with a 10-minute session for testing. Increase duration for longer workflows, but keep it under 60 minutes for optimal IP freshness. ### Rotating Proxy A rotating proxy automatically changes IP addresses either on every request or when the session expires, assigning a new IP to each request or session. **Key Features:** * Different IP for each request (or after session expiry) * Maximum distribution across IP pool * Reduces detection risk * Ideal for high-volume operations **Authentication Format (No Session):** ```bash http://username-country-{code}:password@proxy.mrscraper.com:10000 socks5://username-country-{code}:password@proxy.mrscraper.com:10001 ``` **Example:** ```python import requests # Rotating proxy - new IP each request proxy = { 'http': 'http://user-country-us:pass@proxy.mrscraper.com:10000', 'https': 'http://user-country-us:pass@proxy.mrscraper.com:10000' } # Each request gets a different IP for i in range(10): response = requests.get('https://api.ipify.org', proxies=proxy) print(f"Request {i+1}: {response.text}") # Different IP each time ``` **Perfect For:** | **Large-Scale Scraping** | **Avoiding Rate Limits** | **Reducing Bans** | | ----------------------------- | ---------------------------------- | ------------------------------------ | | ✅ Crawling thousands of pages | ✅ Bypassing per-IP restrictions | ✅ Appearing as different users | | ✅ High-volume data extraction | ✅ Distributing load across IPs | ✅ Avoiding IP-based blacklists | | ✅ Distributed scraping tasks | ✅ Reducing ban risk | ✅ Scraping aggressive anti-bot sites | | ✅ Avoiding rate limits | ✅ Testing rate limiting mechanisms | ✅ Market research at scale | **Note:** Rotating proxies are not suitable for workflows requiring session persistence, such as login flows or multi-step forms. ## Comparison | Feature | Static Proxy | Rotating Proxy | | -------------------- | -------------------------------- | ------------------------------------------ | | **IP Behavior** | Same IP per session | New IP per request | | **Session Support** | ✅ Yes (via session ID) | ❌ No session needed | | **Use Case** | Login flows, stateful operations | High-volume scraping, rate limit avoidance | | **Session Duration** | Configurable (default 10 min) | N/A | | **Best For** | Consistency required | Maximum distribution | ## Which Type Should I Use? }> * Login and authentication flows * Shopping carts and multi-step forms * Sites that track IP changes * Session-based operations }> * Large-scale scraping (1000+ pages) * Bypassing rate limits * Maximum IP distribution * No session persistence needed # Overview import { Globe, Smartphone, Lock, Timer, RotateCcw, Shield } from 'lucide-react'; import { Database, UserCog, Share2 } from 'lucide-react'; import { Step, Steps } from 'fumadocs-ui/components/steps'; MrScraper provides enterprise-grade proxy solutions designed for web scraping, data collection, and anonymous browsing. Static and rotating configurations, and global coverage, that delivers the reliability and flexibility your projects need. ## What is MrScraper's Residential Proxy? MrScraper's Residential Proxy is a premium proxy service offering: * **Residential Proxies**: Access real user IPs from residential ISPs * **Static & Rotating Options**: Choose between persistent IPs or automatic rotation * **Global Coverage**: Support for 100+ countries using ISO 3166 country codes * **Session Management**: Control IP persistence with custom session IDs and durations * **Easy Integration**: Simple authentication format compatible with any HTTP/HTTPS client ## Key Features } href="/docs/residential-proxy/configuration/proxy-types"> Choose between Static and Rotating proxies for different use cases. } href="/docs/residential-proxy/configuration/authentication"> Learn how to configure proxy authentication with username patterns. } href="/docs/residential-proxy/configuration/authentication#3-session-based-static-proxy"> Maintain persistent IPs with custom session IDs and durations. } href="/docs/residential-proxy/configuration/country-codes"> Target specific countries using ISO 3166 country codes. ## Proxy Endpoint All Residential Proxy services connect through a single endpoint: ``` proxy.mrscraper.com:10000 ``` Your authentication credentials (username and password) determine: * Proxy type * Country targeting * Session persistence * Rotation behavior ## Common Use Cases }> Use rotating proxies to avoid rate limits and IP bans when scraping large amounts of data. }> Use static session-based proxies to maintain consistent IPs for account operations and dashboards. }> Use country-specific proxies to view localized content and pricing. ## Quick Start ### Get Your Credentials Obtain your MrScraper username and password from your account dashboard. ### Choose Your Configuration Decide on your [proxy type](/docs/residential-proxy/configuration/proxy-types), [country](/docs/residential-proxy/configuration/country-codes), and whether you need session persistence. ### Format Your Authentication Build your username string according to the [authentication format](/docs/residential-proxy/configuration/authentication). ### Connect and Test Use the format: `username:password@proxy.mrscraper.com:10000` in your application. ## Need Help? If you have questions or need assistance: * Check our [Examples page](/docs/residential-proxy/examples/overview) for common configurations * Review the [Authentication guide](/docs/residential-proxy/configuration/authentication) for username format details * Contact our support team for personalized assistance # Proxy Generator import { Step, Steps } from 'fumadocs-ui/components/steps'; ## Generate A Proxy You generate residential proxy credentials for use outside the Proxy Scraper. For example, in your own scripts, third-party tools, or external applications. Follow these steps to generate a residential proxy: Go to the **Proxies** page by clicking the **globe icon** in the left sidebar. Select the **Proxy Setup** tab. ### Configuration You can configure your proxy here #### Proxy Access Get your `Username` and `Password` for your proxy here. #### Proxy List Copy, Download or Delete your proxy list here. #### Proxy Configuration Main proxy configuration: * `Connection Settings` : `Static` or `Rotating` * `Protocol` : `HTTP` or `SOCKS5` * `Country` : Pick the country you want your request to appear from. Uses ISO 3166-1 alpha-2 codes (e.g. United States → `us`). Click **Generate** to create a new proxy credential. Copy the generated proxy URL and use it in your application or tool. Or use generated code from **Integration Code** panel. supported framework/library : * Puppeteer (JavaScript) * Playwright (JavaScript & Python) * CloakBrowser (JavaScript & Python) * Selenium (Python) # Proxy Scraper import { Step, Steps } from 'fumadocs-ui/components/steps'; The **Proxy Scraper** is a standalone scraping service available on the [Proxies page](https://app.mrscraper.com/proxies). It routes requests through residential proxies, allowing you to specify a target URL, select a proxy country, and retrieve the scraped content directly from a single interface. ## Proxies Page Tabs The [**Proxies page**](https://app.mrscraper.com/proxies) is organized into four tabs: | Tab | Description | | ----------------- | -------------------------------------------------------------------------- | | **Proxy Scraper** | Scrape any URL through a proxy using MrScraper's built-in scraping engine. | | **Proxy Setup** | Generate and configure residential proxy credentials for external use. | | **Analytics** | Monitor proxy bandwidth consumption and request statistics. | | **Proxy Users** | Create sub-user accounts and allocate proxy bandwidth. | * When you switch between tabs, your scrape results are preserved. You can navigate away and return to find your results intact. * If a scrape is in progress, you receive a warning before switching tabs to prevent losing an active request. * Completed proxy scrape results are also available on the Result section. If you navigate away during a long-running scrape, you can find the results there. * Your active tab is saved in the URL parameters. You can bookmark or share a direct link to any tab, and it will open in the same state when revisited. ## Proxy Scraper The **Proxy Scraper** tab is an interactive playground for running on-demand, proxy-powered scraping requests. It shares the same interface layout as the [Playground](/docs/getting-started/playground) but adds a proxy country selector so every request is routed through a residential proxy. Enter a **Target URL**, select a **proxy country** from the dropdown, and click **Run**. MrScraper's scraping engine fetches the page through a residential proxy in the selected region and returns the extracted content. When a proxy scrape finishes, you receive an in-app notification confirming the result is ready. This lets you queue a request and continue working without watching the screen. ### Target URL In the **Target URL** panel, you configure and run the request: * Enter the target URL. * Select **Run** to send the request, or **Reset** to clear the current configuration. * Select a country to load the page from. ### API Token You can copy your secret API token for authenticating requests to the MrScraper API. Keep this value private. ### Request Options The request options panel controls how the proxy scraper handles each request. Adjust these settings to improve reliability and performance. | Setting | Default | Description | | --------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------- | | **Timeout (seconds)** | `90` | Maximum time in seconds to wait for the page to load. | | **Max retries** | `3` | Maximum number of retry attempts if the scrape fails. Failed requests are automatically retried up to this limit. | | **Wait for selector** | None | Pause until a specific CSS selector appears in the DOM. Useful for content that loads asynchronously after the initial page load. | | **Super** | Off | Uses additional resources to improve extraction accuracy on complex or protected websites. | | **From Homepage** | Off | Redirect to the target homepage first before going to the target URL. | | **Browser rendering** | Off | Renders the page in a headless browser, executing JavaScript before extracting content. Required for SPAs and dynamic websites. | | **Block resources** | Off | Skips non-essential assets (images, fonts, CSS) to reduce bandwidth and improve speed. | Our bandwidth is not 1:1 with the bandwidth you see in your browser when opening the target page. This is due to our retry mechanism and techniques used to avoid blocking from the page. ## Proxy Setup The **Proxy Setup** tab lets you generate residential proxy credentials for use outside the Proxy Scraper. For example, in your own scripts, third-party tools, or external applications. Follow these steps to generate a residential proxy: Go to the **Proxies** page by clicking the **globe icon** in the left sidebar. Select the **Proxy Setup** tab. Scroll down to the **Proxy Configuration** section and configure your proxy settings. Click **Generate** to create a new proxy credential. Copy the generated proxy URL and use it in your application or tool. ## Analytics The **Analytics** tab provides a real-time overview of your proxy bandwidth consumption and request performance. It displays the following summary cards: | Metric | Description | | ------------------- | --------------------------------------------------------- | | **Total Bandwidth** | Total proxy bandwidth consumed (displayed in MB or GB). | | **Total Requests** | Number of proxy requests made within the selected period. | | **Success Rate** | Percentage of proxy requests that completed successfully. | | **Success** | Count of successful proxy requests. | | **Failed** | Count of failed proxy requests. | ### Available Filters You can narrow the analytics view using the following filters: | Filter | Description | | -------------- | --------------------------------------------------------------- | | **Time Range** | Select a time window (e.g., 1 Hour, 24 Hours, 7 Days, 30 Days). | | **Interval** | Choose the data granularity (e.g., Hourly, Daily). | | **Username** | Filter by a specific proxy sub-user. | | **Country** | Filter by proxy country/region. | | **Timezone** | View data in your preferred timezone. | Analytics data may take up to 5 minutes to update. The dashboard refreshes automatically, or you can click the refresh icon on the top right manually to fetch the latest data. ## Proxy Users This tab is still in `Beta` phase The **Proxy Users** tab lets you create and manage sub-user accounts for your proxy service. Each sub-user receives their own credentials and a dedicated bandwidth allocation drawn from your main proxy bandwidth pool. This is useful when you need to: * Distribute proxy access across team members or clients * Control and limit bandwidth usage per user * Track individual proxy consumption separately ### Creating a Sub-User Follow these steps to create a new proxy sub-user: Navigate to the **Proxies** page and select the **Proxy Users** tab. Click the **+ Create Sub-User** button in the top-right corner. In the **Create Sub-User** dialog, fill in the following fields: * **Username suffix** *(optional)*: Enter a suffix to append to your account email. Only alphanumeric characters are allowed; symbols and hyphens are not permitted. * **Bandwidth Quota**: Enter the amount of bandwidth (in MB) to allocate to this sub-user. The value is drawn from your unallocated bandwidth pool. Click **OK** to create the sub-user. You can create up to 80 sub-user. ### Renaming username You can rename a username suffix in the Proxy User table by clicking the edit icon beside the username. ### Editing Bandwidth Allocation To change a sub-user's bandwidth limit: Locate the sub-user in the table. Click the **edit icon** (pencil) next to the **Bandwidth Limit** value. Enter the new bandwidth amount (in MB). For user with `Enterprise` plan you can select `Unlimited` to pay as you go. Confirm the change. The updated allocation is reflected immediately. Any bandwidth freed up returns to the unallocated pool. ### Deleting a Sub-User To remove a sub-user, click the **delete icon** (trash) next to their row in the table. The bandwidth that was allocated to the deleted sub-user is returned to your unallocated pool. # Troubleshoot URL Access import { Step, Steps } from 'fumadocs-ui/components/steps'; When a target website doesn't load through your proxy integration, it can be hard to tell whether the issue lies in the proxy configuration, or the target site's defenses. This guide walks you through a structured process to isolate the cause and resolve it. ## Why target websites block proxy requests Websites use several techniques to detect and block non-human traffic. Even with residential proxy IPs, a request can still fail if: * **Web Application Firewalls (WAF)** like Cloudflare, Akamai, or AWS WAF inspect request patterns and block suspicious traffic. * **CAPTCHA challenges** are triggered by rate limiting or behavioral analysis. * **Bot detection systems** flag requests that lack proper browser fingerprints (headers, cookies, JavaScript execution). * **Geo-restrictions** block access from certain countries or regions. * **Rate limiting** blocks IPs that send too many requests in a short period. * **TLS fingerprinting** identifies non-browser HTTP clients by their TLS handshake characteristics. Understanding these causes helps you pick the right fix. The fastest way to start is to use **Proxy Scraper** to test whether the issue is on your side or the proxy side. ## Test with Proxy Scraper [Proxy Scraper](/docs/residential-proxy/getting-started/proxy-scraper) is a standalone scraping tool built into MrScraper. It sends requests through the same residential proxy network. This isolates the proxy path from your code path. Go to [app.mrscraper.com](https://app.mrscraper.com), sign in, and select **Proxy** from the left navigation menu. Then select the [**Proxy Scraper**](https://app.mrscraper.com/proxies?tab=Proxy+Scraper) tab. Enter a simple, reliable URL in the **Target URL** field. For example, `https://www.google.com`. Keep all advanced settings at their defaults. Click **Run**. If Proxy Scraper returns an **HTTP 200** response with page content, the proxy connection is working. Proceed to the next step. If it returns an error, see [When the connectivity test fails](#when-the-connectivity-test-fails) below. Now enter the **real target URL**. Click **Run** with the same default settings. ## Interpret the results Use the table below to determine your next action based on the Proxy Scraper results: | google.com result | Target URL result | What it means | Next action | | ----------------- | ----------------- | ------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | ✅ Success | ✅ Success | The proxy path works. The website target detecting the scraping script. | You can check your proxy configuration or continue using our Proxy Scraper to access your website target. | | ✅ Success | ❌ Failure | The proxy path works, but the target website is blocking the request. | Go to [Bypass target website defenses](#bypass-target-website-defenses). | | ❌ Failure | ❌ Failure | The proxy path itself has an issue. | Go to [When the connectivity test fails](#when-the-connectivity-test-fails). | | ❌ Failure | ✅ Success | Unlikely, but possible if google.com is blocked in the selected proxy country. Try a different proxy country. | Change the proxy country and retest. | ## Check your configuration If Proxy Scraper succeeds but your own code or tool fails, the issue is in how your integration sends requests. Check the following: ### Proxy credentials * Verify your **username** and **password** are correct and match the format described in the [Authentication guide](/docs/residential-proxy/configuration/authentication). * Check for typos, extra spaces, or missing segments (e.g., `-country-`, `-sessid-`). ### Request format * Ensure you're using the correct proxy endpoint: `proxy.mrscraper.com:10000`. * Verify the proxy protocol. MrScraper supports HTTP. Make sure your client is configured to use the HTTP protocol. * Check that the target URL includes the protocol (`http://`). ### HTTP headers * Some websites block requests that lack a proper `User-Agent` header. Make sure your client sends a realistic browser User-Agent. * Include standard headers like `Accept`, `Accept-Language`, and `Accept-Encoding` to reduce the chance of being flagged as a bot. ### Network and firewall * Confirm your network allows outbound connections to `proxy.mrscraper.com` on port `10000`. * If you're behind a corporate firewall or VPN, check whether proxy traffic is being intercepted or blocked. ## Bypass target website defenses If Proxy Scraper can reach google.com but fails on your target URL, the target website is actively blocking the request. Try the following escalation steps **in order**: ### Try a different proxy country Some websites only serve traffic from specific regions. Change the proxy country in Proxy Scraper to one that matches the target website's expected audience (e.g., `us` for a US-based website). ### Enable Browser rendering Toggle **Browser rendering** to **On** in the Proxy Scraper request options. This renders the page in a headless browser, executing JavaScript before extracting content. This is required for: * Single-page applications (SPAs) * Websites that load content dynamically via JavaScript * Pages behind JavaScript-based challenges ### Adjust timeout and retries Increase the **Timeout** beyond the default 90 seconds, and set **Max retries** to 5 or higher. Some heavily protected websites are slow to respond or require multiple attempts. ### Verify the block manually in a browser Before assuming the proxy IPs are at fault, browse the target website through the same proxy in a real browser, arriving from a Google search and navigating to the page like a human would. If the page loads that way, the IPs are fine and the website is detecting your scraper, not your proxy. See [Verify a Block Manually](/docs/residential-proxy/getting-started/verify-block-manually). ### Contact support If none of the above steps resolve the issue, the target website may require custom handling. Contact [us](mailto:support@mrscraper.com) with the following details: * The target URL that fails * The Proxy Scraper result (screenshot or response) * The proxy country you tested with * The settings you've tried (Super, Browser rendering, timeout) ## When the connectivity test fails If Proxy Scraper cannot reach even a simple URL like google.com, check the following: | Possible cause | What to check | | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Account or subscription issue** | Verify your account is active and has remaining proxy bandwidth in the **Analytics** tab. | | **Service outage** | Check if MrScraper is experiencing an outage. Try again in a few minutes. | | **Proxy country unavailable** | Some proxy countries may have limited availability. Try a different country code from the [supported countries list](/docs/residential-proxy/configuration/country-codes). | | **Browser or network issue** | Try accessing [app.mrscraper.com](https://app.mrscraper.com) in an incognito/private window to rule out browser cache or extension interference. | If the issue persists, contact support at [support@mrscraper.com](mailto:support@mrscraper.com). # Verify a Block Manually import { Step, Steps } from 'fumadocs-ui/components/steps'; When a scraper suddenly starts getting blocked, the proxy is the easiest thing to blame. But in most cases the target website has tightened its bot detection, and the same IPs still work fine for ordinary browsing. This guide walks through a manual test: browse the target website through your Residential Proxy in a real browser, exactly the way a human would. The result tells you which side the block is actually coming from. Use this test when your scraper worked before and started failing, or when a scrape is blocked but you're not sure whether the proxy IPs or the website is responsible. If you haven't run the automated check yet, start with [Troubleshoot URL Access](/docs/residential-proxy/getting-started/troubleshoot-url-access) first, it's faster. ## Why direct URL access fails Many websites treat a request for a deep page (a product page, a listing, a search result) with no prior history as a strong bot signal. A real visitor almost never lands there cold. They arrive from a search engine, from the homepage, or from a category page, carrying cookies, a referrer, and a browsing history that the site has already seen. That's why a scraper hitting a product URL directly can be blocked while the exact same IP loads the site normally in a browser. The test below reproduces the natural path so you can compare the two. ## What you need * Your Residential Proxy credentials from the [Proxy Setup tab](https://app.mrscraper.com/proxies?tab=Proxy+Setup). See the [Authentication guide](/docs/residential-proxy/configuration/authentication) for the username format. * Google Chrome with the [FoxyProxy](https://chromewebstore.google.com/detail/foxyproxy/gcknhkkoolaabfmlnjonogaaifnjlfnp) extension installed. * Optional: [CloakBrowser](/docs/residential-proxy/examples/cloakbrowser), an open source stealth browser, for the second round of testing. ## Run the test ### Route Chrome through the proxy Open FoxyProxy, add a new proxy, and fill in the connection details: | Field | Value | | ----------------- | -------------------------------------------------- | | **Proxy type** | HTTP | | **Hostname / IP** | `proxy.mrscraper.com` | | **Port** | `10000` | | **Username** | Your proxy username, for example `user-country-us` | | **Password** | Your proxy password | Set the country in the username to match the target website's audience. For a Brazilian website, use `-country-br`. Save the proxy, then enable it from the FoxyProxy toolbar icon. Confirm the traffic is actually routed by opening [api.ipify.org](https://api.ipify.org) and checking that the IP is not your own. ### Search for the website on Google Go to [google.com](https://www.google.com) and search for the target website's name or homepage, for example `ifood.com.br`. Don't paste the target page URL into the address bar. The point of this test is to arrive at the site the way a visitor does. ### Click through from the search results Click the organic search result that leads to the website's homepage. This gives the session a search engine referrer and lets the site set its cookies on a page it expects first-time visitors to land on. ### Navigate to the target page like a human From the homepage, reach the page you want to scrape using the site's own navigation: menus, category links, the on-site search box, pagination. Take a moment between clicks instead of firing them back to back. If you reach the page and it loads normally, the website is serving your proxy IP without complaint. ### If regular Chrome is blocked, retry with CloakBrowser Regular Chrome still exposes automation and environment signals that stricter sites fingerprint. [CloakBrowser](/docs/residential-proxy/examples/cloakbrowser) is an open source stealth browser built on a patched Chromium that hides far more of them. Launch it in headed mode with the same proxy credentials, then repeat steps 2 to 4 inside it: ```javascript import { launch } from 'cloakbrowser'; const browser = await launch({ headless: false, proxy: { server: 'http://proxy.mrscraper.com:10000', username: 'user-country-br', password: 'pass123' } }); const page = await browser.newPage(); await page.goto('https://www.google.com'); // Now search for the site and click through by hand. ``` ## Interpret the results | Manual browser test | What it means | Next action | | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | ✅ Works in regular Chrome | The proxy IPs are fine. Your scraper is being detected by its request signature or by navigating directly to the target URL. | Reproduce the natural flow in your scraper: start from the homepage or a search result, keep cookies across requests, and enable browser rendering. | | ❌ Blocked in Chrome, ✅ works in CloakBrowser | The IPs are fine. The website is fingerprinting the browser, not the IP. | Move your scraping to a stealth browser. See the [CloakBrowser examples](/docs/residential-proxy/examples/cloakbrowser). | | ❌ Blocked in both, even with natural navigation | The website is rejecting the IP itself, or the whole country or ASN. | Try a different [proxy country](/docs/residential-proxy/configuration/country-codes) and repeat the test. If it still fails, contact support. | | ❌ Blocked in both, and the site is also blocked without the proxy | The website is blocking your region or is down for everyone. | Verify from a different network before investigating the proxy further. | A website that recently changed its protection can block a perfectly healthy residential IP simply because the request didn't look like a human session. Running this test before escalating saves you from swapping proxies that were never the problem. # CloakBrowser Cloakbrowser is an opensource stealth browser developed by CloakHQ that will increase the success rate of the web scraping process, although there still many factors affecting the success rate. CloakBrowser code examples divided by programming language (JavaScript and Python) for stealth browser automation with Residential Proxy. ## JavaScript (Node.js) ### Prerequisites ```bash npm install cloakbrowser proxy-chain npx cloakbrowser install ``` * `cloakbrowser`: Node.js stealth browser library featuring custom Chromium fingerprint evasion and Playwright-like APIs. ### Proxy Format In JavaScript, CloakBrowser accepts a `proxy` object during `launch()`: ```javascript const proxy = { server: 'http://proxy.mrscraper.com:10000', username: 'user-country-us-sessid-session1', password: 'pass123', } ``` ### SOCKS5 Proxy Setup For SOCKS5 proxies, use `proxy-chain` to anonymize the authenticated SOCKS5 connection before passing the local proxy server to CloakBrowser: ```javascript import { launch } from 'cloakbrowser' import { anonymizeProxy, closeAnonymizedProxy } from 'proxy-chain' const username = 'user-country-us' const password = 'pass123' const socks5Proxy = `socks5://${encodeURIComponent(username)}:${encodeURIComponent(password)}` + `@proxy.mrscraper.com:10001` const localProxy = await anonymizeProxy(socks5Proxy) const browser = await launch({ proxy: { server: localProxy, }, }) try { const page = await browser.newPage() await page.goto('https://ipinfo.io/json') console.log(await page.textContent('body')) } finally { await browser.close() await closeAnonymizedProxy(localProxy, true) } ``` Use port `10001` for SOCKS5. Always close the browser before calling `closeAnonymizedProxy()`. ### 1. Basic Stealth Request Demonstrates launching CloakBrowser with an authenticated residential proxy in JavaScript. It navigates to an IP lookup endpoint and prints the verified proxy IP. ```javascript import { launch } from 'cloakbrowser' async function testStealthProxy() { // Launch stealth browser with proxy options const browser = await launch({ headless: true, proxy: { server: 'http://proxy.mrscraper.com:10000', username: 'user-country-us', password: 'pass123', }, }) const page = await browser.newPage() await page.goto('https://api.ipify.org?format=json') const content = await page.textContent('body') console.log('Your Stealth Proxy IP:', content) await browser.close() } testStealthProxy().catch(console.error) ``` **Use Case:** Baseline proxy connectivity check and stealth browser launch in Node.js. ### 2. Sticky Session Navigation Maintains a static IP address across sequential page navigations by reusing a session ID (`sessid`) and extending session time (`sesstime-30`). ```javascript import { launch } from 'cloakbrowser' async function runStickyStealthSession() { const browser = await launch({ headless: true, proxy: { server: 'http://proxy.mrscraper.com:10000', // 30-minute Canadian static session username: 'user-country-ca-sessid-stealth1-sesstime-30', password: 'pass123', }, }) const page = await browser.newPage() const targetUrls = [ 'https://example.com/step1', 'https://example.com/step2', 'https://example.com/step3', ] for (const url of targetUrls) { await page.goto(url) console.log(`Visited ${url} -> Title:`, await page.title()) } await browser.close() } runStickyStealthSession().catch(console.error) ``` **Use Case:** Executing sensitive multi-page JavaScript workflows (logins, checkouts) while bypassing bot detection and retaining IP consistency. ### 3. Parallel Stealth Workers Demonstrates spawning multiple concurrent JavaScript CloakBrowser instances, each configured with a unique sticky session ID for isolated parallel operations. ```javascript import { launch } from 'cloakbrowser' async function runParallelWorkers() { const workerIds = ['worker_1', 'worker_2', 'worker_3'] const workerPromises = workerIds.map(async (workerId) => { const browser = await launch({ headless: true, proxy: { server: 'http://proxy.mrscraper.com:10000', username: `user-country-us-sessid-${workerId}`, password: 'pass123', }, }) const page = await browser.newPage() await page.goto('https://api.ipify.org?format=json') const ipData = await page.textContent('body') console.log(`Stealth Worker [${workerId}] IP:`, ipData) await browser.close() }) await Promise.all(workerPromises) } runParallelWorkers().catch(console.error) ``` **Use Case:** High-concurrency Node.js web scraping where each worker thread requires dedicated stealth fingerprinting and IP isolation. ### 4. Production Error Handling Production-ready error handling pattern for JavaScript CloakBrowser, providing navigation timeout configurations, response status validation, and safe browser closing routines. ```javascript import { launch } from 'cloakbrowser' async function safeStealthScrape(targetUrl) { let browser = null try { browser = await launch({ headless: true, proxy: { server: 'http://proxy.mrscraper.com:10000', username: 'user-country-us-sessid-prod1', password: 'pass123', }, }) const page = await browser.newPage() // Set 30s timeout const response = await page.goto(targetUrl, { timeout: 30000 }) if (!response || !response.ok()) { throw new Error(`HTTP Error ${response?.status()}`) } console.log('Successfully loaded page title:', await page.title()) } catch (error) { console.error(`CloakBrowser execution failed:`, error.message) } finally { if (browser) { await browser.close() } } } safeStealthScrape('https://example.com') ``` **Use Case:** Deploying enterprise Node.js stealth scrapers that handle timeouts and site exceptions safely. *** ## Python ### Prerequisites ```bash pip install cloakbrowser python -m cloakbrowser install ``` * `cloakbrowser`: Python package for stealth browser automation and anti-bot evasions. * Install `sing-box` separately and make sure the `sing-box` command is available on your `PATH` for SOCKS5 proxy bridging. ### Proxy Format In Python, CloakBrowser receives a dictionary with proxy parameters: ```python proxy = { "server": "http://proxy.mrscraper.com:10000", "username": "user-country-us-sessid-session1", "password": "pass123" } ``` ### SOCKS5 Proxy Setup CloakBrowser's Python integration uses `sing-box` to expose the authenticated SOCKS5 proxy as a local HTTP proxy: ```python import json import os import subprocess import tempfile import time from cloakbrowser import launch username = "user-country-us" password = "pass123" local_proxy = "http://127.0.0.1:8080" singbox_config = { "inbounds": [ { "type": "http", "tag": "http-in", "listen": "127.0.0.1", "listen_port": 8080, } ], "outbounds": [ { "type": "socks", "tag": "mrscraper", "server": "proxy.mrscraper.com", "server_port": 10001, "username": username, "password": password, "version": "5", } ], "route": { "final": "mrscraper" }, } with tempfile.NamedTemporaryFile( mode="w", suffix=".json", delete=False, ) as config_file: json.dump(singbox_config, config_file) config_path = config_file.name singbox_process = subprocess.Popen([ "sing-box", "run", "-c", config_path, ]) try: time.sleep(1) browser = launch( headless=True, proxy={ "server": local_proxy, }, ) try: page = browser.new_page() page.goto("https://ipinfo.io/json") print(page.text_content("body")) finally: browser.close() finally: singbox_process.terminate() singbox_process.wait() os.remove(config_path) ``` Use port `10001` for SOCKS5 and always terminate `sing-box` after closing the browser. ### 1. Basic Stealth Request Demonstrates launching CloakBrowser using its Python API to route automated browser requests through a residential proxy. ```python from cloakbrowser import launch def test_stealth_proxy(): # Launch browser with proxy server and credentials browser = launch( headless=True, proxy={ "server": "http://proxy.mrscraper.com:10000", "username": "user-country-us", "password": "pass123" } ) page = browser.new_page() page.goto("https://api.ipify.org?format=json") print("Your Stealth Proxy IP:", page.text_content("body")) browser.close() if __name__ == "__main__": test_stealth_proxy() ``` **Use Case:** Quick stealth proxy verification script in Python for anti-bot testing and scraping. ### 2. Sticky Session Navigation Maintains a static Python stealth browser session across multiple page navigations using a session ID (`sessid`). ```python from cloakbrowser import launch def run_sticky_session(): browser = launch( headless=True, proxy={ "server": "http://proxy.mrscraper.com:10000", "username": "user-country-us-sessid-python1-sesstime-20", "password": "pass123" } ) page = browser.new_page() urls = [ "https://example.com/step1", "https://example.com/step2", "https://example.com/step3" ] for url in urls: page.goto(url) print(f"Scraped {url} -> Title: {page.title()}") browser.close() if __name__ == "__main__": run_sticky_session() ``` **Use Case:** Python multi-step form completion and account crawling requiring anti-bot protection and session IP continuity. ### 3. Geotargeting & Stealth Fingerprinting Demonstrates routing Python CloakBrowser traffic through a German residential proxy while leveraging stealth browser features. ```python from cloakbrowser import launch def geotargeted_stealth(): browser = launch( headless=True, proxy={ "server": "http://proxy.mrscraper.com:10000", "username": "user-country-de-sessid-shop1", "password": "pass123" } ) page = browser.new_page() page.goto("https://httpbin.org/headers") print("Stealth Browser Headers:\n", page.text_content("body")) browser.close() if __name__ == "__main__": geotargeted_stealth() ``` **Use Case:** Accessing heavily protected international websites that combine WAF anti-bot protections with geo-restrictions. ### 4. Production Error Handling & Lifecycle Management Provides robust Python error handling, custom navigation timeout configuration, response validation, and guaranteed teardown. ```python from cloakbrowser import launch def safe_stealth_scrape(target_url): browser = None try: browser = launch( headless=True, proxy={ "server": "http://proxy.mrscraper.com:10000", "username": "user-country-us-sessid-prod1", "password": "pass123" } ) page = browser.new_page() page.set_default_timeout(30000) page.goto(target_url) print("Successfully loaded page title:", page.title()) except Exception as error: print(f"CloakBrowser Python execution failed: {error}") finally: if browser: browser.close() if __name__ == "__main__": safe_stealth_scrape("https://example.com") ``` **Use Case:** Production Python automation scripts requiring clean error logging and reliable browser process cleanup. * CloakBrowser follows standard Playwright page and navigation APIs in both JavaScript and Python. * Run `npx cloakbrowser install` or `python -m cloakbrowser install` prior to execution to download stealth binaries. * Keep sensitive proxy credentials in environment variables (`process.env.PROXY_PASSWORD` or `os.environ.get("PROXY_PASSWORD")`). # cURL Command-line examples for testing Residential Proxy configurations quickly. ## SOCKS5 Proxy Setup Every example on this page uses the HTTP endpoint on port `10000`. SOCKS5 is supported too, on port `10001` — add the scheme to `-x`: ```bash # SOCKS5 with DNS resolved by the proxy curl -x "socks5h://user-country-us:pass123@proxy.mrscraper.com:10001" https://api.ipify.org # Equivalent using the dedicated flag curl --socks5-hostname proxy.mrscraper.com:10001 \ --proxy-user "user-country-us:pass123" \ https://api.ipify.org ``` Use port `10001` for SOCKS5. Prefer `socks5h://` (or `--socks5-hostname`) so hostnames are resolved by the proxy instead of locally. Username parameters such as `-country-` and `-sessid-` behave identically on both protocols. ## 1. Basic Request The simplest way to test Residential Proxy connectivity. This command routes your request through a US proxy and returns the proxy's IP address. ```bash # Test proxy connection curl -x "user-country-us:pass123@proxy.mrscraper.com:10000" https://api.ipify.org ``` **Use Case:** Quick connectivity testing and verifying your proxy credentials are working correctly. ## 2. Static Session Request Demonstrates session-based proxying where the same session ID (`sessid-test1`) maintains the same IP address across multiple requests. ```bash # Same IP across multiple requests curl -x "user-country-us-sessid-test1:pass123@proxy.mrscraper.com:10000" https://example.com # Verify same IP curl -x "user-country-us-sessid-test1:pass123@proxy.mrscraper.com:10000" https://api.ipify.org ``` **Use Case:** Testing scenarios where you need to maintain session state, like logging into websites or simulating a single user's browsing session. ## 3. Rotating Proxy Test This loop demonstrates IP rotation by making multiple requests without a session ID. Each request will come from a different IP address. ```bash # Test IP rotation - each request gets different IP for i in {1..5}; do echo "Request $i:" curl -x "user-country-gb:pass123@proxy.mrscraper.com:10000" https://api.ipify.org echo "" done ``` **Use Case:** Verifying that IP rotation is working correctly and understanding the different IPs available in your proxy pool. ## 4. Custom Headers Shows how to combine proxy usage with custom HTTP headers to appear more like a real browser from a specific region. ```bash # Add custom user agent and headers curl -x "user-country-de:pass123@proxy.mrscraper.com:10000" \ -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)" \ -H "Accept-Language: de-DE" \ https://example.com ``` **Use Case:** Scraping geo-specific content or avoiding detection by matching your headers to your proxy's location. ## 5. Save Response Captures the complete HTTP response body and saves it to a local file for later analysis or processing. ```bash # Save scraped content to file curl -x "user-country-jp-sessid-save1:pass123@proxy.mrscraper.com:10000" \ -o output.html \ https://example.com ``` **Use Case:** Batch downloading of pages, creating local copies of websites, or saving responses for offline analysis. ## 6. Verbose Output Enables detailed logging to troubleshoot connection issues, see SSL handshakes, and understand the complete request/response flow. ```bash # Debug proxy connection with verbose output curl -v -x "user-country-fr:pass123@proxy.mrscraper.com:10000" \ https://api.ipify.org ``` **Use Case:** Debugging proxy connection problems, understanding SSL certificate issues, or verifying that requests are properly routed through the proxy. ## 7. POST Request Demonstrates sending POST requests with JSON data through a proxy, useful for API interactions and form submissions. ```bash # POST request through proxy curl -x "user-country-ca-sessid-post1:pass123@proxy.mrscraper.com:10000" \ -X POST \ -H "Content-Type: application/json" \ -d '{"key":"value"}' \ https://api.example.com/endpoint ``` **Use Case:** Testing APIs through proxies, submitting forms, or sending data to web services while maintaining anonymity or bypassing geo-restrictions. ## 8. Custom Session Duration Controls how long your proxy session remains active using the `sesstime` parameter, extending beyond the default 10-minute session time. ```bash # 30-minute session curl -x "user-country-au-sessid-long1-sesstime-30:pass123@proxy.mrscraper.com:10000" \ https://example.com ``` **Use Case:** Long-running scripts that need to maintain the same IP for extended periods, such as multi-step authentication flows or comprehensive site crawling. ## 9. Multiple Requests Script A practical bash script that demonstrates bulk scraping with status monitoring and rate limiting. Each URL gets a different IP due to proxy rotation. ```bash #!/bin/bash # Script to scrape multiple URLs with rotating proxy PROXY="user-country-us:pass123@proxy.mrscraper.com:10000" urls=( "https://example.com/page1" "https://example.com/page2" "https://example.com/page3" "https://example.com/page4" "https://example.com/page5" ) for url in "${urls[@]}"; do echo "Scraping: $url" curl -x "$PROXY" -s "$url" -o /dev/null -w "Status: %{http_code}\n" sleep 1 # Delay between requests done ``` **Use Case:** Batch processing multiple URLs with different IPs, monitoring success rates, and implementing respectful scraping practices with delays. ## 10. Error Handling Production-ready error handling with timeouts and status code checking. Essential for reliable automated scraping operations. ```bash # Check connection and handle errors if curl -x "user-country-nl:pass123@proxy.mrscraper.com:10000" \ --connect-timeout 30 \ --max-time 60 \ -f -s -o /dev/null https://example.com; then echo "Success" else echo "Failed with exit code: $?" fi ``` **Use Case:** Building robust automation scripts that can detect and handle network failures, timeouts, and HTTP errors gracefully. ## Useful cURL Options | Option | Description | | ------------------- | --------------------------------- | | `-x, --proxy` | Specify proxy server | | `-H, --header` | Add custom header | | `-o, --output` | Save output to file | | `-v, --verbose` | Show detailed connection info | | `-s, --silent` | Silent mode (no progress bar) | | `-f, --fail` | Fail silently on HTTP errors | | `--connect-timeout` | Maximum time for connection | | `--max-time` | Maximum time for entire operation | | `-L, --location` | Follow redirects | | `-A, --user-agent` | Set user agent string | # Go Practical Go examples for integrating Residential Proxy into your scraping applications. ## SOCKS5 Proxy Setup Every example on this page uses the HTTP endpoint on port `10000`. SOCKS5 is supported too, on port `10001`. `net/http` handles it natively — only the scheme and port change: ```go package main import ( "fmt" "io" "net/http" "net/url" ) func main() { // Same credentials, SOCKS5 scheme and port proxyURL, _ := url.Parse("socks5://user-country-us:pass123@proxy.mrscraper.com:10001") client := &http.Client{ Transport: &http.Transport{ Proxy: http.ProxyURL(proxyURL), }, } resp, err := client.Get("https://api.ipify.org") if err != nil { fmt.Println("Error:", err) return } defer resp.Body.Close() body, _ := io.ReadAll(resp.Body) fmt.Println("Your IP:", string(body)) } ``` Use port `10001` for SOCKS5. `http.Transport` recognises the `socks5` scheme and passes the username and password from the URL as SOCKS authentication. Username parameters such as `-country-` and `-sessid-` behave identically on both protocols. ## 1. Basic HTTP Request This example demonstrates the fundamental setup for using Residential Proxy with Go's `net/http` package. It configures a static proxy session and makes a simple GET request. ```go package main import ( "fmt" "io" "net/http" "net/url" ) func main() { // Static proxy proxyURL, _ := url.Parse("http://user-country-us-sessid-go1:pass123@proxy.mrscraper.com:10000") client := &http.Client{ Transport: &http.Transport{ Proxy: http.ProxyURL(proxyURL), }, } resp, err := client.Get("https://api.ipify.org") if err != nil { fmt.Println("Error:", err) return } defer resp.Body.Close() body, _ := io.ReadAll(resp.Body) fmt.Println("Your IP:", string(body)) } ``` **Use Case:** Perfect for simple proxy testing, IP verification, and single-request scenarios where you need a consistent IP address through a static session. ## 2. Rotating Proxy for Scraping Shows how to create a reusable scraping function that uses rotating proxies (no session ID). Each call gets a fresh IP address, ideal for avoiding rate limits. ```go package main import ( "fmt" "io" "net/http" "net/url" ) func scrapeWithProxy(targetURL string) error { // Rotating proxy - new IP per request proxyURL, _ := url.Parse("http://user-country-gb:pass123@proxy.mrscraper.com:10000") client := &http.Client{ Transport: &http.Transport{ Proxy: http.ProxyURL(proxyURL), }, } resp, err := client.Get(targetURL) if err != nil { return fmt.Errorf("failed to fetch %s: %w", targetURL, err) } defer resp.Body.Close() body, _ := io.ReadAll(resp.Body) fmt.Printf("Scraped %s: %d bytes (status: %d)\n", targetURL, len(body), resp.StatusCode) return nil } func main() { urls := []string{ "https://example.com/page1", "https://example.com/page2", "https://example.com/page3", } for _, url := range urls { if err := scrapeWithProxy(url); err != nil { fmt.Println("Error:", err) } } } ``` **Use Case:** High-volume scraping where IP diversity is crucial for avoiding detection and rate limiting. Each page request appears to come from a different visitor. ## 3. Concurrent Scraping Demonstrates parallel scraping using goroutines with different static proxy sessions. Each goroutine maintains its own consistent IP address for the duration of its work. ```go package main import ( "fmt" "io" "net/http" "net/url" "sync" ) func scrapeWithProxy(targetURL string, proxyAuth string, wg *sync.WaitGroup) { defer wg.Done() proxyURL, _ := url.Parse(fmt.Sprintf("http://%s@proxy.mrscraper.com:10000", proxyAuth)) client := &http.Client{ Transport: &http.Transport{ Proxy: http.ProxyURL(proxyURL), }, } resp, err := client.Get(targetURL) if err != nil { fmt.Printf("Error scraping %s: %v\n", targetURL, err) return } defer resp.Body.Close() body, _ := io.ReadAll(resp.Body) fmt.Printf("Scraped %s: %d bytes\n", targetURL, len(body)) } func main() { urls := []string{ "https://example.com/page1", "https://example.com/page2", "https://example.com/page3", } var wg sync.WaitGroup for i, targetURL := range urls { wg.Add(1) proxyAuth := fmt.Sprintf("user-country-fr-sessid-worker%d:pass123", i+1) go scrapeWithProxy(targetURL, proxyAuth, &wg) } wg.Wait() fmt.Println("All scraping completed") } ``` **Use Case:** High-performance scraping that needs to process multiple URLs quickly while maintaining separate identities. Great for scraping sites with per-IP rate limits.**Use Case:** High-performance scraping that needs to process multiple URLs quickly while maintaining separate identities. Great for scraping sites with per-IP rate limits. ## 4. Error Handling & Retries Implements robust error handling with exponential backoff retry logic. Essential for production scraping applications that need to handle network failures gracefully. ```go package main import ( "fmt" "io" "math" "net/http" "net/url" "time" ) func scrapeWithRetry(targetURL string, maxRetries int) ([]byte, error) { proxyURL, _ := url.Parse("http://user-country-de:pass123@proxy.mrscraper.com:10000") client := &http.Client{ Transport: &http.Transport{ Proxy: http.ProxyURL(proxyURL), }, Timeout: 30 * time.Second, } var lastErr error for attempt := 0; attempt < maxRetries; attempt++ { resp, err := client.Get(targetURL) if err != nil { lastErr = err backoff := time.Duration(math.Pow(2, float64(attempt))) * time.Second fmt.Printf("Attempt %d failed: %v. Retrying in %v...\n", attempt+1, err, backoff) time.Sleep(backoff) continue } defer resp.Body.Close() if resp.StatusCode >= 200 && resp.StatusCode < 300 { body, _ := io.ReadAll(resp.Body) return body, nil } lastErr = fmt.Errorf("HTTP %d", resp.StatusCode) backoff := time.Duration(math.Pow(2, float64(attempt))) * time.Second fmt.Printf("Attempt %d returned %d. Retrying in %v...\n", attempt+1, resp.StatusCode, backoff) time.Sleep(backoff) } return nil, fmt.Errorf("failed after %d attempts: %w", maxRetries, lastErr) } func main() { body, err := scrapeWithRetry("https://example.com", 3) if err != nil { fmt.Println("Failed:", err) } else { fmt.Printf("Success: %d bytes\n", len(body)) } } ``` **Use Case:** Production-ready scraping that must handle temporary network issues, server errors, and proxy failures without crashing or losing data. ## 5. Custom Headers Shows how to add realistic browser headers to your requests when using proxies. This helps avoid detection by making requests appear more legitimate. ```go package main import ( "fmt" "io" "net/http" "net/url" ) func scrapeWithHeaders(targetURL string) error { proxyURL, _ := url.Parse("http://user-country-jp:pass123@proxy.mrscraper.com:10000") client := &http.Client{ Transport: &http.Transport{ Proxy: http.ProxyURL(proxyURL), }, } req, _ := http.NewRequest("GET", targetURL, nil) // Add custom headers req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)") req.Header.Set("Accept-Language", "ja-JP,ja;q=0.9") req.Header.Set("Accept", "text/html,application/xhtml+xml") resp, err := client.Do(req) if err != nil { return err } defer resp.Body.Close() body, _ := io.ReadAll(resp.Body) fmt.Printf("Scraped %d bytes with status %d\n", len(body), resp.StatusCode) return nil } func main() { if err := scrapeWithHeaders("https://example.com"); err != nil { fmt.Println("Error:", err) } } ``` **Use Case:** Scraping sites that check for browser-like behavior or when you need to match your headers to your proxy's geographic location for consistency. ## 6. Context with Timeout Demonstrates using Go's context package for proper timeout handling and request cancellation. This prevents requests from hanging indefinitely. ```go package main import ( "context" "fmt" "io" "net/http" "net/url" "time" ) func scrapeWithContext(targetURL string) error { proxyURL, _ := url.Parse("http://user-country-ca-sessid-ctx1:pass123@proxy.mrscraper.com:10000") client := &http.Client{ Transport: &http.Transport{ Proxy: http.ProxyURL(proxyURL), }, } // Create context with 30-second timeout ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel() req, _ := http.NewRequestWithContext(ctx, "GET", targetURL, nil) resp, err := client.Do(req) if err != nil { return fmt.Errorf("request failed: %w", err) } defer resp.Body.Close() body, _ := io.ReadAll(resp.Body) fmt.Printf("Scraped %d bytes\n", len(body)) return nil } func main() { if err := scrapeWithContext("https://example.com"); err != nil { fmt.Println("Error:", err) } } ``` **Use Case:** Applications that need precise control over request timeouts and graceful cancellation, especially in concurrent scenarios or when dealing with unreliable target sites. ## 7. Rate Limiting Implements respectful scraping with controlled request rates using Go's time.Ticker. This prevents overwhelming target servers while maintaining steady throughput. ```go package main import ( "fmt" "io" "net/http" "net/url" "time" ) func scrapeWithRateLimit(urls []string, requestsPerMinute int) { proxyURL, _ := url.Parse("http://user-country-au:pass123@proxy.mrscraper.com:10000") client := &http.Client{ Transport: &http.Transport{ Proxy: http.ProxyURL(proxyURL), }, } delay := time.Minute / time.Duration(requestsPerMinute) ticker := time.NewTicker(delay) defer ticker.Stop() for _, targetURL := range urls { <-ticker.C // Wait for next tick resp, err := client.Get(targetURL) if err != nil { fmt.Printf("Error scraping %s: %v\n", targetURL, err) continue } body, _ := io.ReadAll(resp.Body) resp.Body.Close() fmt.Printf("Scraped %s: %d bytes\n", targetURL, len(body)) } } func main() { urls := []string{ "https://example.com/page1", "https://example.com/page2", "https://example.com/page3", } scrapeWithRateLimit(urls, 20) // 20 requests per minute } ``` **Use Case:** Ethical scraping that respects server resources and terms of service. Essential for long-running scraping operations that need to avoid being blocked. # Node.js Practical Node.js examples for integrating Residential Proxy into your scraping and automation projects. ## Prerequisites ```bash npm install axios cheerio puppeteer ``` * `axios`: Promise-based HTTP client for making requests with built-in proxy support and interceptors. * `cheerio`: Server-side jQuery implementation for parsing and manipulating HTML responses. * `puppeteer`: Headless Chrome automation library for full browser simulation with proxy support. ## SOCKS5 Proxy Setup Every example on this page uses the HTTP endpoint on port `10000`. SOCKS5 is supported too, on port `10001`. The axios `proxy` option only speaks HTTP, so SOCKS5 goes through an agent instead: ```bash npm install socks-proxy-agent ``` ```javascript const axios = require('axios'); const { SocksProxyAgent } = require('socks-proxy-agent'); // Same credentials, SOCKS5 scheme and port const agent = new SocksProxyAgent( 'socks5://user-country-us-sessid-session1:pass123@proxy.mrscraper.com:10001' ); const response = await axios.get('https://api.ipify.org', { httpAgent: agent, httpsAgent: agent, proxy: false, }); console.log('Your IP:', response.data); ``` Use port `10001` for SOCKS5. Keep `proxy: false` so axios does not layer its own HTTP proxy on top of the agent. Username parameters such as `-country-` and `-sessid-` behave identically on both protocols. ## 1. Basic Static Proxy Demonstrates the simplest proxy setup with axios using a static session. The same IP address is maintained across requests using the session ID parameter. ```javascript const axios = require('axios'); // Static proxy configuration const proxy = { host: 'proxy.mrscraper.com', port: 10000, auth: { username: 'user-country-us-sessid-session1', password: 'pass123' } }; // Make a request async function testProxy() { try { const response = await axios.get('https://api.ipify.org', { proxy }); console.log('Your IP:', response.data); } catch (error) { console.error('Error:', error.message); } } testProxy(); ``` **Use Case:** Simple proxy testing and scenarios where you need consistent IP addresses for maintaining login sessions or avoiding duplicate detection. ## 2. Rotating Proxy with Axios Shows how to combine rotating proxies with Cheerio for HTML parsing. Each request gets a fresh IP address, perfect for scraping multiple pages without detection. ```javascript const axios = require('axios'); const cheerio = require('cheerio'); // Rotating proxy - new IP per request const rotatingProxy = { host: 'proxy.mrscraper.com', port: 10000, auth: { username: 'user-country-gb', password: 'pass123' } }; async function scrapePage(url) { try { const response = await axios.get(url, { proxy: rotatingProxy, timeout: 30000 }); const $ = cheerio.load(response.data); const title = $('h1').first().text(); console.log(`Scraped: ${title}`); return response.data; } catch (error) { console.error(`Error scraping ${url}:`, error.message); return null; } } // Scrape multiple pages (each with different IP) async function scrapeMultiple() { const urls = [ 'https://example.com/page1', 'https://example.com/page2', 'https://example.com/page3' ]; for (const url of urls) { await scrapePage(url); } } scrapeMultiple(); ``` **Use Case:** High-volume web scraping where IP rotation helps avoid rate limits and detection. Ideal for scraping e-commerce sites, job boards, or news sites. ## 3. Multiple Sessions (Promise.all) Demonstrates parallel scraping using Promise.all with different static proxy sessions. Each concurrent request maintains its own consistent IP address. ```javascript const axios = require('axios'); // Create array of proxy configs with different sessions const proxies = Array.from({ length: 5 }, (_, i) => ({ host: 'proxy.mrscraper.com', port: 10000, auth: { username: `user-country-de-sessid-worker${i + 1}`, password: 'pass123' } })); async function scrapeWithProxy(proxy, url) { try { const response = await axios.get(url, { proxy, timeout: 30000 }); return { status: response.status, length: response.data.length, url }; } catch (error) { return { error: error.message, url }; } } async function parallelScraping() { const urls = Array.from({ length: 10 }, (_, i) => `https://example.com/page${i + 1}` ); // Scrape in parallel using different sessions const promises = urls.map((url, index) => { const proxy = proxies[index % proxies.length]; return scrapeWithProxy(proxy, url); }); const results = await Promise.all(promises); console.log('Results:', results); } parallelScraping(); ``` **Use Case:** High-performance scraping that needs maximum throughput while maintaining multiple distinct identities. Great for scraping large datasets quickly. ## 4. Puppeteer with Proxy Shows how to use Residential Proxy with Puppeteer for full browser automation. This enables JavaScript rendering, screenshot capture, and interaction with dynamic content. ```javascript const puppeteer = require('puppeteer'); async function scrapeWithPuppeteer() { // Proxy URL format for Puppeteer const proxyUrl = 'http://user-country-jp-sessid-pup1:pass123@proxy.mrscraper.com:10000'; const browser = await puppeteer.launch({ headless: true, args: [ `--proxy-server=${proxyUrl}`, '--no-sandbox', '--disable-setuid-sandbox' ] }); try { const page = await browser.newPage(); // Navigate to page await page.goto('https://example.com', { waitUntil: 'networkidle2' }); // Extract data const title = await page.title(); console.log('Page title:', title); // Take screenshot await page.screenshot({ path: 'screenshot.png' }); } finally { await browser.close(); } } scrapeWithPuppeteer(); ``` **Use Case:** Scraping Single Page Applications (SPAs), taking screenshots for monitoring, or automating complex user interactions through a proxy. ## 5. Error Handling & Retries Implements robust error handling with exponential backoff retry logic. Essential for production applications that need to handle network failures gracefully. ```javascript const axios = require('axios'); const proxy = { host: 'proxy.mrscraper.com', port: 10000, auth: { username: 'user-country-fr', password: 'pass123' } }; async function scrapeWithRetry(url, maxRetries = 3) { for (let attempt = 0; attempt < maxRetries; attempt++) { try { const response = await axios.get(url, { proxy, timeout: 30000 }); return response.data; } catch (error) { console.log(`Attempt ${attempt + 1} failed: ${error.message}`); if (attempt < maxRetries - 1) { const delay = Math.pow(2, attempt) * 1000; // Exponential backoff await new Promise(resolve => setTimeout(resolve, delay)); } } } console.log(`Failed after ${maxRetries} attempts`); return null; } // Use with retry logic scrapeWithRetry('https://example.com') .then(data => console.log('Success:', data ? 'Got data' : 'Failed')); ``` **Use Case:** Production-ready scraping that must handle temporary network issues, server errors, and proxy failures without losing data or crashing. ## 6. Environment Variables Demonstrates secure credential management using environment variables. This keeps sensitive proxy credentials out of your source code. ```javascript const axios = require('axios'); require('dotenv').config(); // Use environment variables for credentials const proxy = { host: 'proxy.mrscraper.com', port: 10000, auth: { username: process.env.MRPROXY_USERNAME, password: process.env.MRPROXY_PASSWORD } }; async function secureScrape(url) { try { const response = await axios.get(url, { proxy }); return response.data; } catch (error) { console.error('Error:', error.message); return null; } } // .env file: // MRPROXY_USERNAME=user-country-us-sessid-session1 // MRPROXY_PASSWORD=pass123 ``` **Use Case:** Production deployments where credentials need to be kept secure and separate from code. Essential for CI/CD pipelines and team development. # Overview import { Code2, Terminal, FileCode, Theater, Bot, Zap, FlaskConical, ShieldCheck, Globe, LineChart, Bug } from 'lucide-react'; Ready-to-use code examples for integrating Residential Proxy into your applications. Choose your programming language to get started. ## Quick Reference ### Example Proxy Configurations Default (US) US Static GB Rotating Custom Session SOCKS5 ```bash # No country specified, falls back to US, rotating IP user:password@proxy.mrscraper.com:10000 ``` ```bash # US proxy with 10-minute session user-country-us-sessid-a1:password@proxy.mrscraper.com:10000 ``` ```bash # GB rotating proxy user-country-gb:password@proxy.mrscraper.com:10000 ``` ```bash # 30-minute session in Japan user-country-jp-sessid-x9-sesstime-30:password@proxy.mrscraper.com:10000 ``` ```bash # Same username format, SOCKS5 scheme and port socks5://user-country-us:password@proxy.mrscraper.com:10001 ``` The snippets on this page and in most language guides use the HTTP endpoint on port `10000` because it needs no extra dependencies. SOCKS5 is fully supported on port `10001` with the `socks5://` prefix, and every username parameter works the same on both. Each language page has a **SOCKS5 Proxy Setup** section with the client-specific setup, and [Proxy Types](/docs/residential-proxy/configuration/proxy-types) compares the two protocols. ## Examples by Language & Framework } href="/docs/residential-proxy/examples/python"> Examples with requests, BeautifulSoup, and asyncio } href="/docs/residential-proxy/examples/nodejs"> Examples with axios, Puppeteer, and native http } href="/docs/residential-proxy/examples/curl"> Command-line examples for quick testing } href="/docs/residential-proxy/examples/php"> Examples with cURL and Guzzle } href="/docs/residential-proxy/examples/ruby"> Examples with Net::HTTP and HTTParty } href="/docs/residential-proxy/examples/go"> Examples with net/http package } href="/docs/residential-proxy/examples/playwright"> Browser automation in JavaScript & Python } href="/docs/residential-proxy/examples/puppeteer"> Headless Chrome automation with page authentication } href="/docs/residential-proxy/examples/selenium"> Browser testing and scraping with selenium-wire } href="/docs/residential-proxy/examples/cloakbrowser"> Stealth browser automation in JS & Python ## Best Practices ### 1. Error Handling Always implement proper error handling and retry logic: * Catch connection timeouts * Handle proxy authentication failures * Retry failed requests with exponential backoff ### 2. Session Management * Use unique session IDs for parallel operations * Set appropriate session durations (10-60 minutes) * Reuse sessions for related requests ### 3. Rate Limiting * Respect target website's rate limits * Use rotating proxies for high-volume scraping * Add delays between requests when appropriate ### 4. Security * Never hardcode credentials in source code * Use environment variables for sensitive data * Rotate credentials regularly ## Common Use Cases }> Use rotating proxies with country targeting to scrape large amounts of data while avoiding rate limits and IP bans. }> Use static proxies with custom session duration to maintain consistent IPs for login flows and account operations. }> Use static or rotating proxies with country targeting to view localized content and pricing across different regions. }> Use static proxies with various country codes to test your application's behavior across different IP addresses and geographic locations. # PHP Practical PHP examples for integrating Residential Proxy into your web scraping projects. ## SOCKS5 Proxy Setup Every example on this page uses the HTTP endpoint on port `10000`. SOCKS5 is supported too, on port `10001` — set the proxy type explicitly with cURL, or use the `socks5h://` scheme with Guzzle: ```php get('https://api.ipify.org', [ 'proxy' => 'socks5h://user-country-us:pass123@proxy.mrscraper.com:10001', ]); ?> ``` Use port `10001` for SOCKS5. `CURLPROXY_SOCKS5_HOSTNAME` and `socks5h://` resolve hostnames through the proxy; `CURLPROXY_SOCKS5` and `socks5://` resolve them locally. Username parameters such as `-country-` and `-sessid-` behave identically on both protocols. ## 1. Basic cURL Request Demonstrates the fundamental cURL setup for using Residential Proxy with a static session. This example shows essential cURL options for proxy configuration and basic error handling. ```php ``` **Use Case:** Simple proxy testing and basic scraping tasks where you need a consistent IP address. Perfect for beginners learning PHP proxy integration. ## 2. Rotating Proxy for Scraping Shows how to create a reusable scraping function with rotating proxies. Each request gets a fresh IP address, and the function returns structured data for easy processing. ```php $httpCode, 'html' => $html ]; } // Scrape multiple pages $urls = [ 'https://example.com/page1', 'https://example.com/page2', 'https://example.com/page3' ]; foreach ($urls as $url) { $result = scrapeWithProxy($url); echo "Scraped {$url}: Status {$result['status']}\n"; } ?> ``` **Use Case:** High-volume scraping where IP diversity helps avoid detection and rate limiting. Ideal for scraping product catalogs, news sites, or job boards. ## 3. Multiple Sessions Implements an object-oriented approach with multiple static proxy sessions. The ProxyScraper class encapsulates proxy configuration and provides clean session management. ```php username = $username; $this->password = $password; } public function scrape($url) { $ch = curl_init($url); curl_setopt($ch, CURLOPT_PROXY, $this->proxy); curl_setopt($ch, CURLOPT_PROXYUSERPWD, "{$this->username}:{$this->password}"); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_TIMEOUT, 30); $response = curl_exec($ch); $info = curl_getinfo($ch); curl_close($ch); return [ 'status' => $info['http_code'], 'content' => $response ]; } } // Create multiple scrapers with different sessions $scrapers = []; for ($i = 1; $i <= 3; $i++) { $scrapers[] = new ProxyScraper("user-country-gb-sessid-worker{$i}", 'pass123'); } // Use different sessions $urls = ['https://example.com/page1', 'https://example.com/page2', 'https://example.com/page3']; foreach ($urls as $index => $url) { $scraper = $scrapers[$index % count($scrapers)]; $result = $scraper->scrape($url); echo "Scraped {$url}: {$result['status']}\n"; } ?> ``` **Use Case:** Organized scraping operations that need multiple consistent identities. Great for maintaining separate sessions for different data sources or user profiles. ## 4. Error Handling & Retries Implements robust retry logic with exponential backoff to handle network failures gracefully. This pattern is essential for production scraping applications. ```php = 200 && $httpCode < 300) { return $response; } echo "Attempt {$attempt} returned HTTP {$httpCode}\n"; sleep(pow(2, $attempt - 1)); } return false; } $result = scrapeWithRetry('https://example.com'); if ($result) { echo "Success!\n"; } else { echo "Failed after retries\n"; } ?> ``` **Use Case:** Production-ready scraping that must handle temporary network issues, server errors, and proxy failures without losing data or crashing the application. ## 5. Guzzle HTTP Client Demonstrates using the popular Guzzle library for more elegant HTTP requests. Guzzle provides better error handling and a more intuitive API compared to raw cURL. ```php 'http://user-country-jp-sessid-guzzle1:pass123@proxy.mrscraper.com:10000', 'timeout' => 30 ]); try { $response = $client->get('https://api.ipify.org'); echo 'Your IP: ' . $response->getBody(); } catch (RequestException $e) { echo 'Error: ' . $e->getMessage(); } ?> ``` **Use Case:** Modern PHP applications that prefer object-oriented HTTP clients with better exception handling and cleaner syntax than raw cURL. ## 6. Async Requests with Guzzle Shows how to make concurrent HTTP requests using Guzzle's promise-based async functionality. This dramatically improves performance for bulk scraping operations. ```php 'http://user-country-au:pass123@proxy.mrscraper.com:10000', 'timeout' => 30 ]); // Create array of promises $promises = [ 'page1' => $client->getAsync('https://example.com/page1'), 'page2' => $client->getAsync('https://example.com/page2'), 'page3' => $client->getAsync('https://example.com/page3'), ]; // Wait for all requests to complete $results = Promise\Utils::settle($promises)->wait(); // Process results foreach ($results as $key => $result) { if ($result['state'] === 'fulfilled') { echo "{$key}: Success\n"; } else { echo "{$key}: Failed - {$result['reason']}\n"; } } ?> ``` **Use Case:** High-performance scraping that needs maximum throughput. Perfect for bulk data collection where you need to process many URLs simultaneously. ## 7. Environment Variables Demonstrates secure credential management by loading proxy credentials from environment variables instead of hardcoding them in your source code. ```php ``` **Use Case:** Production deployments where credentials must be kept secure and separate from code. Essential for team development and CI/CD pipelines. # Playwright Practical Playwright examples for integrating Residential Proxy into browser automation and web scraping projects, separated into JavaScript (Node.js) and Python sections. ## JavaScript (Node.js) ### Prerequisites ```bash npm install playwright proxy-chain npx playwright install chromium ``` * `playwright`: Node.js library for browser automation supporting Chromium, Firefox, and WebKit with native proxy authentication. ### Proxy Format Playwright accepts proxy settings via the `proxy` launch option in JavaScript: ```javascript const proxy = { server: 'http://proxy.mrscraper.com:10000', username: 'user-country-us-sessid-session1', password: 'pass123', } ``` ### SOCKS5 Proxy Setup For SOCKS5 proxies, use `proxy-chain` to anonymize the authenticated SOCKS5 connection before passing the local proxy server to Playwright: ```javascript import { chromium } from 'playwright' import { anonymizeProxy, closeAnonymizedProxy } from 'proxy-chain' const username = 'user-country-us' const password = 'pass123' const socks5Proxy = `socks5://${encodeURIComponent(username)}:${encodeURIComponent(password)}` + `@proxy.mrscraper.com:10001` const localProxy = await anonymizeProxy(socks5Proxy) const browser = await chromium.launch({ proxy: { server: localProxy, }, }) try { const page = await browser.newPage() await page.goto('https://ipinfo.io/json') console.log(await page.textContent('body')) } finally { await browser.close() await closeAnonymizedProxy(localProxy, true) } ``` Use port `10001` for SOCKS5. Do not pass SOCKS5 credentials directly to Playwright; `proxy-chain` handles the authenticated upstream connection. ### 1. Basic Proxy Setup Demonstrates launching Headless Chromium with authenticated proxy credentials in JavaScript, inspecting the resulting external proxy IP. ```javascript import { chromium } from 'playwright' async function testProxy() { // Launch browser with proxy settings const browser = await chromium.launch({ headless: true, proxy: { server: 'http://proxy.mrscraper.com:10000', username: 'user-country-us', password: 'pass123', }, }) const page = await browser.newPage() // Navigate to IP inspection endpoint await page.goto('https://api.ipify.org?format=json') const content = await page.textContent('body') console.log('Your Proxy IP:', content) await browser.close() } testProxy().catch(console.error) ``` **Use Case:** Fundamental proxy connectivity check and baseline browser launch in Node.js. ### 2. Multi-Context Sticky Sessions Shows how to create isolated browser contexts on a single browser instance. Each context maintains its own consistent IP address using unique session IDs (`sessid`). ```javascript import { chromium } from 'playwright' async function runParallelSessions() { const browser = await chromium.launch({ headless: true }) const workerSessions = ['worker1', 'worker2', 'worker3'] // Create isolated context per session for (const session of workerSessions) { const context = await browser.newContext({ proxy: { server: 'http://proxy.mrscraper.com:10000', username: `user-country-us-sessid-${session}`, password: 'pass123', }, }) const page = await context.newPage() await page.goto('https://api.ipify.org?format=json') const ipData = await page.textContent('body') console.log(`Session [${session}]:`, ipData) await context.close() } await browser.close() } runParallelSessions().catch(console.error) ``` **Use Case:** Parallel web scraping where multiple workers maintain independent sticky sessions on a single browser process. ### 3. Geotargeting & Custom Session Duration Demonstrates routing Playwright traffic through a German proxy with a 30-minute static session duration (`sesstime-30`), configuring regional locale settings. ```javascript import { chromium } from 'playwright' async function geotargetedSession() { const browser = await chromium.launch({ headless: true, proxy: { server: 'http://proxy.mrscraper.com:10000', // German IP with 30-minute static duration username: 'user-country-de-sessid-shop1-sesstime-30', password: 'pass123', }, }) const context = await browser.newContext({ userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36', locale: 'de-DE', }) const page = await context.newPage() await page.goto('https://httpbin.org/headers') console.log(await page.textContent('body')) await browser.close() } geotargetedSession().catch(console.error) ``` **Use Case:** Testing localized e-commerce pricing, geo-restricted page content, and multi-step checkouts requiring static IPs. ### 4. Production Error Handling Production-ready error handling pattern for JavaScript Playwright operations, featuring timeout settings, response checks, and guaranteed resource cleanup. ```javascript import { chromium } from 'playwright' async function robustScraping(targetUrl) { let browser = null try { browser = await chromium.launch({ headless: true, proxy: { server: 'http://proxy.mrscraper.com:10000', username: 'user-country-us-sessid-task1', password: 'pass123', }, }) const page = await browser.newPage() page.setDefaultTimeout(30000) // 30s timeout const response = await page.goto(targetUrl, { waitUntil: 'networkidle' }) if (!response || !response.ok()) { throw new Error(`Failed to load page: ${response?.status()}`) } console.log('Page loaded successfully:', await page.title()) } catch (error) { console.error(`Scraping failed for ${targetUrl}:`, error.message) } finally { if (browser) { await browser.close() } } } robustScraping('https://example.com') ``` **Use Case:** Deploying enterprise JavaScript Playwright scrapers that handle timeouts and failures gracefully. *** ## Python ### Prerequisites ```bash pip install playwright playwright install chromium ``` * `playwright`: Python package offering synchronous and asynchronous browser automation bindings. * Install `sing-box` separately and make sure the `sing-box` command is available on your `PATH` for SOCKS5 proxy bridging. ### Proxy Format Playwright for Python passes proxy options via a dictionary to `chromium.launch()`: ```python proxy = { "server": "http://proxy.mrscraper.com:10000", "username": "user-country-us-sessid-session1", "password": "pass123" } ``` ### SOCKS5 Proxy Setup Playwright's Python integration uses `sing-box` to expose the authenticated SOCKS5 proxy as a local HTTP proxy: ```python import json import os import subprocess import tempfile import time from playwright.sync_api import sync_playwright username = "user-country-us" password = "pass123" local_proxy = "http://127.0.0.1:8080" singbox_config = { "inbounds": [ { "type": "http", "tag": "http-in", "listen": "127.0.0.1", "listen_port": 8080, } ], "outbounds": [ { "type": "socks", "tag": "mrscraper", "server": "proxy.mrscraper.com", "server_port": 10001, "username": username, "password": password, "version": "5", } ], "route": { "rules": [ { "action": "route", "outbound": "mrscraper", } ] }, } with tempfile.NamedTemporaryFile( mode="w", suffix=".json", delete=False, ) as config_file: json.dump(singbox_config, config_file) config_path = config_file.name print("Starting sing-box...") singbox_process = subprocess.Popen([ "sing-box", "run", "-c", config_path, ]) try: time.sleep(1) with sync_playwright() as playwright: browser = playwright.chromium.launch( proxy={ "server": local_proxy, } ) try: page = browser.new_page() page.goto( "https://ipinfo.io/json", wait_until="domcontentloaded", timeout=30000, ) print(page.text_content("body")) finally: browser.close() finally: singbox_process.terminate() singbox_process.wait() os.remove(config_path) ``` Use port `10001` for SOCKS5. The browser connects to the local HTTP proxy at `127.0.0.1:8080`; `sing-box` handles the authenticated SOCKS5 upstream. ### 1. Basic Proxy Setup (Sync API) Demonstrates the Python synchronous Playwright API for initializing an authenticated proxy connection and inspecting page output. ```python from playwright.sync_api import sync_playwright def test_proxy(): with sync_playwright() as p: # Launch browser with proxy server and credentials browser = p.chromium.launch( headless=True, proxy={ "server": "http://proxy.mrscraper.com:10000", "username": "user-country-us", "password": "pass123" } ) page = browser.new_page() page.goto("https://api.ipify.org?format=json") print("Your Proxy IP:", page.text_content("body")) browser.close() if __name__ == "__main__": test_proxy() ``` **Use Case:** Quick proxy integration for Python automation scripts, test suites, and data collection utilities. ### 2. Sticky Session Management (Sync API) Maintains a static Python Playwright browser session with a stable IP address across multi-step page navigations. ```python from playwright.sync_api import sync_playwright def run_sticky_session(): with sync_playwright() as p: # Static session IP in Canada browser = p.chromium.launch( headless=True, proxy={ "server": "http://proxy.mrscraper.com:10000", "username": "user-country-ca-sessid-auth1-sesstime-20", "password": "pass123" } ) context = browser.new_context() page = context.new_page() for url in ["https://example.com/step1", "https://example.com/step2"]: page.goto(url) print(f"Visited {url} -> Title: {page.title()}") browser.close() if __name__ == "__main__": run_sticky_session() ``` **Use Case:** Simulating multi-step user interactions and login sequences in Python requiring a consistent residential IP. ### 3. Rotating Proxy Scraping (Async API) Uses an asynchronous Python Playwright setup with rotating proxies to visit multiple target URLs, ensuring each request receives a fresh IP address. ```python import asyncio from playwright.async_api import async_playwright async def scrape_target(url, proxy_config): async with async_playwright() as p: browser = await p.chromium.launch(headless=True, proxy=proxy_config) page = await browser.new_page() try: await page.goto(url, wait_until="domcontentloaded", timeout=30000) title = await page.title() print(f"Scraped '{url}' -> Title: {title}") except Exception as e: print(f"Error scraping {url}: {e}") finally: await browser.close() async def main(): # Rotating proxy (no sessid parameter) proxy_config = { "server": "http://proxy.mrscraper.com:10000", "username": "user-country-gb", "password": "pass123" } urls = [ "https://example.com/page1", "https://example.com/page2", "https://example.com/page3", ] for url in urls: await scrape_target(url, proxy_config) if __name__ == "__main__": asyncio.run(main()) ``` **Use Case:** High-volume automated Python browser scraping where rotating IPs prevent target site rate limits. ### 4. Production Error Handling & Async Teardown Implements defensive Python async error trapping, default page timeout limits, and safe browser closing routines. ```python import asyncio from playwright.async_api import async_playwright async def safe_scrape(url): proxy_config = { "server": "http://proxy.mrscraper.com:10000", "username": "user-country-us-sessid-prod1", "password": "pass123" } async with async_playwright() as p: browser = await p.chromium.launch(headless=True, proxy=proxy_config) try: page = await browser.new_page() page.set_default_timeout(30000) response = await page.goto(url, wait_until="networkidle") if not response or not response.ok: raise RuntimeError(f"HTTP error {response.status if response else 'No response'}") print(f"Successfully scraped {url} - Length: {len(await page.content())}") except Exception as err: print(f"Failed to scrape {url}: {err}") finally: await browser.close() if __name__ == "__main__": asyncio.run(safe_scrape("https://example.com")) ``` **Use Case:** Enterprise Python async web scrapers operating in production environments with strict reliability requirements. * Define proxy credentials when calling `chromium.launch()` or `browser.newContext()` / `browser.new_context()`. * Use browser contexts to handle multiple proxy configurations within a single browser process. * Keep sensitive proxy credentials in environment variables (`process.env.PROXY_PASSWORD` or `os.environ.get("PROXY_PASSWORD")`). # Puppeteer Practical Puppeteer code examples for routing browser automation traffic through Residential Proxies using `page.authenticate()`. ## Prerequisites ```bash npm install puppeteer proxy-chain ``` * `puppeteer`: Node.js library providing a high-level API to control Headless Chrome or Chromium over the DevTools Protocol. ## Proxy Format Puppeteer passes the proxy server address via Chromium launch flags (`--proxy-server`), then authenticates using `page.authenticate()`. ```text username:password@proxy.mrscraper.com:10000 ``` For sticky sessions or targeted countries, format the username with parameters: ```text user-country-us-sessid-session1:password@proxy.mrscraper.com:10000 ``` ### SOCKS5 Proxy Setup For SOCKS5 proxies, use `proxy-chain` to anonymize the authenticated SOCKS5 connection before passing the local proxy server to Puppeteer: ```javascript import puppeteer from 'puppeteer' import { anonymizeProxy, closeAnonymizedProxy } from 'proxy-chain' const username = 'user-country-us' const password = 'pass123' const socks5Proxy = `socks5://${encodeURIComponent(username)}:${encodeURIComponent(password)}` + `@proxy.mrscraper.com:10001` const localProxy = await anonymizeProxy(socks5Proxy) const browser = await puppeteer.launch({ args: [`--proxy-server=${localProxy}`], }) try { const page = await browser.newPage() await page.goto('https://ipinfo.io/json', { waitUntil: 'networkidle2', }) console.log(await page.$eval('body', (el) => el.innerText)) } finally { await browser.close() await closeAnonymizedProxy(localProxy, true) } ``` Use port `10001` for SOCKS5. Always close the browser before calling `closeAnonymizedProxy()`. ## 1. Basic Proxy Setup Demonstrates the standard setup for launching Headless Chrome with proxy flags and authenticating page traffic. ```javascript import puppeteer from 'puppeteer' const proxyHost = 'proxy.mrscraper.com' const proxyPort = '10000' const proxyUsername = 'user-country-us' const proxyPassword = 'pass123' async function testProxy() { const browser = await puppeteer.launch({ headless: true, args: [`--proxy-server=http://${proxyHost}:${proxyPort}`], }) const page = await browser.newPage() // Authenticate page before navigating await page.authenticate({ username: proxyUsername, password: proxyPassword, }) await page.goto('https://api.ipify.org?format=json', { waitUntil: 'networkidle2', }) const content = await page.evaluate(() => document.body.innerText) console.log('Your Proxy IP:', content) await browser.close() } testProxy().catch(console.error) ``` **Use Case:** Quick proxy connectivity check and baseline authentication testing in Puppeteer. ## 2. Sticky Session Management Maintains a stable residential IP address across multi-step browser workflows using a consistent session ID (`sessid`) and explicit session duration (`sesstime-20`). ```javascript import puppeteer from 'puppeteer' async function runStickySession() { const browser = await puppeteer.launch({ headless: true, args: ['--proxy-server=http://proxy.mrscraper.com:10000'], }) const page = await browser.newPage() // 20-minute static session in Canada await page.authenticate({ username: 'user-country-ca-sessid-checkout1-sesstime-20', password: 'pass123', }) const urls = [ 'https://example.com/step1', 'https://example.com/step2', 'https://example.com/step3', ] for (const url of urls) { await page.goto(url, { waitUntil: 'domcontentloaded' }) console.log(`Visited ${url} - Title:`, await page.title()) } await browser.close() } runStickySession().catch(console.error) ``` **Use Case:** Form submissions, login workflows, and multi-page checkouts where maintaining the same IP address is mandatory. ## 3. Rotating Proxy Scraping Demonstrates rotating proxies with Puppeteer. Each request uses rotating proxy credentials to automatically fetch content with a different residential IP address. ```javascript import puppeteer from 'puppeteer' async function scrapeWithRotation(urls) { const browser = await puppeteer.launch({ headless: true, args: ['--proxy-server=http://proxy.mrscraper.com:10000'], }) for (const url of urls) { const page = await browser.newPage() // Rotating proxy - new IP per request await page.authenticate({ username: 'user-country-gb', password: 'pass123', }) try { await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 }) const h1Text = await page .$eval('h1', (el) => el.textContent) .catch(() => 'No H1') console.log(`Scraped [${url}]: ${h1Text}`) } catch (err) { console.error(`Failed to scrape ${url}:`, err.message) } finally { await page.close() } } await browser.close() } scrapeWithRotation([ 'https://example.com/page1', 'https://example.com/page2', 'https://example.com/page3', ]) ``` **Use Case:** High-volume scraping across multiple target pages without triggering IP rate limits or blocklists. ## 4. Geotargeting & Location Testing Demonstrates routing Puppeteer traffic through specific countries (Japan) while customizing HTTP headers and viewport to match target geographic personas. ```javascript import puppeteer from 'puppeteer' async function testLocalizedContent() { const browser = await puppeteer.launch({ headless: true, args: ['--proxy-server=http://proxy.mrscraper.com:10000', '--lang=ja-JP'], }) const page = await browser.newPage() // Target Japanese residential proxy await page.authenticate({ username: 'user-country-jp', password: 'pass123', }) await page.setExtraHTTPHeaders({ 'Accept-Language': 'ja-JP,ja;q=0.9', }) await page.goto('https://httpbin.org/headers') console.log(await page.evaluate(() => document.body.innerText)) await browser.close() } testLocalizedContent().catch(console.error) ``` **Use Case:** Auditing localized e-commerce pricing, geo-specific ad placement, and regional content availability. ## 5. Parallel Isolated Contexts Uses Incognito browser contexts (`createBrowserContext()`) to isolate multiple concurrent worker sessions, each authenticated with a unique proxy session ID. ```javascript import puppeteer from 'puppeteer' async function parallelWorkers() { const browser = await puppeteer.launch({ headless: true, args: ['--proxy-server=http://proxy.mrscraper.com:10000'], }) const workers = ['worker_a', 'worker_b', 'worker_c'] const tasks = workers.map(async (workerId) => { // Isolated incognito context const context = await browser.createBrowserContext() const page = await context.newPage() await page.authenticate({ username: `user-country-us-sessid-${workerId}`, password: 'pass123', }) await page.goto('https://api.ipify.org?format=json') const ip = await page.evaluate(() => document.body.innerText) console.log(`Worker ${workerId} IP:`, ip) await context.close() }) await Promise.all(tasks) await browser.close() } parallelWorkers().catch(console.error) ``` **Use Case:** Scaling automated browser tasks in parallel with dedicated IP addresses per thread without launching multiple browser instances. ## 6. Production Error Handling & Cleanup Provides robust exception handling, custom navigation timeouts, response validation, and safe browser shutdown routines. ```javascript import puppeteer from 'puppeteer' async function safeScrape(url) { let browser = null try { browser = await puppeteer.launch({ headless: true, args: ['--proxy-server=http://proxy.mrscraper.com:10000'], }) const page = await browser.newPage() page.setDefaultNavigationTimeout(30000) await page.authenticate({ username: 'user-country-us-sessid-prod1', password: 'pass123', }) const response = await page.goto(url, { waitUntil: 'networkidle2' }) if (!response || !response.ok()) { throw new Error(`HTTP Error ${response?.status()}`) } console.log('Successfully fetched:', await page.title()) } catch (error) { console.error('Scraping error:', error.message) } finally { if (browser) { await browser.close() } } } safeScrape('https://example.com') ``` **Use Case:** Deploying production Puppeteer bots that handle network glitches, authentication errors, and timeouts gracefully. * Call `page.authenticate()` before invoking `page.goto()`. * Use `--proxy-server=http://...` in `launch.args` to direct Puppeteer traffic. * Use `createBrowserContext()` for parallel isolated sessions with unique proxy usernames. # Python Practical Python examples for integrating Residential Proxy into your scraping and automation projects. ## Prerequisites ```bash pip install requests beautifulsoup4 ``` * `requests` : Human-friendly HTTP client used to send requests through Residential Proxy, control timeouts, and access response data. * `beautifulsoup4` : HTML parser used together with `requests` to turn raw HTML into a searchable object model for extracting elements like titles, links, and product data. ## SOCKS5 Proxy Setup Every example on this page uses the HTTP endpoint on port `10000`. SOCKS5 is supported too, on port `10001`. `requests` routes SOCKS traffic only when the `socks` extra is installed: ```bash pip install "requests[socks]" ``` ```python import requests # Same credentials, SOCKS5 scheme and port proxy = { 'http': 'socks5h://user-country-us-sessid-session1:pass123@proxy.mrscraper.com:10001', 'https': 'socks5h://user-country-us-sessid-session1:pass123@proxy.mrscraper.com:10001' } response = requests.get('https://api.ipify.org', proxies=proxy) print(f"Your IP: {response.text}") ``` Use port `10001` for SOCKS5. Prefer `socks5h://` so hostnames are resolved by the proxy instead of locally. Username parameters such as `-country-` and `-sessid-` behave identically on both protocols. ## 1. Basic Static Proxy This example demonstrates how to use a static proxy with a consistent session. The same IP address will be maintained throughout your requests as long as you use the same session ID. ```python import requests # Static proxy with session proxy = { 'http': 'http://user-country-us-sessid-session1:pass123@proxy.mrscraper.com:10000', 'https': 'http://user-country-us-sessid-session1:pass123@proxy.mrscraper.com:10000' } # Make a request response = requests.get('https://api.ipify.org', proxies=proxy) print(f"Your IP: {response.text}") ``` **Use Case:** Ideal for scenarios where you need to maintain the same IP across multiple requests, such as logging into websites or maintaining session state. ## 2. Rotating Proxy for Web Scraping Rotating proxies automatically change your IP address with each request. This is perfect for scraping multiple pages without being detected or rate-limited. ```python import requests from bs4 import BeautifulSoup # Rotating proxy - new IP each request proxy = { 'http': 'http://user-country-us:pass123@proxy.mrscraper.com:10000', 'https': 'http://user-country-us:pass123@proxy.mrscraper.com:10000' } def scrape_page(url): """Scrape a page using rotating proxy""" try: response = requests.get(url, proxies=proxy, timeout=30) soup = BeautifulSoup(response.text, 'html.parser') # Extract data title = soup.find('h1').text if soup.find('h1') else 'No title' print(f"Scraped: {title}") return soup except Exception as e: print(f"Error: {e}") return None # Scrape multiple pages (each with different IP) urls = [ 'https://example.com/page1', 'https://example.com/page2', 'https://example.com/page3', ] for url in urls: scrape_page(url) ``` **Use Case:** Best for high-volume scraping where you need to avoid IP-based rate limits or blocks. Each page appears to come from a different visitor. ## 3. Multiple Static Sessions (Parallel) This advanced example shows how to scrape multiple pages simultaneously using different static proxy sessions. Each worker thread maintains its own consistent IP address. ```python import requests from concurrent.futures import ThreadPoolExecutor, as_completed # Create multiple session-based proxies proxies = [ { 'http': f'http://user-country-us-sessid-worker{i}:pass123@proxy.mrscraper.com:10000', 'https': f'http://user-country-us-sessid-worker{i}:pass123@proxy.mrscraper.com:10000' } for i in range(1, 6) # 5 different sessions ] def scrape_with_proxy(proxy_config, url): """Scrape using a specific proxy session""" try: response = requests.get(url, proxies=proxy_config, timeout=30) return { 'status': response.status_code, 'length': len(response.text), 'url': url } except Exception as e: return {'error': str(e), 'url': url} # URLs to scrape urls = [f'https://example.com/page{i}' for i in range(1, 11)] # Scrape in parallel using different sessions with ThreadPoolExecutor(max_workers=5) as executor: futures = [] for url in urls: # Rotate through available proxies proxy = proxies[len(futures) % len(proxies)] future = executor.submit(scrape_with_proxy, proxy, url) futures.append(future) # Collect results for future in as_completed(futures): result = future.result() print(f"Result: {result}") ``` **Use Case:** Perfect for high-performance scraping where you need to make many requests quickly while maintaining multiple consistent identities. Great for scraping sites with per-IP rate limits. ## 4. Session with Custom Duration Control how long your proxy session remains active using the `sesstime` parameter. This is useful for extended scraping operations where you need to maintain the same IP for a specific period. ```python import requests import time # 20-minute session for extended scraping long_session_proxy = { 'http': 'http://user-country-ca-sessid-long1-sesstime-20:pass123@proxy.mrscraper.com:10000', 'https': 'http://user-country-ca-sessid-long1-sesstime-20:pass123@proxy.mrscraper.com:10000' } def long_scraping_session(urls): """Scrape multiple pages with same IP over 20 minutes""" print("Starting long scraping session...") for i, url in enumerate(urls): try: response = requests.get(url, proxies=long_session_proxy, timeout=30) print(f"[{i+1}/{len(urls)}] Scraped {url}: {response.status_code}") # Verify IP stays the same ip = requests.get('https://api.ipify.org', proxies=long_session_proxy).text print(f" Current IP: {ip}") # Wait between requests time.sleep(60) # 1 minute between requests except Exception as e: print(f"Error on {url}: {e}") # Scrape 15 pages over 15 minutes (within 20-minute session) urls = [f'https://example.com/page{i}' for i in range(1, 16)] long_scraping_session(urls) ``` **Use Case:** Ideal for scraping workflows that require extended session times, such as crawling paginated content, navigating multi-step forms, or monitoring a site over time without IP changes. ## 5. Error Handling & Retries Production-ready scraping requires robust error handling. This example implements automatic retries with exponential backoff to handle network issues, timeouts, and proxy errors gracefully. ```python import requests from time import sleep proxy = { 'http': 'http://user-country-gb:pass123@proxy.mrscraper.com:10000', 'https': 'http://user-country-gb:pass123@proxy.mrscraper.com:10000' } def scrape_with_retry(url, max_retries=3): """Scrape with automatic retry on failure""" for attempt in range(max_retries): try: response = requests.get(url, proxies=proxy, timeout=30) response.raise_for_status() # Raise exception for bad status codes return response.text except requests.exceptions.ProxyError: print(f"Proxy error on attempt {attempt + 1}") sleep(2 ** attempt) # Exponential backoff except requests.exceptions.Timeout: print(f"Timeout on attempt {attempt + 1}") sleep(2 ** attempt) except Exception as e: print(f"Error on attempt {attempt + 1}: {e}") sleep(2 ** attempt) print(f"Failed after {max_retries} attempts") return None # Use with retry logic result = scrape_with_retry('https://example.com') ``` **Use Case:** Essential for reliable scraping operations. The exponential backoff prevents overwhelming servers while giving temporary issues time to resolve. Always use retry logic in production environments. # Ruby Practical Ruby examples for integrating Residential Proxy into your scraping projects. ## Prerequisites ```bash gem install httparty ``` * **HTTParty**: Popular Ruby gem that provides a simple, elegant API for HTTP requests with built-in proxy configuration. ## SOCKS5 Proxy Setup Every example on this page uses the HTTP endpoint on port `10000`. SOCKS5 is supported too, on port `10001`. `Net::HTTP` has no native SOCKS support, so it needs the `socksify` gem: ```bash gem install socksify ``` ```ruby require 'socksify/http' require 'uri' # Same credentials, SOCKS5 port TCPSocket.socks_username = 'user-country-us' TCPSocket.socks_password = 'pass123' uri = URI.parse('https://api.ipify.org') response = Net::HTTP.SOCKSProxy('proxy.mrscraper.com', 10001) .start(uri.host, uri.port, use_ssl: true) do |http| http.request(Net::HTTP::Get.new(uri.request_uri)) end puts "Your IP: #{response.body}" ``` Use port `10001` for SOCKS5. `socksify` sends the hostname to the proxy for resolution. Username parameters such as `-country-` and `-sessid-` behave identically on both protocols. ## 1. Basic HTTP Request Demonstrates the fundamental setup using Ruby's built-in Net::HTTP library with a static proxy session. This example shows the essential proxy configuration parameters. ```ruby require 'net/http' require 'uri' # Static proxy configuration proxy_uri = URI.parse('http://user-country-us-sessid-ruby1:pass123@proxy.mrscraper.com:10000') uri = URI.parse('https://api.ipify.org') http = Net::HTTP.new(uri.host, uri.port, proxy_uri.host, proxy_uri.port, proxy_uri.user, proxy_uri.password) http.use_ssl = true request = Net::HTTP::Get.new(uri.request_uri) response = http.request(request) puts "Your IP: #{response.body}" ``` **Use Case:** Simple proxy testing and basic scraping tasks where you need a consistent IP address. Perfect for understanding Ruby's proxy configuration fundamentals. ## 2. Rotating Proxy with HTTParty Shows how to use the HTTParty gem for cleaner proxy configuration with rotating IPs. The class-based approach provides reusable scraping functionality with automatic IP rotation. ```ruby require 'httparty' class ProxyScraper include HTTParty # Rotating proxy http_proxy 'proxy.mrscraper.com', 10000, 'user-country-jp', 'pass123' def self.scrape(url) response = get(url, timeout: 30) { status: response.code, body: response.body } rescue => e { error: e.message } end end # Scrape multiple pages urls = [ 'https://example.com/page1', 'https://example.com/page2', 'https://example.com/page3' ] urls.each do |url| result = ProxyScraper.scrape(url) puts "Scraped #{url}: #{result[:status]}" end ``` **Use Case:** High-volume scraping where IP diversity helps avoid detection and rate limiting. HTTParty's clean syntax makes it ideal for production scraping applications. ## 3. Multiple Sessions Implements an object-oriented approach with multiple static proxy sessions. Each SessionScraper instance maintains its own consistent IP address throughout its lifetime. ```ruby require 'net/http' require 'uri' class SessionScraper def initialize(session_id, password) @username = "user-country-gb-sessid-#{session_id}" @password = password @proxy_uri = URI.parse("http://#{@username}:#{@password}@proxy.mrscraper.com:10000") end def scrape(url) uri = URI.parse(url) http = Net::HTTP.new(uri.host, uri.port, @proxy_uri.host, @proxy_uri.port, @proxy_uri.user, @proxy_uri.password) http.use_ssl = true request = Net::HTTP::Get.new(uri.request_uri) response = http.request(request) { status: response.code, body_length: response.body.length } rescue => e { error: e.message } end end # Create multiple scrapers scrapers = (1..3).map { |i| SessionScraper.new("worker#{i}", 'pass123') } urls = [ 'https://example.com/page1', 'https://example.com/page2', 'https://example.com/page3' ] urls.each_with_index do |url, index| scraper = scrapers[index % scrapers.length] result = scraper.scrape(url) puts "Scraped #{url}: #{result}" end ``` **Use Case:** Organized scraping operations that need multiple consistent identities. Great for maintaining separate sessions for different data sources or simulating multiple users. ## 4. Error Handling & Retries Implements robust retry logic with exponential backoff using HTTParty's proxy configuration options. Essential for production applications that need to handle failures gracefully. ```ruby require 'httparty' class RobustScraper include HTTParty def self.scrape_with_retry(url, max_retries = 3) proxy_config = { http_proxyaddr: 'proxy.mrscraper.com', http_proxyport: 10000, http_proxyuser: 'user-country-de', http_proxypass: 'pass123' } max_retries.times do |attempt| begin response = get(url, proxy_config.merge(timeout: 30)) return response if response.success? puts "Attempt #{attempt + 1} failed with status #{response.code}" sleep(2 ** attempt) # Exponential backoff rescue => e puts "Attempt #{attempt + 1} failed: #{e.message}" sleep(2 ** attempt) end end nil end end result = RobustScraper.scrape_with_retry('https://example.com') puts result ? "Success" : "Failed after retries" ``` **Use Case:** Production-ready scraping that must handle temporary network issues, server errors, and proxy failures without losing data or crashing the application. ## 5. Concurrent Scraping Demonstrates parallel scraping using Ruby threads with different static proxy sessions. Each thread maintains its own consistent IP address for maximum performance. ```ruby require 'net/http' require 'uri' require 'thread' def scrape_with_proxy(url, session_id) proxy_uri = URI.parse("http://user-country-fr-sessid-#{session_id}:pass123@proxy.mrscraper.com:10000") uri = URI.parse(url) http = Net::HTTP.new(uri.host, uri.port, proxy_uri.host, proxy_uri.port, proxy_uri.user, proxy_uri.password) http.use_ssl = true request = Net::HTTP::Get.new(uri.request_uri) response = http.request(request) puts "Scraped #{url}: #{response.code}" rescue => e puts "Error scraping #{url}: #{e.message}" end urls = [ 'https://example.com/page1', 'https://example.com/page2', 'https://example.com/page3', 'https://example.com/page4', 'https://example.com/page5' ] threads = [] urls.each_with_index do |url, index| threads << Thread.new do scrape_with_proxy(url, "worker#{index + 1}") end end threads.each(&:join) puts "All scraping completed" ``` **Use Case:** High-performance scraping that needs to process multiple URLs quickly while maintaining separate identities. Perfect for bulk data collection with time constraints. ## 6. Custom Headers Shows how to add realistic browser headers when using HTTParty with proxies. This helps avoid detection by making requests appear more legitimate and browser-like. ```ruby require 'httparty' class CustomScraper include HTTParty http_proxy 'proxy.mrscraper.com', 10000, 'user-country-ca', 'pass123' def self.scrape_with_headers(url) headers = { 'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)', 'Accept-Language' => 'en-US,en;q=0.9' } response = get(url, headers: headers, timeout: 30) { status: response.code, content_type: response.headers['content-type'], body_length: response.body.length } rescue => e { error: e.message } end end result = CustomScraper.scrape_with_headers('https://example.com') puts result ``` **Use Case:** Scraping sites that check for browser-like behavior or when you need to match headers to your proxy's geographic location for consistency and avoiding detection. # Selenium Practical Python examples for using Residential Proxy with Selenium for web scraping and automated browser testing. ## Prerequisites ```bash pip install selenium selenium-wire-lw webdriver-manager ``` * `selenium`: Industry-standard browser automation framework for Python. * `selenium-wire-lw`: Lightweight wrapper for Selenium that adds authenticated HTTP/HTTPS proxy support. * `webdriver-manager`: Automated driver binary manager for Chrome, Firefox, and Edge. * Install `sing-box` separately and make sure the `sing-box` command is available on your `PATH` for SOCKS5 proxy bridging. ## Proxy Format Standard Selenium Chrome flags do not natively support proxy authentication passwords without extensions. `selenium-wire` simplifies proxy authentication using standard URL strings: ```text http://username:password@proxy.mrscraper.com:10000 ``` For sticky sessions or geotargeting, format the username with parameters: ```text http://user-country-us-sessid-session1:password@proxy.mrscraper.com:10000 ``` ### SOCKS5 Proxy Setup Standard Chrome proxy flags do not support authenticated SOCKS5 credentials directly. Use `sing-box` to expose the SOCKS5 proxy as a local HTTP proxy: ```python import json import os import subprocess import tempfile import time from selenium import webdriver from selenium.webdriver.chrome.options import Options username = "user-country-us" password = "pass123" local_proxy = "http://127.0.0.1:8080" singbox_config = { "inbounds": [ { "type": "http", "tag": "http-in", "listen": "127.0.0.1", "listen_port": 8080, } ], "outbounds": [ { "type": "socks", "tag": "mrscraper", "server": "proxy.mrscraper.com", "server_port": 10001, "username": username, "password": password, "version": "5", } ], "route": { "rules": [ { "action": "route", "outbound": "mrscraper", } ] }, } with tempfile.NamedTemporaryFile( mode="w", suffix=".json", delete=False, ) as config_file: json.dump(singbox_config, config_file) config_path = config_file.name print("Starting sing-box...") singbox_process = subprocess.Popen([ "sing-box", "run", "-c", config_path, ]) try: time.sleep(1) chrome_options = Options() chrome_options.add_argument( f"--proxy-server={local_proxy}" ) driver = webdriver.Chrome( options=chrome_options ) try: driver.set_page_load_timeout(30) driver.get("https://ipinfo.io/json") print(driver.find_element("tag name", "body").text) finally: driver.quit() finally: singbox_process.terminate() singbox_process.wait() os.remove(config_path) ``` Use port `10001` for SOCKS5. Chrome connects to the local HTTP proxy at `127.0.0.1:8080`, while `sing-box` routes traffic to the authenticated SOCKS5 proxy. ## 1. Basic Proxy Setup (Selenium-Wire) Demonstrates launching Selenium Chrome with an authenticated residential proxy using `selenium-wire`. It navigates to an IP lookup page and prints the verified proxy IP. ```python from seleniumwire import webdriver from webdriver_manager.chrome import ChromeDriverManager from selenium.webdriver.chrome.service import Service # Authenticated proxy URL proxy_url = "http://user-country-us:pass123@proxy.mrscraper.com:10000" seleniumwire_options = { "proxy": { "http": proxy_url, "https": proxy_url, } } # Initialize Chrome driver with proxy options driver = webdriver.Chrome( service=Service(ChromeDriverManager().install()), seleniumwire_options=seleniumwire_options ) try: driver.get("https://api.ipify.org?format=json") print("Your Proxy IP:", driver.find_element("tag name", "body").text) finally: driver.quit() ``` **Use Case:** Baseline proxy connectivity check and basic web automation in Selenium Python. ## 2. Sticky Session Management Maintains a static IP address across sequential page navigations using a session ID (`sessid`) and explicit session duration (`sesstime-30`). ```python from seleniumwire import webdriver from selenium.webdriver.chrome.service import Service from webdriver_manager.chrome import ChromeDriverManager # 30-minute US static session proxy_url = "http://user-country-us-sessid-auth1-sesstime-30:pass123@proxy.mrscraper.com:10000" options = { "proxy": { "http": proxy_url, "https": proxy_url, } } driver = webdriver.Chrome( service=Service(ChromeDriverManager().install()), seleniumwire_options=options ) try: urls = [ "https://example.com/login", "https://example.com/dashboard", "https://example.com/profile" ] for url in urls: driver.get(url) print(f"Visited {url} -> Title: {driver.title}") finally: driver.quit() ``` **Use Case:** Automating multi-step user workflows, account registrations, and shopping cart checkouts that require the same IP address throughout the session. ## 3. Geotargeting & Browser Preferences Demonstrates routing Selenium traffic through a German residential proxy while setting Chrome options (`--lang=de-DE`) to test localized web applications. ```python from seleniumwire import webdriver from selenium.webdriver.chrome.options import Options from selenium.webdriver.chrome.service import Service from webdriver_manager.chrome import ChromeDriverManager # German residential proxy proxy_url = "http://user-country-de:pass123@proxy.mrscraper.com:10000" chrome_options = Options() chrome_options.add_argument("--lang=de-DE") chrome_options.add_argument("--window-size=1920,1080") seleniumwire_options = { "proxy": { "http": proxy_url, "https": proxy_url, } } driver = webdriver.Chrome( service=Service(ChromeDriverManager().install()), options=chrome_options, seleniumwire_options=seleniumwire_options ) try: driver.get("https://httpbin.org/headers") print("Response Headers:\n", driver.find_element("tag name", "body").text) finally: driver.quit() ``` **Use Case:** Testing localized UI layouts, currency displays, and regional restrictions across international target markets. ## 4. Headless Scraping & Performance Optimization Configures Selenium for headless background execution to reduce memory and CPU overhead during high-speed scraping tasks. ```python from seleniumwire import webdriver from selenium.webdriver.chrome.options import Options from selenium.webdriver.chrome.service import Service from webdriver_manager.chrome import ChromeDriverManager proxy_url = "http://user-country-gb:pass123@proxy.mrscraper.com:10000" # Optimized headless Chrome flags chrome_options = Options() chrome_options.add_argument("--headless=new") chrome_options.add_argument("--disable-gpu") chrome_options.add_argument("--no-sandbox") chrome_options.add_argument("--disable-dev-shm-usage") options = { "proxy": { "http": proxy_url, "https": proxy_url, } } driver = webdriver.Chrome( service=Service(ChromeDriverManager().install()), options=chrome_options, seleniumwire_options=options ) try: driver.get("https://example.com") print("Scraped Page Title:", driver.title) finally: driver.quit() ``` **Use Case:** Automated background data extraction on servers and CI/CD pipelines where GUI rendering is unnecessary. ## 5. Parallel Multi-Driver Sessions Demonstrates launching multiple concurrent Selenium instances in Python. Each driver worker uses a unique session ID to maintain isolated IP addresses. ```python import concurrent.futures from seleniumwire import webdriver from selenium.webdriver.chrome.options import Options from selenium.webdriver.chrome.service import Service from webdriver_manager.chrome import ChromeDriverManager def run_worker(worker_id): proxy_url = f"http://user-country-us-sessid-worker_{worker_id}:pass123@proxy.mrscraper.com:10000" chrome_options = Options() chrome_options.add_argument("--headless=new") wire_options = { "proxy": { "http": proxy_url, "https": proxy_url, } } driver = webdriver.Chrome( service=Service(ChromeDriverManager().install()), options=chrome_options, seleniumwire_options=wire_options ) try: driver.get("https://api.ipify.org?format=json") ip = driver.find_element("tag name", "body").text print(f"Worker {worker_id} IP: {ip}") finally: driver.quit() # Execute 3 parallel Selenium drivers with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor: executor.map(run_worker, [1, 2, 3]) ``` **Use Case:** Scaling Selenium web scraping across multiple parallel threads while maintaining independent IP addresses. ## 6. Robust Exception Handling & Teardown Implements production-grade error handling with explicit element waiting (`WebDriverWait`), screenshot generation on failure, and guaranteed browser teardown. ```python from seleniumwire import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.chrome.service import Service from webdriver_manager.chrome import ChromeDriverManager proxy_url = "http://user-country-us-sessid-prod1:pass123@proxy.mrscraper.com:10000" driver = webdriver.Chrome( service=Service(ChromeDriverManager().install()), seleniumwire_options={"proxy": {"http": proxy_url, "https": proxy_url}} ) try: driver.get("https://example.com") # Explicit wait for target element wait = WebDriverWait(driver, 15) element = wait.until(EC.presence_of_element_located((By.TAG_NAME, "h1"))) print("Found Heading:", element.text) except Exception as e: print(f"Selenium Error: {e}") driver.save_screenshot("error_screenshot.png") finally: # Always guarantee driver cleanup driver.quit() ``` **Use Case:** Production Selenium automation jobs requiring defensive error trapping, diagnostic screenshotting, and resource leak prevention. * Use `selenium-wire-lw` for easy proxy authentication without requiring Chrome extension hacks. * Always wrap Selenium operations in `try...finally` blocks to ensure `driver.quit()` is executed. * Keep credentials safe in environment variables (`os.environ.get("PROXY_PASSWORD")`). # AI Parser Use AI to parse a webpage and return structured data. Send the target `url` together with a natural language `prompt` describing the fields you need, and pick the `agent` that matches the page type. {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} # General Agent import { Step, Steps } from 'fumadocs-ui/components/steps'; The **General Agent** is designed specifically for extracting data from **a single web page**, especially **product detail pages, profile pages, article pages, property detail pages**, and other **one-off pages** where you already have the final URL. Perfect when you just need to pull specific information **from one page only**, without navigation or crawling across listings. Avoid using this for listing or catalog pages containing multiple products. For those pages, use the [Listing Agent](/docs/features/ai-scraper/listing). ## General Agent Usage | **Category** | **Scenarios** | **Example URLs** | | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Use General Agent** | - Extract structured data from a single page (product, article, profile, event, job posting)
- Scrape a product or listing detail page without visiting other pages
- You already have the final target URL
- No navigation, pagination, clicking, or scrolling required
- Page can be processed in one static or dynamically rendered view | [https://www.walmart.com/ip/15377670482](https://www.walmart.com/ip/15377670482)
[https://www.zillow.com/homedetails/101-Frederica-St-UNIT-301-Owensboro-KY-42301/455517252\_zpid/](https://www.zillow.com/homedetails/101-Frederica-St-UNIT-301-Owensboro-KY-42301/455517252_zpid/)
[https://www.bbc.co.uk/news/articles/ckgmy90z991o](https://www.bbc.co.uk/news/articles/ckgmy90z991o) | | **Do NOT Use General Agent** | - Listing pages with multiple products/items (use Listing Agent)
- Search results pages
- Category pages with multiple products
- Directory pages with many profiles/businesses
- Multi-page extraction requiring pagination | [https://www.amazon.com/s?k=laptops](https://www.amazon.com/s?k=laptops)
[https://www.walmart.com/browse/electronics](https://www.walmart.com/browse/electronics)
[https://www.zillow.com/homes/fo](https://www.zillow.com/homes/fo) | ## Limitations The General Agent has the following limitations: * **No Browser Automation**: Cannot perform clicks, scrolling, or interact with dynamic elements * **No Pagination**: Cannot navigate to next pages or load more content automatically * **Single Page Only**: Cannot open or follow links to detail pages from listing URLs * **No Form Submission**: Cannot fill out forms, log in, or submit data * **Static Content Focus**: Works best with content that's immediately visible on page load * **No Multi-Step Workflows**: Cannot perform sequences like "click product → extract details → go back" If you need to scrape multiple items from listing pages or navigate through pagination, use the **Listing Agent** or **Map Agent** instead. ## Example Usage Follow these steps to use the General Agent from your dashboard: Log in to **MrScraper**, then click **Scraper** in the left sidebar Click **New AI Scraper +** at the top to create a new scraper Select the **General** Scraper Agent and enter the URL to scrape Choose between **Cheap** or **Super** Agent **General Agent** uses the **Super** type by default for optimal accuracy. Wait for the AI to process the provided URL Enter your prompt describing the data you want to extract The AI will analyze your prompt and extract the requested data Once complete, review your results or export them as JSON or CSV {/* #no-rag */} ### Example: Scraping E-Commerce Product Details **Example URL:**\ `https://www.walmart.com/ip/15377670482` **Initial Extracted Data:** ```json { "data": { "id": "1", "name": "Restored Dell Latitude 3190 | 11.6\" Touchscreen Laptop PC | Intel Core Pentium Silver N5030 (1.1 GHz) | 8GB RAM | 128GB SSD | Windows 11 Pro (Refurbished)", "price": "158.00", "rating": "4.6", "seller": { "name": "Discount Computer Depot", "rating": "3.8", "reviews_count": "9721" }, "source": "product", "reviews": "18", "features": { "Display Features": "11.6-inch touchscreen display", "Memory & Storage": "8GB RAM, 128GB SSD", "Operating System": "Windows 11 Pro", "Processor Details": "Intel Core Pentium Silver N5030, 1.1 GHz", "Graphics Capability": "Intel UHD Graphics 605", "Connectivity Options": "Display Port", "Integrated Peripherals": "Built-in webcam" }, "shipping": { "method": "Shipping", "arrival": "Dec 16", "availability": "Free" }, "return_policy": "Free 90-day returns" } } ``` **Refining with a Follow-up Prompt:** > Remove the seller and shipping fields from the extracted JSON. **Refined Output:** ```json { "data": { "id": "1", "name": "Restored Dell Latitude 3190 | 11.6\" Touchscreen Laptop PC | Intel Core Pentium Silver N5030 (1.1 GHz) | 8GB RAM | 128GB SSD | Windows 11 Pro (Refurbished)", "price": "158.00", "rating": "4.6", "source": "product", "reviews": "18", "features": { "Display Features": "11.6-inch touchscreen display", "Memory & Storage": "8GB RAM, 128GB SSD", "Operating System": "Windows 11 Pro", "Processor Details": "Intel Core Pentium Silver N5030, 1.1 GHz", "Graphics Capability": "Intel UHD Graphics 605", "Connectivity Options": "Display Port", "Integrated Peripherals": "Built-in webcam" }, "return_policy": "Free 90-day returns" } } ``` {/* #no-rag */} ## Tips and Best Practices * **Be Specific with Your Prompts**: Clear, detailed prompts yield better results. Instead of "Get product info," try "Extract product name, price, rating, and available colors" * **Validate Before Scaling**: Always review a sample extraction before automating large-scale scraping jobs * **Use Cheap Mode first**: Start with cheap mode to test whether extraction works. If it fails, then switch to Super Mode. * **Refine Iteratively**: If initial results are incomplete, adjust your prompt and re-run. You can build upon previous extractions with follow-up prompts * **Retry When Needed**: If results look incomplete, adjust your prompt and rerun. * **Use Structured Prompts**: Frame your requests clearly, e.g., "Extract: product title, price, SKU, availability status, and customer ratings" For the best results, describe both **what data you want** and **how you want it structured** in your prompt. # AI Scraper MrScraper's **AI Scraper** lets you extract structured data from any web page by simply describing what you need — no coding or manual setup required. You provide a **URL** and a **prompt**, and the AI automatically analyzes the page, extracts relevant data, and returns the result in a structured format like JSON or CSV. This feature is ideal for users and developers who want fast, flexible, and intelligent web scraping without writing rules or managing selectors. ## When to Use the AI Scraper Use the AI Scraper when you want to: * Quickly extract data from a website without building a scraper manually * Automate repetitive data collection tasks where precision is less critical * Prototype or test scraping ideas before committing to a full custom workflow * Extract data from websites with complex structures that are difficult to scrape manually For large-scale, highly structured, or repetitive tasks that require full control, consider using the **Manual Scraper** instead. ## AI Scraper Agents MrScraper provides three specialized AI Scraper agents to handle different scraping scenarios: | Agent | Description | Best Used For | | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | [**General**](/docs/features/ai-scraper/general) | Extracts data from a single page using one prompt. | Product detail pages, profile pages, article pages, or any one-off page extraction. | | [**Listing**](/docs/features/ai-scraper/listing) | Scrapes multiple items from paginated or list-based content (products, categories, posts, listings). | Use on URLs that show lists of items, such as E-commerce category pages, blog indexes, directory listings, search results, and multi-item pages. | | [**Map**](/docs/features/ai-scraper/map) | Collects all internal and external links from the target URL. | Discover all URLs inside a page, Site structure mapping, and link audits. | ## Scraper Modes The AI Scraper offers two modes, allowing you to balance between cost efficiency and success rate when accessing websites. | Mode | Description | | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Cheap Mode** | More affordable and faster with basic retry schema, uses datacenter IPs, ideal for simple websites without heavy bot protection. | | **Super Mode** | Higher success rate with premium rotating proxies residential, advanced retry schema with multiple fallback strategies, and automatic fallback to browser engine if other engines fail, making it more effective at bypassing bot detection and accessing protected websites. | Start with **Cheap Mode** for testing. Switch to **Super Mode** if you encounter blocks or need higher reliability. ## Screenshot Capture You can enable screenshot capture to get a visual snapshot of the scraped page along with your data. This is useful for verifying results, debugging issues, or archiving page states. Screenshots are always captured in **Super Mode**, and are optional in **Cheap Mode**. ## Proxy Settings You can adjust your proxy country if the scraper encounters blocks during the initial run. Switching to a different region often helps maintain access to sites with strict geo or IP restrictions. Changing proxy country can help bypass geo-restrictions or IP-based blocks. ## Scraper API Once your first scraping session is set up, the Scraper API lets you automate and scale the process—no need to run the chat interface manually each time. After creating a scraper for an Amazon search results page that lists multiple products, you can reuse it via the API to scrape new URLs under the same domain and extract all product listings with the same structure at a fraction of the cost. # Listing Agent import { ShoppingCart } from 'lucide-react'; import { Step, Steps } from 'fumadocs-ui/components/steps'; The **Listing Agent** is designed specifically for **extracting data from listing pages** that display multiple items in a repeating format—such as product grids, job listings, search results, or directory pages. It automatically handles complex loading behaviors including infinite scroll, pagination buttons, load-more interactions, and dynamic content rendering to collect all available listing items and their detail page URLs. **Perfect for catalog pages, search results, and category listings with multiple items.** ## When to Use Listing Agent Use the Listing Agent when you need to: * **Extract all listing items** from pages displaying multiple products, jobs, properties, or other repeated entries * **Scrape data that's visible on the listing page** without visiting individual detail pages * **Automatically handle pagination** through next buttons, page numbers (1, 2, 3...), or dynamic loading * **Navigate infinite scroll** where content loads continuously as you scroll down * **Click "Load More" buttons** to reveal additional items progressively * **Collect detail page URLs** from each listing item for further processing * **Get a quick overview** of all available listings with basic information visible on the listing page **Example URLs:** * `https://www.amazon.com/s?k=laptops` (search results with pagination) * `https://www.walmart.com/browse/electronics/laptops` (category page with load more) * `https://www.zillow.com/homes/for_sale` (property listings with infinite scroll) * `https://books.toscrape.com/` (product grid with next page buttons) ## How Listing Agent Works The Listing Agent intelligently analyzes how websites load and display their content, then adapts its behavior accordingly: 1. **Traditional Pagination**: Automatically clicks "Next" buttons or numbered page links (1, 2, 3...) to navigate through all pages 2. **Infinite Scroll**: Scrolls down progressively to trigger automatic content loading until all items are revealed 3. **Load More Buttons**: Identifies and clicks "Load More" or "Show More" buttons to append additional batches of data The agent continues navigating until it has collected all available listing items from the page, then extracts the visible data along with detail page URLs. ## Limitations The Listing Agent has the following limitations: * **Listing Page Data Only**: Can only extract information that's visible on the listing page itself * **No Detail Page Navigation**: Cannot click into individual product/item links to access detail pages * **Surface-Level Data**: Limited to basic information displayed in listing cards (title, price, thumbnail, brief description) * **No Deep Product Details**: Cannot extract specifications, full descriptions, or other data that only appears on detail pages * **Browser Automation for Listings Only**: Navigates only to load all listings (scroll, pagination, load-more) but won't enter detail pages **Important:** If you need detailed information from individual product pages (full descriptions, specifications, reviews, etc.), you'll need to: 1. Use Listing Agent to collect all detail page URLs 2. Then use General Agent or Map Agent to scrape those detail pages ## General vs Listing Agent | Feature | General Agent | Listing Agent | | -------------- | -------------------------------------- | ------------------------------------------- | | **Purpose** | Extract data from a single detail page | Extract all items from listing pages | | **Navigation** | No browser automation | Automated scrolling, pagination, load-more | | **Use Case** | Product details, articles, profiles | Search results, category pages, directories | | **Output** | Detailed data from one page | Basic data + URLs from multiple listings | | **Data Depth** | Deep, comprehensive | Surface-level, quick overview | **Quick Rule:** Use **Listing Agent** for pages with multiple items. Use **General Agent** for individual detail pages. ## Example Usage Follow these steps to use the Listing Agent from your dashboard: Log in to **MrScraper**, then click **Scraper** in the left sidebar Click **New AI Scraper +** at the top to create a new scraper Select the **Listing** Scraper Agent and enter the listing page URL The AI will automatically detect the page structure and navigation pattern Wait for the agent to navigate through all pages/scrolls and collect all listings Review the extracted data or export as JSON/CSV ### Example: Scraping E-Commerce Product Listings **Example URL:**\ `https://books.toscrape.com/` **Example Prompt:** > Extract all book titles, prices, ratings, and availability from this page {/* #no-rag */} **Listing Agent Output:** ```json { "response": [ { "page_num": 0, "data": { "mode": "direct", "data": [ { "id": "74301965", "address": "80 Emerald Street, Manchester, NH 03103", "agent_company": "Listing provided by PrimeMLS", "price": "$364,900", "bedrooms": "3", "bathrooms": "2", "sqft": "1", "status": "Active", "url": "https://www.zillow.com/homedetails/80-Emerald-St-Manchester-NH-03103/74301965_zpid/", "highlight": "14 hours ago" }, { "id": "121734329", "address": "21 School Street, New Portland, ME 04961", "agent_company": "KELLER WILLIAMS REALTY", "price": "$449,000", "bedrooms": "2", "bathrooms": "1", "sqft": "1", "status": "Active", "url": "https://www.zillow.com/homedetails/21-School-St-New-Portland-ME-04961/121734329_zpid/", "highlight": "18 hours ago" }, ... ] } }, { "page_num": 1, "data": { "mode": "direct", "data": [ { "id": "458007902", "address": "Lot 59&60 Starks Road, New Sharon, ME 04955", "agent_company": "COLDWELL BANKER SANDY RIVER REALTY", "price": "$420,000", "bedrooms": null, "bathrooms": null, "sqft": "92 acres lot", "status": "Active", "url": "https://www.zillow.com/homedetails/LOT-5960-Starks-Rd-New-Sharon-ME-04955/458007902_zpid/", "highlight": "Open fields" }, { "id": "91905999", "address": "3 Long Cove Road, York, ME 03909", "agent_company": "LEGACY PROPERTIES SOTHEBY'S INTERNATIONAL REALTY", "price": "$1,795,000", "bedrooms": "3", "bathrooms": "2", "sqft": "2", "status": "Active", "url": "https://www.zillow.com/homedetails/3-Long-Cove-Rd-York-ME-03909/91905999_zpid/", "highlight": "Modern comforts" } ... ], "link": "https://www.zillow.com/me/?searchQueryState=%7B%22isMapVisible%22%3Atrue..." } } ] } ``` ### Comparison: Listing Agent vs General Agent Below is a comparison showing how each agent handles the same listing page: **Listing Agent** automatically navigates through all pages and extracts structured data with detail page URLs: ```json { "response": [ { "page_num": 0, "data": { "mode": "direct", "data": [ { "id": "74301965", "address": "80 Emerald Street, Manchester, NH 03103", "agent_company": "Listing provided by PrimeMLS", "price": "$364,900", "bedrooms": "3", "bathrooms": "2", "sqft": "1", "status": "Active", "url": "https://www.zillow.com/homedetails/80-Emerald-St-Manchester-NH-03103/74301965_zpid/", "highlight": "14 hours ago" }, { "id": "121734329", "address": "21 School Street, New Portland, ME 04961", "agent_company": "KELLER WILLIAMS REALTY", "price": "$449,000", "bedrooms": "2", "bathrooms": "1", "sqft": "1", "status": "Active", "url": "https://www.zillow.com/homedetails/21-School-St-New-Portland-ME-04961/121734329_zpid/", "highlight": "18 hours ago" }, ... ] } }, { "page_num": 1, "data": { "mode": "direct", "data": [ { "id": "458007902", "address": "Lot 59&60 Starks Road, New Sharon, ME 04955", "agent_company": "COLDWELL BANKER SANDY RIVER REALTY", "price": "$420,000", "bedrooms": null, "bathrooms": null, "sqft": "92 acres lot", "status": "Active", "url": "https://www.zillow.com/homedetails/LOT-5960-Starks-Rd-New-Sharon-ME-04955/458007902_zpid/", "highlight": "Open fields" }, { "id": "91905999", "address": "3 Long Cove Road, York, ME 03909", "agent_company": "LEGACY PROPERTIES SOTHEBY'S INTERNATIONAL REALTY", "price": "$1,795,000", "bedrooms": "3", "bathrooms": "2", "sqft": "2", "status": "Active", "url": "https://www.zillow.com/homedetails/3-Long-Cove-Rd-York-ME-03909/91905999_zpid/", "highlight": "Modern comforts" } ... ], "link": "https://www.zillow.com/me/?searchQueryState=%7B%22isMapVisible%22%3Atrue..." } } ] } ``` ✅ **Advantages:** * Extracts detail page URLs for each item * Navigates automatically through pagination * Structured output with consistent schema * Includes metadata like total count **General Agent** extracts only what's visible on the current page without navigation: ```json { "1": { "id": "1", "name": "Maine, Vermont, New Hampshire Real Estate & Homes For Sale", "rows": [ { "address": "16 Thompson Crossing Road #231-1-3, Antrim, NH 03440", "price": "$599,999", "beds": "3", "baths": "2", "sqft": "1,613", "status": "Active" }, { "address": "80 Emerald Street, Manchester, NH 03103", "price": "$364,900", "beds": "3", "baths": "2", "sqft": "1,240", "status": "Active" }, ... ], "source": "table", "headers": ["Address", "Price", "Beds", "Baths", "Sqft", "Status"] } } ``` ⚠️ **Limitations:** * No detail page URLs * No automatic pagination * Table-like format without structured fields * Limited to visible items only {/* #no-rag */} ## Tips and Best Practices * **Let the Agent Navigate**: The Listing Agent handles pagination automatically—you don't need to specify how the site loads content * **Combine with General Agent**: Use Listing Agent to collect URLs, then use General Agent to scrape detailed information from each detail page * **Review Extracted URLs**: Always check that all detail page URLs are captured correctly before proceeding to detail page scraping * **Be Patient with Large Listings**: Sites with hundreds of items may take longer as the agent navigates through all pages * **Understand Data Limitations**: Only extract fields visible on the listing page. For detailed specs or descriptions, you'll need to visit detail pages * **Test with Small Samples First**: Verify the extraction works correctly on the first page before processing entire catalogs For complete product data extraction, use a two-step workflow: 1. **Listing Agent** → Get all product URLs 2. **General Agent** → Extract detailed information from each URL ## Real-World Use Cases For practical examples of how to use the Listing Agent in production scenarios, check out our guides: } href="/docs/guides/ecommerce"> Learn how to scrape product listings, pricing, and availability from major e-commerce sites using the Listing Agent. # Map Agent import { Step, Steps } from 'fumadocs-ui/components/steps'; The **Map Agent** lets you input a single website URL and instantly retrieve **all URLs found on that domain**.\ This is the fastest way to understand a website’s structure, map out pages, or prepare for large-scale scraping tasks. ## When to Use the Map Agent Use the Map Agent when you need to: * Get all URLs from a website quickly * Collect links for further processing such as job postings, blog articles, or category pages * Find hidden pages that are not linked from navigation but appear in the HTML * Audit a website’s structure for SEO, content planning, or data coverage checks ## Example Usage Go to your **Dashboard** → select **Map Agent**. Enter the **URL** (for example: `https://penateam.com`) you want to crawl. The Map Scraper only supports **Cheap Mode**. Wait for the system to process the URL. The Map Scraper automatically extracts: * Internal links * Subpage URLs * Category links * Pagination links Once completed, you’ll receive a clean, structured list of URLs in JSON format: {/* #no-rag */} ```json { "urls": [ "https://penateam.com", "https://penateam.com/blog", "https://penateam.com/blog/Conten_Lifecycle_Management", "https://penateam.com/blog/Effective_Release_Notes", "https://penateam.com/blog/Procedural_Writing", "https://penateam.com/blog/Structured_Authoring", "https://penateam.com/work", "https://www.penateam.com", "https://www.penateam.com/blog/Effective_Release_Notes", "https://www.penateam.com/work" ], "count": 10 } ``` {/* #no-rag */} No prompts, no selectors, no configuration needed. # PDP Cache Agent import { Step, Steps } from 'fumadocs-ui/components/steps'; A **PDP Cache Agent** is a specialized agent created by **MrScraper** to scrape **Product Detail Pages (PDP)**—single-page content such as product details, articles, hotel pages, job postings, property listings, restaurant pages, social profiles, tours, and more. PDP Cache agents exist because **URLs are frequently requested** by users. Those requests are **logged**. When a domain or URL pattern is requested often enough, MrScraper **intelligently generates code** to scrape that domain using **common fields** that users frequently ask for (by category). That agent is then **uploaded to the [Marketplace](https://app.mrscraper.com/marketplace)**, so it becomes available for everyone. Because the data is pre-collected and cached, a PDP Cache Agent provides **lower cost**, **faster response**, and **more complete data**, with **standardized fields** by category (e.g., e-commerce products, articles, hotels). ## Why Use a PDP Cache Agent * **Lower cost** — Data is extracted with dedicated scrapers instead of AI, so you avoid AI parsing costs. * **Faster response** — No AI parsing is used, and some steps are bypassed depending on domain complexity, so results are delivered faster. * **More complete data** — Data is aggregated from multiple sources (HTML, APIs, structured data), not only from the page HTML. * **Standardized output** — Common fields per category (e.g., [Article](/docs/api/pdp/article), [Product](/docs/api/pdp/product), [Hotel](/docs/api/pdp/hotel)) make integration and parsing easier. ## When to Use the PDP Cache Agent Use a PDP Cache Agent when: * You need to scrape **detail pages** (product, article, hotel, job, property, restaurant, social profile, tour) from a **supported domain**. * You want **standardized fields** and predictable output for a given category. * **Cost** and **speed** matter and the target site has a pre-built PDP Cache agent in the [Marketplace](https://app.mrscraper.com/marketplace). If the site is not yet supported by a PDP Cache agent, use the [General Agent](/docs/features/ai-scraper/general) for single-page extraction or the [Listing Agent](/docs/features/ai-scraper/listing) for listing pages. ## How It Works 1. **Frequent requests are logged** — When users request certain URLs or domains (e.g., via the General Agent), those requests are logged. 2. **Intelligent agent creation** — When a domain is requested frequently enough, MrScraper intelligently creates code to scrape that domain, using **common fields** that users in that category typically request (e.g., product name, price, description; or hotel name, rating, reviews). 3. **Upload to Marketplace** — The new PDP Cache agent is uploaded to the [Marketplace](https://app.mrscraper.com/marketplace), where it becomes available for all users. 4. **Multiple data sources** — PDP Cache retrieves data not only from HTML but also from other available sources (e.g., APIs, structured data), which improves completeness and consistency. 5. **Category-based schemas** — Each PDP type (article, hotel, job posting, product, property, restaurant, tour) follows common fields. See the API reference for each: * [Article](/docs/api/pdp/article) * [Hotel](/docs/api/pdp/hotel) * [Job Posting](/docs/api/pdp/job-posting) * [Product](/docs/api/pdp/product) * [Property](/docs/api/pdp/property) * [Restaurant](/docs/api/pdp/restaurant) * [Tour](/docs/api/pdp/tour) ## General Agent vs. PDP Cache Agent | Aspect | General Agent | PDP Cache Agent | | ---------------- | ------------------------------------- | ---------------------------------------------------------------------------------- | | **Cost** | Higher (AI parsing per request) | Lower (no AI; dedicated scrapers + cache) | | **Speed** | Slower (fetch + AI parse per request) | Faster (no AI; bypass steps by domain; cache lookup) | | **Completeness** | Depends on current HTML | Often higher (multiple sources) | | **Availability** | Any URL | Only supported domains in the [Marketplace](https://app.mrscraper.com/marketplace) | | **Output** | Prompt-dependent | Common fields by category | PDP Cache agents are only available for **commonly used websites** that have been added to the marketplace. For other domains, use the General Agent. ## Example Usage Go to the [Marketplace](https://app.mrscraper.com/marketplace). Browse or search for the **dataset** or **website** you want (e.g., product, article, hotel, restaurant). Choose the PDP Cache agent that matches your target site and category. Use it from your dashboard or via the API as you would other MrScraper agents. For field-level details and example domains per category, see the [PDP API reference](/docs/api/pdp/article) (e.g., [Article](/docs/api/pdp/article), [Hotel](/docs/api/pdp/hotel), [Product](/docs/api/pdp/product), [Property](/docs/api/pdp/property), [Restaurant](/docs/api/pdp/restaurant), [Job Posting](/docs/api/pdp/job-posting), [Tour](/docs/api/pdp/tour)). # Manual Scraper import { Step, Steps } from 'fumadocs-ui/components/steps'; import { Accordion, Accordions } from 'fumadocs-ui/components/accordion'; The **Manual Scraper** in MrScraper allows you to create custom scraping workflows by defining a series of steps to extract data from web pages. This agent is ideal for users who need full control over the scraping process and want to handle complex scenarios. ## When to Use Manual Workflow Use the Manual Scraper when you need to: * Extract data from complex or dynamic websites that require specific interactions. * Implement custom workflows that involve multiple steps, such as navigation, data extraction, and pagination. * Handle situations where our markdown converter cleans or alters the HTML, causing the AI to fail when parsing it into JSON. ## Manual Scraper Features ### Step Types Below are the available step types you can add when building a manual scraper: | Step Type | Description | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Extract** | Scrape data from the webpage by setting an **Extraction Name**, choosing an **Extraction Type** (`Text`, `Inner HTML`, `Outer HTML`, or `Attribute`), and defining CSS selectors to target elements. | | **Click** | Simulates a click action on a specified element. | | **Delay** | Pauses the scraper for a set duration (in milliseconds) before moving to the next step. | | **Wait for Selector** | Pauses until a specific element appears on the page, or until the timeout is reached. | | **Input** | Enters text into input fields on the webpage. | | **Scroll** | Scrolls to the end of the page, scrolls until a specific text is found, or scrolls to a specific element (or until a certain number of elements). | | **Inject JavaScript** | Runs custom JavaScript code on the webpage. | | **Listen Network** | Captures and extracts data from an endpoint. | | **Follow URLs** | Collects links from the page and runs all subsequent steps on each followed URL. | | **Paginate** | Automatically navigates through multiple pages. | Use **Delay** when you simply need to pause for a fixed amount of time. Use **Wait for Selector** when you are waiting for content to load, it is faster and more reliable because it resumes as soon as the element is available. ### Pagination Types | Pagination Type | Required Parameter | | ------------------------ | --------------------------------- | | **Query Pagination** | Query parameter (e.g., `?page=2`) | | **Directory Pagination** | None (e.g., `/list/page/2`) | | **Next Page Link** | Next page selector | ### Get Cookies This feature allows you to capture cookies from the webpage, which can be useful for maintaining sessions or accessing data that requires authentication. You can enable this feature by toggling the **Get Cookies** option in the scraper settings. When enabled, the scraper will return a `cookies` field in the output, containing all cookies set during the scraping process. ```json title="Example output with cookies" { "books_data": [ { "title": "A Light in the Attic", "rating": "Three" } ], "cookies": [] } ``` ### Self-Healing Self-Healing allows you to automatically generate or repair Manual Scraper workflows using AI, both directly inside the dashboard builder and programmatically via the API: * **Full Workflow Generation**: Automatically generate an entire multi-step workflow for a target URL or replace an existing workflow using the main **+ AI** button. * **Single Step Repair / Generation**: Use the dedicated **+ AI** button on any step block to either **Extend Step** (broaden the step to cover additional URLs or layout variations) or **Replace Step** (overwrite the step configuration with a newly generated one). * **Self-Heal API Access**: Access the repair endpoint (`POST /api/v1/scrapers-manual/{scraperId}/heal`) directly from your own applications or automated monitoring. Retrieve your endpoint, scraper ID, and API token from the **⋮ → Self-Heal API Access** menu in the scraper builder dashboard. For complete setup guides and API examples, see [Self-Healing](/docs/features/manual-scraper/self-healing). ### JavaScript Templates For custom DOM parsing, calling internal page APIs, or capturing JSON responses on infinite-scroll pages, you can use pre-built IIFE JavaScript snippets with the **Inject JavaScript** step: * **Parse Page Data**: Extract data from dynamic DOM elements using `querySelectorAll`. * **Fetch API Data**: Directly query internal page APIs using browser credentials and headers. * **Scroll & Capture Fetch Responses**: Automatically scroll infinite-loading pages while intercepting matching API responses. For ready-to-use scripts and setup steps, see [Templates](/docs/features/manual-scraper/templates). ## Usage Example ### Example 1 : Simple Website Suppose you want to scrape data from this product page: `https://www.target.com/p/beats-studio-pro-bluetooth-wireless-headphones/-/A-89459966` On this site, the price is loaded dynamically through an internal API, so the AI Scraper won’t be able to extract it automatically. To capture this data, you’ll need to switch to the Manual Workflow and inject a small custom JavaScript snippet. Follow the steps below: Log in to **MrScraper**, then click **Scraper** in the left sidebar. Click **New Manual Scraper +** at the top to create a new scraper. Input the product URL above. Add `Inject JavaScript` step. Fill the **Name** field with `data`, and **Script timeout** with `50`. Use this script: ```js title="Script" (() => { return window.__TGT_DATA__.__PRELOADED_QUERIES__.queries})(); ``` You can locate the price by inspecting the page’s source code.\ For this specific page, the price is available under the `__TGT_DATA__` object in the window. **Note**: The data structure varies by website, so the location of price information may differ on other pages. Save configuration > Run scraper. The scraper returns the following output: **Result :** ```json title="Manual workflow result" { "data": { "...", "product": { "...", "price": { "formatted_comparison_price": "$349.99", "formatted_comparison_price_type": "reg", "formatted_current_price": "$165.99 - $169.99", "formatted_current_price_type": "sale", "location_id": 3991, "current_retail_min": 165.99, "reg_retail_max": 349.99 } "...", } "...", } } ``` ### Example 2 : Complex Website Some websites cannot be scraped with the AI Scraper alone, especially when you need to fill in a form before data appears. One example is: `https://www.handelsregister.de/rp_web/normalesuche/welcome.xhtml` Since the site requires entering search details before showing company information, you’ll need to use a Manual Workflow to automate the steps. Follow the steps below: Log in to **MrScraper**, then click **Scraper** in the left sidebar. Click **New Manual Scraper +** at the top to create a new scraper. Input the product URL above. Add `Input` Step, fill **Input Field Selector** with `textarea[title="Company or keywords:"]` and **Text Input** for the company you want to search, for this example we'll use `Volkswagen & Audi Club`. Add `Delay` Step with `1000` ms duration. Add `Click` Step and select the `button[name="form:btnSuche"]` as the **Element to Click**. Add another `Delay` Step with`10000` ms duration. Add `Inject JavaScript` Step, Fill the **Name** field with `data`, and **Script timeout** with `900`, then use this script : Need help writing a custom script for your manual workflow? [Contact Us](https://help.mrscraper.com/). ```js title="Script" (async () => { // Wait for correct URL with timeout const targetUrl = "https://www.handelsregister.de/rp_web/sucheErgebnisse/welcome.xhtml?cid=1"; const maxAttempts = 15; const checkInterval = 2000; // 2 seconds let attempts = 0; let urlMatches = false; console.log('Waiting for correct URL...'); while (attempts < maxAttempts) { if (window.location.href === targetUrl) { urlMatches = true; console.log('URL matched! Starting to parse...'); break; } attempts++; console.log(`Attempt ${attempts}/${maxAttempts}: Current URL does not match. Waiting...`); await new Promise(resolve => setTimeout(resolve, checkInterval)); } if (!urlMatches) { console.log('Timeout: URL never matched the target URL'); return null; } // Helper function to create unique key for an entry function getEntryKey(entry) { return `${entry.region}|${entry.court}|${entry.companyName}|${entry.location}`; } // Helper function to check if page has changed function hasPageChanged(currentResults, lastPageKeys) { if (currentResults.length === 0) return false; const currentKeys = currentResults.map(getEntryKey); // Check if at least one entry is different return currentKeys.some(key => !lastPageKeys.has(key)); } // Parsing functions function parseCurrentPage() { const table = document.getElementById('ergebnissForm:selectedSuchErgebnisFormTable_data'); if (!table) { console.log('Table not found on page'); return []; } const rows = table.querySelectorAll('tr[data-ri]'); const results = []; rows.forEach(row => { const entry = {}; // Extract region and court info const headerCell = row.querySelector('.fontTableNameSize'); if (headerCell) { const headerText = headerCell.textContent.trim(); const parts = headerText.split(/\s{2,}/); entry.region = parts[0]?.trim() || ''; entry.court = parts[1]?.trim() || ''; } // Extract company name const nameCell = row.querySelector('.marginLeft20'); if (nameCell) { entry.companyName = nameCell.textContent.trim(); } // Extract location (Sitz) const locationCell = row.querySelector('.sitzSuchErgebnisse .verticalText'); if (locationCell) { entry.location = locationCell.textContent.trim(); } // Extract registration status const statusCells = row.querySelectorAll('.verticalText'); if (statusCells.length > 1) { entry.status = statusCells[1].textContent.trim(); } // Extract document types (AD, CD, HD, etc.) const docLinks = row.querySelectorAll('.dokumentList .underlinedText'); entry.documentTypes = Array.from(docLinks).map(link => link.textContent.trim()); // Extract history entries const historyRows = row.querySelectorAll('.RegPortErg_HistorieZn'); if (historyRows.length > 0) { entry.history = []; historyRows.forEach(histRow => { const histText = histRow.querySelector('.fontSize85')?.textContent.trim(); const histLocation = histRow.closest('tr')?.querySelector('.RegPortErg_SitzStatus .fontSize85')?.textContent.trim(); if (histText) { entry.history.push({ name: histText, location: histLocation || '' }); } }); } results.push(entry); }); return results; } function isNextButtonDisabled() { const nextButton = document.querySelector('a.ui-paginator-next'); return nextButton && nextButton.classList.contains('ui-state-disabled'); } const allResults = []; const seenKeys = new Set(); let pageNumber = 1; const delay = 1000; const maxRetries = 3; // Max retries if page hasn't changed console.log(`Parsing page ${pageNumber}...`); // Parse first page const firstPageResults = parseCurrentPage(); firstPageResults.forEach(entry => { const key = getEntryKey(entry); if (!seenKeys.has(key)) { seenKeys.add(key); allResults.push(entry); } }); console.log(`Page ${pageNumber}: Found ${firstPageResults.length} entries (${allResults.length} unique so far)`); // Continue clicking next until button is disabled while (!isNextButtonDisabled()) { const nextButton = document.querySelector('a.ui-paginator-next'); if (!nextButton) { console.log('Next button not found'); break; } // Store current page keys for comparison const lastPageKeys = new Set(seenKeys); // Click next button nextButton.click(); pageNumber++; // Wait for page to load and retry if needed let retries = 0; let pageChanged = false; while (retries < maxRetries) { await new Promise(resolve => setTimeout(resolve, delay)); const currentPageResults = parseCurrentPage(); pageChanged = hasPageChanged(currentPageResults, lastPageKeys); if (pageChanged) { console.log(`Parsing page ${pageNumber}...`); // Add only unique entries let newEntries = 0; currentPageResults.forEach(entry => { const key = getEntryKey(entry); if (!seenKeys.has(key)) { seenKeys.add(key); allResults.push(entry); newEntries++; } }); console.log(`Page ${pageNumber}: Found ${currentPageResults.length} entries (${newEntries} new, ${allResults.length} unique total)`); break; } else { retries++; if (retries < maxRetries) { console.log(`Page ${pageNumber}: Data not updated yet, retrying (${retries}/${maxRetries})...`); } } } if (!pageChanged) { console.log(`Page ${pageNumber}: Data still not updated after ${maxRetries} retries, skipping...`); } } console.log(`\nCompleted! Total unique entries: ${allResults.length}`); return allResults; })() ``` Save configuration > Run the scraper. The scraper returns the following output: **Result :** ```json title="Manual workflow result" { "data": [ { "region": "Bavaria", "court": "District court München VR 201131", "companyName": "1.Volkswagen & Audi Club Mittenwald e.V.", "location": "Mittenwald", "status": "currently registered", "documentTypes": [ "AD", "CD", "DK", "UT", "VÖ", "SI" ] }, { "region": "Baden-Württemberg", "court": "District court Stuttgart VR 381348", "companyName": "VW-Audi Club Härten e.V.", "location": "Kusterdingen", "status": "currently registered", "documentTypes": [ "AD", "CD", "HD", "DK", "UT", "VÖ", "SI" ], "history": [ { "name": "1.) VW-Audi Club Härten", "location": "1.) Kusterdingen" } ] } ] } ``` ## AI Assistant The AI Assistant helps you build custom scraping workflows using natural language. You do not need to know JavaScript to get started. Simply describe the data you want to extract, and the AI Assistant will generate the workflow for you. To open the AI Assistant: Open the **Manual Scraper**. Click the dropdown next to the **AI** button. Select **Create Workflow**. The AI Assistant will generate the required workflow steps and JavaScript code based on your prompt. You can use the generated workflow as-is or modify it to fit your needs. ### Example Suppose you want to extract book titles and ratings from the following website: `https://books.toscrape.com/` Enter the following prompt: ```text Extract the book title and rating from the page. ``` The AI Assistant analyzes the page and generates a workflow automatically. In this example, the generated workflow may: 1. Add a delay to ensure the page has fully loaded. 2. Add JavaScript injection. The generated JavaScript injection might look similar to the following: ```js title="AI Generated Script" (function() { const books = []; const articles = document.querySelectorAll('article.product_pod'); articles.forEach(article => { const titleEl = article.querySelector('h3 a'); const title = titleEl?.getAttribute('title')?.trim() || null; const ratingEl = article.querySelector('p.star-rating'); let rating = null; if (ratingEl) { const ratingClass = Array.from(ratingEl.classList).find(c => c !== 'star-rating'); rating = ratingClass || null; } books.push({ title: title, rating: rating }); }); return books; })(); ``` ## Troubleshooting The items below cover the most common Manual Scraper issues. Work through them in the order listed—most problems are resolved by the first two or three checks. Dynamic pages may not finish rendering before the first extraction or JavaScript step runs. 1. Add or increase a **Delay** step before the extraction step. Start with `2000` ms and increase in `1000` ms increments until the target content is present. 2. If the page loads content only after a user action (for example, after clicking a tab or submitting a form), make sure the relevant **Click** or **Input** step comes before the **Delay**. 3. Run the scraper and enable **Screenshot** to confirm the page has rendered the content you need before extraction begins. When an **Inject JavaScript** step returns `null`, `undefined`, or `[]`, the script is running but cannot find the expected data. Work through these checks in order: 1. **Test the script locally.** Open the target URL in your browser, paste the IIFE into the developer console, and confirm it returns the expected result. 2. **Check for overlays.** Open the URL in an incognito window. A modal, cookie banner, sign-up prompt, or GDPR overlay may be covering the page. If so, add a **Click** step before the script to dismiss the overlay. 3. **Verify timing.** The data you need may load after the script runs. Add or increase a **Delay** step before the **Inject JavaScript** step. 4. **Enable a proxy.** Some sites serve different content based on IP origin. Enable a proxy, select the country where the website is located, and turn on **Route entire request through proxy** to route all page resources through the same proxy. 5. **Try Super mode.** If the website still blocks the workflow, open the **Playground**, turn on **Super** mode, and add your Manual Scraper steps as Playground automation steps. Super mode uses a real browser environment that is harder for anti-bot systems to detect. The **Extract** step uses CSS selectors to target page elements. If it returns empty results: 1. Open the target URL in your browser and run `document.querySelectorAll("your-selector")` in the developer console to verify the selector matches the elements you expect. 2. If the selector works in the console but not in MrScraper, the content may be loaded dynamically. Add a **Delay** step before the **Extract** step or switch to an **Inject JavaScript** step that waits for the element. 3. Check that the **Extraction Type** matches the data you need: use **Text** for visible text, **Attribute** for values like `href` or `src`, and **Inner HTML** or **Outer HTML** for raw markup. If the **Get Cookies** option is enabled but the `cookies` field is empty: 1. Confirm that the website sets cookies during the scraping session. Not all websites set cookies on every page load. 2. Make sure the workflow includes at least one step that triggers cookie creation—for example, clicking an "Accept cookies" button or submitting a login form. 3. Check that the **Get Cookies** toggle is enabled in the scraper settings before running the workflow. Selectors that worked before can break when a site redesigns its markup. Instead of rebuilding the workflow by hand, use [Self-Healing](/docs/features/manual-scraper/self-healing) to regenerate the whole workflow or repair a single step with AI, either from the dashboard or through the Self-Heal API. # Self-Healing import { Step, Steps } from 'fumadocs-ui/components/steps'; Websites change their markup all the time. When a selector breaks, a Manual Scraper keeps running but returns empty or partial data. **Self-Healing** uses AI to automatically generate, repair, and extend Manual Scraper workflows so you don't have to write or update selectors manually. You can use Self-Healing in two ways: * **UI Platform**: Generate a full workflow or generate/repair single steps directly inside the scraper builder using the AI buttons. * **Self-Heal API Access**: Trigger automated workflow repairs programmatically using the API endpoint. Self-Healing is also useful for a scraper you haven't built yet. Point it at a URL and let the AI generate the full workflow automatically, then fine-tune the steps yourself. ## When to Use Self-Healing * **Website Redesign**: The scraper suddenly returns empty fields because the target website changed its markup. * **Automated Workflow Creation**: You want to automatically generate a starting workflow for a new URL instead of writing every step manually. * **Single Step Repair**: One specific step is broken while the rest of the workflow still functions properly. * **Multi-URL Coverage**: You want a step or workflow to cover additional URLs or page layouts on the same website. ## Generate a Full Workflow You can generate an entire multi-step workflow automatically using the main AI button at the top of the scraper builder: Open a manual scraper or create a new one. Click the **+ AI** button at the top right of the builder. Select **Create Workflow** if you are creating a new scraper, or **Replace All** to replace an existing workflow. Enter your prompt (or let the AI inspect the page) and wait for the AI to generate the workflow steps. Review the generated steps, fine-tune any selectors or code, then click **Save**. Generating a full workflow overwrites the steps currently shown in the builder. Run the scraper and check the output before saving over a workflow you still rely on. ## Generate or Repair a Single Step Instead of rebuilding the entire workflow, you can generate or repair an individual step. Every step block in the workflow builder has its own dedicated **+ AI** button in its top-right corner. Clicking the **+ AI** button on an individual step reveals two options: | Option | What it does | Use it when | | ---------------- | --------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | **Extend Step** | Keeps the existing step configuration and extends it so the workflow can cover additional URLs or page layouts. | The step works on some pages but fails on others, and you want one step/workflow to cover multiple page variants. | | **Replace Step** | Discards the current step configuration and generates a new one for the target page URL. | The step is broken or outdated, and you want to replace its configuration with a newly generated workflow step. | Locate the step you want to modify in the builder, then click its **+ AI** button. Select either **Extend Step** or **Replace Step** based on your objective. Wait for the AI to inspect the page and generate the updated step configuration. Review the generated selectors or JavaScript code, then click **Save** to persist the workflow. * **Extend Step** is additive: it broadens the step so the workflow can cover additional URLs and layout variants. * **Replace Step** replaces the existing step configuration with a brand new workflow step for the current URL. ## Self-Heal API Access Besides using Self-Healing in the UI platform, MrScraper also provides the **Self-Heal API**. This allows you to trigger automated workflow repairs programmatically from a monitoring script, a scheduled task, or your own application whenever a scraper detects empty or invalid results. ### Accessing Self-Heal API in the Dashboard You can retrieve the API endpoint, your scraper ID, and your API token directly from the scraper dashboard: Open your manual scraper in the builder dashboard. Click the **⋮** (more options) menu icon at the top right of the page. Select **Self-Heal API Access** from the menu. In the popup modal, enter or copy your API token (or click **Generate** to create a new one). The modal provides a ready-made cURL example pre-configured with your `scraperId` and token. See [Authentication](/docs/api/authentication) for more details on managing API tokens. ### Request ```bash title="Self-heal a manual scraper" curl --request POST "https://api.app.mrscraper.com/api/v1/scrapers-manual/{scraperId}/heal" \ --header "accept: application/json" \ --header "x-api-token: YOUR_API_TOKEN" ``` | Parameter | Type | Location | Required | Description | | ------------- | ------ | -------- | -------- | ---------------------------------------------------------------------------------------------- | | `x-api-token` | string | header | Yes | Your MrScraper API token | | `scraperId` | string | path | Yes | The ID of the manual scraper you want to repair (found in the scraper URL or API Access modal) | See [Self-Heal a Manual Scraper](/docs/api/v3/scraper/manual-heal) in the API reference for the complete OpenAPI schema and error handling details. ### Response ```json title="Successful response" { "message": "Successful operation!", "data": { "success": true, "scraperId": "", "workflowUpdated": true } } ``` The endpoint asks the AI engine to repair the current workflow. The scraper's saved workflow is updated **only** when a valid workflow response is returned. If the repair attempt fails, your existing workflow configuration remains unchanged. # Super Mode import { Step, Steps } from 'fumadocs-ui/components/steps'; ## Why Use Super Mode? Some websites enforce strong anti-bot protection. By default, Manual Scraper accesses pages through our cloud infrastructure, which these protections may block. Super mode routes your request through a real device instead, significantly improving the chance of bypassing such protection. ## How to Use Super Mode Super mode is not a toggle inside the scraper builder. To run a Manual Scraper workflow in Super mode, follow these steps : Open the manual scraper you want to run in Super mode. Click the **Ellipsis** ( **⋮** ) button in the top right and click **Manual Scraper API Access**. In the cURL example shown in the modal, copy the `workflow` array. Paste the workflow into the `workflow` field of the request below, then fill in your API token, target URL, and the country you want to access the URL from. ### Request ```bash title="Run a manual scraper workflow in Super mode" curl --location 'https://api.mrscraper.com?token={MRSCRAPER_API_TOKEN}&geoCode=us&html=true&super=true&proxyCountry=us' \ --header 'x-api-token: {MRSCRAPER_API_TOKEN}' \ --header 'Content-Type: application/json' \ --data '{ "url": "{YOUR_TARGET_URL}", "homePage": true, "proxyCountry" : "us", "workflow": [ ] }' ``` | Parameter | Location | Description | | ----------------------- | -------------- | --------------------------------------------------------------------------------------------- | | `super` | query | Set to `true` to route the request through a real device instead of our cloud infrastructure. | | `token` / `x-api-token` | query / header | Your MrScraper API token. | | `geoCode` | query | ISO 3166-1 alpha-2 country code used for proxy routing. | | `html` | query | Set to `true` to include the raw page HTML in the response. | | `proxyCountry` | query / body | Country you want to access the target URL from. | | `url` | body | The target URL to scrape. | | `homePage` | body | Set to `true` to visit the website's home page first, then navigate to the target URL. | | `workflow` | body | The workflow steps copied from the **Manual Scraper API Access** modal. | ### Visiting the Home Page First Some websites block requests that land directly on a deep URL. eBay, for example, expects visitors to arrive from the home page. Setting `homePage` to `true` makes the request load the home page first and then navigate to your target URL, which mimics normal browsing behavior. | Trade-off | Detail | | --------- | ------------------------------------------------------------------------------ | | **Pros** | More reliable and robust against blocks on sites that check how you arrived. | | **Cons** | Higher bandwidth usage and higher latency, since each run loads an extra page. | Leave `homePage` off by default. Enable it when a target URL is blocked or returns empty data on its own, and you are willing to trade extra bandwidth and run time for a higher success rate. See [Authentication](/docs/api/authentication) for more details on managing API tokens. ### Response ```json title="Example response" { "code": null, "screenshots": [], "recording_path": null, "extractions": "{\"Product Link\": [{\"follow_url\": \"https://books.toscrape.com//catalogue/a-light-in-the-attic_1000/index.html\", \"result\": {\"product_name\": \"\\n \\n In stock (22 available)\\n \\n\"}}]}", "data": { "Product Link": [ { "follow_url": "https://books.toscrape.com//catalogue/a-light-in-the-attic_1000/index.html", "result": { "product_name": "\n \n In stock (22 available)\n \n" } } ] }, "data_path": "results/b09aab93-717c-4804-9d37-a37084be4627/data.json", "html_path": "results/b09aab93-717c-4804-9d37-a37084be4627/page.html", "error": null, "residential_proxy_usage": { "received": 0.157379150390625, "sent": 0.0006866455078125, "total": 0.1580657958984375 }, "runtime": 13.358241081237793, "token_usage": 2, "listen_network_data": {}, "html": " ... full page HTML truncated ... ", "markdown": "", "screenshot": "", "retry_count": 0 } ``` You can also review the run and its output on the [Results page](https://app.mrscraper.com/scrapers-results) in the app. # Templates import { Step, Steps } from 'fumadocs-ui/components/steps'; The Manual Scraper combines low-code workflow steps with custom JavaScript. Use low-code steps to click elements, scroll the page, follow URLs, listen to network requests, add delays, and extract text with CSS selectors. For more advanced workflows, the **Inject JavaScript** step runs browser-side code—the same code you would run in the developer console—to extract data with selectors, call page APIs, or intercept network responses. ## Prerequisites Before using these templates: * Replace all placeholder selectors and API endpoints with values from your target website. * Each script is an [Immediately Invoked Function Expression (IIFE)](https://developer.mozilla.org/en-US/docs/Glossary/IIFE) and must return JSON-serializable data. * Test your selectors and scripts in the browser developer console before adding them to a workflow. ## Parse Page Data Use this template when the data you need is already rendered in the page HTML. ### Setup Steps Open the target URL in a Manual Scraper. If the content loads dynamically, add a **Delay** step before extraction. Add an **Inject JavaScript** step. Set an **Extraction Name** (for example, `products`). Paste the script below into the script box and replace the example selectors with selectors from your target page. ```js title="Parse page data with querySelectorAll" (() => { const products = Array.from( document.querySelectorAll(".product-card") ).map((product) => ({ title: product.querySelector(".product-title")?.textContent?.trim() || null, price: product.querySelector(".product-price")?.textContent?.trim() || null, url: product.querySelector("a")?.href || null, })); return products; })(); ``` Test your selectors in the browser console first. Run `document.querySelectorAll(".product-card").length` and confirm it returns the expected number of elements before adding the script to your workflow. ## Fetch API Data Use this template when the page loads its data from an API endpoint. You can identify the endpoint with the browser's **Network → Fetch/XHR** panel or a Manual Scraper **Listen Network** step. ### Setup Steps Open the target URL in a Manual Scraper. Add an **Inject JavaScript** step. Set an **Extraction Name** (for example, `apiData`). Replace `/api/products` with the endpoint used by the target page. ```js title="Fetch data from a page API" (async () => { const apiUrl = new URL("/api/products", window.location.origin); const response = await fetch(apiUrl, { method: "GET", credentials: "include", headers: { Accept: "application/json", }, }); if (!response.ok) { throw new Error(`API request failed with status ${response.status}`); } const data = await response.json(); return data; })(); ``` This template works best for same-origin APIs that use the page's existing cookies. Some endpoints also require request headers, query parameters, or tokens copied from the website's own request. ## Scroll and Capture Fetch Responses Use this template on infinite-scroll pages that load additional data through the browser Fetch API. The script intercepts matching JSON responses while scrolling, then returns all collected data once scrolling completes. ### Setup Steps Open the listing page in a Manual Scraper. Add an **Inject JavaScript** step. Set a **Script timeout** long enough for the scrolling loop to finish (for example, `30000` ms for a page with many scroll loads). Replace `/api/products` with a unique part of the endpoint you want to capture. ```js title="Scroll and capture Fetch API responses" (async () => { const endpointPattern = "/api/products"; const capturedResponses = []; const pendingCaptures = []; const originalFetch = window.fetch; const wait = (milliseconds) => new Promise((resolve) => setTimeout(resolve, milliseconds)); const patchedFetch = async (...args) => { const response = await originalFetch.apply(window, args); const requestUrl = typeof args[0] === "string" ? args[0] : args[0]?.url || ""; if (requestUrl.includes(endpointPattern)) { const capture = response .clone() .json() .then((data) => { capturedResponses.push({ url: requestUrl, data, }); }) .catch(() => { // Ignore matching responses that are not JSON. }); pendingCaptures.push(capture); } return response; }; window.fetch = patchedFetch; try { let previousHeight = document.body.scrollHeight; let stableRounds = 0; for (let attempt = 0; attempt < 20; attempt += 1) { window.scrollTo(0, document.body.scrollHeight); await wait(1500); const currentHeight = document.body.scrollHeight; stableRounds = currentHeight === previousHeight ? stableRounds + 1 : 0; previousHeight = currentHeight; if (stableRounds >= 2) { break; } } await wait(1000); await Promise.allSettled(pendingCaptures); return capturedResponses; } finally { if (window.fetch === patchedFetch) { window.fetch = originalFetch; } } })(); ``` This template captures requests made with `window.fetch` after the script starts. It does not capture `XMLHttpRequest`, requests made before injection, or non-JSON responses. Use the **Listen Network** step when you need broader network capture. ## Run the Workflow After adding a template to your workflow: Replace all placeholder selectors, endpoints, and timeouts with values from the target website. Click **Save**. Run the scraper and inspect the returned field that matches the extraction name you set. If the result is empty or the website blocks the request, see [Troubleshooting](/docs/features/manual-scraper#troubleshooting) on the Manual Scraper page. # Get Screenshot ## Overview Selecting **Get Screenshot** in the Playground Use Case panel pre-configures the Playground environment specifically for visual page capture. When this preset is active, the Playground enables real browser rendering (`browserRendering=true`) by default so you can visually verify how dynamic web pages, client-side scripts, and layout elements render. You can toggle between capturing the visible viewport (`top`) or rendering the full scrollable page (`full`). ### Practical Applications * **Visual QA & Deployment Audits**: Verify visual layout integrity across dynamic responsive viewports after deployments or design updates. * **Compliance & Legal Archiving**: Maintain time-stamped visual proof of pricing, terms of service, disclosures, or ad placements. * **Competitor UI & Campaign Tracking**: Monitor visual changes on competitor landing pages, promotional banners, and checkout flows over time. * **Scraper & Extraction Debugging**: Inspect the exact page state when data extraction fails—instantly spot CAPTCHAs, cookie banners, bot blocks, or unexpected layouts. * **Automated Previews & Dashboards**: Generate real-time page thumbnails and visual previews for internal dashboards, link previews, or reports. ## Default Parameters | Parameter | Description | | --------------------- | ---------------------------------------------------- | | html=true | Used by this endpoint to return html in the response | | browserRendering=true | In this use case, `Render JavaScript` default to On | | waitUntil | Default to `DOM Content Loaded` | | timeout | Default to `300` | ## Example ### Request Scrape `https://www.walmart.com/search?q=table%20lamps` with `Record` toggled on, `Render JavaScript` toggled on, and `Screenshot area` set to `Full` ```sh title="Request" curl --location --request POST 'https://api.mrscraper.com?token={MRSCRAPER_API_TOKEN}&browserRendering=true&waitUntil=networkidle0&timeout=300&geoCode=us&html=true&screenshot=full&record=true&saveResult=true&proxyCountry=us' \ -H 'x-api-token: {MRSCRAPER_API_TOKEN}' \ -H 'Content-Type: application/json' \ -d '{ "url": "https://www.walmart.com/search?q=table%20lamps" }' ``` ### Response Images : Recording :