Bulk Scraping With Custom Scraper
Run a custom scraper workflow against many URLs in a single bulk submission, then poll for progress and results.
The Bulk API lets you run one custom scraper workflow against a list of URLs in a single submission, instead of sending a separate request for every page.
Bulk runs are asynchronous. You submit the workflow and its URLs, receive a submission ID immediately, then poll that ID for progress and results.
What Is a Custom Scraper?
A custom scraper is a Playground request that carries its own manual workflow. Instead of pointing at a saved scraper, you send the workflow steps inline with the request, so the same extraction logic can be applied to any URL you pass in.
Submit a Bulk Run
Copy the body response from your custom manual scraper.
Paste it into the "workflow": {} of the request below.
Fill in your API key, the proxy country, and the list of URLs you want to scrape.
Send the request and save the submission_id from the response.
Request
curl --location 'https://bulk.mrscraper.com/api/gateway/v1/bulk-submissions/run' \
--header 'Content-Type: application/json' \
--header 'x-api-key: {MRSCRAPER_API_KEY}' \
--header 'Cookie: sl-session=P89TLJQZiGoIfOVndtHSPg==' \
--data '{
"engine": "http",
"workflow": {
"url": "https://books.toscrape.com/",
"homePage": true,
"proxyCountry" : "us",
"workflow": [
{
"type": "script",
"data": {
"name": "first_book",
"timeout": 30,
"code": "(function() {\n const firstArticle = document.querySelector('\''article.product_pod'\'');\n if (!firstArticle) {\n return { title: null, price: null };\n }\n \n const titleLink = firstArticle.querySelector('\''h3 a'\'');\n const title = titleLink ? (titleLink.getAttribute('\''title'\'') || titleLink.textContent).trim() : null;\n \n const priceEl = firstArticle.querySelector('\''p.price_color'\'');\n const price = priceEl ? priceEl.textContent.trim() : null;\n \n return { title: title || null, price: price || null };\n})();"
}
}
]
},
"request_method": "POST",
"proxy_country": "us",
"urls": [
"https://books.toscrape.com/",
"https://books.toscrape.com/catalogue/page-2.html",
"https://books.toscrape.com/catalogue/page-3.html"
]
}'| Parameter | Location | Description |
|---|---|---|
x-api-key | header | Your MrScraper API key. |
Cookie | header | Session cookie for the bulk gateway. Send the default value shown above. |
engine | body | Execution engine used for the run. Use http. |
workflow | body | The scraper configuration applied to every URL in the run. |
workflow.url | body | The first page to scrape. |
workflow.homePage | body | Set to true to visit the website's home page first, then navigate to the target URL. |
workflow.proxyCountry | body | Country the workflow accesses the pages from. |
workflow.workflow | body | The workflow steps copied from your Playground scraper. |
request_method | body | HTTP method used to request each URL. |
proxy_country | body | Proxy country for the bulk submission. |
urls | body | Array of URLs to run the workflow against. |
Response
The submission is accepted and queued immediately. The run itself continues in the background.
{
"data": {
"submission_id": "01M0HD8JB6YXN2RZM2AC48391M",
"engine": "http",
"priority": 1,
"status": "queued",
"total_urls": 3,
"succeeded": 0,
"failed": 0,
"dead": 0,
"pending": 3,
"percent_complete": 0,
"created_at": "2026-08-21T05:38:52.902189Z",
"started_at": "2026-08-21T05:38:52.939086Z",
"finished_at": null
},
"message": "submission running"
}Save the submission ID
The submission_id is the only way to retrieve the results of a bulk run. Store it before moving on.
Check Progress and Results
Use the submission ID to poll the run. The same endpoint returns both the current progress counters and any results extracted so far.
Request
curl --location 'https://bulk.mrscraper.com/api/results/v1/bulk-submissions/{SUBMISSION_ID}?limit=25' \
--header 'x-api-key: {MRSCRAPER_API_KEY}' \
--header 'Cookie: sl-session=P89TLJQZiGoIfOVndtHSPg=='| Parameter | Location | Description |
|---|---|---|
x-api-key | header | Your MrScraper API key. |
Cookie | header | Session cookie for the bulk gateway. Send the default value shown above. |
SUBMISSION_ID | path | The ID returned when you submitted the bulk run. |
limit | query | Number of results to return per page. Maximum is 100. |
Response While Running
While the run is still in progress, results is empty or partially filled and percent_complete is below 100.
{
"data": {
"submission_id": "01M0HDS6DSWSSFTCT1JCDTA0A2",
"engine": "http",
"status": "queued",
"total_urls": 3,
"succeeded": 0,
"failed": 0,
"dead": 0,
"pending": 3,
"percent_complete": 0,
"created_at": "2026-08-21T05:47:57.753168Z",
"started_at": "2026-08-21T05:47:57.783051Z",
"finished_at": null,
"results": [],
"next_cursor": null
},
"message": "run found"
}Response When Finished
Once every URL has been processed, percent_complete reaches 100 and results contains one entry per URL.
{
"data": {
"submission_id": "01M0HD8JB6YXN2RZM2AC48391M",
"engine": "http",
"status": "queued",
"total_urls": 3,
"succeeded": 3,
"failed": 0,
"dead": 0,
"pending": 0,
"percent_complete": 100,
"created_at": "2026-08-21T05:38:52.902189Z",
"started_at": "2026-08-21T05:38:52.939086Z",
"finished_at": null,
"results": [
{
"id": "01M0HD8JBQKH2WFKWZ0KKQ15BB:0",
"job_id": "01M0HD8JBQKH2WFKWZ0KKQ15BB",
"url": "https://books.toscrape.com/catalogue/page-2.html",
"http_status": 200,
"data": {
"first_book": {
"price": "£12.84",
"title": "In Her Wake"
}
},
"raw_ref": "2026/08/21/01M0HD8JBQKH2WFKWZ0KKQ15BB-a0.html.zst",
"screenshot_ref": null,
"engine": "http",
"extracted_at": "2026-08-21T05:39:15.841670Z"
},
{
"id": "01M0HD8JBQAS5DN2ZDWQ7M7MGN:0",
"job_id": "01M0HD8JBQAS5DN2ZDWQ7M7MGN",
"url": "https://books.toscrape.com/catalogue/page-3.html",
"http_status": 200,
"data": {
"first_book": {
"price": "£57.31",
"title": "Slow States of Collapse: Poems"
}
},
"raw_ref": "2026/08/21/01M0HD8JBQAS5DN2ZDWQ7M7MGN-a0.html.zst",
"screenshot_ref": null,
"engine": "http",
"extracted_at": "2026-08-21T05:39:14.414348Z"
},
{
"id": "01M0HD8JBQJ2TRGA8CNKQC9Z2B:0",
"job_id": "01M0HD8JBQJ2TRGA8CNKQC9Z2B",
"url": "https://books.toscrape.com/",
"http_status": 200,
"data": {
"first_book": {
"price": "£51.77",
"title": "A Light in the Attic"
}
},
"raw_ref": "2026/08/21/01M0HD8JBQJ2TRGA8CNKQC9Z2B-a0.html.zst",
"screenshot_ref": null,
"engine": "http",
"extracted_at": "2026-08-21T05:39:14.089989Z"
}
],
"next_cursor": null
},
"message": "run found"
}Result Fields
| Field | Description |
|---|---|
total_urls | Number of URLs in the submission. |
succeeded / failed / dead / pending | Per-URL status counters for the run. |
percent_complete | Progress of the run, from 0 to 100. |
results[].url | The URL this result was extracted from. |
results[].http_status | HTTP status code returned by the target page. |
results[].data | Data extracted by your workflow steps, keyed by step name. |
results[].raw_ref | Reference to the stored raw page content. |
results[].screenshot_ref | Reference to the stored screenshot, if one was captured. |
next_cursor | Cursor for the next page of results. null when there are no more. |
Result order
Results are not returned in the order the URLs were submitted. Match each result to its source page using the url field rather than its position in the array.