Bulk Scraping
Learn how to efficiently scrape multiple URLs using our bulk scraping features.
MrScraper's Bulk Scraping feature allows you to extract data from multiple URLs in a single operation. Instead of scraping URLs one at a time, you can upload a list and apply the same scraping configuration to all of them, making it easy to gather large datasets efficiently.
Best For
Scraping multiple pages with similar structures, such as product listings, article archives, user profiles, or directory entries.
How to Configure Bulk Scraping
Step 1: Prepare Your URLs
You have two options for providing URLs:
Option A: Excel File Upload
Create an Excel file (.xlsx or .xls) with a column header named url or URL. Each row should contain one URL.
Excel Format Example
Your spreadsheet should look like this:
Option B: Direct Input
Paste URLs directly into the text area, one URL per line:
- https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
- https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html
- https://books.toscrape.com/catalogue/soumission_998/index.html
- https://books.toscrape.com/catalogue/sharp-objects_997/index.html
Step 2: Upload Your URLs
- Open a new or existing scraper in your MrScraper dashboard
- Click the Multiple URLs button in the top section
- Choose your input method:
- Click Upload File to select an Excel file from your computer
- Or paste URLs directly into the text area
- Click Save to confirm your URL list
Step 3: Run the Bulk Scrape
- Click Run All to start the bulk scraping process
- Click Result to view the scraper’s progress and results in real time
Token Usage
Each URL in your bulk scrape consumes tokens. Ensure you have sufficient tokens in your account before starting a large bulk operation. Check your token balance in your account settings.
Cancel Bulk Scraping
You can cancel a running bulk scraping job at any time. For more details, refer to the Bulk Scraping API.
Step 4: Monitor Progress
The Result page displays real-time progress information:
While Scraping:
{
"mergedData": null,
"urlDetails": [],
"summary": {
"totalUrls": 4,
"successfulUrls": 0,
"failedUrls": 0,
"scrapedCount": 0,
"totalTokenUsage": 0,
"estimatedFinishAt": null
}
}Progress Indicators:
| Field | Description |
|---|---|
totalUrls | Total number of URLs being scraped |
successfulUrls | Number of URLs scraped successfully |
failedUrls | Number of URLs that encountered errors |
scrapedCount | Current number of completed scrapes |
totalTokenUsage | Total tokens consumed so far |
estimatedFinishAt | Estimated completion time |
Once scraping completes, the Result page displays all extracted data:
[
{
"1": {
"id": "a897fe39b1053632",
"name": "A Light in the Attic",
"price": "£51.77",
"rating": null,
"source": "product",
"features": [
"Classic collection of poetry and drawings from Shel Silverstein",
"20th anniversary special edition",
"Humorous and creative verse"
],
"return_policy": null,
"shipping_info": {
"availability": "In stock (22 available)",
"shipping_arrival": null
},
"specifications": {
"UPC": "a897fe39b1053632",
"Product Type": "Books",
"Price (excl. tax)": "£51.77",
"Price (incl. tax)": "£51.77",
"Tax": "£0.00",
"Availability": "In stock (22 available)",
"Number of reviews": "0"
}
}
},
{
"1": {
"id": "90fa61229261140a",
"name": "Tipping the Velvet",
"price": "£53.74",
"rating": null,
"source": "product",
"features": [
"Erotic and absorbing...Written with starling power.",
"Nan King, an oyster girl, is captivated by the music hall phenomenon Kitty Butler"
],
"shipping_info": {
"availability": "In stock (20 available)",
"shipping_arrival": null
},
"specifications": {
"UPC": "90fa61229261140a",
"Product Type": "Books",
"Price (excl. tax)": "£53.74",
"Price (incl. tax)": "£53.74",
"Tax": "£0.00",
"Availability": "In stock (20 available)",
"Number of reviews": "0"
}
}
}
]For more details, refer to the Bulk Scraping API.