S3 Storage

S3 Storage saves your scraper results as files instead of returning all the data in each API response. When you enable it, MrScraper uploads each result to S3 storage and returns a signed URL you can use to download the file.

Why Use S3 Storage

Returning a large result directly in an API response can be slow and may exceed response size limits. S3 Storage helps you:

• Keep a durable copy. Store each result as a file you can download or reprocess later.
• Fit results into a pipeline. Pass the signed URL to another service, or queue results to process in a separate step.

Enable S3 Storage

To save your scraped data to an S3 bucket, follow these steps:

1. Open your scraper detail page

2. Click the ellipsis button in the top right corner and click Result Storage.

3. Toggle the Enable Result Storage option.

   Note

   Results are stored as JSON files matching your scraper's output format. The signed URL is returned in the data field of each result.

4. Save your settings.

5. The scraper will returns a signed URL:

   {
       "value": "https://api.app.mrscraper.com/api/v1/attachments/results/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxxxx.json"
   }