MrScraper Documentation

Use the MrScraper CLI to scrape, crawl, and extract data from the web directly from your terminal.

The MrScraper CLI allows you to fetch rendered HTML, create AI scrapers, rerun existing scrapers, and retrieve results directly from your terminal. Perfect for automation, testing, and integration with AI coding agents.

Requirements

Node.js 18 or higher
A MrScraper API key from the dashboard

Package Information

See @mrscraper/cli on npm for the latest release and version information.

Installation

Global Installation (Recommended)

npm install -g @mrscraper/cli
mrscraper --help

Run Without Installing

npx @mrscraper/cli --help
npx @mrscraper/cli scrape "https://example.com/"

Install from Source

git clone https://github.com/mrscraper-com/cli.git
cd cli
npm install
node bin/mrscraper.js --help

Authentication

Before using the CLI, you need to authenticate with your MrScraper API key.

mrscraper login

The CLI will prompt you to enter your API key securely.

For CI/CD pipelines or scripts:

mrscraper login --api-key YOUR_API_KEY

Environment Variable

Set your API key in your shell:

export MRSCRAPER_API_KEY=YOUR_API_KEY

Per-Command Authentication

Override authentication for a single command:

mrscraper scrape "https://example.com" --token YOUR_API_KEY

Authentication Priority

The CLI uses authentication in this order:

--token flag (highest priority)
Environment variables (MRSCRAPER_API_KEY or MRSCRAPER_API_TOKEN)
Saved credentials from mrscraper login
No authentication (command will fail)

Quick Start

# 1. Install globally
npm install -g @mrscraper/cli

# 2. Login
mrscraper login

# 3. Scrape a webpage
mrscraper scrape "https://www.ebay.com/itm/266727555514" \
  --prompt "get name, price, features/description, images, seller, and shipping"

Commands

Save your API key to the credential file for persistent authentication.

# Interactive login
mrscraper login

# Non-interactive (for CI/CD)
mrscraper login --api-key YOUR_API_KEY

Options

Option	Description
`--api-key <key>`	Provide API key non-interactively
`--token <key>`	Deprecated alias for --api-key on this command only

Credential Storage

Your API key is stored securely:

macOS/Linux: ~/.config/mrscraper/credentials.json
Windows: %LOCALAPPDATA%\mrscraper\credentials.json

The file contains: {"api_key": "..."}

Init

Same as login, but prints a welcome message first. Useful for first-time setup.

mrscraper init

Logout

Delete the saved credential file.

mrscraper logout

Scrape

Scrape a single URL with AI-powered extraction or fetch raw HTML.

Two Modes:

HTML Mode - Fetch rendered HTML without AI extraction
AI Mode - Create an AI scraper with natural language instructions

HTML Mode

Fetch rendered HTML without creating a scraper:

# Basic HTML fetch
mrscraper scrape "https://example.com"

# With geo-targeting
mrscraper scrape "https://example.com" --geo-code US --timeout 120

# Block resources for faster loading
mrscraper scrape "https://example.com" --block-resources

HTML Mode Options

Option	Default	Required	Description
`--geo-code <code>`	`US`	Yes	ISO country code for proxy
`--timeout <seconds>`	`120`	Yes	Maximum wait time
`--block-resources`	`false`	Yes	Block images, CSS, fonts for faster loading

AI Mode

Create an AI scraper with natural language extraction:

# General agent (default)
mrscraper scrape "https://example.com/product" \
  --prompt "Extract product name, price, description, and availability"

# Listing agent for multi-page content
mrscraper scrape "https://example.com/products" \
  --agent listing \
  --prompt "Extract title, price, and URL" \
  --max-pages 5

# Map agent for URL discovery
mrscraper scrape "https://example.com" \
  --agent map \
  --max-depth 2 \
  --max-pages 50 \
  --limit 1000

AI Mode Options

Option	Default	Required	Description
`--prompt <text>`	`"Get all data as complete as possible"`	Yes	Natural language extraction instructions
`--agent <type>`	`general`	Yes	Agent type: `general`, `listing`, or `map`
`--proxy-country <code>`	-	Yes	Country code for proxy (e.g., `US`, `UK`)
`--token <key>`	-	No	Override saved key / env
`--max-pages <n>`	`1` (listing), `50` (map)	Yes	Maximum pages to scrape
`--max-depth <n>`	`2`	Yes	Crawl depth for map agent
`--limit <n>`	`1000`	Yes	Maximum results for map agent
`--include-patterns <regex>`	`""`	Yes	URL patterns to include (map agent)
`--exclude-patterns <regex>`	`""`	Yes	URL patterns to exclude (map agent)

Examples:

# Product page extraction
mrscraper scrape "https://www.ebay.com/itm/266727555514" \
  --prompt "get name, price, features, images, seller, and shipping"

# Multi-page listing
mrscraper scrape "https://books.toscrape.com/catalogue/category/books_1/index.html" \
  --agent listing \
  --prompt "Extract book title, price, and availability" \
  --max-pages 3

# Website mapping
mrscraper scrape "https://docs.mrscraper.com" \
  --agent map \
  --include-patterns "/docs/.*" \
  --exclude-patterns "/admin"

Rerun

Rerun an existing scraper on new URLs.

# Single URL Rerun
# Rerun AI scraper
mrscraper rerun "https://example.com/new-page" \
  --type ai \
  --scraper-id SCRAPER_UUID

# Rerun manual scraper
mrscraper rerun "https://example.com/new-page" \
  --type manual \
  --scraper-id SCRAPER_UUID

# Bulk Rerun
# Bulk rerun (comma-separated)
mrscraper rerun "https://a.com,https://b.com,https://c.com" \
  --bulk \
  --type ai \
  --id SCRAPER_UUID

# Bulk rerun (newline-separated)
mrscraper rerun "https://a.com
https://b.com
https://c.com" \
  --bulk \
  --type manual \
  --id SCRAPER_UUID

Options

Option	Default	Required	Description
`--type <type>`	-	Yes	Scraper type: `ai` or `manual`
`--scraper-id <uuid>`	-	Yes (single)	Scraper ID for single URL rerun
`--id <uuid>`	-	Yes (bulk)	Scraper ID for bulk rerun
`--bulk`	Yes	-	Enable bulk mode for multiple URLs
`--token <key>`	-	No	Override saved key / env
`--max-depth <n>`	`2`	No	For AI scrapers: crawl depth
`--max-pages <n>`	`50`	No	For AI scrapers: maximum pages
`--limit <n>`	`1000`	For AI scrapers: maximum results
`--include-patterns <regex>`	`""`	No	URL patterns to include
`--exclude-patterns <regex>`	`""`	No	URL patterns to exclude

Important

Manual scrapers ignore map-style options (max-depth, max-pages, etc.). These options only apply to AI scrapers.
Bulk rerun is asynchronous. To check the results, use results command as explained below.

Results

List all scraping results with pagination and filtering.

# List recent results
mrscraper results

# Paginated results
mrscraper results --page-size 20 --page 2

# Sort by creation date
mrscraper results --sort-field createdAt --sort-order ASC

# Search results
mrscraper results --search "example.com"

# Filter by date range
mrscraper results \
  --date-range-column updatedAt \
  --start-at "2024-01-01" \
  --end-at "2024-12-31"

Options

Option	Default	Description
`--sort-field <field>`	`updatedAt`	Sort by: `createdAt`, `updatedAt`, `id`, `type`, `url`, `status`, `error`, `tokenUsage`, `runtime`
`--sort-order <order>`	`DESC`	Sort direction: `ASC` or `DESC`
`--page-size <n>`	`10`	Results per page
`--page <n>`	`1`	Page number (1-based)
`--search <query>`	-	Search filter
`--date-range-column <col>`	-	Date column for filtering
`--start-at <date>`	-	Range start (ISO 8601)
`--end-at <date>`	-	Range end (ISO 8601)

Result

Fetch a single result by ID.

# Get result by ID
mrscraper result RESULT_UUID

# Alternative syntax
mrscraper result --id RESULT_UUID

Options

Option	Description
`--id <uuid>`	Result UUID (alternative to positional argument)

Global Options

These options are available for the root program (before a subcommand):

Option	Description
`-v`, `--version`	Print the CLI version and exit.
`-h`, `--help`	Print help and exit.

Note

Running mrscraper with no subcommand prints the same help.

Typical Workflow

Here's a common workflow for using the MrScraper CLI:

Authenticate

mrscraper login

Create a Scraper

mrscraper scrape "https://example.com/product" \
  --agent general \
  --prompt "Extract product name, price, and description"

The command returns JSON with a scraper_id.

Rerun on New URLs

mrscraper rerun "https://example.com/another-product" \
  --type ai \
  --scraper-id SCRAPER_UUID

Use the same scraper configuration on different URLs.

Check Results

# List all results
mrscraper results

# Get specific result
mrscraper result RESULT_UUID

Poll until the status indicates completion.

Environment Variables

The CLI supports environment variables for configuration:

Variable	Description
`MRSCRAPER_API_KEY`	Your API key (preferred)
`MRSCRAPER_API_TOKEN`	Alternative name for API key

.env File Support:

The CLI automatically loads .env files from your current directory:

.env

MRSCRAPER_API_KEY=your_api_key_here

Programmatic Use

The CLI package exports helpers for Node.js applications:

import {
  VERSION,
  loadSavedApiKey,
  createAiScraperApi,
  fetchHtmlApi,
} from "@mrscraper/cli";

// Load saved credentials
const apiKey = await loadSavedApiKey();

// Create AI scraper
const scraper = await createAiScraperApi({
  url: "https://example.com",
  message: "Extract all products",
  agent: "general",
  token: apiKey,
});

// Fetch HTML
const html = await fetchHtmlApi({
  url: "https://example.com",
  token: apiKey,
});

Examples

Product Scraping

mrscraper scrape "https://www.amazon.com/dp/B08N5WRWNW" \
  --prompt "Extract product name, price, rating, features, and images"

Multi-Page Listing

mrscraper scrape "https://www.realtor.com/realestateandhomes-search/San-Francisco_CA" \
  --agent listing \
  --prompt "Extract property address, price, bedrooms, bathrooms, and listing URL" \
  --max-pages 5

Website Crawling

mrscraper scrape "https://docs.mrscraper.com" \
  --agent map \
  --max-depth 3 \
  --include-patterns "/docs/.*" \
  --limit 500

Bulk Processing

# Create a list of URLs
echo "https://example.com/product1
https://example.com/product2
https://example.com/product3" > urls.txt

# Bulk rerun
mrscraper rerun "$(cat urls.txt)" \
  --bulk \
  --type ai \
  --id SCRAPER_UUID

CI/CD Integration

name: Scrape Data
on:
  schedule:
    - cron: '0 0 * * *'  # Daily at midnight

jobs:
  scrape:
    runs-on: ubuntu-latest
    steps:
      - name: Install CLI
        run: npm install -g @mrscraper/cli
      
      - name: Login
        run: mrscraper login --api-key ${{ secrets.MRSCRAPER_API_KEY }}
      
      - name: Scrape Data
        run: |
          mrscraper scrape "https://example.com" \
            --agent general \
            --prompt "Extract all data" > results.json
      
      - name: Upload Results
        uses: actions/upload-artifact@v2
        with:
          name: scrape-results
          path: results.json

CLI

On this page