Token Plan
Learn how tokens work in MrScraper and how to create your own API token for authentication.
MrScraper uses a token-based system for both usage and authentication.
- Plan Tokens measure how much you can use MrScraper’s features, such as running or rerunning scrapers.
- API Tokens allow you to securely access the MrScraper API, for example, to rerun an existing scraper programmatically.
Understanding Plan Tokens
Each MrScraper billing plan includes a specific number of tokens. Tokens represent the compute resources your scraper consumes—similar to how minutes or data are counted in a phone plan.
The number of tokens used depends on the type of scraper you run.
AI Scraper Token Usage
AI Scrapers consume tokens based on several factors:
| Usage Type | Description | Conversion Rate |
|---|---|---|
| Run Tracking | Each scraper run consumes 1 token for trace tracking. | 1 token per run |
| Runtime | Time spent processing your task. | 1 token per 30 seconds of runtime |
| Input Tokens | Amount of text data (prompts or instructions) sent to the AI model. | 1 token per 30,000 input tokens |
| Output Tokens | Amount of text generated by the AI model (responses or extracted results). | 1 token per 7,500 output tokens |
Example
If your AI scraper runs for 40 seconds, processes 40,000 input tokens, and generates 14,000 output tokens, your total token usage would be:
| Component | Calculation | Tokens |
|---|---|---|
| Runtime | 40 ÷ 30 | 2 |
| Input | 40,000 ÷ 30,000 | 2 |
| Output | 14,000 ÷ 7,500 | 2 |
| Run Trace | Fixed | 1 |
| Total | 7 |
Note
- Each run automatically includes trace tracking for debugging and performance monitoring.
- Input and output tokens are based on how much text the AI model processes and generates.
- AI Scrapers tend to consume more tokens due to model processing.
- Runtime tokens are rounded up using the ceiling rule. For example, if your scraper runs for 40 seconds, it will count as 2 tokens (not 1).
Manual Scraper Token Usage
Manual Scrapers (non-AI) use tokens based on runtime and bandwidth.
| Usage Type | Description | Conversion Rate |
|---|---|---|
| Runtime | Time your scraper runs. | 1 token per 30 seconds |
| Bandwidth | Amount of data downloaded or uploaded. | 1 token per 2 MB |
Example
If your manual scraper runs for 40 seconds and uses 11 MB of bandwidth:
| Component | Calculation | Tokens |
|---|---|---|
| Runtime | 40 ÷ 30 | 2 |
| Bandwidth | 11 ÷ 2 | 6 |
| Total | 8 |
Note
Runtime tokens are rounded up using the ceiling rule. For example, if your scraper runs for 40 seconds, it will count as 2 tokens (not 1).
Why Token Usage Varies
Several factors affect how many tokens a scraper consumes:
- Website Complexity: Websites with heavy JavaScript, dynamic content, or multiple requests take longer to load and process, increasing runtime and token usage.
- Page Size and Content Density: Larger pages with long articles, complex html layout, or extensive metadata produce more input text for AI models to process.
- Data Volume: Extracting large sets of data (for example, multiple listings or paginated results) requires more compute and bandwidth.
- AI Processing Load: For AI Scrapers, complex instructions or long-structured expected output data could increase input and output token usage, since the model must interpret and generate more text.
Generating an API Token
An API Token lets your applications securely interact with the MrScraper API. It serves as your authentication key, allowing you to rerun scrapers that you’ve already created in the dashboard.
Follow these steps to generate your API token:
- Click your User Profile at the top-right corner of the page.
- Select API Tokens.
- Click New Token.
- Enter a name and set an expiration date.
- Click Create.
- Copy your new API token and store it securely.
Keep It Secret!
Never expose your API token in client-side code (like browsers or apps).
Always store it securely in an environment variable or server configuration.