LangChain SDK

Use MrScraper as LangChain tools via the langchain-mrscraper package to fetch web pages, search the web, and check account usage from agents.

The langchain-mrscraper package exposes the MrScraper API as LangChain BaseTool instances, so AI agents can fetch web pages as HTML, Markdown, or screenshots, search the web, and check account usage. It wraps the Python SDK (mrscraper-sdk), which is installed as a dependency. MrScraper is also listed in LangChain's web browsing tools.

Tools-First Integration

This integration provides tools that agents can explicitly call—not a document loader. For deterministic "URL → documents" ingestion into vector stores, a document loader is typically a better fit. MrScraper may offer that in a separate package later.

Requirements

Python 3.9 or higher
mrscraper-sdk 0.9.x and langchain-core 0.3 or later (installed as dependencies)

Package Information

This page documents version 0.5.0. See langchain-mrscraper on PyPI for the latest release, version history, and installation details.

Installation

Install the package from PyPI using pip:

pip install -U langchain-mrscraper

This installs both the LangChain integration and the underlying MrScraper SDK client.

Authentication

Set your API token from the MrScraper dashboard as an environment variable:

import os

os.environ["MRSCRAPER_API_KEY"] = "MRSCRAPER_API_TOKEN"

Authentication Options

The tools resolve the token in this order:

The token parameter, such as MrScraperToolkit(token="...")
The mrscraper_api_key parameter, such as load_mrscraper_tools(mrscraper_api_key="...")
The MRSCRAPER_API_KEY environment variable
The MRSCRAPER_API_TOKEN environment variable

You can also pass an existing mrscraper.MrScraper client with client=....

For more information on generating API tokens, see the Generate Token guide.

Quick Start

Using MrScraperToolkit

Load all three tools with the toolkit:

from langchain_mrscraper import MrScraperToolkit

# Using environment variable
tools = MrScraperToolkit().get_tools()

# Or with explicit token
tools = MrScraperToolkit(token="MRSCRAPER_API_TOKEN").get_tools()

Using Convenience Loader

Alternatively, use the convenience function. It accepts the same configuration as the toolkit:

from langchain_mrscraper import load_mrscraper_tools

tools = load_mrscraper_tools()

Using with AI Agents

from langgraph.prebuilt import create_react_agent
from langchain_openai import ChatOpenAI
from langchain_mrscraper import MrScraperToolkit

# Load MrScraper tools
tools = MrScraperToolkit(token="MRSCRAPER_API_TOKEN").get_tools()

# Create agent with OpenAI
agent = create_react_agent(ChatOpenAI(model="gpt-4o-mini"), tools)

# Use the agent
response = agent.invoke({
    "messages": [{"role": "user", "content": "Search the web for MrScraper and summarize the top result."}]
})

Prerequisites

This example requires langgraph and langchain-openai packages:

pip install langgraph langchain-openai

Available Tools

Class	Tool name	SDK method	Purpose
`MrScraperScraperAPI`	`mrscraper_scraper_api`	`scraper_api`	Fetch a public web page as rendered HTML, Markdown, screenshot, or a combination
`MrScraperWebSearch`	`mrscraper_web_search`	`web_search`	Search the web to retrieve real-time search results, including relevant website URLs, titles, and descriptions
`MrScraperAccount`	`mrscraper_account`	`account`	Read account identity, subscription, quota, token usage, and rate limits

Tool Outputs

Every tool returns a pretty-printed JSON string containing the SDK's status_code, data, and headers.

Core Methods

Scraper API

Fetch a public web page as rendered HTML, Markdown, screenshot, or a combination.

import base64
import json
from langchain_mrscraper import MrScraperScraperAPI

tool = MrScraperScraperAPI(token="MRSCRAPER_API_TOKEN")

# HTML only (the default) returns the HTML string in data
page = json.loads(tool.invoke({"url": "https://example.com"}))
print(page["data"][:200])

# Markdown or a screenshot returns an object with one entry per output
combined = json.loads(tool.invoke({
    "url": "https://example.com",
    "html": True,
    "markdown": True,
    "screenshot": True,
}))
print(combined["data"]["markdown"][:200])
image = base64.b64decode(combined["data"]["screenshot"])

At least one of html, markdown, or screenshot must be True. The screenshot is a base64 string. Setting screenshot=True enables browser rendering, so combining it with browser_rendering=False is rejected.

Parameters:

Parameter	Type	Required	Default	Description
`url`	string	Yes	-	HTTP or HTTPS URL to fetch
`html`	boolean	No	`True`	Return the rendered page HTML
`markdown`	boolean	No	`False`	Return the page as Markdown
`screenshot`	boolean	No	`False`	Return a base64 screenshot. Enables browser rendering
`timeout`	integer	No	`120`	Page timeout in seconds (1-300)
`geo_code`	string	No	`"US"`	Geographic country code
`proxy_country`	string	No	`None`	Proxy country code
`block_resources`	boolean	No	`False`	Block images, CSS, and fonts
`browser_rendering`	boolean	No	`None`	`True` forces JavaScript rendering, `False` disables it, `None` lets the service decide
`super`	boolean	No	`False`	Use real-device routing
`use_proxy`	boolean	No	`True`	Use a proxy for this request
`proxy`	string	No	`None`	Custom proxy in `username:password@host:port` format
`wait_for_selector`	string	No	`None`	CSS selector to wait for
`wait_until`	string	No	`None`	Page lifecycle event to wait for, such as `"domcontentloaded"`
`home_page`	boolean	No	`False`	Visit the site's root page first
`bypass_proxy`	boolean	No	`False`	Load static assets such as images, fonts and stylesheets outside the residential proxy to save bandwidth
`cookie_jar`	string	No	`None`	Cookie configuration string
`headers`	dict \| string	No	`None`	Target request headers, as a dictionary or JSON string
`max_retries`	integer	No	`3`	Maximum retries, 0 or greater
`token_cap`	integer	No	`None`	Maximum retry-token budget, 0 or greater
`stream`	boolean	No	`None`	Request a streamed response
`discard_token_secret`	string	No	`""`	Discard-token secret, if required

For more examples of what you can fetch, see Scrape HTML, Scrape Markdown, and Get Screenshot.

Web Search

Search the web to retrieve real-time search results, including relevant website URLs, titles, and descriptions.

import json
from langchain_mrscraper import MrScraperWebSearch

tool = MrScraperWebSearch(token="MRSCRAPER_API_TOKEN")

result = json.loads(tool.invoke({
    "query": "python web scraping",
    "country": "us",
    "num_results": 10,
}))
for item in result["data"]["data"]:
    print(item["rank"], item["title"], item["url"])

Parameters:

Parameter	Type	Required	Default	Description
`query`	string	Yes	-	Search query. Must not be blank
`country`	string	No	`None`	Country code for localized results, such as `"us"`. Omit to use the service default
`num_results`	integer	No	`10`	Number of results to return (10-100)
`timeout`	float	No	`600.0`	HTTP timeout in seconds

The response data contains success, message, and data, a list of ranked results. Each result has rank, title, url, description, and source.

Account

Read account identity, subscription status, token usage and limits, remaining quota, and rate limits.

from langchain_mrscraper import MrScraperAccount

tool = MrScraperAccount(token="MRSCRAPER_API_TOKEN")
print(tool.invoke({}))

Parameters:

Parameter	Type	Required	Default	Description
`timeout`	float	No	`30.0`	HTTP timeout in seconds

See Account in the Python SDK documentation for the fields in the account summary.

Tool Loading Options

You can obtain tools in several ways:

1. Toolkit (Recommended)

from langchain_mrscraper import MrScraperToolkit

tools = MrScraperToolkit(token="MRSCRAPER_API_TOKEN").get_tools()

2. Convenience Loader

from langchain_mrscraper import load_mrscraper_tools

tools = load_mrscraper_tools(token="MRSCRAPER_API_TOKEN")

3. Selective Tool Loading

tool_names filters by the exact tool names. An empty list returns no tools.

from langchain_mrscraper import MrScraperToolkit

tools = MrScraperToolkit(
    token="MRSCRAPER_API_TOKEN",
    tool_names=["mrscraper_scraper_api", "mrscraper_web_search"],
).get_tools()

4. Per-Tool Constructors

from langchain_mrscraper import MrScraperScraperAPI

scraper_api = MrScraperScraperAPI(token="MRSCRAPER_API_TOKEN")

5. Shared SDK Client

from mrscraper import MrScraper
from langchain_mrscraper import MrScraperToolkit

client = MrScraper(token="MRSCRAPER_API_TOKEN")
tools = MrScraperToolkit(client=client).get_tools()

Async Usage

Every tool supports both .invoke(...) and await .ainvoke(...):

import asyncio
from langchain_mrscraper import MrScraperScraperAPI

async def async_scrape():
    tool = MrScraperScraperAPI(token="MRSCRAPER_API_TOKEN")
    output = await tool.ainvoke({
        "url": "https://example.com",
        "timeout": 120,
    })
    print(output)

asyncio.run(async_scrape())

Synchronous .invoke(...) calls also work inside notebooks that already have a running event loop.

Best Practices

Token Management

Store your API token securely:

import os

# Load from .env file
from dotenv import load_dotenv
load_dotenv()

token = os.getenv("MRSCRAPER_API_KEY")

Error Handling

Each tool validates its input before sending a request. Invalid URLs, out-of-range limits, and unsupported fields are rejected by the tool schema. API, authentication, and network errors are raised from the Python SDK.

from langchain_mrscraper import MrScraperScraperAPI

try:
    tool = MrScraperScraperAPI(token="MRSCRAPER_API_TOKEN")
    output = tool.invoke({"url": "https://example.com"})
except Exception as e:
    print(f"Scraping error: {e}")

Tool Selection

Only load the tools your agent needs:

from langchain_mrscraper import MrScraperToolkit

# Good: Only load needed tools
tools = MrScraperToolkit(tool_names=["mrscraper_web_search"]).get_tools()

# Loads all three tools
tools = MrScraperToolkit().get_tools()

Migrating from 0.3.x

Version 0.5.0 requires mrscraper-sdk>=0.9.0,<0.10.0, which exposes exactly three public methods. The toolkit now returns three tools instead of ten.

0.3.x tool	0.5.0 replacement
`MrScraperHTML` (`mrscraper_scraper_html`)	`MrScraperScraperAPI` with `{"url": ...}`
`MrScraperMarkdown` (`mrscraper_scraper_markdown`)	`MrScraperScraperAPI` with `{"url": ..., "html": False, "markdown": True}`
`MrScraperScreenshot` (`mrscraper_get_screenshot`)	`MrScraperScraperAPI` with `{"url": ..., "html": False, "screenshot": True}`
`MrScraperJSON`, `MrScraperMultiplePage`, `MrScraperSitemap`, `MrScraperBrand`, `MrScraperGetBrand`	Removed. Fetch pages with `MrScraperScraperAPI` and let your model extract data, or pin `langchain-mrscraper<0.4`
`MrScraperWebSearch` (`url`, `raw`, `format`, `region`, `language`, `page`, `render_js`)	`MrScraperWebSearch` (`query`, `country`, `num_results`, `timeout`)
`MrScraperAccount`	Unchanged
`MrScraperFetchHTML` import alias	Removed. Use `MrScraperScraperAPI`

Screenshots now come back as a base64 string in data["screenshot"], together with any other requested outputs. The 0.3.x screenshot full_page option and its timeout=300 and wait_until="domcontentloaded" defaults no longer exist; pass timeout and wait_until explicitly when needed.
Update tool_names filters to mrscraper_scraper_api, mrscraper_web_search, and mrscraper_account.
The scraper and result tools from earlier versions (create, rerun, and bulk-rerun scrapers, and result listing) remain removed. Call the REST API directly for scrapers and results.

LangChain SDK

On this page