Dify

Connect MrScraper to Dify to fetch web pages, search the web, and check account usage from agents, chatflows, and workflows.

The MrScraper plugin for Dify adds web retrieval and search to your Dify applications. Use its tools as workflow nodes or let an agent choose when to call them.

Available tools

Tool	Tool ID	Purpose
Account	`mrscraper_account`	Check your MrScraper account, token usage, and limits.
Scraper API	`mrscraper_scraper_api`	Retrieve a page as HTML, Markdown, a screenshot, or a combination.
Web Search	`mrscraper_web_search`	Search the web for real-time results, including website URLs, titles, and descriptions.

This guide covers plugin version 0.2.0, which requires Dify 1.17.0 or later. See the marketplace listing for releases and compatibility details.

Prerequisites

A Dify workspace with permission to install tool plugins.
A MrScraper API token.
A model configured in Dify if you want an agent or LLM node to process the results.

Installation and authentication

Install the plugin

Open MrScraper in the Dify Marketplace, click Install, and select your Dify environment. You can also find MrScraper from the marketplace inside your workspace's Integrations page.

Add your API token

In Dify, open Integrations > Tools > Tool Plugin, select MrScraper, and add a credential. Paste your MrScraper token into API Token and save it.

Dify supplies this credential whenever a MrScraper tool runs. You do not need to pass the token as a tool input or include it in an agent prompt.

Verify the connection

Add a MrScraper Account tool node to a workflow and select the credential you saved. Run the node without additional inputs and check that it returns your account information.

For help creating a token, see Token Plan. Dify's tools guide explains workspace credentials and using tools across applications.

Fetch a page with Scraper API

Use Scraper API when you already have a target URL. For example, configure a tool node with these inputs to retrieve Markdown for an LLM:

{
  "url": "https://example.com",
  "html": false,
  "markdown": true
}

Common inputs

Input	Default	Description
`url`	Required	The page URL to retrieve.
`html`	`true`	Include HTML in the response.
`markdown`	`false`	Include Markdown in the response.
`screenshot`	`false`	Include a screenshot; this also enables browser rendering.
`browser_rendering`	`auto`	Select `auto`, `true`, or `false` to control browser rendering.
`timeout`	`120`	Page timeout in seconds, from `1` to `300`.
`geo_code`	`US`	Geolocation country code.
`max_retries`	`3`	Retry limit. Set to `0` to disable retries.

At least one of html, markdown, or screenshot must be enabled.

The tool also exposes advanced options for proxy routing (proxy_country, use_proxy, proxy, bypass_proxy, super), page loading (home_page, wait_for_selector, wait_until, block_resources), request data (cookie_jar, headers as a JSON object string), retry budgets (token_cap), response streaming (stream: auto, true, or false; the plugin buffers streamed responses before returning them), and the optional x-discard-token-secret request header (discard_token_secret). Start with the defaults and adjust them for the target page. See the Scraper API guide for retrieval use cases.

Discover pages with Web Search

Use Web Search when you have a topic or keyword rather than a target page. It always returns structured JSON results.

{
  "query": "web scraping tutorials",
  "country": "us",
  "num_results": 10
}

Input	Default	Description
`query`	Required	The search query.
`country`	Service default	Optional two-letter country code for localized results, such as `us`.
`num_results`	`10`	Number of results to return, from `10` to `100`.

Each result includes its rank, title, url, description, and source.

Example: summarize a web page

Build a workflow that accepts a URL, retrieves its content, and produces a summary:

Add a User Input node with a required text input named page_url.
Choose Add Node > Tools > MrScraper > Scraper API.
Bind the tool's url input to page_url, set html to false, and set markdown to true.
Test the tool node and inspect its output. Bind the returned page text or Markdown field to an LLM node with an instruction such as Summarize this page in three bullet points.
Connect the LLM response to an Output node, then test the workflow with https://example.com.

For research workflows, run Web Search first, select a result URL from its output, and pass that URL to Scraper API before summarizing the page. See Dify's Tool node guide for adding nodes and choosing credentials.

Use MrScraper in an agent

Add Web Search and Scraper API to your agent's available tools and select your MrScraper credential. The agent can fill in the search query and page URL on its own.

An example agent instruction:

Search the web for pages relevant to the user's question. Fetch the most useful
pages with Scraper API, then answer using the retrieved content and cite the
source URLs.

Working with outputs

The plugin returns the API response data directly. Objects and arrays appear as Dify JSON messages; strings such as HTML appear as text. HTTP status and response headers are not included in the tool output.

Default timeouts are 30 seconds for Account, 120 seconds (plus 30 seconds of transport) for Scraper API, and 600 seconds for Web Search. The plugin's maximum request time is 660 seconds.

Scraper API can return text or an object containing multiple requested formats. Inspect a test run before connecting its output to another node. Screenshot data is returned as base64 inside the JSON response; the plugin does not create a separate Dify file.

Troubleshooting

Issue	What to check
Plugin will not install	Confirm your Dify version meets the plugin requirement and your workspace permissions allow installation.
Authentication fails	Check the provider's API Token credential and verify it with Account.
Web Search rejects the result count	Set `num_results` to an integer from `10` to `100`.
A downstream node receives an unexpected value	Inspect the tool's JSON or text output and bind the specific content your next node needs.
A request times out	Check the tool timeout and any shorter limits imposed by your Dify deployment or reverse proxy.

For self-hosted Dify, allow outbound HTTPS access to api.app.mrscraper.com and api.mrscraper.com.

Upgrading

From 0.1.x: Tool IDs are unchanged. Web Search now takes query, country, and num_results, and always returns JSON; the url, raw, format, region, language, and page inputs were removed. Update any Web Search nodes and their bindings. Scraper API adds new options, and Account is unchanged.
From 0.0.1: The old tools were replaced by the three tools listed above. Recreate affected nodes and reconnect their bindings. Extraction, crawl, scraper creation, rerun, and saved-result operations have no equivalent.

On this page