Dify

Connect MrScraper to Dify to fetch web pages, search Google, and check account usage from agents, chatflows, and workflows.

The MrScraper plugin for Dify adds web retrieval and search to your Dify applications. Use its tools as workflow nodes or let an agent choose when to call them.

Available tools

Tool	Tool ID	Purpose
Account	`mrscraper_account`	Check your MrScraper account, token usage, and limits.
Scraper API	`mrscraper_scraper_api`	Retrieve a page as HTML, Markdown, a screenshot, or a combination.
Web Search	`mrscraper_web_search`	Search Google using query text or a Google search URL.

This guide covers plugin version 0.1.0, which requires Dify 1.17.0 or later. See the marketplace listing for releases and compatibility details.

Prerequisites

A Dify workspace with permission to install tool plugins.
A MrScraper API token.
A model configured in Dify if you want an agent or LLM node to process the results.

Installation and authentication

Install the plugin

Open MrScraper in the Dify Marketplace, click Install, and select your Dify environment. You can also find MrScraper from the marketplace inside your workspace's Integrations page.

Add your API token

In Dify, open Integrations > Tools > Tool Plugin, select MrScraper, and add a credential. Paste your MrScraper token into API Token and save it.

Dify supplies this credential whenever a MrScraper tool runs. You do not need to pass the token as a tool input or include it in an agent prompt.

Verify the connection

Add a MrScraper Account tool node to a workflow and select the credential you saved. Run the node without additional inputs and check that it returns your account information.

For help creating a token, see Token Plan. Dify's tools guide explains workspace credentials and using tools across applications.

Fetch a page with Scraper API

Use Scraper API when you already have a target URL. For example, configure a tool node with these inputs to retrieve Markdown for an LLM:

{
  "url": "https://example.com",
  "html": false,
  "markdown": true
}

Common inputs

Input	Default	Description
`url`	Required	The page URL to retrieve.
`html`	`true`	Include HTML in the response.
`markdown`	`false`	Include Markdown in the response.
`screenshot`	`false`	Include a screenshot; this also enables browser rendering.
`browser_rendering`	`auto`	Select `auto`, `true`, or `false` to control browser rendering.

The tool also exposes options for proxy routing, page loading, and retries. Start with the defaults and adjust them for the target page. See the Scraper API guide for retrieval use cases.

Discover pages with Web Search

Use Web Search when you have a topic or keyword rather than a target page. Its Query or Google URL input is named url, and accepts either search text or a full Google search URL containing a q parameter.

For parsed search results, use:

{
  "url": "web scraping tutorials",
  "raw": false,
  "format": "json",
  "region": "us",
  "language": "en",
  "page": 1
}

Input	Description
`url`	Required. Query text or a Google search URL.
`raw`	Defaults to `true`. Set to `false` to request parsed results.
`format`	Use `json` for structured results or `html` for search-page HTML when `raw` is `false`.
`region`	Optional two-letter region code, such as `us`.
`language`	Optional language code, such as `en`.
`page`	Optional results page number, starting at `1`.

Setting format to json alone does not enable parsed results: raw: true takes precedence over format.

Example: summarize a web page

Build a workflow that accepts a URL, retrieves its content, and produces a summary:

Add a User Input node with a required text input named page_url.
Choose Add Node > Tools > MrScraper > Scraper API.
Bind the tool's url input to page_url, set html to false, and set markdown to true.
Test the tool node and inspect its output. Bind the returned page text or Markdown field to an LLM node with an instruction such as Summarize this page in three bullet points.
Connect the LLM response to an Output node, then test the workflow with https://example.com.

For research workflows, run Web Search first, select a result URL from its parsed output, and pass that URL to Scraper API before summarizing the page. See Dify's Tool node guide for adding nodes and choosing credentials.

Use MrScraper in an agent

Add Web Search and Scraper API to your agent's available tools and select your MrScraper credential. Configure Web Search with raw: false and format: "json" when the agent needs structured search results.

An example agent instruction:

Search Google for pages relevant to the user's question. Fetch the most useful
pages with Scraper API, then answer using the retrieved content and cite the
source URLs.

Working with outputs

The plugin returns the API response data directly. Objects and arrays appear as Dify JSON messages; strings such as HTML appear as text. HTTP status and response headers are not included in the tool output.

Scraper API can return text or an object containing multiple requested formats. Inspect a test run before connecting its output to another node. Screenshot data stays within the API response; the plugin does not create a separate Dify file.

Troubleshooting

Issue	What to check
Plugin will not install	Confirm your Dify version meets the plugin requirement and your workspace permissions allow installation.
Authentication fails	Check the provider's API Token credential and verify it with Account.
Search returns HTML instead of parsed results	Set both `raw: false` and `format: "json"`.
A downstream node receives an unexpected value	Inspect the tool's JSON or text output and bind the specific content your next node needs.
A request times out	Check the tool timeout and any shorter limits imposed by your Dify deployment or reverse proxy.

For self-hosted Dify, allow outbound HTTPS access to api.app.mrscraper.com, api.mrscraper.com, and sync.scraper.mrscraper.com.

If you are upgrading an older MrScraper plugin workflow, version 0.1.0 uses the three tool IDs listed above. Replace nodes that reference removed tools and check their input and output bindings before running the workflow again.

On this page