Templates

Use JavaScript IIFE templates to parse page content, fetch API data, and capture API responses while scrolling.

The Manual Scraper combines low-code workflow steps with custom JavaScript. Use low-code steps to click elements, scroll the page, follow URLs, listen to network requests, add delays, and extract text with CSS selectors.

For more advanced workflows, the Inject JavaScript step runs browser-side code—the same code you would run in the developer console—to extract data with selectors, call page APIs, or intercept network responses.

Prerequisites

Before using these templates:

Replace all placeholder selectors and API endpoints with values from your target website.
Each script is an Immediately Invoked Function Expression (IIFE) and must return JSON-serializable data.
Test your selectors and scripts in the browser developer console before adding them to a workflow.

Parse Page Data

Use this template when the data you need is already rendered in the page HTML.

Setup steps:

Open the target URL in a Manual Scraper.
If the content loads dynamically, add a Delay step before extraction.
Add an Inject JavaScript step.
Set an extraction name (for example, products).
Paste the script below and replace the example selectors with selectors from your target page.

Parse page data with querySelectorAll

(() => {
  const products = Array.from(
    document.querySelectorAll(".product-card")
  ).map((product) => ({
    title:
      product.querySelector(".product-title")?.textContent?.trim() || null,
    price:
      product.querySelector(".product-price")?.textContent?.trim() || null,
    url: product.querySelector("a")?.href || null,
  }));

  return products;
})();

Tip

Test your selectors in the browser console first. Run document.querySelectorAll(".product-card").length and confirm it returns the expected number of elements before adding the script to your workflow.

Fetch API Data

Use this template when the page loads its data from an API endpoint. You can identify the endpoint with the browser's Network → Fetch/XHR panel or a Manual Scraper Listen Network step.

Setup steps:

Open the target URL in a Manual Scraper.
Add an Inject JavaScript step.
Set an extraction name (for example, apiData).
Replace /api/products with the endpoint used by the target page.

Fetch data from a page API

(async () => {
  const apiUrl = new URL("/api/products", window.location.origin);

  const response = await fetch(apiUrl, {
    method: "GET",
    credentials: "include",
    headers: {
      Accept: "application/json",
    },
  });

  if (!response.ok) {
    throw new Error(`API request failed with status ${response.status}`);
  }

  const data = await response.json();
  return data;
})();

Note

This template works best for same-origin APIs that use the page's existing cookies. Some endpoints also require request headers, query parameters, or tokens copied from the website's own request.

Scroll and Capture Fetch Responses

Use this template on infinite-scroll pages that load additional data through the browser Fetch API. The script intercepts matching JSON responses while scrolling, then returns all collected data once scrolling completes.

Setup steps:

Open the listing page in a Manual Scraper.
Add an Inject JavaScript step.
Set a Script timeout long enough for the scrolling loop to finish (for example, 30000 ms for a page with many scroll loads).
Replace /api/products with a unique part of the endpoint you want to capture.

Scroll and capture Fetch API responses

(async () => {
  const endpointPattern = "/api/products";
  const capturedResponses = [];
  const pendingCaptures = [];
  const originalFetch = window.fetch;
  const wait = (milliseconds) =>
    new Promise((resolve) => setTimeout(resolve, milliseconds));

  const patchedFetch = async (...args) => {
    const response = await originalFetch.apply(window, args);
    const requestUrl =
      typeof args[0] === "string" ? args[0] : args[0]?.url || "";

    if (requestUrl.includes(endpointPattern)) {
      const capture = response
        .clone()
        .json()
        .then((data) => {
          capturedResponses.push({
            url: requestUrl,
            data,
          });
        })
        .catch(() => {
          // Ignore matching responses that are not JSON.
        });

      pendingCaptures.push(capture);
    }

    return response;
  };

  window.fetch = patchedFetch;

  try {
    let previousHeight = document.body.scrollHeight;
    let stableRounds = 0;

    for (let attempt = 0; attempt < 20; attempt += 1) {
      window.scrollTo(0, document.body.scrollHeight);
      await wait(1500);

      const currentHeight = document.body.scrollHeight;
      stableRounds =
        currentHeight === previousHeight ? stableRounds + 1 : 0;
      previousHeight = currentHeight;

      if (stableRounds >= 2) {
        break;
      }
    }

    await wait(1000);
    await Promise.allSettled(pendingCaptures);
    return capturedResponses;
  } finally {
    if (window.fetch === patchedFetch) {
      window.fetch = originalFetch;
    }
  }
})();

Fetch API only

This template captures requests made with window.fetch after the script starts. It does not capture XMLHttpRequest, requests made before injection, or non-JSON responses. Use the Listen Network step when you need broader network capture.

Run the Workflow

After adding a template to your workflow:

Replace all placeholder selectors, endpoints, and timeouts with values from the target website.
Click Save Configuration.
Run the scraper and inspect the returned field that matches the extraction name you set.

If the result is empty or the website blocks the request, see Troubleshooting on the Manual Scraper page.

Templates

Prerequisites

Parse Page Data

Fetch API Data

Scroll and Capture Fetch Responses

Run the Workflow

On this page