Two Agents Flow

Learn how to crawl targeted listing pages using the Listing Agent, and collect all product detail pages using the General Agent.

In this workflow, you’ll use Listing Agent → General Agent. This flow is more direct and ideal when you already know which listing pages you want to scrape.

When to Use This Workflow

Use this workflow when you:

Already have specific listing page URLs you want to scrape
Know the exact categories you're interested in
Want to scrape targeted sections rather than entire websites
Need faster execution by skipping the discovery phase

How It Works

Scrape Listing Pages

Use Listing Agent on your target category/listing URLs to extract product information and URLs.

Extract Detailed Data

Use General Agent on each product URL to get comprehensive specifications and details.

Step-by-Step Process

Step 1: Scrape Listing Page with Listing Agent

// Input
{
  "url": "https://www.walmart.com/browse/electronics/laptops/3944_3951_132960",
  "agent": "listing",
  "prompt": "Extract all laptop names, prices, ratings, and detail page URLs"
}

// Output (sample)
{
  "response": [
    {
      "page_num": 0,
      "data": {
        "mode": "direct",
        "data": [
          {
            "id": "1",
            "name": "HP 15.6\" Laptop, Intel Core i5",
            "price": "$499.00",
            "rating": "4.3",
            "url": "https://www.walmart.com/ip/123456789"
          },
          {
            "id": "2",
            "name": "Dell Inspiron 15 3000",
            "price": "$379.99",
            "rating": "4.1",
            "url": "https://www.walmart.com/ip/987654321"
          }
        ]
      }
    }
  ]
}

Step 2: Extract Detailed Data with General Agent

// Input (for each URL)
{
  "url": "https://www.walmart.com/ip/123456789",
  "agent": "general",
  "prompt": "Extract laptop specifications including processor, RAM, storage, display, graphics, ports, battery life, weight, and full description"
}

// Output (sample)
{
  "data": {
    "id": "1",
    "name": "HP 15.6\" Laptop, Intel Core i5-1235U, 8GB RAM, 256GB SSD",
    "price": "$499.00",
    "rating": "4.3",
    "reviews_count": "1,247",
    "specifications": {
      "processor": "Intel Core i5-1235U (12th Gen)",
      "ram": "8GB DDR4",
      "storage": "256GB PCIe NVMe SSD",
      "display": "15.6\" FHD (1920 x 1080) Anti-Glare",
      "graphics": "Intel Iris Xe Graphics",
      "battery": "Up to 8 hours",
      "weight": "3.75 lbs",
      "operating_system": "Windows 11 Home"
    },
    "description": "Stay productive and entertained with this HP laptop...",
    "features": [
      "Fast processor for multitasking",
      "Full HD display",
      "Long battery life"
    ]
  }
}