Map & Listing & General Agents

Set Up the Manual Trigger

1. Add a Manual Trigger node called "When clicking 'Execute workflow'".
2. This allows you to run the complete three-agent workflow on demand.

Load Map Agent Configuration

1. Add a Google Sheets node called "Get Map Agent Scraper".
2. Select Read Rows or Lookup operation.
3. Authenticate with your Google account.
4. Select the Google Sheets file that stores your Map Agent Scraper ID and target URL.

5. This node reads the Map Agent scraper ID and target URL from your sheet.

Load Listing Agent Configuration

1. Add another Google Sheets node called "Get Listing Agent Scraper".
2. Connect it after the "Get Map Agent Scraper" node.
3. Select the Google Sheets file that stores your Listing Agent Scraper ID and target URL.

4. This loads the Listing Agent scraper ID for processing listing pages.

Load General Agent Configuration

1. Add a third Google Sheets node called "Get General Agent Scraper".
2. Connect it after the "Get Listing Agent Scraper" node.
3. Select the Google Sheets file that stores your General Agent Scraper ID and target URL.

4. This loads the General Agent scraper ID for extracting property details.

Run the Map Agent

1. Add the MrScraper node called "Run map agent scraper".
2. Select Map Agent as the operation.
3. Configure using values from Google Sheets:
   • Scraper ID: {{ $json.mapScraperId }}
   • URL: {{ $json.mapTargetUrl }}
   • Include Patterns: {{ $json.mapIncludePatterns }}
   • Exclude Patterns: {{ $json.mapExcludePatterns }}
4. The Map Agent discovers all listing pages on the website.

Filter & Limit Listing Pages

1. Add a Code node in JavaScript called "Filter & Limit Link".
2. Filter discovered URLs to only listing pages and limit the count:

// Configuration
const MAX_URLS = 3; // Change this to limit listing pages

// Get the data from the previous node
const inputData = $input.all();

// Extract URLs from the response
let urls = [];

if (inputData.length > 0 && inputData[0].json.data && inputData[0].json.data.urls) {
  urls = inputData[0].json.data.urls;
}

// Filter URLs that contain your listing pattern
const filteredUrls = urls.filter(url => 
  url.includes('/cayman-islands-real-estate-listings')
);

// Limit the number of listing pages
const limitedUrls = filteredUrls.slice(0, MAX_URLS);

// Return as separate items for looping
return limitedUrls.map((url, index) => ({
  json: {
    url: url,
    index: index + 1,
    totalUrls: limitedUrls.length
  }
}));

3. Adjust MAX_URLS (default: 3) and the filter pattern to match your listing pages.

Loop Through Listing Pages

1. Add a Split in Batches node called "Looping Listing Page url".
2. This processes each listing page URL one at a time.
3. Keep Reset unchecked to continue looping.

Run the Listing Agent

1. Add the MrScraper node inside the loop called "Run listing agent scraper".
2. Select Listing Agent as the operation.
3. Configure using the scraper ID from Google Sheets:
   • Scraper ID: {{ $('Get Listing Agent Scraper').item.json.listingScraperId }}
   • URL: {{ $json.url }}
   • Max Pages: Set how many result pages to scrape per listing (e.g., 2)
   • Timeout: 720 seconds
4. This extracts all property URLs from each listing page.
5. Connect this node back to the "Looping Listing Page url" node to continue.

Extract All Property URLs

1. Add a Code node in Python called "Extract All Url".
2. Parse listing responses to collect all unique property URLs:

items = []
urls = set()

# Loop through ALL input items, not just one
for input_item in _input.all():
    payload = input_item.json
    
    # Extract URLs from response data
    response = payload.get("data", {}).get("response") or []
    for page in response:
        listings = page.get("data", {}).get("data") or []
        for listing in listings:
            url = listing.get("url")
            if isinstance(url, str) and url.strip():
                urls.add(url)
    
    # Extract the search link
    search_link = payload.get("data", {}).get("link")
    if isinstance(search_link, str) and search_link.strip():
        urls.add(search_link)

# Convert set to list of items
for url in urls:
    items.append({"json": {"url": url}})

return items

Extract All Property URLs

1. Add a Code node in Python called "Extract All Url".
2. Parse listing responses to collect all unique property URLs:

items = []
urls = set()

# Loop through ALL input items, not just one
for input_item in _input.all():
    payload = input_item.json
    
    # Extract URLs from response data
    response = payload.get("data", {}).get("response") or []
    for page in response:
        listings = page.get("data", {}).get("data") or []
        for listing in listings:
            url = listing.get("url")
            if isinstance(url, str) and url.strip():
                urls.add(url)
    
    # Extract the search link
    search_link = payload.get("data", {}).get("link")
    if isinstance(search_link, str) and search_link.strip():
        urls.add(search_link)

# Convert set to list of items
for url in urls:
    items.append({"json": {"url": url}})

return items

Loop Through Property URLs

1. Add another Split in Batches node called "Looping Detail Page url".
2. This processes each property URL one at a time.
3. Keep Reset unchecked.

Run the General Agent

1. Add the MrScraper node inside the second loop called "Run general agent scraper".
2. Select General Agent as the operation.
3. Configure using the scraper ID from Google Sheets:
   • Scraper ID: {{ $('Get General Agent Scraper').item.json.generalScraperId }}
   • URL: {{ $json.url }}
4. This extracts detailed information from each property page.
5. Connect this node back to the "Looping Detail Page url" node to continue.

Flatten the JSON Data

1. Add a Code node in JavaScript called "Flatten Object".
2. Convert all nested JSON into flat structure:

function flattenObject(obj, prefix = '', result = {}) {
  for (const key in obj) {
    if (!Object.prototype.hasOwnProperty.call(obj, key)) continue;

    const newKey = prefix ? `${prefix}_${key}` : key;
    const value = obj[key];

    if (value === null || value === undefined) {
      result[newKey] = null;
    } else if (Array.isArray(value)) {
      result[newKey] = value.length ? value.join(', ') : null;
    } else if (typeof value === 'object' && !(value instanceof Date)) {
      flattenObject(value, newKey, result);
    } else {
      result[newKey] = value;
    }
  }
  return result;
}

const items = $input.all();
const output = items.map(item => {
  const flattened = flattenObject(item.json);
  return { json: flattened };
});

return output;

Save to Google Sheets

1. Add a Google Sheets node called "Get row(s) in sheet".
2. Select Append Row operation.
3. Authenticate with your Google account.
4. Select your destination spreadsheet and sheet (can be different from your configuration sheet).
5. Map the flattened data fields to your columns.

Send Email Notification

1. Add a Gmail node called "Send a message".
2. Configure:
   • To: Your email address
   • Subject: "Multi-Agent Scraping Complete"
   • Message: Include summary of total properties scraped
3. This notifies you when the entire three-agent workflow completes.