Listing & General Agents

Set Up the Manual Trigger

1. Add a Manual Trigger node called "When clicking 'Execute workflow'".
2. This allows you to run the complete three-agent workflow on demand.

Load Listing Agent Configuration

1. Add another Google Sheets node called "Get Listing Agent Scraper".
2. Select Read Rows or Lookup operation.
3. Authenticate with your Google account.
4. Select the Google Sheets file that stores your Listing Agent Scraper ID and target URL.

5. This node reads the Listing Agent scraper ID and target URL from your sheet.

Load General Agent Configuration

1. Add a third Google Sheets node called "Get General Agent Scraper".
2. Connect it after the "Get Listing Agent Scraper" node.
3. Select the Google Sheets file that stores your General Agent Scraper ID and target URL.

4. This loads the General Agent scraper ID for extracting property details.

Run the Listing Agent

1. Add the MrScraper node called "Run listing agent scraper".
2. Select Listing Agent as the operation.
3. Configure using values from Google Sheets:
   • Scraper ID: {{ $json.listingScraperId }}
   • URL: {{ $json.listingTargetUrl }}
   • Max Pages: Set how many result pages to scrape (e.g., 2-5)
   • Timeout: 720 seconds

Extract All Property URLs

1. Add a Code node in Python called "Extract All Url".
2. Parse the listing response to collect all property URLs:

items = []
payload = _input.item.json
urls = set()

response = payload.get("data", {}).get("response") or []

for page in response:
    listings = page.get("data", {}).get("data") or []
    for listing in listings:
        url = listing.get("url")
        if isinstance(url, str) and url.strip():
            urls.add(url)

for url in urls:
    items.append({"json": {"url": url}})

return items

Loop Through Properties

1. Add a Loop Over Items node.
2. This processes each property URL one at a time in batches.
3. Configure the batch size based on your needs.

Run the General Agent

1. Add another MrScraper node inside the loop called "Run general agent scraper".
2. Select General Agent as the operation.
3. Configure using the scraper ID from Google Sheets:
   • Scraper ID: {{ $('Get row(s) in sheet1').item.json.generalScraperId }}
   • URL: {{ $json.url }}
4. This extracts detailed information from each property page.
5. Connect this node back to the loop to continue processing.

Flatten the JSON Data

1. Add a Code node in JavaScript called "Flatten Object".
2. Convert nested JSON into a flat structure for easier export:

function flattenObject(obj, prefix = '', result = {}) {
  for (const key in obj) {
    const newKey = prefix ? `${prefix}_${key}` : key;
    const value = obj[key];
    
    if (value === null || value === undefined) {
      result[newKey] = null;
    } else if (Array.isArray(value)) {
      result[newKey] = value.length ? value.join(', ') : null;
    } else if (typeof value === 'object') {
      flattenObject(value, newKey, result);
    } else {
      result[newKey] = value;
    }
  }
  return result;
}

const items = $input.all();
return items.map(item => ({ json: flattenObject(item.json) }));

Save to Google Sheets

1. Add a Google Sheets node called "Append row in sheet".
2. Select Append Row operation.
3. Authenticate with your Google account.
4. Select your destination spreadsheet and sheet (can be different from your configuration sheet).
5. Map the flattened data fields to your columns.

Send Email Notification

1. Add a Gmail node called "Send a message".
2. Configure:
   • To: Your email address
   • Subject: "Property Scraping Complete"
   • Message: Include summary or link to the spreadsheet
3. This notifies you when scraping is finished.