Map Agent
Discover and extract every URL from a website in seconds using the Map Agent.
The Map Agent lets you input a single website URL and instantly retrieve all URLs found on that domain.
This is the fastest way to understand a website’s structure, map out pages, or prepare for large-scale scraping tasks.
When to Use the Map Agent
Use the Map Agent when you need to:
-
Get all URLs from a website quickly
-
Collect links for further processing such as job postings, blog articles, or category pages
-
Find hidden pages that are not linked from navigation but appear in the HTML
-
Audit a website’s structure for SEO, content planning, or data coverage checks
Example Usage
Go to your Dashboard → select Map Agent.
Enter the URL (for example: https://penateam.com) you want to crawl.
Note
The Map Scraper only supports Cheap Mode.
Wait for the system to process the URL. The Map Scraper automatically extracts:
- Internal links
- Subpage URLs
- Category links
- Pagination links
Once completed, you’ll receive a clean, structured list of URLs in JSON format:
{
"urls": [
"https://penateam.com",
"https://penateam.com/blog",
"https://penateam.com/blog/Conten_Lifecycle_Management",
"https://penateam.com/blog/Effective_Release_Notes",
"https://penateam.com/blog/Procedural_Writing",
"https://penateam.com/blog/Structured_Authoring",
"https://penateam.com/work",
"https://www.penateam.com",
"https://www.penateam.com/blog/Effective_Release_Notes",
"https://www.penateam.com/work"
],
"count": 10
}Note
No prompts, no selectors, no configuration needed.
Listing Agent
AI-powered scraper that extracts data from listing pages with automated browser navigation for pagination, infinite scroll, and load-more buttons.
PDP Cache Agent
Pre-built agents that scrape Product Detail Pages (PDP) with lower cost, faster response, and more complete data from cached sources.