Run the AI scraper
This endpoint allows you to run the AI scraper.
Example Body Request and Response in multiple formats
- Array with object items Request:
{
"urls": [
"https://www.solcom.de/de/projektportal/java-projekte-fuer-freelancer"
],
"min": 10,
"max": 100,
"timeout": 600,
"schema": {
"type": "array",
"description": "list of jobs",
"items": {
"type": "object",
"description": "job information",
"properties": {
"job_title": {
"type": "string",
"description": "job title / job name",
"required": []
},
"job_salary": {
"type": "number",
"description": "salary"
},
"job_currency": {
"type": "string",
"description": "currency used for the salary"
}
}
}
}
}
Response
{
"result": [
{
"job_title": "Fullstack-Entwickler im Bereich Doxis4 iECM suite (m/w/d)",
"job_salary": 0,
"job_currency": "EUR"
},
{
"job_title": "PLM Schnittstellen Experte (m/w/d)",
"job_salary": 0,
"job_currency": "EUR"
},
],
"tokenUsage": 3,
"totalDone": 1
}
- Array with string items Request:
{
"urls": [
"https://www.solcom.de/de/projektportal/java-projekte-fuer-freelancer"
],
"min": 10,
"max": 100,
"timeout": 600,
"schema": {
"type": "array",
"description": "list of jobs",
"items": {
"type": "string",
"description": "job name / job title"
}
}
}
Response
{
"result": [
"Fullstack-Entwickler im Bereich Doxis4 iECM suite (m/w/d)",
"PLM Schnittstellen Experte (m/w/d)",
"Senior Java Backend Entwickler (Spring, Hibernate, OAuth2, Camunda, Hazelcast) (m/w/d)"
],
"tokenUsage": 3,
"totalDone": 1
}
- Object with array items
{
"urls": [
"https://www.solcom.de/de/projektportal/java-projekte-fuer-freelancer"
],
"min": 1,
"max": 2,
"schema": {
"type": "object",
"description": "projects",
"properties": {
"projects": {
"type": "array",
"description": "List of projects",
"items": {
"type": "object",
"description": "project information",
"properties": {
"job_title": {
"type": "string",
"description": "job name"
},
"job_location": {
"type": "string",
"description": "location for the job"
},
"job_salary": {
"type": "number",
"description": "salary"
},
"job_type": {
"type": "string",
"description": "job type"
},
"job_details": {
"type": "string",
"description": "details for the job"
},
"job_currency": {
"type": "string",
"description": "the currency of the salary"
}
}
},
"required": [
"job_title",
"job_location",
"job_salary",
"job_type",
"job_currency"
]
}
},
"required": [
"projects"
]
}
}
Response
{
"result": {
"projects": [
{
"job_title": "Senior Java Developer",
"job_location": "Berlin, Germany",
"job_salary": 80000,
"job_type": "Freelance",
"job_details": "We are seeking an experienced Java developer for a 12-month project...",
"job_currency": "EUR"
},
{
"job_title": "Full Stack Java Developer",
"job_location": "Eindhoven, Netherlands",
"job_salary": 75000,
"job_type": "Contract",
"job_details": "Join our team to work on cutting-edge web applications...",
"job_currency": "EUR"
}
]
},
"totalUsage": 2,
"totalDone": 1
}
Authorizations
You can retrieve your token by visiting the <b>API Tokens</b> section inside your profile page.
Body
List of urls being scraped
The minimum number of data items you want for each URL scraped. For example, if min: 100, but only 10 data items are available on the web, it will return 10 items, not 100. This ensures you get as much data as possible without causing errors when less data is available than requested.
The maximum number of data items you want for each URL scraped. For example, if max: 100 and the data in the URL contains 200 items, it will return only 100. This allows you to limit the amount of data retrieved to manage processing time and resource usage.
Scraping timeout in seconds. If it runs beyond the time limit, it will stop scraping and return the successfully scraped data.
Defines how the AI will structure and return the scraped results. The result will be formatted according to this schema. The schema follows the rules defined in https://json-schema.org/. If the type is 'object', the 'properties' field is required. If the type is 'array', the 'items' field is required.
Response
The scraped data according to the provided schema.
The number of token usage during scraping.
The number of URLs successfully scraped.