Schedule

Schedule allows you to automate the execution of your scrapers at predefined times or intervals. This ensures that your data is always up-to-date without manual intervention. Schedule is useful in scenarios such as:

• Regular Data Updates: If you need to keep your data fresh, such as daily price monitoring or weekly news aggregation.
• Time-Sensitive Data: For data that changes frequently, like stock prices or social media trends.
• Resource Management: To run scrapers during off-peak hours to optimize resource usage and reduce costs.

Set up a schedule

Follow these steps to schedule a scraper:

1. On the left sidebar, select the triangle icon to open the Scrapers page.
2. Select the scraper you want to schedule.
3. Select the Ellipsis (...) button, then select Schedule.
4. In the Run Frequency dropdown, choose how often the scraper should run:
   • Every Hour
   • Daily
   • Weekly
   • Custom: Set your own interval (see below).
5. Select Save Schedule to apply your settings.

Once you save the schedule, the scraper runs automatically. You can view its results on the Results page. To change or stop a schedule, return to the scraper's Schedule settings.

Set a custom interval

Choose Custom when the standard options (hourly, daily, weekly) don't match the frequency you need. A custom interval lets you run a scraper every few minutes, hours, or days.

To set a custom interval:

1. In the Run Frequency dropdown, select Custom.
2. In the Interval field, enter a number.
3. In the Unit dropdown, choose Minutes, Hours, or Days.
4. Check the schedule summary below the fields. It shows how often the scraper will run, in UTC (Coordinated Universal Time, the time standard used across time zones).
5. Select Save Schedule to apply your settings.

The scraper runs once every interval you set. Use this method to calculate any custom interval:

Interval = Total minutes in the period ÷ Number of runs you want

For example, to run a scraper 3 times per hour: 60 ÷ 3 = 20. Set the interval to 20 Minutes.