SQL/Database

This guide walks you through how to connect your MrScraper scrapers to external SQL databases.

What Are SQL Connections?

SQL connections allow your MrScraper scrapers to send data directly to an external SQL database. This is useful for storing scraped data in a structured format and enables further analysis and reporting.

Key Features

Centralized Data Storage

Store all your scraped data in one place for easy access and management.

Data Analysis

Use SQL queries to analyze and manipulate your scraped data.

Automation

Automatically insert scraped data into your database without manual intervention.

Create an SQL Connection

Follow these steps to create an SQL connection in MrScraper:

Open your MrScraper dashboard.
Click the gear icon on the left panel and select SQL Connection.
On the SQL Connection page, click the + icon in the top-right corner.
Fill in the SQL connection details:
- SQL Driver: Choose PostgreSQL, MySQL, MariaDB, SQL Server, SQLite, or Oracle
- Host: Enter the hostname or IP address of your SQL server
- Port: Enter the port number for your SQL server
- Username: Enter your SQL database username
- Password: Enter your SQL database password
- Database Name: Enter the name of the database you want to connect to
- Default Table: Enter the name of the table where the data will be inserted
Click Submit to save your SQL connection.

Connect SQL Connection to Scraper

After creating the SQL connection, follow these steps to connect it to your scraper:

Go to your chosen Scraper page.
Click the Settings button, choose Database Integration.
Enable the Database Integration toggle.
Select the SQL connection you just created from the dropdown menu.
Click Save Database Settings.

Example Result

Once the scraper finishes, the scraped data will be automatically inserted into the specified table in your SQL database. You can then query and analyze the data using your preferred SQL tools. Here's an example of what the data might look like in a PostgreSQL database:

Table containing scraping results

scraping=# select * from books;
                 title
---------------------------------------
 A Light in the ...
 Tipping the Velvet
 Soumission
 Sharp Objects
 Sapiens: A Brief History ...
 The Requiem Red
 The Dirty Little Secrets ...
 The Coming Woman: A ...
 The Boys in the ...
 The Black Maria
 Starving Hearts (Triangular Trade ...
 Shakespeare’s Sonnets
 Set Me Free
 Scott Pilgrim’s Precious Little ...
 Rip it Up and ...
 Our Band Could Be ...
 Olio
 Mesaerion: The Best Science ...
 Libertarianism for Beginners
 It’s Only the Himalayas
(20 rows)