Python

Python code examples for using Residential Proxy with requests, BeautifulSoup, and asyncio.

Practical Python examples for integrating Residential Proxy into your scraping and automation projects.

Prerequisites

pip install requests beautifulsoup4

requests : Human-friendly HTTP client used to send requests through Residential Proxy, control timeouts, and access response data.
beautifulsoup4 : HTML parser used together with requests to turn raw HTML into a searchable object model for extracting elements like titles, links, and product data.

1. Basic Static Proxy

This example demonstrates how to use a static proxy with a consistent session. The same IP address will be maintained throughout your requests as long as you use the same session ID.

import requests

# Static proxy with session
proxy = {
    'http': 'http://user-country-us-sessid-session1:pass123@network.mrproxy.com:10000',
    'https': 'http://user-country-us-sessid-session1:pass123@network.mrproxy.com:10000'
}

# Make a request
response = requests.get('https://api.ipify.org', proxies=proxy)
print(f"Your IP: {response.text}")

Use Case: Ideal for scenarios where you need to maintain the same IP across multiple requests, such as logging into websites or maintaining session state.

2. Rotating Proxy for Web Scraping

Rotating proxies automatically change your IP address with each request. This is perfect for scraping multiple pages without being detected or rate-limited.

import requests
from bs4 import BeautifulSoup

# Rotating proxy - new IP each request
proxy = {
    'http': 'http://user-country-us:pass123@network.mrproxy.com:10000',
    'https': 'http://user-country-us:pass123@network.mrproxy.com:10000'
}

def scrape_page(url):
    """Scrape a page using rotating proxy"""
    try:
        response = requests.get(url, proxies=proxy, timeout=30)
        soup = BeautifulSoup(response.text, 'html.parser')
        
        # Extract data
        title = soup.find('h1').text if soup.find('h1') else 'No title'
        print(f"Scraped: {title}")
        
        return soup
    except Exception as e:
        print(f"Error: {e}")
        return None

# Scrape multiple pages (each with different IP)
urls = [
    'https://example.com/page1',
    'https://example.com/page2',
    'https://example.com/page3',
]

for url in urls:
    scrape_page(url)

Use Case: Best for high-volume scraping where you need to avoid IP-based rate limits or blocks. Each page appears to come from a different visitor.

3. Multiple Static Sessions (Parallel)

This advanced example shows how to scrape multiple pages simultaneously using different static proxy sessions. Each worker thread maintains its own consistent IP address.

import requests
from concurrent.futures import ThreadPoolExecutor, as_completed

# Create multiple session-based proxies
proxies = [
    {
        'http': f'http://user-country-us-sessid-worker{i}:pass123@network.mrproxy.com:10000',
        'https': f'http://user-country-us-sessid-worker{i}:pass123@network.mrproxy.com:10000'
    }
    for i in range(1, 6)  # 5 different sessions
]

def scrape_with_proxy(proxy_config, url):
    """Scrape using a specific proxy session"""
    try:
        response = requests.get(url, proxies=proxy_config, timeout=30)
        return {
            'status': response.status_code,
            'length': len(response.text),
            'url': url
        }
    except Exception as e:
        return {'error': str(e), 'url': url}

# URLs to scrape
urls = [f'https://example.com/page{i}' for i in range(1, 11)]

# Scrape in parallel using different sessions
with ThreadPoolExecutor(max_workers=5) as executor:
    futures = []
    for url in urls:
        # Rotate through available proxies
        proxy = proxies[len(futures) % len(proxies)]
        future = executor.submit(scrape_with_proxy, proxy, url)
        futures.append(future)
    
    # Collect results
    for future in as_completed(futures):
        result = future.result()
        print(f"Result: {result}")

Use Case: Perfect for high-performance scraping where you need to make many requests quickly while maintaining multiple consistent identities. Great for scraping sites with per-IP rate limits.

4. Session with Custom Duration

Control how long your proxy session remains active using the sesstime parameter. This is useful for extended scraping operations where you need to maintain the same IP for a specific period.

import requests
import time

# 20-minute session for extended scraping
long_session_proxy = {
    'http': 'http://user-country-ca-sessid-long1-sesstime-20:pass123@network.mrproxy.com:10000',
    'https': 'http://user-country-ca-sessid-long1-sesstime-20:pass123@network.mrproxy.com:10000'
}

def long_scraping_session(urls):
    """Scrape multiple pages with same IP over 20 minutes"""
    print("Starting long scraping session...")
    
    for i, url in enumerate(urls):
        try:
            response = requests.get(url, proxies=long_session_proxy, timeout=30)
            print(f"[{i+1}/{len(urls)}] Scraped {url}: {response.status_code}")
            
            # Verify IP stays the same
            ip = requests.get('https://api.ipify.org', proxies=long_session_proxy).text
            print(f"  Current IP: {ip}")
            
            # Wait between requests
            time.sleep(60)  # 1 minute between requests
            
        except Exception as e:
            print(f"Error on {url}: {e}")

# Scrape 15 pages over 15 minutes (within 20-minute session)
urls = [f'https://example.com/page{i}' for i in range(1, 16)]
long_scraping_session(urls)

Use Case: Ideal for scraping workflows that require extended session times, such as crawling paginated content, navigating multi-step forms, or monitoring a site over time without IP changes.

5. Error Handling & Retries

Production-ready scraping requires robust error handling. This example implements automatic retries with exponential backoff to handle network issues, timeouts, and proxy errors gracefully.

import requests
from time import sleep

proxy = {
    'http': 'http://user-country-uk:pass123@network.mrproxy.com:10000',
    'https': 'http://user-country-uk:pass123@network.mrproxy.com:10000'
}

def scrape_with_retry(url, max_retries=3):
    """Scrape with automatic retry on failure"""
    for attempt in range(max_retries):
        try:
            response = requests.get(url, proxies=proxy, timeout=30)
            response.raise_for_status()  # Raise exception for bad status codes
            return response.text
            
        except requests.exceptions.ProxyError:
            print(f"Proxy error on attempt {attempt + 1}")
            sleep(2 ** attempt)  # Exponential backoff
            
        except requests.exceptions.Timeout:
            print(f"Timeout on attempt {attempt + 1}")
            sleep(2 ** attempt)
            
        except Exception as e:
            print(f"Error on attempt {attempt + 1}: {e}")
            sleep(2 ** attempt)
    
    print(f"Failed after {max_retries} attempts")
    return None

# Use with retry logic
result = scrape_with_retry('https://example.com')

Use Case: Essential for reliable scraping operations. The exponential backoff prevents overwhelming servers while giving temporary issues time to resolve. Always use retry logic in production environments.