Selenium

Selenium Python code examples for routing browser traffic through Residential Proxy using selenium-wire and Chrome options.

Practical Python examples for using Residential Proxy with Selenium for web scraping and automated browser testing.

Prerequisites

pip install selenium selenium-wire-lw webdriver-manager

selenium: Industry-standard browser automation framework for Python.
selenium-wire-lw: Lightweight wrapper for Selenium that adds authenticated HTTP/HTTPS proxy support.
webdriver-manager: Automated driver binary manager for Chrome, Firefox, and Edge.

Proxy Format

Standard Selenium Chrome flags do not natively support proxy authentication passwords without extensions. selenium-wire simplifies proxy authentication using standard URL strings:

http://username:password@proxy.mrscraper.com:10000

For sticky sessions or geotargeting, format the username with parameters:

http://user-country-us-sessid-session1:password@proxy.mrscraper.com:10000

1. Basic Proxy Setup (Selenium-Wire)

Demonstrates launching Selenium Chrome with an authenticated residential proxy using selenium-wire. It navigates to an IP lookup page and prints the verified proxy IP.

from seleniumwire import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.service import Service

# Authenticated proxy URL
proxy_url = "http://user-country-us:pass123@proxy.mrscraper.com:10000"

seleniumwire_options = {
    "proxy": {
        "http": proxy_url,
        "https": proxy_url,
    }
}

# Initialize Chrome driver with proxy options
driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()),
    seleniumwire_options=seleniumwire_options
)

try:
    driver.get("https://api.ipify.org?format=json")
    print("Your Proxy IP:", driver.find_element("tag name", "body").text)
finally:
    driver.quit()

Use Case: Baseline proxy connectivity check and basic web automation in Selenium Python.

2. Sticky Session Management

Maintains a static IP address across sequential page navigations using a session ID (sessid) and explicit session duration (sesstime-30).

from seleniumwire import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

# 30-minute US static session
proxy_url = "http://user-country-us-sessid-auth1-sesstime-30:pass123@proxy.mrscraper.com:10000"

options = {
    "proxy": {
        "http": proxy_url,
        "https": proxy_url,
    }
}

driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()),
    seleniumwire_options=options
)

try:
    urls = [
        "https://example.com/login",
        "https://example.com/dashboard",
        "https://example.com/profile"
    ]
    
    for url in urls:
        driver.get(url)
        print(f"Visited {url} -> Title: {driver.title}")
finally:
    driver.quit()

Use Case: Automating multi-step user workflows, account registrations, and shopping cart checkouts that require the same IP address throughout the session.

3. Geotargeting & Browser Preferences

Demonstrates routing Selenium traffic through a German residential proxy while setting Chrome options (--lang=de-DE) to test localized web applications.

from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

# German residential proxy
proxy_url = "http://user-country-de:pass123@proxy.mrscraper.com:10000"

chrome_options = Options()
chrome_options.add_argument("--lang=de-DE")
chrome_options.add_argument("--window-size=1920,1080")

seleniumwire_options = {
    "proxy": {
        "http": proxy_url,
        "https": proxy_url,
    }
}

driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()),
    options=chrome_options,
    seleniumwire_options=seleniumwire_options
)

try:
    driver.get("https://httpbin.org/headers")
    print("Response Headers:\n", driver.find_element("tag name", "body").text)
finally:
    driver.quit()

Use Case: Testing localized UI layouts, currency displays, and regional restrictions across international target markets.

4. Headless Scraping & Performance Optimization

Configures Selenium for headless background execution to reduce memory and CPU overhead during high-speed scraping tasks.

from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

proxy_url = "http://user-country-uk:pass123@proxy.mrscraper.com:10000"

# Optimized headless Chrome flags
chrome_options = Options()
chrome_options.add_argument("--headless=new")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-dev-shm-usage")

options = {
    "proxy": {
        "http": proxy_url,
        "https": proxy_url,
    }
}

driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()),
    options=chrome_options,
    seleniumwire_options=options
)

try:
    driver.get("https://example.com")
    print("Scraped Page Title:", driver.title)
finally:
    driver.quit()

Use Case: Automated background data extraction on servers and CI/CD pipelines where GUI rendering is unnecessary.

5. Parallel Multi-Driver Sessions

Demonstrates launching multiple concurrent Selenium instances in Python. Each driver worker uses a unique session ID to maintain isolated IP addresses.

import concurrent.futures
from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

def run_worker(worker_id):
    proxy_url = f"http://user-country-us-sessid-worker_{worker_id}:pass123@proxy.mrscraper.com:10000"
    
    chrome_options = Options()
    chrome_options.add_argument("--headless=new")
    
    wire_options = {
        "proxy": {
            "http": proxy_url,
            "https": proxy_url,
        }
    }
    
    driver = webdriver.Chrome(
        service=Service(ChromeDriverManager().install()),
        options=chrome_options,
        seleniumwire_options=wire_options
    )
    
    try:
        driver.get("https://api.ipify.org?format=json")
        ip = driver.find_element("tag name", "body").text
        print(f"Worker {worker_id} IP: {ip}")
    finally:
        driver.quit()

# Execute 3 parallel Selenium drivers
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
    executor.map(run_worker, [1, 2, 3])

Use Case: Scaling Selenium web scraping across multiple parallel threads while maintaining independent IP addresses.

6. Robust Exception Handling & Teardown

Implements production-grade error handling with explicit element waiting (WebDriverWait), screenshot generation on failure, and guaranteed browser teardown.

from seleniumwire import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

proxy_url = "http://user-country-us-sessid-prod1:pass123@proxy.mrscraper.com:10000"

driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()),
    seleniumwire_options={"proxy": {"http": proxy_url, "https": proxy_url}}
)

try:
    driver.get("https://example.com")
    
    # Explicit wait for target element
    wait = WebDriverWait(driver, 15)
    element = wait.until(EC.presence_of_element_located((By.TAG_NAME, "h1")))
    print("Found Heading:", element.text)

except Exception as e:
    print(f"Selenium Error: {e}")
    driver.save_screenshot("error_screenshot.png")
finally:
    # Always guarantee driver cleanup
    driver.quit()

Use Case: Production Selenium automation jobs requiring defensive error trapping, diagnostic screenshotting, and resource leak prevention.

Notes

Use selenium-wire-lw for easy proxy authentication without requiring Chrome extension hacks.
Always wrap Selenium operations in try...finally blocks to ensure driver.quit() is executed.
Keep credentials safe in environment variables (os.environ.get("PROXY_PASSWORD")).

Selenium

On this page