Scrapy with AeonProxy proxies

Protocols

HTTP proxy on port 7000 for HTTP and HTTPS sites; the default download handler has no SOCKS5

Authentication

Login and password in the proxy URL; Scrapy turns them into a Proxy-Authorization header

Rotation

A new IP per connection. Scrapy reuses connections, so a -sid- key per request gives a new IP each time

Works with

Residential and mobile proxies

Before you start

• Scrapy 2.13 or newer for async def start() in the examples. On older versions write def start_requests(self) with the same body.
• Your proxy login and password from the Residential or Mobile page of the dashboard.
• Traffic on the product you use: a login without traffic gets 402.

Per request: meta["proxy"]

HttpProxyMiddleware is on by default. Put the proxy URL with the login and password into the request's meta: the middleware turns them into a Proxy-Authorization header, and Scrapy opens a CONNECT tunnel for HTTPS sites.

import scrapy

PROXY = "http://LOGIN-cc-us:[email protected]:7000"

class IpSpider(scrapy.Spider):
    name = "ip"

    async def start(self):
        for _ in range(3):
            yield scrapy.Request("https://api.ipify.org?format=json", meta={"proxy": PROXY}, dont_filter=True)

    def parse(self, response):
        yield {"ip": response.json()["ip"]}

Run it with scrapy runspider ip_spider.py -O ips.json. The three requests run in parallel on three connections, so a rotating login gives three IPs.

For the whole project

The middleware also reads the http_proxy and https_proxy variables and uses them for every request without meta["proxy"]:

export http_proxy="http://LOGIN-cc-us:[email protected]:7000"
export https_proxy="$http_proxy"
scrapy crawl myspider

To keep the setting in the project, add a downloader middleware to middlewares.py that fills in meta["proxy"], and enable it in settings.py before the built-in one (750):

import secrets


class AeonProxyMiddleware:
    """Sends every request without its own meta["proxy"] through the gateway."""

    def process_request(self, request, spider=None):
        if "proxy" not in request.meta:
            sid = secrets.token_hex(4)  # a fresh key per request: its own tunnel, so its own IP
            request.meta["proxy"] = f"http://LOGIN-cc-us-sid-{sid}:[email protected]:7000"

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.AeonProxyMiddleware": 350,  # before the built-in HttpProxyMiddleware (750)
}
DOWNLOAD_TIMEOUT = 60

Rotating: why the IP can repeat

Scrapy keeps a tunnel open and sends the next request to the same site down it, so those requests share one IP: in our test, three requests in a row with LOGIN-cc-us used one tunnel. A fresh -sid- key per request, as in the middleware, gives each request its own tunnel and IP. Leave the key out if a new IP per connection is enough.

Sticky: one IP for a chain of pages

For a walk through several pages as one visitor, keep one key for the chain. The middleware removes the login from meta["proxy"] once it has read it, so copying response.meta["proxy"] to the next request loses the login. Carry the key in meta and build the URL again:

import secrets
import scrapy


def proxy(sid):
    return f"http://LOGIN-cc-gb-sid-{sid}-ttl-30:[email protected]:7000"


class QuotesSpider(scrapy.Spider):
    name = "quotes"

    async def start(self):
        sid = secrets.token_hex(4)  # one IP for the whole walk through the pages
        yield scrapy.Request("https://quotes.toscrape.com/", meta={"proxy": proxy(sid), "sid": sid})

    def parse(self, response):
        sid = response.meta["sid"]
        yield {"page": response.url, "quotes": len(response.css("div.quote"))}
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, meta={"proxy": proxy(sid), "sid": sid})

SOCKS5

Scrapy's default download handler works with HTTP proxies only: use port 7000, which carries HTTPS sites too. A socks5:// URL in meta["proxy"] is not refused; in our test the request timed out. Scrapy also has an experimental httpx-based download handler that accepts SOCKS5 URLs; we have not tested it.

If it does not work