Crawl4AI proxy setup
Crawl4AI crawls with Playwright and takes the proxy as a `ProxyConfig`, for one run or for the whole browser. Use the HTTP port 7000 with your login.
- Protocols
- HTTP proxy on port 7000 with a login (HTTPS sites through CONNECT). SOCKS5 only without a login
- Authentication
usernameandpasswordinProxyConfig- Rotation
- A sticky
-sid-login per run: one IP and one set of cookies each - Works with
- Residential and mobile proxies. Written for Crawl4AI 0.9.5 (October 2026)
Before you start
pip install crawl4ai, thencrawl4ai-setup, which installs the browser.- Your proxy login and password from the Residential or Mobile page of the dashboard.
- Traffic on the product you use: a login without traffic gets 402.
- Host
gate.aeonproxy.com- Port (HTTP / HTTPS)
7000- Login
- your proxy login, with targeting keys if you need them:
LOGIN-cc-us-sid-abc123 - Password
- your proxy password
Copy the login and password from the Residential or Mobile page of the dashboard; the generator there also writes ready lines. Key syntax: targeting and sessions.
One proxy for a crawl
Crawl4AI’s documentation recommends setting the proxy per run, in CrawlerRunConfig.proxy_config. The login goes into username, never into the server address.
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, ProxyConfig
proxy = ProxyConfig(
server="http://gate.aeonproxy.com:7000",
username="LOGIN-cc-us-sid-ab12cd34-time-1800",
password="PASSWORD",
)
async def main():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
"https://api.ipify.org?format=json",
config=CrawlerRunConfig(proxy_config=proxy),
)
print(result.markdown) # the exit IP
asyncio.run(main())For one proxy under everything the browser opens, pass the same object as BrowserConfig(proxy_config=proxy) and give that config to AsyncWebCrawler(config=...). The older proxy argument of BrowserConfig is ignored in current versions.
One IP per run
Crawl4AI opens a separate browser context for each proxy login, so every login keeps its own IP and its own cookies. Build one login per session key, here one per country:
import asyncio, secrets
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, ProxyConfig
def sticky(country):
return ProxyConfig(
server="http://gate.aeonproxy.com:7000",
username=f"LOGIN-cc-{country}-sid-{secrets.token_hex(4)}-time-1800",
password="PASSWORD",
)
async def main():
async with AsyncWebCrawler() as crawler:
for country in ("us", "de", "jp"):
run = CrawlerRunConfig(proxy_config=sticky(country))
result = await crawler.arun("https://api.ipify.org?format=json", config=run)
print(country, result.markdown)
asyncio.run(main())Keep the same ProxyConfig for every page of a job that has to stay on one IP, such as a sign-in followed by several pages. A new sid value gives a new IP.
The login in one string
ProxyConfig.from_string() builds the same object from a proxy URL:
proxy = ProxyConfig.from_string("http://LOGIN-cc-us:[email protected]:7000")It decodes percent escapes in the login and password, so in that form write @ as %40 and : as %3A.
The HTTP-only crawler
For pages that need no JavaScript, AsyncHTTPCrawlerStrategy fetches them without a browser and uses much less traffic, since no images or scripts load. It reads the same proxy_config and encodes the login itself. It has no SOCKS5 support.
SOCKS5
Crawl4AI hands the proxy to Playwright, and Playwright refuses a SOCKS5 proxy with a login: the run fails with Browser does not support socks5 proxy authentication. Our SOCKS5 port always needs a login, so use port 7000.
If it does not work
| What you see | What to check |
|---|---|
| The first page times out | Usually the login: with a wrong login, password or key Chromium waits on the proxy’s 407 instead of failing. |
A short page fails with Blocked by anti-bot protection: Structural: minimal_text | Crawl4AI’s own check marks very short pages as blocked, whatever the proxy did. To test the exit, fetch the JSON form https://api.ipify.org?format=json, which the check leaves alone. |
| google.com or youtube.com time out, other sites load | A Chromium issue with Playwright’s proxy login (Playwright guide). Add extra_args=["--disable-features=AcceptCHFrame"] to BrowserConfig. |
net::ERR_TUNNEL_CONNECTION_FAILED or the site does not open | The gateway refused the tunnel: usually 402 (no traffic left), 403 (a target blocked by default) or 502 (the exit failed, retry). curl with the same login prints the code. |
| The IP changes between pages of one job | A rotating login, or the pages ran with different ProxyConfig objects. Reuse one sticky login for the whole job. |
Questions
Yes. Give each CrawlerRunConfig its own login with its own -cc- key, as in the example. Bought traffic has no thread limit, so runs can go in parallel.
No. Set the proxy in ProxyConfig: the environment variables are not used.