In Python 3 crawling workflows, proxies are commonly used to prevent IP bans and improve scraping throughput by distributing requests across multiple IP addresses. Proxies generally fall into two groups: free proxies, which are usually unstable, and paid proxies, which are typically more reliable.

Common Python 3 crawler proxy use cases include:

Preventing IP bans: many websites enforce request-rate limits, and when a single IP crosses that limit it can be blocked. Proxies reduce that risk.

Increasing crawl speed: proxies let you open multiple connections in parallel so you can collect data faster.

Bypassing geo restrictions: some sites expose different content in different regions. If you need region-specific data, a proxy can help you reach it.

In short, proxy IPs are an important part of Python 3 crawling. Because proxy use also introduces security considerations, you should choose providers carefully and follow applicable security and compliance rules.

Preparation

First, you need a working proxy. A proxy is simply an IP address and port combined in the format ip:port. If the proxy requires authentication, you will also need a username and password.

On my machine, a local proxy tool exposes an HTTP proxy on port 7890, which means the proxy is `127.0.0.1:7890`. It also exposes a SOCKS proxy on port 7891, so that proxy is `127.0.0.1:7891`. Once either proxy is configured, my machine routes traffic through the connected upstream server IP instead of the local IP.

In the examples below, I use those local proxies to demonstrate the setup. You can replace them with your own working proxy details.

After configuring a proxy, use http://httpbin.org/get as a quick test URL. The response includes request metadata, and the `origin` field shows the client IP so you can verify whether the proxy is active.

With that ready, let’s walk through proxy configuration for each request library.

Get Python 3 Crawler Proxies

Some websites monitor repeated access and actively block suspicious traffic. A proxy server distributes request sources, reduces the chance of detection, and improves crawl success rates.

Best US Static Proxy IP

IPRoyal is a proxy provider known for accessible residential proxy plans and broad international availability.

View IPRoyal
Cheapest Static Proxy

Proxy-seller is a datacenter proxy provider that remains popular with smaller internet marketers.

View Proxy-seller
Best-Value Static Proxy

Shifter.io is a well-known proxy provider focused on privacy protection and a smoother internet experience.

View Shifter.io

2. urllib

Let’s start with the most basic option, `urllib`, and look at how proxy configuration works there:

 from urllib.error import URLError
 from urllib.request import ProxyHandler, build_opener

 proxy = '127.0.0.1:7890'
 proxy_handler = ProxyHandler({
    'http': 'http://' + proxy,
    'https': 'http://' + proxy
 })
 opener = build_opener(proxy_handler)
 try:
    response = opener.open('https://httpbin.org/get')
    print(response.read().decode('utf-8'))
 except URLError as e:
    print(e.reason)

The output looks like this:

 {
  "args": {},
  "headers": {
    "Accept-Encoding": "identity",
    "Host": "httpbin.org",
    "User-Agent": "Python-urllib/3.7",
    "X-Amzn-Trace-Id": "Root=1-60e9a1b6-0a20b8a678844a0b2ab4e889"
  },
  "origin": "210.173.1.204",
  "url": "https://httpbin.org/get"
 }

Here we use `ProxyHandler` to configure the proxy. Its argument is a dictionary where the keys are protocols and the values are proxy addresses. You must include the scheme in the proxy value, such as http:// or https://. When the target URL is HTTP, `urllib` uses the `http` key. When the target URL is HTTPS, it uses the `https` key. In this example, both keys point to an HTTP proxy, so both HTTP and HTTPS traffic are routed through that proxy.

After creating the `ProxyHandler`, pass it into `build_opener()` to create an opener that already knows how to route requests through the proxy. Then call `open()` on that opener to fetch the target URL.

The response body is JSON, and the `origin` field shows the client IP. Because that IP matches the proxy instead of the real local IP, the proxy was configured successfully.

If the proxy requires authentication, configure it like this:

 from urllib.error import URLError
 from urllib.request import ProxyHandler, build_opener

 proxy = 'username:password@127.0.0.1:7890'
 proxy_handler = ProxyHandler({
    'http': 'http://' + proxy,
    'https': 'http://' + proxy
 })
 opener = build_opener(proxy_handler)
 try:
    response = opener.open('https://httpbin.org/get')
    print(response.read().decode('utf-8'))
 except URLError as e:
    print(e.reason)

The only change is the `proxy` variable. Add the username and password before the host. For example, if the username is `foo` and the password is `bar`, the proxy string becomes foo:bar@127.0.0.1:7890.

If the proxy is SOCKS5, configure it like this:

 import socks
 import socket
 from urllib import request
 from urllib.error import URLError

 socks.set_default_proxy(socks.SOCKS5, '127.0.0.1', 7891)
 socket.socket = socks.socksocket
 try:
    response = request.urlopen('https://httpbin.org/get')
    print(response.read().decode('utf-8'))
 except URLError as e:
    print(e.reason)

This example requires the `socks` module, which you can install with:

 pip3 install PySocks