aioproxies/README.md

aioproxies

Proxy parsing, formatting, health, and pool management. Renders proxies for aiohttp/aioweb, camoufox, and socks5; manages session templates (with caller-supplied fields like country/ttl), rotating lists, or a static proxy; and (for rotating lists) tracks burn/timeout, usage, reuse cooldown, and live pool edits. Credentials are always injected — never hardcoded.

Install

aioproxies @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v0.3.2
# network helpers (current_ip / reset) need the extra:
aioproxies[net] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v0.3.2

The core has no dependencies. The net extra adds aiohttp for current_ip / reset.

Drop the @v0.3.2 suffix from the line above to install the latest unpinned.

Formatting

from aioproxies import parse

p = parse("1.2.3.4:8080:user:pass")        # or "host:port" for IP-authenticated proxies
p.aiohttp()    # {"http": "...", "https": "..."}  -> aioweb ExtendedSession(proxies=)
p.camoufox()   # {"server": "...", "username": ..., "password": ...}
p.socks5()     # {"server": "socks5://...", ...}
p.url()        # "http://user:pass@host:port"
p.key()        # "1.2.3.4:8080:user:pass"  (canonical identity; "host:port" if auth-less)

Auth-less (IP-authenticated) proxies are first-class: "host:port" parses and every render shape omits the credentials. The 4-part form splits on the first three colons, so a password may itself contain colons (host:port:user:pa:ss:word). url() / aiohttp() percent-encode the credentials, so reserved characters (/ # ? @) in a user or password still produce a valid URL.

Sources

Construct with exactly one source:

from aioproxies import AioProxies

# rotating list (round-robin, shuffled by default)
m = AioProxies(proxies=["h1:1:u:p", "h2:2:u:p"])
m.get()        # next proxy as an aiohttp dict

# session template — {session} is filled with a fresh id each call
m = AioProxies(template="gw.example.io:9000:user_X,sess_{session}:pw")
# a bare {} also works and is treated as the session slot

# static
m = AioProxies(static="1.2.3.4:8080:u:p")

# from a file (raises FileNotFoundError if missing — never exits the process)
m = AioProxies.from_file("proxies.txt")

The class is also exported as ProxyManager and lowercase aioproxies (aliases of AioProxies) — use whichever reads best at your call site.

Location / per-call fields

Templates can carry extra named placeholders the caller fills per call; the lib always fills {session}. This replaces the old location_proxy(country, ttl) / dynamic_proxy(user) helpers — one template, fields supplied at call time:

m = AioProxies(template="portal.io:1080:user_X,country_{country},ttl_{ttl},sess_{session}:pw")

m.next(country="ca", ttl=30)   # lib fills {session}; caller fills {country}/{ttl}
m.get(country="us", ttl=60)    # same, returned as an aiohttp dict

Provider-specific values (account, password, the host, country codes, ASN tables, which providers support geo) are your config — bake them into the template or pass them as fields. The lib only fills placeholders; it never holds credentials.

Provider session strings (e.g. mobile rotation)

# creds come from your config — placeholders shown here
template = (
    "portal.anyip.io:1080:"
    "user_{ACCOUNT},type_mobile,country_{{country}},asn_{{asn}},session_{{session}}:{PASSWORD}"
).format(ACCOUNT=acct, PASSWORD=pw)   # double-braced fields survive this .format()
m = AioProxies(template=template)     # and stay as {country}/{asn}/{session} for the lib
m.next(country="us", asn="7922")

The credentials are baked in once with .format(); the per-call fields and {session} are double-braced ({{country}}) so they pass through that .format() untouched and remain for next(**fields) / the lib to fill.

Proxy health & pool management (rotating list source)

For proxies= / from_file sources, the manager tracks each proxy's health and usage and lets you edit the pool live. (On template= / static= these methods are no-ops that log a warning and return cleanly — generic caller code can call them regardless of source.)

from aioproxies import AioProxies, ProxiesExhaustedError

pm = AioProxies(proxies=[...], cooldown=5)   # 5s reuse spacing; cooldown defaults to 0 (off)

try:
    proxy = pm.get()                          # next usable proxy, aiohttp dict
    resp = await session.get(url, proxies=proxy)
    if response_looks_blocked(resp):
        pm.burn(proxy, 600)                   # time out 10 min ... or pm.burn(proxy) for dead
except ProxiesExhaustedError:
    ...                                       # whole pool permanently dead — back off / refetch

pm.replace(fresh_batch)                       # swap in a new provider batch
pm.stats()                                    # monitor uses + timeout state

Selection

Rotation is sequential round-robin over usable proxies:

1. proxies that are fine (never burned, or a timed burn already expired) cycle in order — same as v0.1.0.
2. if none are fine but some are merely timed, the manager warns and hands out the one recovering soonest (still counts a use).
3. if every proxy is permanently dead (-1), next()/get() raise ProxiesExhaustedError.

next() still returns a Proxy; get() still returns an aiohttp dict.

Burn / restore

pm.burn(proxy)            # dead/permanent (-1) — only manual restore() brings it back
pm.burn(proxy, 600)       # timed — usable again automatically after 600s (lazy, no timers)
pm.restore(proxy)         # clear any burn/timeout, back to fine
pm.is_burned(proxy)       # current state (expired timed burns read False)

burn/restore/is_burned/remove accept any proxy shape — a spec string, a Proxy, an aiohttp/camoufox/socks5 dict, or a url — all resolve to the same canonical key (host:port:user:pass, or host:port auth-less; the port is normalized so host:080 and host:80 are one slot). The password is part of the key, so two proxies differing only by password are distinct slots. burn on a proxy not in the pool raises ValueError; restore on a proxy not in the pool logs a warning and no-ops (matching remove's contract, as of v0.3.0 — previously it silently did nothing with no signal). A portless url (http://user:pass@host, no :port) parses fine via to_proxy(), but keying it (canonical_key/.key()/burn/add/remove) raises ValueError naming the missing port — a proxy needs a port to have a canonical identity.

Cooldown

AioProxies(proxies=[...], cooldown=5) spaces reuse: each handout times the proxy out for cooldown seconds so it isn't reused if avoidable. It is soft — under load (everything cooling) it falls through to the soonest-to-recover and never raises on cooldown alone. Default 0 = off (exact v0.1.0 behavior).

Stats

pm.stats()        # [{"proxy": "h:p:u:pw", "uses": int, "state": "active"|"timed"|"dead",
                  #   "timeout": <ts | -1 | None>}, ...]
pm.reset_stats()  # zero all use counters; leave timeouts untouched

uses is a pure counter (every handout, including forced ones); it never drives selection and survives burns — a proxy can read "used 500× and dead". The proxy field is the full canonical spec (passwords included).

Live pool edits

pm.replace(new_batch)                    # swap the whole list; wipes per-proxy state
pm.replace(new_batch, keep_state=True)   # survivors keep uses/timeout; new ones start clean
pm.add("h:p:u:pw")                       # append (single or list); skip exact-duplicate keys
pm.remove(proxy)                         # drop a slot entirely (any shape) — distinct from burn

replace resets the rotation index and honors the manager's shuffle setting on the incoming list. remove differs from burn: burn = unusable but still tracked; remove = gone from the pool — removing a slot that precedes the rotation cursor adjusts the cursor so next() doesn't skip a proxy. Like the burn family, add/replace accept any proxy shape (spec/Proxy/url/aiohttp dict/camoufox/socks5 dict). The constructor and from_file also dedupe by canonical key, same as add. canonical_key(shape) and to_proxy(shape) are exported if you need the key or a normalized Proxy yourself.

Network helpers (optional)

from aioproxies.net import current_ip, reset

ip = await current_ip("1.2.3.4:8080:u:p")   # egress ip through the proxy (ipify by default)
await reset("https://provider/reset-url")    # rotate upstream ip

current_ip defaults to ipify and is opt-in; pass test_url= to point elsewhere.

Notes

• No module-level globals; rotation state is per-instance.
• A missing proxy file raises, it does not exit the process.
• Country/ASN tables, provider accounts, and reset URLs are project config — inject them; do not hardcode credentials in shared code.
• aioweb integration is the manual loop shown above (get → use → burn on block). A provider-protocol auto-rotation is a possible later enhancement, not in this lib.

Changelog

v0.3.2

• Portless proxy url now fails loud and legible. canonical_key/.key() (and therefore burn/add/remove) on a proxy url with no :port (http://user:pass@host) used to raise a bare ValueError("invalid literal for int() with base 10: ''") from normalize_port('') — an unrelated int() error that broke burn()'s documented "raises ValueError naming the key if not in pool" contract. normalize_port now raises ValueError("missing port: ...") naming the problem instead. to_proxy() on a portless url still succeeds (construction tolerates a missing port); only keying it raises.

v0.3.1

• Docs-only de-bloat pass. Compressed module/internal docstrings and comments, replaced mojibake em-dashes with plain hyphens in source. No behavior change.

v0.3.0

• Stored-shape change: int unix deadlines. burn(proxy, seconds) and the cooldown timeout write int(time.time()) + n instead of a float. Comparisons (is_burned/stats/selection) are unaffected — this only tightens what gets persisted into per-proxy state.
• Port key normalization: key() strips leading zeros from the port, so host:080 and host:80 are the same canonical slot. Previously a zero-padded port could break a get() → burn() pairing (the burn would raise "not in pool" against the proxy that was just handed out).
• Constructor / from_file dedupe by canonical key, matching add(). A proxy list with repeated entries (e.g. same host:port:user:pass twice) no longer inflates the pool or double-weights rotation.
• remove() no longer skews rotation. Removing a proxy that precedes the rotation cursor now decrements the cursor, so the next next() call doesn't skip or double-serve a proxy.
• restore() on an unknown proxy now logs a warning and no-ops, matching remove()'s contract (previously it silently did nothing, with burn() raising for the same precondition and remove() warning — restore() was the odd one out).
• Genuine timed burns under cooldown>0 now log at WARNING, not DEBUG. A real burn(proxy, seconds) is now tracked separately from the manager's own cooldown resting, so it no longer gets buried as routine cooldown noise when every proxy in the pool happens to also be cooling down.
• Source truthiness → presence. The constructor's exactly-one-source check now uses is not None instead of truthiness, so template="" (or another falsy-but-explicitly-provided source) is accepted rather than silently rejected as "no source given."
• Escape-aware bare-{} template normalization. A template's bare {} is still filled as the session slot, but an escaped {{}} (str.format's own convention for a literal {} in the output) now survives untouched instead of being corrupted into {{session}}.

v0.2.1

• Legible missing-template-field error: a template= placeholder not supplied to next(**fields) now raises a clear ValueError naming the field, instead of leaking a bare KeyError from str.format.

v0.2.0

• Proxy health for rotating lists: burn/restore/is_burned (dead -1 vs timed), stats/reset_stats, and the new ProxiesExhaustedError (all-dead pool).
• Cooldown: new cooldown= constructor arg spaces reuse; default 0 = off.
• Live pool edits: replace (with keep_state=), add, remove, keyed by a canonical proxy key that accepts every input shape (incl. auth-less / IP-auth).
• {session} default is now 8-char alphanumeric (was 10-digit numeric); session_len default is 8. Templates that set session_len explicitly are unaffected by the length change; the charset is now alphanumeric regardless.
• Backward-compatible: a v0.1.0-style manager (no burns, cooldown=0) behaves byte-for-byte identically — sequential round-robin, next()→Proxy, get()→aiohttp dict, never raises.