rethink-public/aioproxies
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: proxy health & pool management (v0.2.0)

Browse Source 
burn/timeout (dead -1 vs timed, lazy expiry), usage counters, reuse cooldown,
and live pool management (replace/add/remove) for the rotating list source.
template/static sources treat these as no-ops that log a warning.

- canonical key (host:port:user:pass, or host:port auth-less) identifies a proxy
  across every input shape (spec/Proxy/url/aiohttp/camoufox/socks5 dict); host is
  lowercased (DNS-caseless), password included, 4-part split on first 3 colons so
  colon passwords survive
- ProxiesExhaustedError when the whole pool is permanently dead
- cooldown soft (falls through to soonest-recovering, never raises); default 0 = off
- soonest-recovering fallthrough logs warning on a genuine burn, debug on cooldown
- {session} now 8-char alphanumeric (was 10-digit numeric); session_len default 8
- backward-compatible: a v0.1.0 manager (no burns, cooldown=0) is byte-for-byte
  identical — sequential round-robin, next()->Proxy, get()->aiohttp dict, never raises

Signed-off-by: disqualifier <dev@disqualifier.me>
This commit is contained in:
dsql
2026-06-25 14:23:20 -04:00
parent 993dd12fc2
commit ca708191c8
5 changed files with 477 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+116 -6
View File
@@ -1,16 +1,17 @@
# aioproxies
Proxy parsing, formatting, and source management. Renders proxies for
Proxy parsing, formatting, health, and pool management. Renders proxies for
aiohttp/aioweb, camoufox, and socks5; manages session templates (with
caller-supplied fields like country/ttl), rotating lists, or a static proxy.
**Credentials are always injected — never hardcoded.**
caller-supplied fields like country/ttl), rotating lists, or a static proxy; and
(for rotating lists) tracks burn/timeout, usage, reuse cooldown, and live pool
edits. **Credentials are always injected — never hardcoded.**
## Install
```
aioproxies @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v0.1.0
aioproxies @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v0.2.0
# network helpers (current_ip / reset) need the extra:
aioproxies[net] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v0.1.0
aioproxies[net] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v0.2.0
```
The core has no dependencies. The `net` extra adds `aiohttp` for `current_ip` /
@@ -21,13 +22,18 @@ The core has no dependencies. The `net` extra adds `aiohttp` for `current_ip` /
```python
from aioproxies import parse
p = parse("1.2.3.4:8080:user:pass") # or "host:port"
p = parse("1.2.3.4:8080:user:pass") # or "host:port" for IP-authenticated proxies
p.aiohttp() # {"http": "...", "https": "..."} -> aioweb ExtendedSession(proxies=)
p.camoufox() # {"server": "...", "username": ..., "password": ...}
p.socks5() # {"server": "socks5://...", ...}
p.url() # "http://user:pass@host:port"
p.key() # "1.2.3.4:8080:user:pass" (canonical identity; "host:port" if auth-less)
```
Auth-less (IP-authenticated) proxies are first-class: `"host:port"` parses and
every render shape omits the credentials. The 4-part form splits on the first three
colons, so a password may itself contain colons (`host:port:user:pa:ss:word`).
## Sources
Construct with exactly one source:
@@ -86,6 +92,92 @@ The credentials are baked in once with `.format()`; the per-call fields and `{se
are double-braced (`{{country}}`) so they pass through that `.format()` untouched and
remain for `next(**fields)` / the lib to fill.
## Proxy health & pool management (rotating list source)
For `proxies=` / `from_file` sources, the manager tracks each proxy's health and
usage and lets you edit the pool live. (On `template=` / `static=` these methods are
**no-ops that log a warning** and return cleanly — generic caller code can call them
regardless of source.)
```python
from aioproxies import AioProxies, ProxiesExhaustedError
pm = AioProxies(proxies=[...], cooldown=5) # 5s reuse spacing; cooldown defaults to 0 (off)
try:
proxy = pm.get() # next usable proxy, aiohttp dict
resp = await session.get(url, proxies=proxy)
if response_looks_blocked(resp):
pm.burn(proxy, 600) # time out 10 min ... or pm.burn(proxy) for dead
except ProxiesExhaustedError:
... # whole pool permanently dead — back off / refetch
pm.replace(fresh_batch) # swap in a new provider batch
pm.stats() # monitor uses + timeout state
```
### Selection
Rotation is **sequential round-robin over usable proxies**:
1. proxies that are fine (never burned, or a timed burn already expired) cycle in
order — same as v0.1.0.
2. if none are fine but some are merely timed, the manager **warns** and hands out
the one recovering soonest (still counts a use).
3. if every proxy is permanently dead (`-1`), `next()`/`get()` raise
`ProxiesExhaustedError`.
`next()` still returns a `Proxy`; `get()` still returns an aiohttp dict.
### Burn / restore
```python
pm.burn(proxy) # dead/permanent (-1) — only manual restore() brings it back
pm.burn(proxy, 600) # timed — usable again automatically after 600s (lazy, no timers)
pm.restore(proxy) # clear any burn/timeout, back to fine
pm.is_burned(proxy) # current state (expired timed burns read False)
```
`burn`/`restore`/`is_burned`/`remove` accept **any proxy shape** — a spec string, a
`Proxy`, an aiohttp/camoufox/socks5 dict, or a url — all resolve to the same canonical
key (`host:port:user:pass`, or `host:port` auth-less). The password is part of the key,
so two proxies differing only by password are distinct slots. `burn` on a proxy not in
the pool raises `ValueError`.
### Cooldown
`AioProxies(proxies=[...], cooldown=5)` spaces reuse: each handout times the proxy out
for `cooldown` seconds so it isn't reused if avoidable. It is **soft** — under load
(everything cooling) it falls through to the soonest-to-recover and never raises on
cooldown alone. Default `0` = off (exact v0.1.0 behavior).
### Stats
```python
pm.stats() # [{"proxy": "h:p:u:pw", "uses": int, "state": "active"|"timed"|"dead",
# "timeout": <ts | -1 | None>}, ...]
pm.reset_stats() # zero all use counters; leave timeouts untouched
```
`uses` is a pure counter (every handout, including forced ones); it never drives
selection and survives burns — a proxy can read "used 500× and dead". The `proxy`
field is the full canonical spec (passwords included).
### Live pool edits
```python
pm.replace(new_batch) # swap the whole list; wipes per-proxy state
pm.replace(new_batch, keep_state=True) # survivors keep uses/timeout; new ones start clean
pm.add("h:p:u:pw") # append (single or list); skip exact-duplicate keys
pm.remove(proxy) # drop a slot entirely (any shape) — distinct from burn
```
`replace` resets the rotation index and honors the manager's `shuffle` setting on the
incoming list. `remove` differs from `burn`: burn = unusable but still tracked; remove =
gone from the pool. Like the burn family, `add`/`replace` accept **any proxy shape**
(spec/`Proxy`/url/aiohttp dict/camoufox/socks5 dict). `canonical_key(shape)` and
`to_proxy(shape)` are exported if you need the key or a normalized `Proxy` yourself.
## Network helpers (optional)
```python
@@ -103,3 +195,21 @@ await reset("https://provider/reset-url") # rotate upstream ip
- A missing proxy file raises, it does not exit the process.
- Country/ASN tables, provider accounts, and reset URLs are project config —
inject them; do not hardcode credentials in shared code.
- aioweb integration is the manual loop shown above (get → use → burn on block).
A provider-protocol auto-rotation is a possible later enhancement, not in this lib.
## Changelog
### v0.2.0
- **Proxy health for rotating lists:** `burn`/`restore`/`is_burned` (dead `-1` vs
timed), `stats`/`reset_stats`, and the new `ProxiesExhaustedError` (all-dead pool).
- **Cooldown:** new `cooldown=` constructor arg spaces reuse; default `0` = off.
- **Live pool edits:** `replace` (with `keep_state=`), `add`, `remove`, keyed by a
canonical proxy key that accepts every input shape (incl. auth-less / IP-auth).
- **`{session}` default is now 8-char alphanumeric** (was 10-digit numeric);
`session_len` default is `8`. Templates that set `session_len` explicitly are
unaffected by the length change; the charset is now alphanumeric regardless.
- **Backward-compatible:** a v0.1.0-style manager (no burns, `cooldown=0`) behaves
byte-for-byte identically — sequential round-robin, `next()`→`Proxy`,
`get()`→aiohttp dict, never raises.