21 Commits
Author SHA1 Message Date
dsql 4d5caac8d8 chore: bump to 1.1.0 (logging-discipline audit)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-08-10 23:00:50 -04:00
dsql 5699aa9dde fix: mask the provider reset url before logging it
net.reset() logged the full reset_url at INFO on the success path; a provider reset url
can carry a rotation token in its query string, so this leaked it to logs. strip the
query + fragment before logging (stdlib urlsplit, no new dependency - the core stays
dependency-free) so the token cannot reach a log sink; scheme/host/path are kept for
diagnostics.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-08-09 02:10:45 -04:00
dsql e44c8950ee release: 1.0.0
first stable release. pre-1.0.0 verification complete: all surviving MED regressions and
gaps resolved and independently re-fired, tree audited clean across the suite.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-09 18:53:15 -04:00
dsql 0b6b6b440b fix: cooldown handout never truncates a longer active burn deadline
A forced soonest-recovering handout unconditionally set timeout = now + cooldown,
silently collapsing an explicit burn(proxy, 3600) to the cooldown window. Cooldown now
only EXTENDS availability delay: it leaves a still-running longer timed burn untouched
and never resurrects a permanent _DEAD (-1) burn into a future timestamp, while a fine
proxy and a burn shorter than cooldown are cooled as before.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:26:31 -04:00
dsql 1adc8139b9 docs: fix README health-check snippet's invalid aiohttp kwarg; widen add()/replace() type hints to accept dict shapes
the flagship health-check snippet called session.get(url, proxies=proxy) on a
plain aiohttp session; aiohttp's ClientSession.get/_request has no `proxies=`
kwarg (only singular `proxy:`), so the snippet TypeErrors immediately if
followed verbatim. pm.get()'s dict output is documented (aiohttp()'s own
docstring) as being for aioweb's ExtendedSession(proxies=...), so the snippet
now names that session type instead of a bare aiohttp one.

add()/replace() type hints omitted Dict[str, str] even though both already
call to_proxy() at runtime and accept dict-shaped proxy specs, same as the
burn()/restore()/is_burned()/remove() family. purely annotation, no behavior
change.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 00:15:29 -04:00
dsql a4a0227d5d fix: URL-aware parse() at static/template/net.current_ip call sites; self-clear stale burn flag; guard replace([])
static=/template=/net.current_ip() fed URL-shaped input straight into parse(),
which only understands host:port(:user:pass) specs and mis-splits a url's
colons into a garbage Proxy (e.g. embedded-auth urls) instead of raising or
normalizing. all three now route through to_proxy(), the url-aware normalizer
add()/replace() already use.

the per-state `burned` flag (drives WARNING vs DEBUG on the cooldown
fallthrough log) was only ever cleared by restore(); a timed burn() that
lazily expires (is_burned() correctly reads False again) left `burned` stuck
True, permanently upgrading routine cooldown noise to WARNING. `burned` now
self-clears wherever the existing lazy-expiry check already finds the proxy
available (_next_from_list's fine-tier scan, is_burned()).

replace([]) silently wiped the entire pool with no error, even though the
constructor already rejects an empty proxies=[] list. replace() now raises
the same ValueError for an empty list on list sources.

parse()'s docstring corrected: it is spec-string only and does not understand
url shapes; points callers at to_proxy() for url/dict/spec input.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 00:15:06 -04:00
dsql 9649eb77c7 refactor: derive __version__ from package metadata (single source)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:59:41 -04:00
dsql 1ca3144245 fix: sync __version__ to 0.3.2 (drifted behind pyproject)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:25:17 -04:00
dsql e6663926c0 docs: pin install line to v0.3.2 (pyproject already bumped)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:23:52 -04:00
dsql bac459c5b5 fix: normalize_port raises a clear error on missing/empty port
canonical_key/key()/burn on a portless proxy url used to raise a bare
ValueError from int('') instead of naming the problem, breaking burn()'s
documented not-in-pool contract. normalize_port now raises a legible
'missing port' ValueError; to_proxy() on a portless url still succeeds.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:15:06 -04:00
dsql 84127a93ff docs: compress prose/module docstrings, em-dash->hyphen (de-bloat wave 1)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 00:12:54 -04:00
dsql 147341d38a fix: canonical port key, remove()/restore() drift, escape-aware templates
Port keys no longer drift on zero-padding (host:080 vs host:80 collapsed
one canonical key), so a get() -> burn() pairing on a padded port no
longer raises a false "not in pool". Constructor/from_file now dedupe by
canonical key like add(). remove() decrements the rotation cursor when
the removed slot precedes it, fixing a skip/double-serve. restore() on
an unknown key now warns instead of silently no-op, matching remove()'s
contract. A genuine timed burn() under cooldown>0 is now tracked
separately from routine cooldown resting, so it logs WARNING instead of
being buried as DEBUG. Source validation uses is-not-None instead of
truthiness, so a falsy-but-provided source (e.g. template="") is
accepted. Bare "{}" template normalization is now escape-aware, so an
intentional "{{}}" literal survives instead of being corrupted.

Stored-shape change: burn()/cooldown deadlines now write int(time.time())
instead of float; comparisons are unaffected.

Docstrings compressed (module + next()); no behavior change.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 23:25:52 -04:00
dsql eef4b25f07 chore: ignore .claude/ dir (CLAUDE.md now lives under .claude/)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:55:13 -04:00
dsql a5e36544d4 docs: note url()/aiohttp() percent-encode proxy credentials
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:41:36 -04:00
dsql 72c5342a6b fix: AP-1 percent-encode url() creds, AP-2 reject next(session=) collision
AP-1: url()/aiohttp() percent-encode user/password so reserved chars (/ # ? @) produce a
valid url, mirroring the parse-side unquote. AP-2: next(session=...) raises a clear
ValueError instead of an opaque str.format TypeError. net.current_ip uses
json(content_type=None) + dict guard.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:34:36 -04:00
dsql 932fb71c95 docs: pin install line to release, note unpinned-latest option
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 18:13:33 -04:00
dsql 0abc071f14 docs: show unpinned install line; note tag-pinning for reproducibility
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 18:07:17 -04:00
dsql fc27d77000 fix: preserve URL-embedded proxy auth; clearer empty-list + malformed-template errors (v0.2.2)
- _proxy_from_dict server branch falls back to auth embedded in the server URL when no
  explicit username/password keys are given, so it isn't dropped and the proxy keys
  with its credentials instead of colliding auth-less (L5)
- AioProxies(proxies=[]) now raises a clear 'empty' error, not the misleading
  'provide exactly one of' (nit)
- a malformed template re-raises with a naming message instead of a bare str.format
  ValueError (nit).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 17:57:37 -04:00
dsql 260b92b66a docs: add v0.2.1 changelog entry (legible missing-template-field error)
the README changelog jumped from v0.2.0 to nothing; added the v0.2.1 entry documenting the legible ValueError for a template placeholder not supplied to next(**fields).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 01:39:21 -04:00
dsql f618b6a6a1 style: drop inline comments in the template-field error path
the rationale they carried is already in the ValueError message; the lib convention is docstrings only.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-28 17:18:28 -04:00
dsql aa661bd6de fix: legible error on missing template placeholder (v0.2.1)
next() on a template source did template.format(...) directly, so a placeholder not
supplied in next(**fields) leaked a bare KeyError. catch KeyError/IndexError and
raise a clear ValueError naming the missing placeholder.

verified: next() without a required field -> ValueError naming it; happy path intact.
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-28 15:50:33 -04:00
7 changed files with 278 additions and 127 deletions
+1 -1
View File
@@ -1,5 +1,5 @@
# claude
CLAUDE.md
.claude/
# python
__pycache__/
+78 -9
View File
@@ -9,14 +9,16 @@ edits. **Credentials are always injected — never hardcoded.**
## Install
```
aioproxies @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v0.2.0
aioproxies @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v1.0.0
# network helpers (current_ip / reset) need the extra:
aioproxies[net] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v0.2.0
aioproxies[net] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v1.0.0
```
The core has no dependencies. The `net` extra adds `aiohttp` for `current_ip` /
`reset`.
Drop the `@v1.0.0` suffix from the line above to install the latest unpinned.
## Formatting
```python
@@ -33,6 +35,8 @@ p.key() # "1.2.3.4:8080:user:pass" (canonical identity; "host:port" if a
Auth-less (IP-authenticated) proxies are first-class: `"host:port"` parses and
every render shape omits the credentials. The 4-part form splits on the first three
colons, so a password may itself contain colons (`host:port:user:pa:ss:word`).
`url()` / `aiohttp()` percent-encode the credentials, so reserved characters
(`/ # ? @`) in a user or password still produce a valid URL.
## Sources
@@ -101,12 +105,14 @@ regardless of source.)
```python
from aioproxies import AioProxies, ProxiesExhaustedError
from aioweb import ExtendedSession
pm = AioProxies(proxies=[...], cooldown=5) # 5s reuse spacing; cooldown defaults to 0 (off)
try:
proxy = pm.get() # next usable proxy, aiohttp dict
resp = await session.get(url, proxies=proxy)
async with ExtendedSession(proxies=proxy) as session:
resp = await session.get(url)
if response_looks_blocked(resp):
pm.burn(proxy, 600) # time out 10 min ... or pm.burn(proxy) for dead
except ProxiesExhaustedError:
@@ -140,9 +146,14 @@ pm.is_burned(proxy) # current state (expired timed burns read False)
`burn`/`restore`/`is_burned`/`remove` accept **any proxy shape** — a spec string, a
`Proxy`, an aiohttp/camoufox/socks5 dict, or a url — all resolve to the same canonical
key (`host:port:user:pass`, or `host:port` auth-less). The password is part of the key,
so two proxies differing only by password are distinct slots. `burn` on a proxy not in
the pool raises `ValueError`.
key (`host:port:user:pass`, or `host:port` auth-less; the port is normalized so
`host:080` and `host:80` are one slot). The password is part of the key, so two proxies
differing only by password are distinct slots. `burn` on a proxy not in the pool raises
`ValueError`; `restore` on a proxy not in the pool logs a warning and no-ops (matching
`remove`'s contract, as of v0.3.0 — previously it silently did nothing with no signal).
A **portless** url (`http://user:pass@host`, no `:port`) parses fine via `to_proxy()`,
but keying it (`canonical_key`/`.key()`/`burn`/`add`/`remove`) raises `ValueError`
naming the missing port — a proxy needs a port to have a canonical identity.
### Cooldown
@@ -174,9 +185,12 @@ pm.remove(proxy) # drop a slot entirely (any shape) —
`replace` resets the rotation index and honors the manager's `shuffle` setting on the
incoming list. `remove` differs from `burn`: burn = unusable but still tracked; remove =
gone from the pool. Like the burn family, `add`/`replace` accept **any proxy shape**
(spec/`Proxy`/url/aiohttp dict/camoufox/socks5 dict). `canonical_key(shape)` and
`to_proxy(shape)` are exported if you need the key or a normalized `Proxy` yourself.
gone from the pool — removing a slot that precedes the rotation cursor adjusts the
cursor so `next()` doesn't skip a proxy. Like the burn family, `add`/`replace` accept
**any proxy shape** (spec/`Proxy`/url/aiohttp dict/camoufox/socks5 dict). The
constructor and `from_file` also dedupe by canonical key, same as `add`.
`canonical_key(shape)` and `to_proxy(shape)` are exported if you need the key or a
normalized `Proxy` yourself.
## Network helpers (optional)
@@ -200,6 +214,61 @@ await reset("https://provider/reset-url") # rotate upstream ip
## Changelog
### v0.3.2
- **Portless proxy url now fails loud and legible.** `canonical_key`/`.key()`
(and therefore `burn`/`add`/`remove`) on a proxy url with no `:port`
(`http://user:pass@host`) used to raise a bare `ValueError("invalid literal
for int() with base 10: ''")` from `normalize_port('')` — an unrelated `int()`
error that broke `burn()`'s documented "raises ValueError naming the key if
not in pool" contract. `normalize_port` now raises `ValueError("missing port:
...")` naming the problem instead. `to_proxy()` on a portless url still
succeeds (construction tolerates a missing port); only keying it raises.
### v0.3.1
- **Docs-only de-bloat pass.** Compressed module/internal docstrings and comments,
replaced mojibake em-dashes with plain hyphens in source. No behavior change.
### v0.3.0
- **Stored-shape change: int unix deadlines.** `burn(proxy, seconds)` and the
cooldown timeout write `int(time.time()) + n` instead of a `float`. Comparisons
(`is_burned`/`stats`/selection) are unaffected — this only tightens what gets
persisted into per-proxy state.
- **Port key normalization:** `key()` strips leading zeros from the port, so
`host:080` and `host:80` are the same canonical slot. Previously a zero-padded
port could break a `get()``burn()` pairing (the burn would raise "not in
pool" against the proxy that was just handed out).
- **Constructor / `from_file` dedupe by canonical key**, matching `add()`. A
proxy list with repeated entries (e.g. same host:port:user:pass twice) no
longer inflates the pool or double-weights rotation.
- **`remove()` no longer skews rotation.** Removing a proxy that precedes the
rotation cursor now decrements the cursor, so the next `next()` call doesn't
skip or double-serve a proxy.
- **`restore()` on an unknown proxy now logs a warning and no-ops**, matching
`remove()`'s contract (previously it silently did nothing, with `burn()`
raising for the same precondition and `remove()` warning — `restore()` was the
odd one out).
- **Genuine timed burns under `cooldown>0` now log at WARNING**, not DEBUG. A
real `burn(proxy, seconds)` is now tracked separately from the manager's own
cooldown resting, so it no longer gets buried as routine cooldown noise when
every proxy in the pool happens to also be cooling down.
- **Source truthiness → presence.** The constructor's exactly-one-source check
now uses `is not None` instead of truthiness, so `template=""` (or another
falsy-but-explicitly-provided source) is accepted rather than silently
rejected as "no source given."
- **Escape-aware bare-`{}` template normalization.** A template's bare `{}` is
still filled as the session slot, but an escaped `{{}}` (str.format's own
convention for a literal `{}` in the output) now survives untouched instead of
being corrupted into `{{session}}`.
### v0.2.1
- **Legible missing-template-field error:** a `template=` placeholder not supplied to
`next(**fields)` now raises a clear `ValueError` naming the field, instead of leaking
a bare `KeyError` from `str.format`.
### v0.2.0
- **Proxy health for rotating lists:** `burn`/`restore`/`is_burned` (dead `-1` vs
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "aioproxies"
version = "0.2.0"
version = "1.1.0"
description = "proxy parsing, formatting, health, and pool management for aiohttp/aioweb, camoufox, and socks5"
requires-python = ">=3.10"
dependencies = []
+6 -6
View File
@@ -1,9 +1,6 @@
"""aioproxies proxy parsing, formatting, and source management.
"""aioproxies - proxy parsing, formatting, and source management. see README for usage."""
from importlib.metadata import version, PackageNotFoundError
renders proxies for aiohttp/aioweb, camoufox, and socks5; manages session
templates (with caller-supplied fields like country/ttl), rotating lists, or a
static proxy. credentials are always injected, never hardcoded.
"""
from .manager import AioProxies, ProxiesExhaustedError, ProxyManager, aioproxies
from .proxy import Proxy, canonical_key, parse, to_proxy
@@ -18,4 +15,7 @@ __all__ = [
"to_proxy",
]
__version__ = "0.2.0"
try:
__version__ = version("aioproxies")
except PackageNotFoundError:
__version__ = "0.0.0+unknown"
+125 -73
View File
@@ -1,35 +1,23 @@
"""proxy source management: session templates, rotation, static.
`AioProxies` is constructed with exactly one source and hands out `Proxy`
objects. it carries no module-level globals (rotation state is per-instance) and
never exits the process — a missing proxy file raises, it does not `sys.exit`.
`AioProxies` takes exactly one source (template / proxies / static) and hands out
`Proxy` objects via `next()`/`get()`. no module-level globals; a missing proxy file
raises, never `sys.exit`. see README for the full source/method reference.
sources:
- template: a format string with named placeholders. `{session}` is filled with
a fresh session id on each `next()`; any other placeholder (e.g. `{country}`,
`{ttl}`) is filled from keyword args passed to `next(**fields)`. a bare `{}` is
also accepted and treated as the session slot (back-compat with simple templates).
- proxies: a list of specs cycled round-robin
- static: one fixed proxy
v0.2.0 adds proxy health to the rotating list source only — burn/timeout, usage
counters, reuse cooldown, and pool management (replace/add/remove). these are
keyed by each proxy's canonical key (`host:port:user:pass`, or `host:port` for
auth-less / IP-authenticated proxies). on template/static sources they are no-ops
that log a warning and return cleanly, so generic caller code can call them
proxy health (rotating list source only) is keyed by each proxy's canonical key
(`host:port:user:pass`, or `host:port` auth-less; port is normalized so `:080` and
`:80` collapse to one key). on template/static sources health/pool methods are
no-ops that log a warning and return, so generic caller code can call them
regardless of source.
per-proxy state (keyed by canonical key):
- `uses`: pure counter, incremented on every handout. never drives selection;
survives burns (a proxy can read "used 500x and dead").
- `timeout`: availability state. `None`/`0` = fine; `-1` = dead/permanent (manual
restore only); a future unix ts = timed out until then (lazy, checked against
`time.time()`, no timers). cooldown and timed burns share this field. durations
(seconds) only ever exist as arguments — converted to `now + seconds` and
discarded; the lib never stores a raw duration.
per-proxy `timeout` (the availability field) is shared by cooldown and timed
burns: `None`/`0` fine, `-1` dead/permanent, a future unix ts = timed out until
then; durations only ever exist as call arguments, converted to `now + seconds`
and discarded, never stored raw.
"""
import logging
import random
import re
import string
import time
from typing import Dict, List, Optional, Union
@@ -39,6 +27,7 @@ from .proxy import Proxy, canonical_key, parse, to_proxy
log = logging.getLogger(__name__)
_DEAD = -1
_BARE_SESSION_SLOT = re.compile(r"(?<!\{)\{\}(?!\})")
class ProxiesExhaustedError(Exception):
@@ -58,27 +47,47 @@ class AioProxies:
shuffle: bool = True,
cooldown: int = 0,
):
sources = [s for s in (template, proxies, static) if s]
if proxies is not None and len(proxies) == 0:
raise ValueError("proxies list is empty; provide at least one proxy")
sources = [s for s in (template, proxies, static) if s is not None]
if len(sources) != 1:
raise ValueError("provide exactly one of: template, proxies, static")
self.template = template.replace("{}", "{session}") if template else None
self.template = self._normalize_template(template) if template is not None else None
self.session_len = session_len
self.cooldown = cooldown
self._shuffle = shuffle
self._static = parse(static) if static else None
self._proxies = [parse(p) for p in proxies] if proxies else []
self._static = to_proxy(static) if static is not None else None
self._proxies = self._dedupe([parse(p) for p in proxies]) if proxies is not None else []
if self._proxies and shuffle:
random.shuffle(self._proxies)
self._state: Dict[str, Dict[str, object]] = {}
for proxy in self._proxies:
self._state.setdefault(proxy.key(), {"uses": 0, "timeout": None})
self._state.setdefault(proxy.key(), self._fresh_state())
self._index = 0
@staticmethod
def _dedupe(proxies: List[Proxy]) -> List[Proxy]:
"""drop later entries sharing a canonical key, preserving first-seen order"""
seen: Dict[str, Proxy] = {}
for proxy in proxies:
seen.setdefault(proxy.key(), proxy)
return list(seen.values())
@staticmethod
def _fresh_state() -> Dict[str, object]:
"""a clean per-proxy state entry: no uses, no timeout, not burned"""
return {"uses": 0, "timeout": None, "burned": False}
@staticmethod
def _normalize_template(template: str) -> str:
"""fill a bare `{}` session slot with `{session}`; leave escaped `{{}}` untouched"""
return _BARE_SESSION_SLOT.sub("{session}", template)
@classmethod
def from_file(cls, path: str, **kwargs) -> "AioProxies":
"""build a rotating manager from a newline-delimited proxy file
raises FileNotFoundError if the path is missing never exits the process.
raises FileNotFoundError if the path is missing - never exits the process.
"""
with open(path, "r", encoding="utf-8") as fh:
lines = [ln.strip() for ln in fh if ln.strip()]
@@ -107,23 +116,47 @@ class AioProxies:
return False
return now >= timeout
@staticmethod
def _clear_stale_burn(state: Dict[str, object]) -> None:
"""self-clear a lazily-expired timed burn's `burned` flag once found available
mirrors the lazy expiry already applied to `timeout` itself: a timed burn()
that has simply run out its clock should stop upgrading routine cooldown
fallthrough logs to warning, without requiring an explicit restore().
"""
if state["timeout"] not in (None, 0, _DEAD):
state["burned"] = False
def next(self, **fields: object) -> Proxy:
"""return the next proxy from the configured source
for template sources, `{session}` is always filled with a fresh id and any
other named placeholder is filled from `fields` (e.g. next(country="ca",
ttl=30)). fields are ignored by list/static sources.
template sources: `{session}` is always filled with a fresh id, other
placeholders come from `fields` (e.g. next(country="ca", ttl=30)). fields
are ignored by list/static sources.
for list sources, skips proxies whose timeout is active (-1 dead, or a
future ts not yet passed), increments `uses` on handout, and applies the
manager's cooldown. if none are fine but some are merely timed, hands out
the one recovering soonest (with a warning); raises ProxiesExhaustedError
if every proxy is permanently dead.
list sources: skips proxies whose timeout is active (-1 dead, or a future
ts not yet passed), increments `uses` on handout, applies the manager's
cooldown. if none are fine but some are merely timed, hands out the one
recovering soonest (warns); raises ProxiesExhaustedError if all are dead.
"""
if self._static is not None:
return self._static
if self.template is not None:
return parse(self.template.format(session=self.session_id(), **fields))
if "session" in fields:
# `session` is auto-filled with a fresh id; a caller-supplied one would
# collide in str.format with an opaque TypeError - reject it clearly
raise ValueError("'session' is filled automatically; do not pass it to next()")
try:
filled = self.template.format(session=self.session_id(), **fields)
except (KeyError, IndexError) as exc:
raise ValueError(
f"template placeholder {exc} not provided; pass it to next(**fields)"
) from exc
except ValueError as exc:
# a malformed template (e.g. an unmatched '{') makes str.format raise a
# bare ValueError - re-raise naming the cause so it isn't cryptic
raise ValueError(f"malformed proxy template {self.template!r}: {exc}") from exc
return to_proxy(filled)
return self._next_from_list()
def _next_from_list(self) -> Proxy:
@@ -136,9 +169,10 @@ class AioProxies:
fine: List[int] = []
for offset in range(count):
idx = (self._index + offset) % count
timeout = self._state[self._proxies[idx].key()]["timeout"]
if self._available(timeout, now):
state = self._state[self._proxies[idx].key()]
if self._available(state["timeout"], now):
fine.append(idx)
self._clear_stale_burn(state)
if fine:
chosen = fine[0]
@@ -156,24 +190,22 @@ class AioProxies:
state = self._state[proxy.key()]
state["uses"] = int(state["uses"]) + 1
if self.cooldown > 0:
state["timeout"] = now + self.cooldown
# cooldown only ever EXTENDS availability delay - never shorten a longer
# active timeout (a still-running timed burn) down to now+cooldown. _DEAD
# (-1) is a permanent burn and must not be resurrected into a future ts.
cooled = int(now) + self.cooldown
current = state["timeout"]
if current == _DEAD:
pass
elif isinstance(current, (int, float)) and current > cooled:
pass
else:
state["timeout"] = cooled
return proxy
def _has_burned(self) -> bool:
"""whether the empty 'fine' tier reflects a genuine burn, not just cooldown
a dead (-1) proxy is always a real burn. a future-ts timeout is a real timed
burn only when cooldown is off; with cooldown on, future-ts entries are the
manager's own resting and not a pool-health signal. used to pick warning
(real trouble) vs debug (normal cooldown) on the soonest-recovering path.
"""
for state in self._state.values():
timeout = state["timeout"]
if timeout == _DEAD:
return True
if self.cooldown == 0 and timeout not in (None, 0):
return True
return False
"""whether any proxy carries a genuine burn (vs just cooldown resting), for log level choice"""
return any(state["burned"] for state in self._state.values())
def _soonest_recovering(self) -> Optional[int]:
"""index of the timed (non-dead) proxy recovering soonest, or None"""
@@ -208,16 +240,24 @@ class AioProxies:
if seconds is None:
self._state[key]["timeout"] = _DEAD
else:
self._state[key]["timeout"] = time.time() + seconds
self._state[key]["timeout"] = int(time.time()) + seconds
self._state[key]["burned"] = True
def restore(self, proxy: Union[str, Proxy, Dict[str, str]]) -> None:
"""clear any burn/timeout on a proxy (back to fine). no-op if already fine"""
"""clear any burn/timeout on a proxy (back to fine). no-op if already fine
matches remove()'s contract for an unknown proxy: logs a warning and returns
rather than silently doing nothing.
"""
if not self._is_list_source():
self._warn_non_list("restore()")
return
key = canonical_key(proxy)
if key in self._state:
self._state[key]["timeout"] = None
if key not in self._state:
log.warning("restore(): proxy not in pool: %s", key)
return
self._state[key]["timeout"] = None
self._state[key]["burned"] = False
def is_burned(self, proxy: Union[str, Proxy, Dict[str, str]]) -> bool:
"""whether a proxy is currently unavailable (lazy expiry of timed burns)
@@ -230,8 +270,11 @@ class AioProxies:
key = canonical_key(proxy)
if key not in self._state:
return False
timeout = self._state[key]["timeout"]
return not self._available(timeout, time.time())
state = self._state[key]
available = self._available(state["timeout"], time.time())
if available:
self._clear_stale_burn(state)
return not available
def stats(self) -> List[Dict[str, object]]:
"""per-proxy usage + state snapshot (list sources only, else empty)
@@ -269,17 +312,21 @@ class AioProxies:
for state in self._state.values():
state["uses"] = 0
def replace(self, proxies: List[Union[str, Proxy]], *, keep_state: bool = False) -> None:
def replace(self, proxies: List[Union[str, Proxy, Dict[str, str]]], *, keep_state: bool = False) -> None:
"""swap the entire proxy list
keep_state=False (default) wipes all per-proxy state (fresh batch).
keep_state=True preserves uses/timeout for proxies whose canonical key
survives the swap; new proxies start clean, dropped ones are forgotten.
resets the rotation index and honors the manager's shuffle setting.
resets the rotation index and honors the manager's shuffle setting. raises
ValueError on an empty list, matching the constructor's guard - use remove()
to drop proxies one at a time if you actually want a smaller pool.
"""
if not self._is_list_source():
self._warn_non_list("replace()")
return
if len(proxies) == 0:
raise ValueError("proxies list is empty; provide at least one proxy")
old_state = self._state
incoming = [to_proxy(p) for p in proxies]
if self._shuffle:
@@ -294,12 +341,12 @@ class AioProxies:
if keep_state and key in old_state:
new_state[key] = old_state[key]
else:
new_state[key] = {"uses": 0, "timeout": None}
new_state[key] = self._fresh_state()
self._proxies = new_proxies
self._state = new_state
self._index = 0
def add(self, proxies: Union[str, Proxy, List[Union[str, Proxy]]]) -> None:
def add(self, proxies: Union[str, Proxy, Dict[str, str], List[Union[str, Proxy, Dict[str, str]]]]) -> None:
"""append proxies to the pool, keeping existing state; skip duplicate keys"""
if not self._is_list_source():
self._warn_non_list("add()")
@@ -311,13 +358,15 @@ class AioProxies:
if key in self._state:
continue
self._proxies.append(proxy)
self._state[key] = {"uses": 0, "timeout": None}
self._state[key] = self._fresh_state()
def remove(self, proxy: Union[str, Proxy, Dict[str, str]]) -> None:
"""drop a proxy from the pool entirely (by canonical key, any shape)
distinct from burn (burn = unusable but tracked; remove = gone). clamps the
rotation index if needed. no-op + warning if the proxy is not present.
distinct from burn (burn = unusable but tracked; remove = gone). decrements
the rotation index when the removed slot precedes it, so the next() cursor
still lands on the same upcoming proxy instead of skipping one. no-op +
warning if the proxy is not present.
"""
if not self._is_list_source():
self._warn_non_list("remove()")
@@ -326,14 +375,17 @@ class AioProxies:
if key not in self._state:
log.warning("remove(): proxy not in pool: %s", key)
return
removed_idx = next(i for i, p in enumerate(self._proxies) if p.key() == key)
self._proxies = [p for p in self._proxies if p.key() != key]
del self._state[key]
if self._proxies:
self._index %= len(self._proxies)
else:
if not self._proxies:
self._index = 0
else:
if removed_idx < self._index:
self._index -= 1
self._index %= len(self._proxies)
# name aliases same class, call it whichever reads best at your call site
# name aliases - same class, call it whichever reads best at your call site
ProxyManager = AioProxies
aioproxies = AioProxies
+20 -5
View File
@@ -6,11 +6,24 @@ calling a function without it raises a clear error.
"""
import logging
from typing import Optional, Union
from urllib.parse import urlsplit, urlunsplit
from .proxy import Proxy, parse
from .proxy import Proxy, to_proxy
log = logging.getLogger(__name__)
def _safe_url(url: str) -> str:
"""strip query + fragment from a url for logging - a provider reset url can carry a
rotation token in its query/path; this drops the query/fragment (the usual token spot)
so the log line can't leak it. falls back to the raw url only if it won't parse"""
try:
parts = urlsplit(url)
except ValueError:
return url
return urlunsplit((parts.scheme, parts.netloc, parts.path, "", ""))
try:
import aiohttp
@@ -32,13 +45,15 @@ async def current_ip(
"""
if not _HAVE_AIOHTTP:
raise RuntimeError(_MISSING)
p = parse(proxy)
p = to_proxy(proxy)
t = aiohttp.ClientTimeout(total=timeout)
try:
async with aiohttp.ClientSession(timeout=t) as session:
async with session.get(test_url, proxy=p.url()) as resp:
data = await resp.json()
return data.get("ip")
# content_type=None: an echo endpoint may return text/plain json; the
# default would raise ContentTypeError and silently return None
data = await resp.json(content_type=None)
return data.get("ip") if isinstance(data, dict) else None
except Exception as exc:
log.warning("ip check failed: %s", exc)
return None
@@ -53,7 +68,7 @@ async def reset(reset_url: str, *, timeout: float = 15.0) -> bool:
async with aiohttp.ClientSession(timeout=t) as session:
async with session.get(reset_url) as resp:
ok = resp.status == 200
log.info("proxy reset %s -> %s", reset_url, resp.status)
log.info("proxy reset %s -> %s", _safe_url(reset_url), resp.status)
return ok
except Exception as exc:
log.warning("proxy reset failed: %s", exc)
+47 -32
View File
@@ -1,24 +1,33 @@
"""proxy parsing and formatting (pure, no network IO).
a `Proxy` holds host/port/optional-auth and renders the shapes different clients
want: an aiohttp/aioweb proxies dict, a camoufox proxy dict, a socks5 dict, or a
plain url. `parse` accepts the common "host:port" and "host:port:user:pass" string
forms (the user field may itself contain commas, e.g. session-param proxies; the
password field may itself contain colons). auth-less (IP-authenticated) proxies
with no creds are first-class — every render shape handles them via `has_auth`.
`Proxy` holds host/port/optional-auth and renders aiohttp/aioweb, camoufox, socks5,
and url shapes; auth-less (IP-authenticated) proxies are first-class throughout.
`parse` accepts "host:port" and "host:port:user:pass" - the user field may itself
contain commas and the password may itself contain colons.
`key()` produces a stable canonical identity (`host:port:user:pass`, or
`host:port` when auth-less) so burn/remove/stats recognize the same proxy from any
input shape. `canonical_key()` extends that to dict/url forms.
`key()` / `canonical_key()` produce a stable canonical identity so burn/remove/stats
recognize the same proxy from any input shape (spec string, dict, or url).
"""
from dataclasses import dataclass
from typing import Dict, Optional, Union
from urllib.parse import unquote, urlsplit
from urllib.parse import quote, unquote, urlsplit
SCHEME_HTTP = "http"
SCHEME_SOCKS5 = "socks5"
def normalize_port(port: Union[str, int]) -> str:
"""canonical port string: parsed to int, rendered without zero-padding
keeps host:080 and host:80 as one canonical key. raises ValueError naming the
missing port if empty/None (e.g. a portless proxy url), or on a non-integer
port, rather than silently keying it as-is.
"""
if port is None or port == "":
raise ValueError("missing port: proxy url or spec has no port")
return str(int(port))
@dataclass
class Proxy:
"""a single proxy endpoint with optional auth"""
@@ -38,19 +47,27 @@ class Proxy:
the host is lowercased (hostnames are case-insensitive per DNS, so
PROXY.example.com and proxy.example.com are the same host and collapse to one
key); port and credentials are kept verbatim. the password is included in
full (two proxies differing only by password are distinct slots). auth-less
proxies collapse to host:port with no trailing colons.
key); credentials are kept verbatim. the port is normalized (leading zeros
stripped) so host:080 and host:80 collapse to one key. the password is
included in full (two proxies differing only by password are distinct
slots). auth-less proxies collapse to host:port with no trailing colons.
"""
host = self.host.lower()
port = normalize_port(self.port)
if self.has_auth:
return f"{host}:{self.port}:{self.user}:{self.password}"
return f"{host}:{self.port}"
return f"{host}:{port}:{self.user}:{self.password}"
return f"{host}:{port}"
def url(self, scheme: str = SCHEME_HTTP) -> str:
"""render as a url, embedding auth when present"""
"""render as a url, embedding auth when present
credentials are percent-encoded so reserved chars (/ # ? @ :) in a user or
password produce a valid url; this mirrors the unquote() on the parse side.
"""
if self.has_auth:
return f"{scheme}://{self.user}:{self.password}@{self.host}:{self.port}"
user = quote(str(self.user), safe="")
password = quote(str(self.password), safe="")
return f"{scheme}://{user}:{password}@{self.host}:{self.port}"
return f"{scheme}://{self.host}:{self.port}"
def aiohttp(self) -> Dict[str, str]:
@@ -80,12 +97,15 @@ class Proxy:
def parse(spec: Union[str, Proxy]) -> Proxy:
"""parse a proxy spec into a Proxy
"""parse a colon-delimited proxy spec into a Proxy
accepts an existing Proxy (returned as-is) or a colon-delimited string in
`host:port` or `host:port:user:pass` form. the 4-part form splits on the first
three colons only, so a password may itself contain colons. raises ValueError
on anything else rather than guessing.
three colons only, so a password may itself contain colons. this is spec-string
only - it does not understand url shapes (`scheme://user:pass@host:port`); a
url has 3+ colons and would be misparsed as a 4-part spec rather than rejected.
use `to_proxy()` for url / dict / spec input, or anywhere the shape isn't
guaranteed to be a bare spec string.
"""
if isinstance(spec, Proxy):
return spec
@@ -100,14 +120,7 @@ def parse(spec: Union[str, Proxy]) -> Proxy:
def canonical_key(spec: Union[str, Proxy, Dict[str, str]]) -> str:
"""canonical identity key for any supported proxy shape
accepts a spec string, a Proxy, a url string (`http://user:pass@host:port`,
`socks5://...`), an aiohttp dict (`{"http": url, "https": url}`), or a
camoufox/socks5 dict (`{"server": ..., "username": ..., "password": ...}`).
all forms collapse to the same `host:port:user:pass` (or `host:port` auth-less)
key, so burn/remove/stats recognize one proxy regardless of how it was passed.
"""
"""`to_proxy(spec).key()` - the canonical identity key for any supported proxy shape"""
return to_proxy(spec).key()
@@ -115,8 +128,8 @@ def to_proxy(spec: Union[str, Proxy, Dict[str, str]]) -> Proxy:
"""normalize any supported proxy shape into a Proxy
accepts a spec string, a Proxy, a url string, an aiohttp dict, or a
camoufox/socks5 dict — the same shapes `canonical_key` accepts. used both for
keying and for adding proxies to a pool from any shape.
camoufox/socks5 dict. used both for keying and for adding proxies to a pool
from any shape.
"""
if isinstance(spec, Proxy):
return spec
@@ -133,8 +146,10 @@ def _proxy_from_dict(spec: Dict[str, str]) -> Proxy:
"""normalize an aiohttp / camoufox / socks5 dict into a Proxy"""
if "server" in spec:
proxy = _proxy_from_url(spec["server"])
user = spec.get("username")
password = spec.get("password")
# prefer explicit dict auth; fall back to auth embedded in the server URL so
# it isn't silently dropped, which would key the proxy auth-less
user = spec.get("username") or proxy.user
password = spec.get("password") or proxy.password
if user and password:
return Proxy(proxy.host, proxy.port, user, password)
return Proxy(proxy.host, proxy.port)