10 Commits
Author SHA1 Message Date
dsql e44c8950ee release: 1.0.0
first stable release. pre-1.0.0 verification complete: all surviving MED regressions and
gaps resolved and independently re-fired, tree audited clean across the suite.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-09 18:53:15 -04:00
dsql 0b6b6b440b fix: cooldown handout never truncates a longer active burn deadline
A forced soonest-recovering handout unconditionally set timeout = now + cooldown,
silently collapsing an explicit burn(proxy, 3600) to the cooldown window. Cooldown now
only EXTENDS availability delay: it leaves a still-running longer timed burn untouched
and never resurrects a permanent _DEAD (-1) burn into a future timestamp, while a fine
proxy and a burn shorter than cooldown are cooled as before.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:26:31 -04:00
dsql 1adc8139b9 docs: fix README health-check snippet's invalid aiohttp kwarg; widen add()/replace() type hints to accept dict shapes
the flagship health-check snippet called session.get(url, proxies=proxy) on a
plain aiohttp session; aiohttp's ClientSession.get/_request has no `proxies=`
kwarg (only singular `proxy:`), so the snippet TypeErrors immediately if
followed verbatim. pm.get()'s dict output is documented (aiohttp()'s own
docstring) as being for aioweb's ExtendedSession(proxies=...), so the snippet
now names that session type instead of a bare aiohttp one.

add()/replace() type hints omitted Dict[str, str] even though both already
call to_proxy() at runtime and accept dict-shaped proxy specs, same as the
burn()/restore()/is_burned()/remove() family. purely annotation, no behavior
change.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 00:15:29 -04:00
dsql a4a0227d5d fix: URL-aware parse() at static/template/net.current_ip call sites; self-clear stale burn flag; guard replace([])
static=/template=/net.current_ip() fed URL-shaped input straight into parse(),
which only understands host:port(:user:pass) specs and mis-splits a url's
colons into a garbage Proxy (e.g. embedded-auth urls) instead of raising or
normalizing. all three now route through to_proxy(), the url-aware normalizer
add()/replace() already use.

the per-state `burned` flag (drives WARNING vs DEBUG on the cooldown
fallthrough log) was only ever cleared by restore(); a timed burn() that
lazily expires (is_burned() correctly reads False again) left `burned` stuck
True, permanently upgrading routine cooldown noise to WARNING. `burned` now
self-clears wherever the existing lazy-expiry check already finds the proxy
available (_next_from_list's fine-tier scan, is_burned()).

replace([]) silently wiped the entire pool with no error, even though the
constructor already rejects an empty proxies=[] list. replace() now raises
the same ValueError for an empty list on list sources.

parse()'s docstring corrected: it is spec-string only and does not understand
url shapes; points callers at to_proxy() for url/dict/spec input.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 00:15:06 -04:00
dsql 9649eb77c7 refactor: derive __version__ from package metadata (single source)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:59:41 -04:00
dsql 1ca3144245 fix: sync __version__ to 0.3.2 (drifted behind pyproject)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:25:17 -04:00
dsql e6663926c0 docs: pin install line to v0.3.2 (pyproject already bumped)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:23:52 -04:00
dsql bac459c5b5 fix: normalize_port raises a clear error on missing/empty port
canonical_key/key()/burn on a portless proxy url used to raise a bare
ValueError from int('') instead of naming the problem, breaking burn()'s
documented not-in-pool contract. normalize_port now raises a legible
'missing port' ValueError; to_proxy() on a portless url still succeeds.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:15:06 -04:00
dsql 84127a93ff docs: compress prose/module docstrings, em-dash->hyphen (de-bloat wave 1)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 00:12:54 -04:00
dsql 147341d38a fix: canonical port key, remove()/restore() drift, escape-aware templates
Port keys no longer drift on zero-padding (host:080 vs host:80 collapsed
one canonical key), so a get() -> burn() pairing on a padded port no
longer raises a false "not in pool". Constructor/from_file now dedupe by
canonical key like add(). remove() decrements the rotation cursor when
the removed slot precedes it, fixing a skip/double-serve. restore() on
an unknown key now warns instead of silently no-op, matching remove()'s
contract. A genuine timed burn() under cooldown>0 is now tracked
separately from routine cooldown resting, so it logs WARNING instead of
being buried as DEBUG. Source validation uses is-not-None instead of
truthiness, so a falsy-but-provided source (e.g. template="") is
accepted. Bare "{}" template normalization is now escape-aware, so an
intentional "{{}}" literal survives instead of being corrupted.

Stored-shape change: burn()/cooldown deadlines now write int(time.time())
instead of float; comparisons are unaffected.

Docstrings compressed (module + next()); no behavior change.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 23:25:52 -04:00
6 changed files with 225 additions and 124 deletions
+69 -10
View File
@@ -9,15 +9,15 @@ edits. **Credentials are always injected — never hardcoded.**
## Install
```
aioproxies @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v0.2.2
aioproxies @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v1.0.0
# network helpers (current_ip / reset) need the extra:
aioproxies[net] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v0.2.2
aioproxies[net] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v1.0.0
```
The core has no dependencies. The `net` extra adds `aiohttp` for `current_ip` /
`reset`.
Drop the `@v0.2.2` suffix from the line above to install the latest unpinned.
Drop the `@v1.0.0` suffix from the line above to install the latest unpinned.
## Formatting
@@ -105,12 +105,14 @@ regardless of source.)
```python
from aioproxies import AioProxies, ProxiesExhaustedError
from aioweb import ExtendedSession
pm = AioProxies(proxies=[...], cooldown=5) # 5s reuse spacing; cooldown defaults to 0 (off)
try:
proxy = pm.get() # next usable proxy, aiohttp dict
resp = await session.get(url, proxies=proxy)
async with ExtendedSession(proxies=proxy) as session:
resp = await session.get(url)
if response_looks_blocked(resp):
pm.burn(proxy, 600) # time out 10 min ... or pm.burn(proxy) for dead
except ProxiesExhaustedError:
@@ -144,9 +146,14 @@ pm.is_burned(proxy) # current state (expired timed burns read False)
`burn`/`restore`/`is_burned`/`remove` accept **any proxy shape** — a spec string, a
`Proxy`, an aiohttp/camoufox/socks5 dict, or a url — all resolve to the same canonical
key (`host:port:user:pass`, or `host:port` auth-less). The password is part of the key,
so two proxies differing only by password are distinct slots. `burn` on a proxy not in
the pool raises `ValueError`.
key (`host:port:user:pass`, or `host:port` auth-less; the port is normalized so
`host:080` and `host:80` are one slot). The password is part of the key, so two proxies
differing only by password are distinct slots. `burn` on a proxy not in the pool raises
`ValueError`; `restore` on a proxy not in the pool logs a warning and no-ops (matching
`remove`'s contract, as of v0.3.0 — previously it silently did nothing with no signal).
A **portless** url (`http://user:pass@host`, no `:port`) parses fine via `to_proxy()`,
but keying it (`canonical_key`/`.key()`/`burn`/`add`/`remove`) raises `ValueError`
naming the missing port — a proxy needs a port to have a canonical identity.
### Cooldown
@@ -178,9 +185,12 @@ pm.remove(proxy) # drop a slot entirely (any shape) —
`replace` resets the rotation index and honors the manager's `shuffle` setting on the
incoming list. `remove` differs from `burn`: burn = unusable but still tracked; remove =
gone from the pool. Like the burn family, `add`/`replace` accept **any proxy shape**
(spec/`Proxy`/url/aiohttp dict/camoufox/socks5 dict). `canonical_key(shape)` and
`to_proxy(shape)` are exported if you need the key or a normalized `Proxy` yourself.
gone from the pool — removing a slot that precedes the rotation cursor adjusts the
cursor so `next()` doesn't skip a proxy. Like the burn family, `add`/`replace` accept
**any proxy shape** (spec/`Proxy`/url/aiohttp dict/camoufox/socks5 dict). The
constructor and `from_file` also dedupe by canonical key, same as `add`.
`canonical_key(shape)` and `to_proxy(shape)` are exported if you need the key or a
normalized `Proxy` yourself.
## Network helpers (optional)
@@ -204,6 +214,55 @@ await reset("https://provider/reset-url") # rotate upstream ip
## Changelog
### v0.3.2
- **Portless proxy url now fails loud and legible.** `canonical_key`/`.key()`
(and therefore `burn`/`add`/`remove`) on a proxy url with no `:port`
(`http://user:pass@host`) used to raise a bare `ValueError("invalid literal
for int() with base 10: ''")` from `normalize_port('')` — an unrelated `int()`
error that broke `burn()`'s documented "raises ValueError naming the key if
not in pool" contract. `normalize_port` now raises `ValueError("missing port:
...")` naming the problem instead. `to_proxy()` on a portless url still
succeeds (construction tolerates a missing port); only keying it raises.
### v0.3.1
- **Docs-only de-bloat pass.** Compressed module/internal docstrings and comments,
replaced mojibake em-dashes with plain hyphens in source. No behavior change.
### v0.3.0
- **Stored-shape change: int unix deadlines.** `burn(proxy, seconds)` and the
cooldown timeout write `int(time.time()) + n` instead of a `float`. Comparisons
(`is_burned`/`stats`/selection) are unaffected — this only tightens what gets
persisted into per-proxy state.
- **Port key normalization:** `key()` strips leading zeros from the port, so
`host:080` and `host:80` are the same canonical slot. Previously a zero-padded
port could break a `get()``burn()` pairing (the burn would raise "not in
pool" against the proxy that was just handed out).
- **Constructor / `from_file` dedupe by canonical key**, matching `add()`. A
proxy list with repeated entries (e.g. same host:port:user:pass twice) no
longer inflates the pool or double-weights rotation.
- **`remove()` no longer skews rotation.** Removing a proxy that precedes the
rotation cursor now decrements the cursor, so the next `next()` call doesn't
skip or double-serve a proxy.
- **`restore()` on an unknown proxy now logs a warning and no-ops**, matching
`remove()`'s contract (previously it silently did nothing, with `burn()`
raising for the same precondition and `remove()` warning — `restore()` was the
odd one out).
- **Genuine timed burns under `cooldown>0` now log at WARNING**, not DEBUG. A
real `burn(proxy, seconds)` is now tracked separately from the manager's own
cooldown resting, so it no longer gets buried as routine cooldown noise when
every proxy in the pool happens to also be cooling down.
- **Source truthiness → presence.** The constructor's exactly-one-source check
now uses `is not None` instead of truthiness, so `template=""` (or another
falsy-but-explicitly-provided source) is accepted rather than silently
rejected as "no source given."
- **Escape-aware bare-`{}` template normalization.** A template's bare `{}` is
still filled as the session slot, but an escaped `{{}}` (str.format's own
convention for a literal `{}` in the output) now survives untouched instead of
being corrupted into `{{session}}`.
### v0.2.1
- **Legible missing-template-field error:** a `template=` placeholder not supplied to
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "aioproxies"
version = "0.2.2"
version = "1.0.0"
description = "proxy parsing, formatting, health, and pool management for aiohttp/aioweb, camoufox, and socks5"
requires-python = ">=3.10"
dependencies = []
+6 -6
View File
@@ -1,9 +1,6 @@
"""aioproxies proxy parsing, formatting, and source management.
"""aioproxies - proxy parsing, formatting, and source management. see README for usage."""
from importlib.metadata import version, PackageNotFoundError
renders proxies for aiohttp/aioweb, camoufox, and socks5; manages session
templates (with caller-supplied fields like country/ttl), rotating lists, or a
static proxy. credentials are always injected, never hardcoded.
"""
from .manager import AioProxies, ProxiesExhaustedError, ProxyManager, aioproxies
from .proxy import Proxy, canonical_key, parse, to_proxy
@@ -18,4 +15,7 @@ __all__ = [
"to_proxy",
]
__version__ = "0.2.2"
try:
__version__ = version("aioproxies")
except PackageNotFoundError:
__version__ = "0.0.0+unknown"
+110 -74
View File
@@ -1,35 +1,23 @@
"""proxy source management: session templates, rotation, static.
`AioProxies` is constructed with exactly one source and hands out `Proxy`
objects. it carries no module-level globals (rotation state is per-instance) and
never exits the process — a missing proxy file raises, it does not `sys.exit`.
`AioProxies` takes exactly one source (template / proxies / static) and hands out
`Proxy` objects via `next()`/`get()`. no module-level globals; a missing proxy file
raises, never `sys.exit`. see README for the full source/method reference.
sources:
- template: a format string with named placeholders. `{session}` is filled with
a fresh session id on each `next()`; any other placeholder (e.g. `{country}`,
`{ttl}`) is filled from keyword args passed to `next(**fields)`. a bare `{}` is
also accepted and treated as the session slot (back-compat with simple templates).
- proxies: a list of specs cycled round-robin
- static: one fixed proxy
v0.2.0 adds proxy health to the rotating list source only — burn/timeout, usage
counters, reuse cooldown, and pool management (replace/add/remove). these are
keyed by each proxy's canonical key (`host:port:user:pass`, or `host:port` for
auth-less / IP-authenticated proxies). on template/static sources they are no-ops
that log a warning and return cleanly, so generic caller code can call them
proxy health (rotating list source only) is keyed by each proxy's canonical key
(`host:port:user:pass`, or `host:port` auth-less; port is normalized so `:080` and
`:80` collapse to one key). on template/static sources health/pool methods are
no-ops that log a warning and return, so generic caller code can call them
regardless of source.
per-proxy state (keyed by canonical key):
- `uses`: pure counter, incremented on every handout. never drives selection;
survives burns (a proxy can read "used 500x and dead").
- `timeout`: availability state. `None`/`0` = fine; `-1` = dead/permanent (manual
restore only); a future unix ts = timed out until then (lazy, checked against
`time.time()`, no timers). cooldown and timed burns share this field. durations
(seconds) only ever exist as arguments — converted to `now + seconds` and
discarded; the lib never stores a raw duration.
per-proxy `timeout` (the availability field) is shared by cooldown and timed
burns: `None`/`0` fine, `-1` dead/permanent, a future unix ts = timed out until
then; durations only ever exist as call arguments, converted to `now + seconds`
and discarded, never stored raw.
"""
import logging
import random
import re
import string
import time
from typing import Dict, List, Optional, Union
@@ -39,6 +27,7 @@ from .proxy import Proxy, canonical_key, parse, to_proxy
log = logging.getLogger(__name__)
_DEAD = -1
_BARE_SESSION_SLOT = re.compile(r"(?<!\{)\{\}(?!\})")
class ProxiesExhaustedError(Exception):
@@ -60,27 +49,45 @@ class AioProxies:
):
if proxies is not None and len(proxies) == 0:
raise ValueError("proxies list is empty; provide at least one proxy")
sources = [s for s in (template, proxies, static) if s]
sources = [s for s in (template, proxies, static) if s is not None]
if len(sources) != 1:
raise ValueError("provide exactly one of: template, proxies, static")
self.template = template.replace("{}", "{session}") if template else None
self.template = self._normalize_template(template) if template is not None else None
self.session_len = session_len
self.cooldown = cooldown
self._shuffle = shuffle
self._static = parse(static) if static else None
self._proxies = [parse(p) for p in proxies] if proxies else []
self._static = to_proxy(static) if static is not None else None
self._proxies = self._dedupe([parse(p) for p in proxies]) if proxies is not None else []
if self._proxies and shuffle:
random.shuffle(self._proxies)
self._state: Dict[str, Dict[str, object]] = {}
for proxy in self._proxies:
self._state.setdefault(proxy.key(), {"uses": 0, "timeout": None})
self._state.setdefault(proxy.key(), self._fresh_state())
self._index = 0
@staticmethod
def _dedupe(proxies: List[Proxy]) -> List[Proxy]:
"""drop later entries sharing a canonical key, preserving first-seen order"""
seen: Dict[str, Proxy] = {}
for proxy in proxies:
seen.setdefault(proxy.key(), proxy)
return list(seen.values())
@staticmethod
def _fresh_state() -> Dict[str, object]:
"""a clean per-proxy state entry: no uses, no timeout, not burned"""
return {"uses": 0, "timeout": None, "burned": False}
@staticmethod
def _normalize_template(template: str) -> str:
"""fill a bare `{}` session slot with `{session}`; leave escaped `{{}}` untouched"""
return _BARE_SESSION_SLOT.sub("{session}", template)
@classmethod
def from_file(cls, path: str, **kwargs) -> "AioProxies":
"""build a rotating manager from a newline-delimited proxy file
raises FileNotFoundError if the path is missing never exits the process.
raises FileNotFoundError if the path is missing - never exits the process.
"""
with open(path, "r", encoding="utf-8") as fh:
lines = [ln.strip() for ln in fh if ln.strip()]
@@ -109,25 +116,35 @@ class AioProxies:
return False
return now >= timeout
@staticmethod
def _clear_stale_burn(state: Dict[str, object]) -> None:
"""self-clear a lazily-expired timed burn's `burned` flag once found available
mirrors the lazy expiry already applied to `timeout` itself: a timed burn()
that has simply run out its clock should stop upgrading routine cooldown
fallthrough logs to warning, without requiring an explicit restore().
"""
if state["timeout"] not in (None, 0, _DEAD):
state["burned"] = False
def next(self, **fields: object) -> Proxy:
"""return the next proxy from the configured source
for template sources, `{session}` is always filled with a fresh id and any
other named placeholder is filled from `fields` (e.g. next(country="ca",
ttl=30)). fields are ignored by list/static sources.
template sources: `{session}` is always filled with a fresh id, other
placeholders come from `fields` (e.g. next(country="ca", ttl=30)). fields
are ignored by list/static sources.
for list sources, skips proxies whose timeout is active (-1 dead, or a
future ts not yet passed), increments `uses` on handout, and applies the
manager's cooldown. if none are fine but some are merely timed, hands out
the one recovering soonest (with a warning); raises ProxiesExhaustedError
if every proxy is permanently dead.
list sources: skips proxies whose timeout is active (-1 dead, or a future
ts not yet passed), increments `uses` on handout, applies the manager's
cooldown. if none are fine but some are merely timed, hands out the one
recovering soonest (warns); raises ProxiesExhaustedError if all are dead.
"""
if self._static is not None:
return self._static
if self.template is not None:
if "session" in fields:
# `session` is auto-filled with a fresh id; a caller-supplied one would
# collide in str.format with an opaque TypeError reject it clearly
# collide in str.format with an opaque TypeError - reject it clearly
raise ValueError("'session' is filled automatically; do not pass it to next()")
try:
filled = self.template.format(session=self.session_id(), **fields)
@@ -137,9 +154,9 @@ class AioProxies:
) from exc
except ValueError as exc:
# a malformed template (e.g. an unmatched '{') makes str.format raise a
# bare ValueError; re-raise naming the cause so it isn't cryptic
# bare ValueError - re-raise naming the cause so it isn't cryptic
raise ValueError(f"malformed proxy template {self.template!r}: {exc}") from exc
return parse(filled)
return to_proxy(filled)
return self._next_from_list()
def _next_from_list(self) -> Proxy:
@@ -152,9 +169,10 @@ class AioProxies:
fine: List[int] = []
for offset in range(count):
idx = (self._index + offset) % count
timeout = self._state[self._proxies[idx].key()]["timeout"]
if self._available(timeout, now):
state = self._state[self._proxies[idx].key()]
if self._available(state["timeout"], now):
fine.append(idx)
self._clear_stale_burn(state)
if fine:
chosen = fine[0]
@@ -172,24 +190,22 @@ class AioProxies:
state = self._state[proxy.key()]
state["uses"] = int(state["uses"]) + 1
if self.cooldown > 0:
state["timeout"] = now + self.cooldown
# cooldown only ever EXTENDS availability delay - never shorten a longer
# active timeout (a still-running timed burn) down to now+cooldown. _DEAD
# (-1) is a permanent burn and must not be resurrected into a future ts.
cooled = int(now) + self.cooldown
current = state["timeout"]
if current == _DEAD:
pass
elif isinstance(current, (int, float)) and current > cooled:
pass
else:
state["timeout"] = cooled
return proxy
def _has_burned(self) -> bool:
"""whether the empty 'fine' tier reflects a genuine burn, not just cooldown
a dead (-1) proxy is always a real burn. a future-ts timeout is a real timed
burn only when cooldown is off; with cooldown on, future-ts entries are the
manager's own resting and not a pool-health signal. used to pick warning
(real trouble) vs debug (normal cooldown) on the soonest-recovering path.
"""
for state in self._state.values():
timeout = state["timeout"]
if timeout == _DEAD:
return True
if self.cooldown == 0 and timeout not in (None, 0):
return True
return False
"""whether any proxy carries a genuine burn (vs just cooldown resting), for log level choice"""
return any(state["burned"] for state in self._state.values())
def _soonest_recovering(self) -> Optional[int]:
"""index of the timed (non-dead) proxy recovering soonest, or None"""
@@ -224,16 +240,24 @@ class AioProxies:
if seconds is None:
self._state[key]["timeout"] = _DEAD
else:
self._state[key]["timeout"] = time.time() + seconds
self._state[key]["timeout"] = int(time.time()) + seconds
self._state[key]["burned"] = True
def restore(self, proxy: Union[str, Proxy, Dict[str, str]]) -> None:
"""clear any burn/timeout on a proxy (back to fine). no-op if already fine"""
"""clear any burn/timeout on a proxy (back to fine). no-op if already fine
matches remove()'s contract for an unknown proxy: logs a warning and returns
rather than silently doing nothing.
"""
if not self._is_list_source():
self._warn_non_list("restore()")
return
key = canonical_key(proxy)
if key in self._state:
if key not in self._state:
log.warning("restore(): proxy not in pool: %s", key)
return
self._state[key]["timeout"] = None
self._state[key]["burned"] = False
def is_burned(self, proxy: Union[str, Proxy, Dict[str, str]]) -> bool:
"""whether a proxy is currently unavailable (lazy expiry of timed burns)
@@ -246,8 +270,11 @@ class AioProxies:
key = canonical_key(proxy)
if key not in self._state:
return False
timeout = self._state[key]["timeout"]
return not self._available(timeout, time.time())
state = self._state[key]
available = self._available(state["timeout"], time.time())
if available:
self._clear_stale_burn(state)
return not available
def stats(self) -> List[Dict[str, object]]:
"""per-proxy usage + state snapshot (list sources only, else empty)
@@ -285,17 +312,21 @@ class AioProxies:
for state in self._state.values():
state["uses"] = 0
def replace(self, proxies: List[Union[str, Proxy]], *, keep_state: bool = False) -> None:
def replace(self, proxies: List[Union[str, Proxy, Dict[str, str]]], *, keep_state: bool = False) -> None:
"""swap the entire proxy list
keep_state=False (default) wipes all per-proxy state (fresh batch).
keep_state=True preserves uses/timeout for proxies whose canonical key
survives the swap; new proxies start clean, dropped ones are forgotten.
resets the rotation index and honors the manager's shuffle setting.
resets the rotation index and honors the manager's shuffle setting. raises
ValueError on an empty list, matching the constructor's guard - use remove()
to drop proxies one at a time if you actually want a smaller pool.
"""
if not self._is_list_source():
self._warn_non_list("replace()")
return
if len(proxies) == 0:
raise ValueError("proxies list is empty; provide at least one proxy")
old_state = self._state
incoming = [to_proxy(p) for p in proxies]
if self._shuffle:
@@ -310,12 +341,12 @@ class AioProxies:
if keep_state and key in old_state:
new_state[key] = old_state[key]
else:
new_state[key] = {"uses": 0, "timeout": None}
new_state[key] = self._fresh_state()
self._proxies = new_proxies
self._state = new_state
self._index = 0
def add(self, proxies: Union[str, Proxy, List[Union[str, Proxy]]]) -> None:
def add(self, proxies: Union[str, Proxy, Dict[str, str], List[Union[str, Proxy, Dict[str, str]]]]) -> None:
"""append proxies to the pool, keeping existing state; skip duplicate keys"""
if not self._is_list_source():
self._warn_non_list("add()")
@@ -327,13 +358,15 @@ class AioProxies:
if key in self._state:
continue
self._proxies.append(proxy)
self._state[key] = {"uses": 0, "timeout": None}
self._state[key] = self._fresh_state()
def remove(self, proxy: Union[str, Proxy, Dict[str, str]]) -> None:
"""drop a proxy from the pool entirely (by canonical key, any shape)
distinct from burn (burn = unusable but tracked; remove = gone). clamps the
rotation index if needed. no-op + warning if the proxy is not present.
distinct from burn (burn = unusable but tracked; remove = gone). decrements
the rotation index when the removed slot precedes it, so the next() cursor
still lands on the same upcoming proxy instead of skipping one. no-op +
warning if the proxy is not present.
"""
if not self._is_list_source():
self._warn_non_list("remove()")
@@ -342,14 +375,17 @@ class AioProxies:
if key not in self._state:
log.warning("remove(): proxy not in pool: %s", key)
return
removed_idx = next(i for i, p in enumerate(self._proxies) if p.key() == key)
self._proxies = [p for p in self._proxies if p.key() != key]
del self._state[key]
if self._proxies:
self._index %= len(self._proxies)
else:
if not self._proxies:
self._index = 0
else:
if removed_idx < self._index:
self._index -= 1
self._index %= len(self._proxies)
# name aliases same class, call it whichever reads best at your call site
# name aliases - same class, call it whichever reads best at your call site
ProxyManager = AioProxies
aioproxies = AioProxies
+2 -2
View File
@@ -7,7 +7,7 @@ calling a function without it raises a clear error.
import logging
from typing import Optional, Union
from .proxy import Proxy, parse
from .proxy import Proxy, to_proxy
log = logging.getLogger(__name__)
@@ -32,7 +32,7 @@ async def current_ip(
"""
if not _HAVE_AIOHTTP:
raise RuntimeError(_MISSING)
p = parse(proxy)
p = to_proxy(proxy)
t = aiohttp.ClientTimeout(total=timeout)
try:
async with aiohttp.ClientSession(timeout=t) as session:
+36 -30
View File
@@ -1,15 +1,12 @@
"""proxy parsing and formatting (pure, no network IO).
a `Proxy` holds host/port/optional-auth and renders the shapes different clients
want: an aiohttp/aioweb proxies dict, a camoufox proxy dict, a socks5 dict, or a
plain url. `parse` accepts the common "host:port" and "host:port:user:pass" string
forms (the user field may itself contain commas, e.g. session-param proxies; the
password field may itself contain colons). auth-less (IP-authenticated) proxies
with no creds are first-class — every render shape handles them via `has_auth`.
`Proxy` holds host/port/optional-auth and renders aiohttp/aioweb, camoufox, socks5,
and url shapes; auth-less (IP-authenticated) proxies are first-class throughout.
`parse` accepts "host:port" and "host:port:user:pass" - the user field may itself
contain commas and the password may itself contain colons.
`key()` produces a stable canonical identity (`host:port:user:pass`, or
`host:port` when auth-less) so burn/remove/stats recognize the same proxy from any
input shape. `canonical_key()` extends that to dict/url forms.
`key()` / `canonical_key()` produce a stable canonical identity so burn/remove/stats
recognize the same proxy from any input shape (spec string, dict, or url).
"""
from dataclasses import dataclass
from typing import Dict, Optional, Union
@@ -19,6 +16,18 @@ SCHEME_HTTP = "http"
SCHEME_SOCKS5 = "socks5"
def normalize_port(port: Union[str, int]) -> str:
"""canonical port string: parsed to int, rendered without zero-padding
keeps host:080 and host:80 as one canonical key. raises ValueError naming the
missing port if empty/None (e.g. a portless proxy url), or on a non-integer
port, rather than silently keying it as-is.
"""
if port is None or port == "":
raise ValueError("missing port: proxy url or spec has no port")
return str(int(port))
@dataclass
class Proxy:
"""a single proxy endpoint with optional auth"""
@@ -38,14 +47,16 @@ class Proxy:
the host is lowercased (hostnames are case-insensitive per DNS, so
PROXY.example.com and proxy.example.com are the same host and collapse to one
key); port and credentials are kept verbatim. the password is included in
full (two proxies differing only by password are distinct slots). auth-less
proxies collapse to host:port with no trailing colons.
key); credentials are kept verbatim. the port is normalized (leading zeros
stripped) so host:080 and host:80 collapse to one key. the password is
included in full (two proxies differing only by password are distinct
slots). auth-less proxies collapse to host:port with no trailing colons.
"""
host = self.host.lower()
port = normalize_port(self.port)
if self.has_auth:
return f"{host}:{self.port}:{self.user}:{self.password}"
return f"{host}:{self.port}"
return f"{host}:{port}:{self.user}:{self.password}"
return f"{host}:{port}"
def url(self, scheme: str = SCHEME_HTTP) -> str:
"""render as a url, embedding auth when present
@@ -86,12 +97,15 @@ class Proxy:
def parse(spec: Union[str, Proxy]) -> Proxy:
"""parse a proxy spec into a Proxy
"""parse a colon-delimited proxy spec into a Proxy
accepts an existing Proxy (returned as-is) or a colon-delimited string in
`host:port` or `host:port:user:pass` form. the 4-part form splits on the first
three colons only, so a password may itself contain colons. raises ValueError
on anything else rather than guessing.
three colons only, so a password may itself contain colons. this is spec-string
only - it does not understand url shapes (`scheme://user:pass@host:port`); a
url has 3+ colons and would be misparsed as a 4-part spec rather than rejected.
use `to_proxy()` for url / dict / spec input, or anywhere the shape isn't
guaranteed to be a bare spec string.
"""
if isinstance(spec, Proxy):
return spec
@@ -106,14 +120,7 @@ def parse(spec: Union[str, Proxy]) -> Proxy:
def canonical_key(spec: Union[str, Proxy, Dict[str, str]]) -> str:
"""canonical identity key for any supported proxy shape
accepts a spec string, a Proxy, a url string (`http://user:pass@host:port`,
`socks5://...`), an aiohttp dict (`{"http": url, "https": url}`), or a
camoufox/socks5 dict (`{"server": ..., "username": ..., "password": ...}`).
all forms collapse to the same `host:port:user:pass` (or `host:port` auth-less)
key, so burn/remove/stats recognize one proxy regardless of how it was passed.
"""
"""`to_proxy(spec).key()` - the canonical identity key for any supported proxy shape"""
return to_proxy(spec).key()
@@ -121,8 +128,8 @@ def to_proxy(spec: Union[str, Proxy, Dict[str, str]]) -> Proxy:
"""normalize any supported proxy shape into a Proxy
accepts a spec string, a Proxy, a url string, an aiohttp dict, or a
camoufox/socks5 dict — the same shapes `canonical_key` accepts. used both for
keying and for adding proxies to a pool from any shape.
camoufox/socks5 dict. used both for keying and for adding proxies to a pool
from any shape.
"""
if isinstance(spec, Proxy):
return spec
@@ -139,9 +146,8 @@ def _proxy_from_dict(spec: Dict[str, str]) -> Proxy:
"""normalize an aiohttp / camoufox / socks5 dict into a Proxy"""
if "server" in spec:
proxy = _proxy_from_url(spec["server"])
# prefer explicit dict auth; otherwise fall back to auth embedded in the server
# URL (http://user:pass@host:port) so it isn't silently dropped which would
# key the proxy auth-less and collide with a genuinely auth-less one
# prefer explicit dict auth; fall back to auth embedded in the server URL so
# it isn't silently dropped, which would key the proxy auth-less
user = spec.get("username") or proxy.user
password = spec.get("password") or proxy.password
if user and password: