diff --git a/README.md b/README.md index 9efaefa..b2b5446 100644 --- a/README.md +++ b/README.md @@ -9,15 +9,15 @@ edits. **Credentials are always injected — never hardcoded.** ## Install ``` -aioproxies @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v0.2.2 +aioproxies @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v0.3.0 # network helpers (current_ip / reset) need the extra: -aioproxies[net] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v0.2.2 +aioproxies[net] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v0.3.0 ``` The core has no dependencies. The `net` extra adds `aiohttp` for `current_ip` / `reset`. -Drop the `@v0.2.2` suffix from the line above to install the latest unpinned. +Drop the `@v0.3.0` suffix from the line above to install the latest unpinned. ## Formatting @@ -144,9 +144,11 @@ pm.is_burned(proxy) # current state (expired timed burns read False) `burn`/`restore`/`is_burned`/`remove` accept **any proxy shape** — a spec string, a `Proxy`, an aiohttp/camoufox/socks5 dict, or a url — all resolve to the same canonical -key (`host:port:user:pass`, or `host:port` auth-less). The password is part of the key, -so two proxies differing only by password are distinct slots. `burn` on a proxy not in -the pool raises `ValueError`. +key (`host:port:user:pass`, or `host:port` auth-less; the port is normalized so +`host:080` and `host:80` are one slot). The password is part of the key, so two proxies +differing only by password are distinct slots. `burn` on a proxy not in the pool raises +`ValueError`; `restore` on a proxy not in the pool logs a warning and no-ops (matching +`remove`'s contract, as of v0.3.0 — previously it silently did nothing with no signal). ### Cooldown @@ -178,9 +180,12 @@ pm.remove(proxy) # drop a slot entirely (any shape) — `replace` resets the rotation index and honors the manager's `shuffle` setting on the incoming list. `remove` differs from `burn`: burn = unusable but still tracked; remove = -gone from the pool. Like the burn family, `add`/`replace` accept **any proxy shape** -(spec/`Proxy`/url/aiohttp dict/camoufox/socks5 dict). `canonical_key(shape)` and -`to_proxy(shape)` are exported if you need the key or a normalized `Proxy` yourself. +gone from the pool — removing a slot that precedes the rotation cursor adjusts the +cursor so `next()` doesn't skip a proxy. Like the burn family, `add`/`replace` accept +**any proxy shape** (spec/`Proxy`/url/aiohttp dict/camoufox/socks5 dict). The +constructor and `from_file` also dedupe by canonical key, same as `add`. +`canonical_key(shape)` and `to_proxy(shape)` are exported if you need the key or a +normalized `Proxy` yourself. ## Network helpers (optional) @@ -204,6 +209,39 @@ await reset("https://provider/reset-url") # rotate upstream ip ## Changelog +### v0.3.0 + +- **Stored-shape change: int unix deadlines.** `burn(proxy, seconds)` and the + cooldown timeout write `int(time.time()) + n` instead of a `float`. Comparisons + (`is_burned`/`stats`/selection) are unaffected — this only tightens what gets + persisted into per-proxy state. +- **Port key normalization:** `key()` strips leading zeros from the port, so + `host:080` and `host:80` are the same canonical slot. Previously a zero-padded + port could break a `get()` → `burn()` pairing (the burn would raise "not in + pool" against the proxy that was just handed out). +- **Constructor / `from_file` dedupe by canonical key**, matching `add()`. A + proxy list with repeated entries (e.g. same host:port:user:pass twice) no + longer inflates the pool or double-weights rotation. +- **`remove()` no longer skews rotation.** Removing a proxy that precedes the + rotation cursor now decrements the cursor, so the next `next()` call doesn't + skip or double-serve a proxy. +- **`restore()` on an unknown proxy now logs a warning and no-ops**, matching + `remove()`'s contract (previously it silently did nothing, with `burn()` + raising for the same precondition and `remove()` warning — `restore()` was the + odd one out). +- **Genuine timed burns under `cooldown>0` now log at WARNING**, not DEBUG. A + real `burn(proxy, seconds)` is now tracked separately from the manager's own + cooldown resting, so it no longer gets buried as routine cooldown noise when + every proxy in the pool happens to also be cooling down. +- **Source truthiness → presence.** The constructor's exactly-one-source check + now uses `is not None` instead of truthiness, so `template=""` (or another + falsy-but-explicitly-provided source) is accepted rather than silently + rejected as "no source given." +- **Escape-aware bare-`{}` template normalization.** A template's bare `{}` is + still filled as the session slot, but an escaped `{{}}` (str.format's own + convention for a literal `{}` in the output) now survives untouched instead of + being corrupted into `{{session}}`. + ### v0.2.1 - **Legible missing-template-field error:** a `template=` placeholder not supplied to diff --git a/pyproject.toml b/pyproject.toml index 4bcaab2..567e267 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "aioproxies" -version = "0.2.2" +version = "0.3.0" description = "proxy parsing, formatting, health, and pool management for aiohttp/aioweb, camoufox, and socks5" requires-python = ">=3.10" dependencies = [] diff --git a/src/aioproxies/__init__.py b/src/aioproxies/__init__.py index 899b7ef..7409262 100644 --- a/src/aioproxies/__init__.py +++ b/src/aioproxies/__init__.py @@ -18,4 +18,4 @@ __all__ = [ "to_proxy", ] -__version__ = "0.2.2" +__version__ = "0.3.0" diff --git a/src/aioproxies/manager.py b/src/aioproxies/manager.py index 9103291..0c44f2b 100644 --- a/src/aioproxies/manager.py +++ b/src/aioproxies/manager.py @@ -1,35 +1,35 @@ """proxy source management: session templates, rotation, static. -`AioProxies` is constructed with exactly one source and hands out `Proxy` -objects. it carries no module-level globals (rotation state is per-instance) and -never exits the process — a missing proxy file raises, it does not `sys.exit`. +`AioProxies` is constructed with exactly one source and hands out `Proxy` objects. +no module-level globals (rotation state is per-instance); a missing proxy file +raises, never `sys.exit`. sources: -- template: a format string with named placeholders. `{session}` is filled with - a fresh session id on each `next()`; any other placeholder (e.g. `{country}`, - `{ttl}`) is filled from keyword args passed to `next(**fields)`. a bare `{}` is - also accepted and treated as the session slot (back-compat with simple templates). -- proxies: a list of specs cycled round-robin +- template: a format string. `{session}` is filled with a fresh id on each + `next()`; other placeholders (`{country}`, `{ttl}`, ...) come from + `next(**fields)`. a bare `{}` is treated as the session slot unless escaped as + `{{}}` (back-compat with simple templates). +- proxies: a list of specs cycled round-robin (deduped by canonical key) - static: one fixed proxy -v0.2.0 adds proxy health to the rotating list source only — burn/timeout, usage -counters, reuse cooldown, and pool management (replace/add/remove). these are -keyed by each proxy's canonical key (`host:port:user:pass`, or `host:port` for -auth-less / IP-authenticated proxies). on template/static sources they are no-ops -that log a warning and return cleanly, so generic caller code can call them -regardless of source. +proxy health (rotating list source only) — burn/timeout, usage counters, reuse +cooldown, pool edits (replace/add/remove) — keyed by each proxy's canonical key +(`host:port:user:pass`, or `host:port` auth-less; port is normalized so `:080` +and `:80` collapse to one key). on template/static sources these are no-ops that +log a warning and return, so generic caller code can call them regardless of +source. -per-proxy state (keyed by canonical key): -- `uses`: pure counter, incremented on every handout. never drives selection; - survives burns (a proxy can read "used 500x and dead"). -- `timeout`: availability state. `None`/`0` = fine; `-1` = dead/permanent (manual - restore only); a future unix ts = timed out until then (lazy, checked against - `time.time()`, no timers). cooldown and timed burns share this field. durations - (seconds) only ever exist as arguments — converted to `now + seconds` and - discarded; the lib never stores a raw duration. +per-proxy state (keyed by canonical key): `uses` is a pure counter, incremented on +every handout, never drives selection, survives burns. `timeout` is the +availability state — `None`/`0` fine, `-1` dead/permanent (manual restore only), a +future unix ts = timed out until then. this core is sync: no timers, lazy expiry +checked only against `time.time()` at call time. cooldown and timed burns share +this field; durations only ever exist as call arguments, converted to `now + +seconds` and discarded — a raw duration is never stored. """ import logging import random +import re import string import time from typing import Dict, List, Optional, Union @@ -39,6 +39,7 @@ from .proxy import Proxy, canonical_key, parse, to_proxy log = logging.getLogger(__name__) _DEAD = -1 +_BARE_SESSION_SLOT = re.compile(r"(? List[Proxy]: + """drop later entries sharing a canonical key, preserving first-seen order""" + seen: Dict[str, Proxy] = {} + for proxy in proxies: + seen.setdefault(proxy.key(), proxy) + return list(seen.values()) + + @staticmethod + def _fresh_state() -> Dict[str, object]: + """a clean per-proxy state entry: no uses, no timeout, not burned""" + return {"uses": 0, "timeout": None, "burned": False} + + @staticmethod + def _normalize_template(template: str) -> str: + """fill a bare `{}` session slot, escape-aware + + a lone `{}` is treated as `{session}` (back-compat with simple templates). + an escaped literal `{{}}` — str.format's own convention for a literal `{}` + in the output — is left untouched so it survives to render as `{}`, not + `{session}`. + """ + return _BARE_SESSION_SLOT.sub("{session}", template) + @classmethod def from_file(cls, path: str, **kwargs) -> "AioProxies": """build a rotating manager from a newline-delimited proxy file @@ -112,15 +137,14 @@ class AioProxies: def next(self, **fields: object) -> Proxy: """return the next proxy from the configured source - for template sources, `{session}` is always filled with a fresh id and any - other named placeholder is filled from `fields` (e.g. next(country="ca", - ttl=30)). fields are ignored by list/static sources. + template sources: `{session}` is always filled with a fresh id, other + placeholders come from `fields` (e.g. next(country="ca", ttl=30)). fields + are ignored by list/static sources. - for list sources, skips proxies whose timeout is active (-1 dead, or a - future ts not yet passed), increments `uses` on handout, and applies the - manager's cooldown. if none are fine but some are merely timed, hands out - the one recovering soonest (with a warning); raises ProxiesExhaustedError - if every proxy is permanently dead. + list sources: skips proxies whose timeout is active (-1 dead, or a future + ts not yet passed), increments `uses` on handout, applies the manager's + cooldown. if none are fine but some are merely timed, hands out the one + recovering soonest (warns); raises ProxiesExhaustedError if all are dead. """ if self._static is not None: return self._static @@ -172,24 +196,19 @@ class AioProxies: state = self._state[proxy.key()] state["uses"] = int(state["uses"]) + 1 if self.cooldown > 0: - state["timeout"] = now + self.cooldown + state["timeout"] = int(now) + self.cooldown return proxy def _has_burned(self) -> bool: - """whether the empty 'fine' tier reflects a genuine burn, not just cooldown + """whether the empty 'fine' tier reflects a genuine burn(), not just cooldown - a dead (-1) proxy is always a real burn. a future-ts timeout is a real timed - burn only when cooldown is off; with cooldown on, future-ts entries are the - manager's own resting and not a pool-health signal. used to pick warning - (real trouble) vs debug (normal cooldown) on the soonest-recovering path. + tracked directly via each state's `burned` flag (set by burn(), cleared by + restore()) so a real burn is never masked by the manager's own cooldown + resting on the same `timeout` field, regardless of whether cooldown is on. + used to pick warning (real trouble) vs debug (normal cooldown) on the + soonest-recovering path. """ - for state in self._state.values(): - timeout = state["timeout"] - if timeout == _DEAD: - return True - if self.cooldown == 0 and timeout not in (None, 0): - return True - return False + return any(state["burned"] for state in self._state.values()) def _soonest_recovering(self) -> Optional[int]: """index of the timed (non-dead) proxy recovering soonest, or None""" @@ -224,16 +243,24 @@ class AioProxies: if seconds is None: self._state[key]["timeout"] = _DEAD else: - self._state[key]["timeout"] = time.time() + seconds + self._state[key]["timeout"] = int(time.time()) + seconds + self._state[key]["burned"] = True def restore(self, proxy: Union[str, Proxy, Dict[str, str]]) -> None: - """clear any burn/timeout on a proxy (back to fine). no-op if already fine""" + """clear any burn/timeout on a proxy (back to fine). no-op if already fine + + matches remove()'s contract for an unknown proxy: logs a warning and returns + rather than silently doing nothing. + """ if not self._is_list_source(): self._warn_non_list("restore()") return key = canonical_key(proxy) - if key in self._state: - self._state[key]["timeout"] = None + if key not in self._state: + log.warning("restore(): proxy not in pool: %s", key) + return + self._state[key]["timeout"] = None + self._state[key]["burned"] = False def is_burned(self, proxy: Union[str, Proxy, Dict[str, str]]) -> bool: """whether a proxy is currently unavailable (lazy expiry of timed burns) @@ -310,7 +337,7 @@ class AioProxies: if keep_state and key in old_state: new_state[key] = old_state[key] else: - new_state[key] = {"uses": 0, "timeout": None} + new_state[key] = self._fresh_state() self._proxies = new_proxies self._state = new_state self._index = 0 @@ -327,13 +354,15 @@ class AioProxies: if key in self._state: continue self._proxies.append(proxy) - self._state[key] = {"uses": 0, "timeout": None} + self._state[key] = self._fresh_state() def remove(self, proxy: Union[str, Proxy, Dict[str, str]]) -> None: """drop a proxy from the pool entirely (by canonical key, any shape) - distinct from burn (burn = unusable but tracked; remove = gone). clamps the - rotation index if needed. no-op + warning if the proxy is not present. + distinct from burn (burn = unusable but tracked; remove = gone). decrements + the rotation index when the removed slot precedes it, so the next() cursor + still lands on the same upcoming proxy instead of skipping one. no-op + + warning if the proxy is not present. """ if not self._is_list_source(): self._warn_non_list("remove()") @@ -342,12 +371,15 @@ class AioProxies: if key not in self._state: log.warning("remove(): proxy not in pool: %s", key) return + removed_idx = next(i for i, p in enumerate(self._proxies) if p.key() == key) self._proxies = [p for p in self._proxies if p.key() != key] del self._state[key] - if self._proxies: - self._index %= len(self._proxies) - else: + if not self._proxies: self._index = 0 + else: + if removed_idx < self._index: + self._index -= 1 + self._index %= len(self._proxies) # name aliases — same class, call it whichever reads best at your call site diff --git a/src/aioproxies/proxy.py b/src/aioproxies/proxy.py index d9d17cf..3616dba 100644 --- a/src/aioproxies/proxy.py +++ b/src/aioproxies/proxy.py @@ -19,6 +19,15 @@ SCHEME_HTTP = "http" SCHEME_SOCKS5 = "socks5" +def normalize_port(port: Union[str, int]) -> str: + """canonical port string: parsed to int, rendered without zero-padding + + keeps host:080 and host:80 as one canonical key. raises ValueError on a + non-integer port rather than silently keying it as-is. + """ + return str(int(port)) + + @dataclass class Proxy: """a single proxy endpoint with optional auth""" @@ -38,14 +47,16 @@ class Proxy: the host is lowercased (hostnames are case-insensitive per DNS, so PROXY.example.com and proxy.example.com are the same host and collapse to one - key); port and credentials are kept verbatim. the password is included in - full (two proxies differing only by password are distinct slots). auth-less - proxies collapse to host:port with no trailing colons. + key); credentials are kept verbatim. the port is normalized (leading zeros + stripped) so host:080 and host:80 collapse to one key. the password is + included in full (two proxies differing only by password are distinct + slots). auth-less proxies collapse to host:port with no trailing colons. """ host = self.host.lower() + port = normalize_port(self.port) if self.has_auth: - return f"{host}:{self.port}:{self.user}:{self.password}" - return f"{host}:{self.port}" + return f"{host}:{port}:{self.user}:{self.password}" + return f"{host}:{port}" def url(self, scheme: str = SCHEME_HTTP) -> str: """render as a url, embedding auth when present