Compare commits
12
Commits
v0.2.2
...
4d5caac8d8
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4d5caac8d8 | ||
|
|
5699aa9dde | ||
|
|
e44c8950ee | ||
|
|
0b6b6b440b | ||
|
|
1adc8139b9 | ||
|
|
a4a0227d5d | ||
|
|
9649eb77c7 | ||
|
|
1ca3144245 | ||
|
|
e6663926c0 | ||
|
|
bac459c5b5 | ||
|
|
84127a93ff | ||
|
|
147341d38a |
@@ -9,15 +9,15 @@ edits. **Credentials are always injected — never hardcoded.**
|
|||||||
## Install
|
## Install
|
||||||
|
|
||||||
```
|
```
|
||||||
aioproxies @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v0.2.2
|
aioproxies @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v1.0.0
|
||||||
# network helpers (current_ip / reset) need the extra:
|
# network helpers (current_ip / reset) need the extra:
|
||||||
aioproxies[net] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v0.2.2
|
aioproxies[net] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioproxies.git@v1.0.0
|
||||||
```
|
```
|
||||||
|
|
||||||
The core has no dependencies. The `net` extra adds `aiohttp` for `current_ip` /
|
The core has no dependencies. The `net` extra adds `aiohttp` for `current_ip` /
|
||||||
`reset`.
|
`reset`.
|
||||||
|
|
||||||
Drop the `@v0.2.2` suffix from the line above to install the latest unpinned.
|
Drop the `@v1.0.0` suffix from the line above to install the latest unpinned.
|
||||||
|
|
||||||
## Formatting
|
## Formatting
|
||||||
|
|
||||||
@@ -105,12 +105,14 @@ regardless of source.)
|
|||||||
|
|
||||||
```python
|
```python
|
||||||
from aioproxies import AioProxies, ProxiesExhaustedError
|
from aioproxies import AioProxies, ProxiesExhaustedError
|
||||||
|
from aioweb import ExtendedSession
|
||||||
|
|
||||||
pm = AioProxies(proxies=[...], cooldown=5) # 5s reuse spacing; cooldown defaults to 0 (off)
|
pm = AioProxies(proxies=[...], cooldown=5) # 5s reuse spacing; cooldown defaults to 0 (off)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
proxy = pm.get() # next usable proxy, aiohttp dict
|
proxy = pm.get() # next usable proxy, aiohttp dict
|
||||||
resp = await session.get(url, proxies=proxy)
|
async with ExtendedSession(proxies=proxy) as session:
|
||||||
|
resp = await session.get(url)
|
||||||
if response_looks_blocked(resp):
|
if response_looks_blocked(resp):
|
||||||
pm.burn(proxy, 600) # time out 10 min ... or pm.burn(proxy) for dead
|
pm.burn(proxy, 600) # time out 10 min ... or pm.burn(proxy) for dead
|
||||||
except ProxiesExhaustedError:
|
except ProxiesExhaustedError:
|
||||||
@@ -144,9 +146,14 @@ pm.is_burned(proxy) # current state (expired timed burns read False)
|
|||||||
|
|
||||||
`burn`/`restore`/`is_burned`/`remove` accept **any proxy shape** — a spec string, a
|
`burn`/`restore`/`is_burned`/`remove` accept **any proxy shape** — a spec string, a
|
||||||
`Proxy`, an aiohttp/camoufox/socks5 dict, or a url — all resolve to the same canonical
|
`Proxy`, an aiohttp/camoufox/socks5 dict, or a url — all resolve to the same canonical
|
||||||
key (`host:port:user:pass`, or `host:port` auth-less). The password is part of the key,
|
key (`host:port:user:pass`, or `host:port` auth-less; the port is normalized so
|
||||||
so two proxies differing only by password are distinct slots. `burn` on a proxy not in
|
`host:080` and `host:80` are one slot). The password is part of the key, so two proxies
|
||||||
the pool raises `ValueError`.
|
differing only by password are distinct slots. `burn` on a proxy not in the pool raises
|
||||||
|
`ValueError`; `restore` on a proxy not in the pool logs a warning and no-ops (matching
|
||||||
|
`remove`'s contract, as of v0.3.0 — previously it silently did nothing with no signal).
|
||||||
|
A **portless** url (`http://user:pass@host`, no `:port`) parses fine via `to_proxy()`,
|
||||||
|
but keying it (`canonical_key`/`.key()`/`burn`/`add`/`remove`) raises `ValueError`
|
||||||
|
naming the missing port — a proxy needs a port to have a canonical identity.
|
||||||
|
|
||||||
### Cooldown
|
### Cooldown
|
||||||
|
|
||||||
@@ -178,9 +185,12 @@ pm.remove(proxy) # drop a slot entirely (any shape) —
|
|||||||
|
|
||||||
`replace` resets the rotation index and honors the manager's `shuffle` setting on the
|
`replace` resets the rotation index and honors the manager's `shuffle` setting on the
|
||||||
incoming list. `remove` differs from `burn`: burn = unusable but still tracked; remove =
|
incoming list. `remove` differs from `burn`: burn = unusable but still tracked; remove =
|
||||||
gone from the pool. Like the burn family, `add`/`replace` accept **any proxy shape**
|
gone from the pool — removing a slot that precedes the rotation cursor adjusts the
|
||||||
(spec/`Proxy`/url/aiohttp dict/camoufox/socks5 dict). `canonical_key(shape)` and
|
cursor so `next()` doesn't skip a proxy. Like the burn family, `add`/`replace` accept
|
||||||
`to_proxy(shape)` are exported if you need the key or a normalized `Proxy` yourself.
|
**any proxy shape** (spec/`Proxy`/url/aiohttp dict/camoufox/socks5 dict). The
|
||||||
|
constructor and `from_file` also dedupe by canonical key, same as `add`.
|
||||||
|
`canonical_key(shape)` and `to_proxy(shape)` are exported if you need the key or a
|
||||||
|
normalized `Proxy` yourself.
|
||||||
|
|
||||||
## Network helpers (optional)
|
## Network helpers (optional)
|
||||||
|
|
||||||
@@ -204,6 +214,55 @@ await reset("https://provider/reset-url") # rotate upstream ip
|
|||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
### v0.3.2
|
||||||
|
|
||||||
|
- **Portless proxy url now fails loud and legible.** `canonical_key`/`.key()`
|
||||||
|
(and therefore `burn`/`add`/`remove`) on a proxy url with no `:port`
|
||||||
|
(`http://user:pass@host`) used to raise a bare `ValueError("invalid literal
|
||||||
|
for int() with base 10: ''")` from `normalize_port('')` — an unrelated `int()`
|
||||||
|
error that broke `burn()`'s documented "raises ValueError naming the key if
|
||||||
|
not in pool" contract. `normalize_port` now raises `ValueError("missing port:
|
||||||
|
...")` naming the problem instead. `to_proxy()` on a portless url still
|
||||||
|
succeeds (construction tolerates a missing port); only keying it raises.
|
||||||
|
|
||||||
|
### v0.3.1
|
||||||
|
|
||||||
|
- **Docs-only de-bloat pass.** Compressed module/internal docstrings and comments,
|
||||||
|
replaced mojibake em-dashes with plain hyphens in source. No behavior change.
|
||||||
|
|
||||||
|
### v0.3.0
|
||||||
|
|
||||||
|
- **Stored-shape change: int unix deadlines.** `burn(proxy, seconds)` and the
|
||||||
|
cooldown timeout write `int(time.time()) + n` instead of a `float`. Comparisons
|
||||||
|
(`is_burned`/`stats`/selection) are unaffected — this only tightens what gets
|
||||||
|
persisted into per-proxy state.
|
||||||
|
- **Port key normalization:** `key()` strips leading zeros from the port, so
|
||||||
|
`host:080` and `host:80` are the same canonical slot. Previously a zero-padded
|
||||||
|
port could break a `get()` → `burn()` pairing (the burn would raise "not in
|
||||||
|
pool" against the proxy that was just handed out).
|
||||||
|
- **Constructor / `from_file` dedupe by canonical key**, matching `add()`. A
|
||||||
|
proxy list with repeated entries (e.g. same host:port:user:pass twice) no
|
||||||
|
longer inflates the pool or double-weights rotation.
|
||||||
|
- **`remove()` no longer skews rotation.** Removing a proxy that precedes the
|
||||||
|
rotation cursor now decrements the cursor, so the next `next()` call doesn't
|
||||||
|
skip or double-serve a proxy.
|
||||||
|
- **`restore()` on an unknown proxy now logs a warning and no-ops**, matching
|
||||||
|
`remove()`'s contract (previously it silently did nothing, with `burn()`
|
||||||
|
raising for the same precondition and `remove()` warning — `restore()` was the
|
||||||
|
odd one out).
|
||||||
|
- **Genuine timed burns under `cooldown>0` now log at WARNING**, not DEBUG. A
|
||||||
|
real `burn(proxy, seconds)` is now tracked separately from the manager's own
|
||||||
|
cooldown resting, so it no longer gets buried as routine cooldown noise when
|
||||||
|
every proxy in the pool happens to also be cooling down.
|
||||||
|
- **Source truthiness → presence.** The constructor's exactly-one-source check
|
||||||
|
now uses `is not None` instead of truthiness, so `template=""` (or another
|
||||||
|
falsy-but-explicitly-provided source) is accepted rather than silently
|
||||||
|
rejected as "no source given."
|
||||||
|
- **Escape-aware bare-`{}` template normalization.** A template's bare `{}` is
|
||||||
|
still filled as the session slot, but an escaped `{{}}` (str.format's own
|
||||||
|
convention for a literal `{}` in the output) now survives untouched instead of
|
||||||
|
being corrupted into `{{session}}`.
|
||||||
|
|
||||||
### v0.2.1
|
### v0.2.1
|
||||||
|
|
||||||
- **Legible missing-template-field error:** a `template=` placeholder not supplied to
|
- **Legible missing-template-field error:** a `template=` placeholder not supplied to
|
||||||
|
|||||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "aioproxies"
|
name = "aioproxies"
|
||||||
version = "0.2.2"
|
version = "1.1.0"
|
||||||
description = "proxy parsing, formatting, health, and pool management for aiohttp/aioweb, camoufox, and socks5"
|
description = "proxy parsing, formatting, health, and pool management for aiohttp/aioweb, camoufox, and socks5"
|
||||||
requires-python = ">=3.10"
|
requires-python = ">=3.10"
|
||||||
dependencies = []
|
dependencies = []
|
||||||
|
|||||||
@@ -1,9 +1,6 @@
|
|||||||
"""aioproxies — proxy parsing, formatting, and source management.
|
"""aioproxies - proxy parsing, formatting, and source management. see README for usage."""
|
||||||
|
from importlib.metadata import version, PackageNotFoundError
|
||||||
|
|
||||||
renders proxies for aiohttp/aioweb, camoufox, and socks5; manages session
|
|
||||||
templates (with caller-supplied fields like country/ttl), rotating lists, or a
|
|
||||||
static proxy. credentials are always injected, never hardcoded.
|
|
||||||
"""
|
|
||||||
from .manager import AioProxies, ProxiesExhaustedError, ProxyManager, aioproxies
|
from .manager import AioProxies, ProxiesExhaustedError, ProxyManager, aioproxies
|
||||||
from .proxy import Proxy, canonical_key, parse, to_proxy
|
from .proxy import Proxy, canonical_key, parse, to_proxy
|
||||||
|
|
||||||
@@ -18,4 +15,7 @@ __all__ = [
|
|||||||
"to_proxy",
|
"to_proxy",
|
||||||
]
|
]
|
||||||
|
|
||||||
__version__ = "0.2.2"
|
try:
|
||||||
|
__version__ = version("aioproxies")
|
||||||
|
except PackageNotFoundError:
|
||||||
|
__version__ = "0.0.0+unknown"
|
||||||
|
|||||||
+110
-74
@@ -1,35 +1,23 @@
|
|||||||
"""proxy source management: session templates, rotation, static.
|
"""proxy source management: session templates, rotation, static.
|
||||||
|
|
||||||
`AioProxies` is constructed with exactly one source and hands out `Proxy`
|
`AioProxies` takes exactly one source (template / proxies / static) and hands out
|
||||||
objects. it carries no module-level globals (rotation state is per-instance) and
|
`Proxy` objects via `next()`/`get()`. no module-level globals; a missing proxy file
|
||||||
never exits the process — a missing proxy file raises, it does not `sys.exit`.
|
raises, never `sys.exit`. see README for the full source/method reference.
|
||||||
|
|
||||||
sources:
|
proxy health (rotating list source only) is keyed by each proxy's canonical key
|
||||||
- template: a format string with named placeholders. `{session}` is filled with
|
(`host:port:user:pass`, or `host:port` auth-less; port is normalized so `:080` and
|
||||||
a fresh session id on each `next()`; any other placeholder (e.g. `{country}`,
|
`:80` collapse to one key). on template/static sources health/pool methods are
|
||||||
`{ttl}`) is filled from keyword args passed to `next(**fields)`. a bare `{}` is
|
no-ops that log a warning and return, so generic caller code can call them
|
||||||
also accepted and treated as the session slot (back-compat with simple templates).
|
|
||||||
- proxies: a list of specs cycled round-robin
|
|
||||||
- static: one fixed proxy
|
|
||||||
|
|
||||||
v0.2.0 adds proxy health to the rotating list source only — burn/timeout, usage
|
|
||||||
counters, reuse cooldown, and pool management (replace/add/remove). these are
|
|
||||||
keyed by each proxy's canonical key (`host:port:user:pass`, or `host:port` for
|
|
||||||
auth-less / IP-authenticated proxies). on template/static sources they are no-ops
|
|
||||||
that log a warning and return cleanly, so generic caller code can call them
|
|
||||||
regardless of source.
|
regardless of source.
|
||||||
|
|
||||||
per-proxy state (keyed by canonical key):
|
per-proxy `timeout` (the availability field) is shared by cooldown and timed
|
||||||
- `uses`: pure counter, incremented on every handout. never drives selection;
|
burns: `None`/`0` fine, `-1` dead/permanent, a future unix ts = timed out until
|
||||||
survives burns (a proxy can read "used 500x and dead").
|
then; durations only ever exist as call arguments, converted to `now + seconds`
|
||||||
- `timeout`: availability state. `None`/`0` = fine; `-1` = dead/permanent (manual
|
and discarded, never stored raw.
|
||||||
restore only); a future unix ts = timed out until then (lazy, checked against
|
|
||||||
`time.time()`, no timers). cooldown and timed burns share this field. durations
|
|
||||||
(seconds) only ever exist as arguments — converted to `now + seconds` and
|
|
||||||
discarded; the lib never stores a raw duration.
|
|
||||||
"""
|
"""
|
||||||
import logging
|
import logging
|
||||||
import random
|
import random
|
||||||
|
import re
|
||||||
import string
|
import string
|
||||||
import time
|
import time
|
||||||
from typing import Dict, List, Optional, Union
|
from typing import Dict, List, Optional, Union
|
||||||
@@ -39,6 +27,7 @@ from .proxy import Proxy, canonical_key, parse, to_proxy
|
|||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
_DEAD = -1
|
_DEAD = -1
|
||||||
|
_BARE_SESSION_SLOT = re.compile(r"(?<!\{)\{\}(?!\})")
|
||||||
|
|
||||||
|
|
||||||
class ProxiesExhaustedError(Exception):
|
class ProxiesExhaustedError(Exception):
|
||||||
@@ -60,27 +49,45 @@ class AioProxies:
|
|||||||
):
|
):
|
||||||
if proxies is not None and len(proxies) == 0:
|
if proxies is not None and len(proxies) == 0:
|
||||||
raise ValueError("proxies list is empty; provide at least one proxy")
|
raise ValueError("proxies list is empty; provide at least one proxy")
|
||||||
sources = [s for s in (template, proxies, static) if s]
|
sources = [s for s in (template, proxies, static) if s is not None]
|
||||||
if len(sources) != 1:
|
if len(sources) != 1:
|
||||||
raise ValueError("provide exactly one of: template, proxies, static")
|
raise ValueError("provide exactly one of: template, proxies, static")
|
||||||
self.template = template.replace("{}", "{session}") if template else None
|
self.template = self._normalize_template(template) if template is not None else None
|
||||||
self.session_len = session_len
|
self.session_len = session_len
|
||||||
self.cooldown = cooldown
|
self.cooldown = cooldown
|
||||||
self._shuffle = shuffle
|
self._shuffle = shuffle
|
||||||
self._static = parse(static) if static else None
|
self._static = to_proxy(static) if static is not None else None
|
||||||
self._proxies = [parse(p) for p in proxies] if proxies else []
|
self._proxies = self._dedupe([parse(p) for p in proxies]) if proxies is not None else []
|
||||||
if self._proxies and shuffle:
|
if self._proxies and shuffle:
|
||||||
random.shuffle(self._proxies)
|
random.shuffle(self._proxies)
|
||||||
self._state: Dict[str, Dict[str, object]] = {}
|
self._state: Dict[str, Dict[str, object]] = {}
|
||||||
for proxy in self._proxies:
|
for proxy in self._proxies:
|
||||||
self._state.setdefault(proxy.key(), {"uses": 0, "timeout": None})
|
self._state.setdefault(proxy.key(), self._fresh_state())
|
||||||
self._index = 0
|
self._index = 0
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _dedupe(proxies: List[Proxy]) -> List[Proxy]:
|
||||||
|
"""drop later entries sharing a canonical key, preserving first-seen order"""
|
||||||
|
seen: Dict[str, Proxy] = {}
|
||||||
|
for proxy in proxies:
|
||||||
|
seen.setdefault(proxy.key(), proxy)
|
||||||
|
return list(seen.values())
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _fresh_state() -> Dict[str, object]:
|
||||||
|
"""a clean per-proxy state entry: no uses, no timeout, not burned"""
|
||||||
|
return {"uses": 0, "timeout": None, "burned": False}
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _normalize_template(template: str) -> str:
|
||||||
|
"""fill a bare `{}` session slot with `{session}`; leave escaped `{{}}` untouched"""
|
||||||
|
return _BARE_SESSION_SLOT.sub("{session}", template)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def from_file(cls, path: str, **kwargs) -> "AioProxies":
|
def from_file(cls, path: str, **kwargs) -> "AioProxies":
|
||||||
"""build a rotating manager from a newline-delimited proxy file
|
"""build a rotating manager from a newline-delimited proxy file
|
||||||
|
|
||||||
raises FileNotFoundError if the path is missing — never exits the process.
|
raises FileNotFoundError if the path is missing - never exits the process.
|
||||||
"""
|
"""
|
||||||
with open(path, "r", encoding="utf-8") as fh:
|
with open(path, "r", encoding="utf-8") as fh:
|
||||||
lines = [ln.strip() for ln in fh if ln.strip()]
|
lines = [ln.strip() for ln in fh if ln.strip()]
|
||||||
@@ -109,25 +116,35 @@ class AioProxies:
|
|||||||
return False
|
return False
|
||||||
return now >= timeout
|
return now >= timeout
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _clear_stale_burn(state: Dict[str, object]) -> None:
|
||||||
|
"""self-clear a lazily-expired timed burn's `burned` flag once found available
|
||||||
|
|
||||||
|
mirrors the lazy expiry already applied to `timeout` itself: a timed burn()
|
||||||
|
that has simply run out its clock should stop upgrading routine cooldown
|
||||||
|
fallthrough logs to warning, without requiring an explicit restore().
|
||||||
|
"""
|
||||||
|
if state["timeout"] not in (None, 0, _DEAD):
|
||||||
|
state["burned"] = False
|
||||||
|
|
||||||
def next(self, **fields: object) -> Proxy:
|
def next(self, **fields: object) -> Proxy:
|
||||||
"""return the next proxy from the configured source
|
"""return the next proxy from the configured source
|
||||||
|
|
||||||
for template sources, `{session}` is always filled with a fresh id and any
|
template sources: `{session}` is always filled with a fresh id, other
|
||||||
other named placeholder is filled from `fields` (e.g. next(country="ca",
|
placeholders come from `fields` (e.g. next(country="ca", ttl=30)). fields
|
||||||
ttl=30)). fields are ignored by list/static sources.
|
are ignored by list/static sources.
|
||||||
|
|
||||||
for list sources, skips proxies whose timeout is active (-1 dead, or a
|
list sources: skips proxies whose timeout is active (-1 dead, or a future
|
||||||
future ts not yet passed), increments `uses` on handout, and applies the
|
ts not yet passed), increments `uses` on handout, applies the manager's
|
||||||
manager's cooldown. if none are fine but some are merely timed, hands out
|
cooldown. if none are fine but some are merely timed, hands out the one
|
||||||
the one recovering soonest (with a warning); raises ProxiesExhaustedError
|
recovering soonest (warns); raises ProxiesExhaustedError if all are dead.
|
||||||
if every proxy is permanently dead.
|
|
||||||
"""
|
"""
|
||||||
if self._static is not None:
|
if self._static is not None:
|
||||||
return self._static
|
return self._static
|
||||||
if self.template is not None:
|
if self.template is not None:
|
||||||
if "session" in fields:
|
if "session" in fields:
|
||||||
# `session` is auto-filled with a fresh id; a caller-supplied one would
|
# `session` is auto-filled with a fresh id; a caller-supplied one would
|
||||||
# collide in str.format with an opaque TypeError — reject it clearly
|
# collide in str.format with an opaque TypeError - reject it clearly
|
||||||
raise ValueError("'session' is filled automatically; do not pass it to next()")
|
raise ValueError("'session' is filled automatically; do not pass it to next()")
|
||||||
try:
|
try:
|
||||||
filled = self.template.format(session=self.session_id(), **fields)
|
filled = self.template.format(session=self.session_id(), **fields)
|
||||||
@@ -137,9 +154,9 @@ class AioProxies:
|
|||||||
) from exc
|
) from exc
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
# a malformed template (e.g. an unmatched '{') makes str.format raise a
|
# a malformed template (e.g. an unmatched '{') makes str.format raise a
|
||||||
# bare ValueError; re-raise naming the cause so it isn't cryptic
|
# bare ValueError - re-raise naming the cause so it isn't cryptic
|
||||||
raise ValueError(f"malformed proxy template {self.template!r}: {exc}") from exc
|
raise ValueError(f"malformed proxy template {self.template!r}: {exc}") from exc
|
||||||
return parse(filled)
|
return to_proxy(filled)
|
||||||
return self._next_from_list()
|
return self._next_from_list()
|
||||||
|
|
||||||
def _next_from_list(self) -> Proxy:
|
def _next_from_list(self) -> Proxy:
|
||||||
@@ -152,9 +169,10 @@ class AioProxies:
|
|||||||
fine: List[int] = []
|
fine: List[int] = []
|
||||||
for offset in range(count):
|
for offset in range(count):
|
||||||
idx = (self._index + offset) % count
|
idx = (self._index + offset) % count
|
||||||
timeout = self._state[self._proxies[idx].key()]["timeout"]
|
state = self._state[self._proxies[idx].key()]
|
||||||
if self._available(timeout, now):
|
if self._available(state["timeout"], now):
|
||||||
fine.append(idx)
|
fine.append(idx)
|
||||||
|
self._clear_stale_burn(state)
|
||||||
|
|
||||||
if fine:
|
if fine:
|
||||||
chosen = fine[0]
|
chosen = fine[0]
|
||||||
@@ -172,24 +190,22 @@ class AioProxies:
|
|||||||
state = self._state[proxy.key()]
|
state = self._state[proxy.key()]
|
||||||
state["uses"] = int(state["uses"]) + 1
|
state["uses"] = int(state["uses"]) + 1
|
||||||
if self.cooldown > 0:
|
if self.cooldown > 0:
|
||||||
state["timeout"] = now + self.cooldown
|
# cooldown only ever EXTENDS availability delay - never shorten a longer
|
||||||
|
# active timeout (a still-running timed burn) down to now+cooldown. _DEAD
|
||||||
|
# (-1) is a permanent burn and must not be resurrected into a future ts.
|
||||||
|
cooled = int(now) + self.cooldown
|
||||||
|
current = state["timeout"]
|
||||||
|
if current == _DEAD:
|
||||||
|
pass
|
||||||
|
elif isinstance(current, (int, float)) and current > cooled:
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
state["timeout"] = cooled
|
||||||
return proxy
|
return proxy
|
||||||
|
|
||||||
def _has_burned(self) -> bool:
|
def _has_burned(self) -> bool:
|
||||||
"""whether the empty 'fine' tier reflects a genuine burn, not just cooldown
|
"""whether any proxy carries a genuine burn (vs just cooldown resting), for log level choice"""
|
||||||
|
return any(state["burned"] for state in self._state.values())
|
||||||
a dead (-1) proxy is always a real burn. a future-ts timeout is a real timed
|
|
||||||
burn only when cooldown is off; with cooldown on, future-ts entries are the
|
|
||||||
manager's own resting and not a pool-health signal. used to pick warning
|
|
||||||
(real trouble) vs debug (normal cooldown) on the soonest-recovering path.
|
|
||||||
"""
|
|
||||||
for state in self._state.values():
|
|
||||||
timeout = state["timeout"]
|
|
||||||
if timeout == _DEAD:
|
|
||||||
return True
|
|
||||||
if self.cooldown == 0 and timeout not in (None, 0):
|
|
||||||
return True
|
|
||||||
return False
|
|
||||||
|
|
||||||
def _soonest_recovering(self) -> Optional[int]:
|
def _soonest_recovering(self) -> Optional[int]:
|
||||||
"""index of the timed (non-dead) proxy recovering soonest, or None"""
|
"""index of the timed (non-dead) proxy recovering soonest, or None"""
|
||||||
@@ -224,16 +240,24 @@ class AioProxies:
|
|||||||
if seconds is None:
|
if seconds is None:
|
||||||
self._state[key]["timeout"] = _DEAD
|
self._state[key]["timeout"] = _DEAD
|
||||||
else:
|
else:
|
||||||
self._state[key]["timeout"] = time.time() + seconds
|
self._state[key]["timeout"] = int(time.time()) + seconds
|
||||||
|
self._state[key]["burned"] = True
|
||||||
|
|
||||||
def restore(self, proxy: Union[str, Proxy, Dict[str, str]]) -> None:
|
def restore(self, proxy: Union[str, Proxy, Dict[str, str]]) -> None:
|
||||||
"""clear any burn/timeout on a proxy (back to fine). no-op if already fine"""
|
"""clear any burn/timeout on a proxy (back to fine). no-op if already fine
|
||||||
|
|
||||||
|
matches remove()'s contract for an unknown proxy: logs a warning and returns
|
||||||
|
rather than silently doing nothing.
|
||||||
|
"""
|
||||||
if not self._is_list_source():
|
if not self._is_list_source():
|
||||||
self._warn_non_list("restore()")
|
self._warn_non_list("restore()")
|
||||||
return
|
return
|
||||||
key = canonical_key(proxy)
|
key = canonical_key(proxy)
|
||||||
if key in self._state:
|
if key not in self._state:
|
||||||
|
log.warning("restore(): proxy not in pool: %s", key)
|
||||||
|
return
|
||||||
self._state[key]["timeout"] = None
|
self._state[key]["timeout"] = None
|
||||||
|
self._state[key]["burned"] = False
|
||||||
|
|
||||||
def is_burned(self, proxy: Union[str, Proxy, Dict[str, str]]) -> bool:
|
def is_burned(self, proxy: Union[str, Proxy, Dict[str, str]]) -> bool:
|
||||||
"""whether a proxy is currently unavailable (lazy expiry of timed burns)
|
"""whether a proxy is currently unavailable (lazy expiry of timed burns)
|
||||||
@@ -246,8 +270,11 @@ class AioProxies:
|
|||||||
key = canonical_key(proxy)
|
key = canonical_key(proxy)
|
||||||
if key not in self._state:
|
if key not in self._state:
|
||||||
return False
|
return False
|
||||||
timeout = self._state[key]["timeout"]
|
state = self._state[key]
|
||||||
return not self._available(timeout, time.time())
|
available = self._available(state["timeout"], time.time())
|
||||||
|
if available:
|
||||||
|
self._clear_stale_burn(state)
|
||||||
|
return not available
|
||||||
|
|
||||||
def stats(self) -> List[Dict[str, object]]:
|
def stats(self) -> List[Dict[str, object]]:
|
||||||
"""per-proxy usage + state snapshot (list sources only, else empty)
|
"""per-proxy usage + state snapshot (list sources only, else empty)
|
||||||
@@ -285,17 +312,21 @@ class AioProxies:
|
|||||||
for state in self._state.values():
|
for state in self._state.values():
|
||||||
state["uses"] = 0
|
state["uses"] = 0
|
||||||
|
|
||||||
def replace(self, proxies: List[Union[str, Proxy]], *, keep_state: bool = False) -> None:
|
def replace(self, proxies: List[Union[str, Proxy, Dict[str, str]]], *, keep_state: bool = False) -> None:
|
||||||
"""swap the entire proxy list
|
"""swap the entire proxy list
|
||||||
|
|
||||||
keep_state=False (default) wipes all per-proxy state (fresh batch).
|
keep_state=False (default) wipes all per-proxy state (fresh batch).
|
||||||
keep_state=True preserves uses/timeout for proxies whose canonical key
|
keep_state=True preserves uses/timeout for proxies whose canonical key
|
||||||
survives the swap; new proxies start clean, dropped ones are forgotten.
|
survives the swap; new proxies start clean, dropped ones are forgotten.
|
||||||
resets the rotation index and honors the manager's shuffle setting.
|
resets the rotation index and honors the manager's shuffle setting. raises
|
||||||
|
ValueError on an empty list, matching the constructor's guard - use remove()
|
||||||
|
to drop proxies one at a time if you actually want a smaller pool.
|
||||||
"""
|
"""
|
||||||
if not self._is_list_source():
|
if not self._is_list_source():
|
||||||
self._warn_non_list("replace()")
|
self._warn_non_list("replace()")
|
||||||
return
|
return
|
||||||
|
if len(proxies) == 0:
|
||||||
|
raise ValueError("proxies list is empty; provide at least one proxy")
|
||||||
old_state = self._state
|
old_state = self._state
|
||||||
incoming = [to_proxy(p) for p in proxies]
|
incoming = [to_proxy(p) for p in proxies]
|
||||||
if self._shuffle:
|
if self._shuffle:
|
||||||
@@ -310,12 +341,12 @@ class AioProxies:
|
|||||||
if keep_state and key in old_state:
|
if keep_state and key in old_state:
|
||||||
new_state[key] = old_state[key]
|
new_state[key] = old_state[key]
|
||||||
else:
|
else:
|
||||||
new_state[key] = {"uses": 0, "timeout": None}
|
new_state[key] = self._fresh_state()
|
||||||
self._proxies = new_proxies
|
self._proxies = new_proxies
|
||||||
self._state = new_state
|
self._state = new_state
|
||||||
self._index = 0
|
self._index = 0
|
||||||
|
|
||||||
def add(self, proxies: Union[str, Proxy, List[Union[str, Proxy]]]) -> None:
|
def add(self, proxies: Union[str, Proxy, Dict[str, str], List[Union[str, Proxy, Dict[str, str]]]]) -> None:
|
||||||
"""append proxies to the pool, keeping existing state; skip duplicate keys"""
|
"""append proxies to the pool, keeping existing state; skip duplicate keys"""
|
||||||
if not self._is_list_source():
|
if not self._is_list_source():
|
||||||
self._warn_non_list("add()")
|
self._warn_non_list("add()")
|
||||||
@@ -327,13 +358,15 @@ class AioProxies:
|
|||||||
if key in self._state:
|
if key in self._state:
|
||||||
continue
|
continue
|
||||||
self._proxies.append(proxy)
|
self._proxies.append(proxy)
|
||||||
self._state[key] = {"uses": 0, "timeout": None}
|
self._state[key] = self._fresh_state()
|
||||||
|
|
||||||
def remove(self, proxy: Union[str, Proxy, Dict[str, str]]) -> None:
|
def remove(self, proxy: Union[str, Proxy, Dict[str, str]]) -> None:
|
||||||
"""drop a proxy from the pool entirely (by canonical key, any shape)
|
"""drop a proxy from the pool entirely (by canonical key, any shape)
|
||||||
|
|
||||||
distinct from burn (burn = unusable but tracked; remove = gone). clamps the
|
distinct from burn (burn = unusable but tracked; remove = gone). decrements
|
||||||
rotation index if needed. no-op + warning if the proxy is not present.
|
the rotation index when the removed slot precedes it, so the next() cursor
|
||||||
|
still lands on the same upcoming proxy instead of skipping one. no-op +
|
||||||
|
warning if the proxy is not present.
|
||||||
"""
|
"""
|
||||||
if not self._is_list_source():
|
if not self._is_list_source():
|
||||||
self._warn_non_list("remove()")
|
self._warn_non_list("remove()")
|
||||||
@@ -342,14 +375,17 @@ class AioProxies:
|
|||||||
if key not in self._state:
|
if key not in self._state:
|
||||||
log.warning("remove(): proxy not in pool: %s", key)
|
log.warning("remove(): proxy not in pool: %s", key)
|
||||||
return
|
return
|
||||||
|
removed_idx = next(i for i, p in enumerate(self._proxies) if p.key() == key)
|
||||||
self._proxies = [p for p in self._proxies if p.key() != key]
|
self._proxies = [p for p in self._proxies if p.key() != key]
|
||||||
del self._state[key]
|
del self._state[key]
|
||||||
if self._proxies:
|
if not self._proxies:
|
||||||
self._index %= len(self._proxies)
|
|
||||||
else:
|
|
||||||
self._index = 0
|
self._index = 0
|
||||||
|
else:
|
||||||
|
if removed_idx < self._index:
|
||||||
|
self._index -= 1
|
||||||
|
self._index %= len(self._proxies)
|
||||||
|
|
||||||
|
|
||||||
# name aliases — same class, call it whichever reads best at your call site
|
# name aliases - same class, call it whichever reads best at your call site
|
||||||
ProxyManager = AioProxies
|
ProxyManager = AioProxies
|
||||||
aioproxies = AioProxies
|
aioproxies = AioProxies
|
||||||
|
|||||||
+16
-3
@@ -6,11 +6,24 @@ calling a function without it raises a clear error.
|
|||||||
"""
|
"""
|
||||||
import logging
|
import logging
|
||||||
from typing import Optional, Union
|
from typing import Optional, Union
|
||||||
|
from urllib.parse import urlsplit, urlunsplit
|
||||||
|
|
||||||
from .proxy import Proxy, parse
|
from .proxy import Proxy, to_proxy
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def _safe_url(url: str) -> str:
|
||||||
|
"""strip query + fragment from a url for logging - a provider reset url can carry a
|
||||||
|
rotation token in its query/path; this drops the query/fragment (the usual token spot)
|
||||||
|
so the log line can't leak it. falls back to the raw url only if it won't parse"""
|
||||||
|
try:
|
||||||
|
parts = urlsplit(url)
|
||||||
|
except ValueError:
|
||||||
|
return url
|
||||||
|
return urlunsplit((parts.scheme, parts.netloc, parts.path, "", ""))
|
||||||
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
import aiohttp
|
import aiohttp
|
||||||
|
|
||||||
@@ -32,7 +45,7 @@ async def current_ip(
|
|||||||
"""
|
"""
|
||||||
if not _HAVE_AIOHTTP:
|
if not _HAVE_AIOHTTP:
|
||||||
raise RuntimeError(_MISSING)
|
raise RuntimeError(_MISSING)
|
||||||
p = parse(proxy)
|
p = to_proxy(proxy)
|
||||||
t = aiohttp.ClientTimeout(total=timeout)
|
t = aiohttp.ClientTimeout(total=timeout)
|
||||||
try:
|
try:
|
||||||
async with aiohttp.ClientSession(timeout=t) as session:
|
async with aiohttp.ClientSession(timeout=t) as session:
|
||||||
@@ -55,7 +68,7 @@ async def reset(reset_url: str, *, timeout: float = 15.0) -> bool:
|
|||||||
async with aiohttp.ClientSession(timeout=t) as session:
|
async with aiohttp.ClientSession(timeout=t) as session:
|
||||||
async with session.get(reset_url) as resp:
|
async with session.get(reset_url) as resp:
|
||||||
ok = resp.status == 200
|
ok = resp.status == 200
|
||||||
log.info("proxy reset %s -> %s", reset_url, resp.status)
|
log.info("proxy reset %s -> %s", _safe_url(reset_url), resp.status)
|
||||||
return ok
|
return ok
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
log.warning("proxy reset failed: %s", exc)
|
log.warning("proxy reset failed: %s", exc)
|
||||||
|
|||||||
+36
-30
@@ -1,15 +1,12 @@
|
|||||||
"""proxy parsing and formatting (pure, no network IO).
|
"""proxy parsing and formatting (pure, no network IO).
|
||||||
|
|
||||||
a `Proxy` holds host/port/optional-auth and renders the shapes different clients
|
`Proxy` holds host/port/optional-auth and renders aiohttp/aioweb, camoufox, socks5,
|
||||||
want: an aiohttp/aioweb proxies dict, a camoufox proxy dict, a socks5 dict, or a
|
and url shapes; auth-less (IP-authenticated) proxies are first-class throughout.
|
||||||
plain url. `parse` accepts the common "host:port" and "host:port:user:pass" string
|
`parse` accepts "host:port" and "host:port:user:pass" - the user field may itself
|
||||||
forms (the user field may itself contain commas, e.g. session-param proxies; the
|
contain commas and the password may itself contain colons.
|
||||||
password field may itself contain colons). auth-less (IP-authenticated) proxies
|
|
||||||
with no creds are first-class — every render shape handles them via `has_auth`.
|
|
||||||
|
|
||||||
`key()` produces a stable canonical identity (`host:port:user:pass`, or
|
`key()` / `canonical_key()` produce a stable canonical identity so burn/remove/stats
|
||||||
`host:port` when auth-less) so burn/remove/stats recognize the same proxy from any
|
recognize the same proxy from any input shape (spec string, dict, or url).
|
||||||
input shape. `canonical_key()` extends that to dict/url forms.
|
|
||||||
"""
|
"""
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from typing import Dict, Optional, Union
|
from typing import Dict, Optional, Union
|
||||||
@@ -19,6 +16,18 @@ SCHEME_HTTP = "http"
|
|||||||
SCHEME_SOCKS5 = "socks5"
|
SCHEME_SOCKS5 = "socks5"
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_port(port: Union[str, int]) -> str:
|
||||||
|
"""canonical port string: parsed to int, rendered without zero-padding
|
||||||
|
|
||||||
|
keeps host:080 and host:80 as one canonical key. raises ValueError naming the
|
||||||
|
missing port if empty/None (e.g. a portless proxy url), or on a non-integer
|
||||||
|
port, rather than silently keying it as-is.
|
||||||
|
"""
|
||||||
|
if port is None or port == "":
|
||||||
|
raise ValueError("missing port: proxy url or spec has no port")
|
||||||
|
return str(int(port))
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class Proxy:
|
class Proxy:
|
||||||
"""a single proxy endpoint with optional auth"""
|
"""a single proxy endpoint with optional auth"""
|
||||||
@@ -38,14 +47,16 @@ class Proxy:
|
|||||||
|
|
||||||
the host is lowercased (hostnames are case-insensitive per DNS, so
|
the host is lowercased (hostnames are case-insensitive per DNS, so
|
||||||
PROXY.example.com and proxy.example.com are the same host and collapse to one
|
PROXY.example.com and proxy.example.com are the same host and collapse to one
|
||||||
key); port and credentials are kept verbatim. the password is included in
|
key); credentials are kept verbatim. the port is normalized (leading zeros
|
||||||
full (two proxies differing only by password are distinct slots). auth-less
|
stripped) so host:080 and host:80 collapse to one key. the password is
|
||||||
proxies collapse to host:port with no trailing colons.
|
included in full (two proxies differing only by password are distinct
|
||||||
|
slots). auth-less proxies collapse to host:port with no trailing colons.
|
||||||
"""
|
"""
|
||||||
host = self.host.lower()
|
host = self.host.lower()
|
||||||
|
port = normalize_port(self.port)
|
||||||
if self.has_auth:
|
if self.has_auth:
|
||||||
return f"{host}:{self.port}:{self.user}:{self.password}"
|
return f"{host}:{port}:{self.user}:{self.password}"
|
||||||
return f"{host}:{self.port}"
|
return f"{host}:{port}"
|
||||||
|
|
||||||
def url(self, scheme: str = SCHEME_HTTP) -> str:
|
def url(self, scheme: str = SCHEME_HTTP) -> str:
|
||||||
"""render as a url, embedding auth when present
|
"""render as a url, embedding auth when present
|
||||||
@@ -86,12 +97,15 @@ class Proxy:
|
|||||||
|
|
||||||
|
|
||||||
def parse(spec: Union[str, Proxy]) -> Proxy:
|
def parse(spec: Union[str, Proxy]) -> Proxy:
|
||||||
"""parse a proxy spec into a Proxy
|
"""parse a colon-delimited proxy spec into a Proxy
|
||||||
|
|
||||||
accepts an existing Proxy (returned as-is) or a colon-delimited string in
|
accepts an existing Proxy (returned as-is) or a colon-delimited string in
|
||||||
`host:port` or `host:port:user:pass` form. the 4-part form splits on the first
|
`host:port` or `host:port:user:pass` form. the 4-part form splits on the first
|
||||||
three colons only, so a password may itself contain colons. raises ValueError
|
three colons only, so a password may itself contain colons. this is spec-string
|
||||||
on anything else rather than guessing.
|
only - it does not understand url shapes (`scheme://user:pass@host:port`); a
|
||||||
|
url has 3+ colons and would be misparsed as a 4-part spec rather than rejected.
|
||||||
|
use `to_proxy()` for url / dict / spec input, or anywhere the shape isn't
|
||||||
|
guaranteed to be a bare spec string.
|
||||||
"""
|
"""
|
||||||
if isinstance(spec, Proxy):
|
if isinstance(spec, Proxy):
|
||||||
return spec
|
return spec
|
||||||
@@ -106,14 +120,7 @@ def parse(spec: Union[str, Proxy]) -> Proxy:
|
|||||||
|
|
||||||
|
|
||||||
def canonical_key(spec: Union[str, Proxy, Dict[str, str]]) -> str:
|
def canonical_key(spec: Union[str, Proxy, Dict[str, str]]) -> str:
|
||||||
"""canonical identity key for any supported proxy shape
|
"""`to_proxy(spec).key()` - the canonical identity key for any supported proxy shape"""
|
||||||
|
|
||||||
accepts a spec string, a Proxy, a url string (`http://user:pass@host:port`,
|
|
||||||
`socks5://...`), an aiohttp dict (`{"http": url, "https": url}`), or a
|
|
||||||
camoufox/socks5 dict (`{"server": ..., "username": ..., "password": ...}`).
|
|
||||||
all forms collapse to the same `host:port:user:pass` (or `host:port` auth-less)
|
|
||||||
key, so burn/remove/stats recognize one proxy regardless of how it was passed.
|
|
||||||
"""
|
|
||||||
return to_proxy(spec).key()
|
return to_proxy(spec).key()
|
||||||
|
|
||||||
|
|
||||||
@@ -121,8 +128,8 @@ def to_proxy(spec: Union[str, Proxy, Dict[str, str]]) -> Proxy:
|
|||||||
"""normalize any supported proxy shape into a Proxy
|
"""normalize any supported proxy shape into a Proxy
|
||||||
|
|
||||||
accepts a spec string, a Proxy, a url string, an aiohttp dict, or a
|
accepts a spec string, a Proxy, a url string, an aiohttp dict, or a
|
||||||
camoufox/socks5 dict — the same shapes `canonical_key` accepts. used both for
|
camoufox/socks5 dict. used both for keying and for adding proxies to a pool
|
||||||
keying and for adding proxies to a pool from any shape.
|
from any shape.
|
||||||
"""
|
"""
|
||||||
if isinstance(spec, Proxy):
|
if isinstance(spec, Proxy):
|
||||||
return spec
|
return spec
|
||||||
@@ -139,9 +146,8 @@ def _proxy_from_dict(spec: Dict[str, str]) -> Proxy:
|
|||||||
"""normalize an aiohttp / camoufox / socks5 dict into a Proxy"""
|
"""normalize an aiohttp / camoufox / socks5 dict into a Proxy"""
|
||||||
if "server" in spec:
|
if "server" in spec:
|
||||||
proxy = _proxy_from_url(spec["server"])
|
proxy = _proxy_from_url(spec["server"])
|
||||||
# prefer explicit dict auth; otherwise fall back to auth embedded in the server
|
# prefer explicit dict auth; fall back to auth embedded in the server URL so
|
||||||
# URL (http://user:pass@host:port) so it isn't silently dropped — which would
|
# it isn't silently dropped, which would key the proxy auth-less
|
||||||
# key the proxy auth-less and collide with a genuinely auth-less one
|
|
||||||
user = spec.get("username") or proxy.user
|
user = spec.get("username") or proxy.user
|
||||||
password = spec.get("password") or proxy.password
|
password = spec.get("password") or proxy.password
|
||||||
if user and password:
|
if user and password:
|
||||||
|
|||||||
Reference in New Issue
Block a user