18 Commits
Author SHA1 Message Date
dsql 952fe8a00f feat: hang-guard request() with an outer asyncio deadline
a backend can lose both its transfer wakeup and its own timeout enforcement in one
failure (observed: a curl_cffi request that parked an aioweb consumer's event loop for
24h+ - loop idle, nothing in flight, the await never completing). wrap _raw_request in
asyncio.wait_for at the effective timeout + 5s slack: the backend's own timeout still
fires first in every healthy failure, the envelope only trips when the backend's timer
is dead, and its cancellation unwedges the orphaned transfer. deadline = per-call numeric
timeout else the session timeout (default 10); timeout=None from request_with_retries
guards on the session bound (that was the wedge path). explicit Session(timeout=None)
stays unguarded - the documented opt-out for genuinely unbounded calls. additive, no new
knob. fired: a never-completing backend hangs forever without this, dies at deadline+slack
with it, and request_with_retries returns a FailureResponse instead of hanging.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-27 11:33:35 -04:00
dsql 6ec132f97c build: use git+https for inter-lib deps (docker ssh limitation)
docker builds can't use git+ssh (no ssh key / agent in the build), so the inter-lib
dependency references move to git+https (repos are public, anonymous clone). pins are
unchanged in target; bump to 1.0.2 so the https dependency spec ships under a new tag.
README install lines intentionally keep the ssh form for local/dev use.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-20 22:25:06 -04:00
dsql 6633d6cba3 fix: pin inter-lib dependencies to their v1.0.0 tags
the v1.0.0 release still pinned pre-1.0.0 sibling tags, so a fresh install dragged in
stale transitive deps. update the pin(s) to the current v1.0.x release and bump this lib
to 1.0.1 so the corrected dependency chain ships under a new tag (v1.0.0 left intact).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-17 17:46:10 -04:00
dsql 1deb05ce4a release: 1.0.0
first stable release. pre-1.0.0 verification complete: all surviving MED regressions and
gaps resolved and independently re-fired, tree audited clean across the suite.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-09 18:53:15 -04:00
dsql 45d8830833 fix: as_curl form-encodes a dict data= body with doseq=True
urlencode(data) without doseq rendered a list-valued form field as its urlencoded python
repr (x=%5B%271%27...) instead of aiohttp FormData's repeated-pair wire form (x=1&x=2), so
a debug curl for a list-valued body replayed a different request than request() sent. add
doseq=True to match the wire. preview/debug-only; scalar values and non-dict bodies are
byte-unchanged.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 21:16:08 -04:00
dsql eed607b08c fix: an IPv6 domain overwrite applies only on a whole-host match, never a substring splice
overwrite_domain accepted a bare IPv6 replacement, but _apply_domain_overwrites spliced it
as a substring into any host CONTAINING the target, composing an invalid host ('internal::1'
from '.local'->'::1' on internal.local) that exploded deep in yarl at request time. An IPv6
replacement now applies ONLY when the target equals the whole request host; a substring match
is left unrewritten instead of building an invalid host. Hostname/IPv4 substring splices are
unchanged. (Registration can't distinguish whole-host from substring for a hostname target
without the request host, so the guard lives at compose time where the host is known.)

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:56:52 -04:00
dsql ed773e2c1b docs+fix: as_curl form-encodes a dict data= body; README notes the commons dependency
as_curl() rendered a dict data= body as its Python repr ({'a': 1}), but aiohttp
form-encodes a dict data= as application/x-www-form-urlencoded (a=1&b=two), so the emitted
curl replayed a different body - now urlencode()s a dict (str bodies unchanged). README
install section omitted the hard sibling commons dependency (a private git+ssh package).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:37:09 -04:00
dsql 85735023f5 fix: proxy=None no longer drops an explicit proxies= (IP unmask); cookie views work off-loop
request(proxy=None, proxies={...}) silently sent direct because setdefault('proxy', ...)
is a no-op when the None key already exists, unmasking the caller's real IP - now a bare
proxy=None is replaced by the resolved proxies= (the both-set guard still rejects two real
proxies). get_cookies/clear_cookies return {}/no-op when no session was built yet (mirroring
preview) instead of forcing a build that needs a running loop; set_cookie, which genuinely
needs the jar, now raises a clear actionable error off-loop instead of aiohttp's opaque one.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:23:31 -04:00
dsql 51c421c9a2 fix: overwrite_domain accepts a bare IPv6 replacement instead of rejecting any ':'
the guard rejected any replacement containing ':', which over-rejected a valid bare IPv6
literal host ('::1', '2001:db8::1') that yarl's URL.with_host() accepts and brackets. it now
delegates the check to yarl (try with_host, catch ValueError), so a bare IPv6 is accepted
(rewrites to http://[::1]/...) while a genuine host:port or pre-bracketed form still raises.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 16:47:35 -04:00
dsql 59eadabe16 fix: fail loud on unreachable domain overwrites and jar-dropped IP cookies
overwrite_domain() now rejects a port-bearing replacement immediately at
registration time with a clear ValueError, instead of only surfacing a
deep yarl error from inside with_host() the first time a matching request
is made, far from the misconfiguration site.

set_cookie() now raises ValueError when domain resolves to a bare IP host
and the jar is the default CookieJar(unsafe=False), which silently drops
IP-bound cookies with no store, no send, and no log. Non-IP domains and a
caller-supplied unsafe=True jar are unaffected.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 00:09:37 -04:00
dsql 78dfba0962 fix: drop session-default sock_read that poisons pooled keep-alive connections
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 19:05:51 -04:00
dsql a7302a6356 refactor: derive __version__ from package metadata (single source)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 17:00:19 -04:00
dsql 397d74efe3 docs: restore FailureResponse public-method docstrings stripped in de-bloat
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:46:03 -04:00
dsql 7f73f21c93 fix: preview() works without a running loop; set_cookie honors path when domain=None
preview()'s default cookie lookup reached the backend session and could force-
build it, raising RuntimeError('no running event loop') off-loop and defeating
the v0.1.7 lazy-session pre-loop use case; it's now skipped when the session
isn't built yet. set_cookie(domain=None) returned before setting the morsel's
path, so a shared cookie always stored path='/' regardless of the path= arg;
the morsel path is now set before that early return.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:17:46 -04:00
dsql 27f0e49341 docs: compress prose/module docstrings, em-dash->hyphen (de-bloat wave 1)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 00:15:24 -04:00
dsql ca23099e06 fix: proxy/attempts truthiness bugs, stale text() cache, preview cookie order (v0.1.9)
_get_proxy() and request() checked proxies/proxy with truthiness instead of
is None, so a per-call proxies={} silently fell back to session proxies, and a
native proxy= kwarg was clobbered by resolved session proxies (unmasking the
caller's real IP) instead of raising on conflicting sources. Response.text()
cached the first decode regardless of a later explicit encoding= argument.
preview() computed the Cookie header from the pre-rewrite url instead of the
rewritten one actually used by request(). request_with_retries(attempts=0)
became DEFAULT_ATTEMPTS via truthiness instead of flooring to 1.

Also compresses docstrings/comments across the module (no behavior change).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 23:25:43 -04:00
dsql 0768d643b1 fix: preserve exception subtype in request(), render params/timeout in as_curl()
request() re-raised every aiohttp.ClientError subclass as the bare base class,
losing ClientConnectorError/ClientProxyConnectionError/ClientResponseError
(.status/.headers)/TooManyRedirects and their attributes; direct callers
branching by type never matched. Now the original exception is bare-raised,
preserving subtype, attributes, and __cause__. request_with_retries (which
catches the base ClientError) is unaffected.

as_curl() silently dropped params and timeout, so a debug=True replay of a
params-driven request hit a different URL than the one actually sent. params
are now merged into the URL via yarl before quoting, and timeout renders as
--max-time.

Bumps 0.1.7 -> 0.1.8.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 16:57:04 -04:00
dsql e1ab5d38a0 fix: cookie methods actually work; session builds lazily (v0.1.7)
get_cookies() called filter_cookies() with no URL and always returned {}
for domain-bound cookies; now iterates the jar directly. set_cookie()
ignored path and leaked a shared cookie to every host when given a bare
domain string; scheme is now normalized and path honored, with
domain=None made an intentionally shared cookie instead of a silent
localhost-only no-op.

ExtendedSession also built its aiohttp.ClientSession synchronously in
__init__, which requires a running event loop under aiohttp>=3.14 and
crashed the common construct-before-the-loop-starts host pattern. The
session now builds lazily on first access, preserving the
_create_session subclass override seam used by aioweb_tls.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 16:42:30 -04:00
6 changed files with 420 additions and 136 deletions
+140 -5
View File
@@ -11,18 +11,19 @@ and swap the HTTP client while inheriting everything else.
`requirements.txt`:
```
aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.6
aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v1.1.0
```
Direct:
```bash
pip install "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.6"
pip install "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v1.1.0"
```
Requires `aiohttp` and `yarl` (pulled transitively).
Requires `aiohttp` and `yarl` (pulled transitively), plus the sibling `commons` library
(a private `git+ssh` package — the install needs access to the `rethink-public` org).
Drop the `@v0.1.5` suffix from the line above to install the latest unpinned.
Drop the `@v1.1.0` suffix from the line above to install the latest unpinned.
## Usage
@@ -65,6 +66,15 @@ resp = await s.request_with_retries(
Returns a `FailureResponse` (falsy) if every attempt fails.
`request()` (the non-retrying call) raises on failure: a total timeout raises
`aiohttp.ServerTimeoutError`, and any other network/protocol failure raises its real
`aiohttp.ClientError` subtype as-is (`ClientConnectorError`, `ClientProxyConnectionError`,
`ClientResponseError` with `.status`/`.headers`, `TooManyRedirects`, ...) — it is not
flattened into the base `ClientError`, so direct callers can branch by type or read
subtype attributes. `request_with_retries` catches the base `aiohttp.ClientError` (and
`asyncio.TimeoutError`) across all attempts and returns a falsy `FailureResponse` instead
of raising.
## Header overwrites & ephemeral headers
```python
@@ -88,7 +98,15 @@ s.overwrite_domain("internal.local", "127.0.0.1") # host-substring rewrite
print(s.preview("POST", url, json={"a": 1}).as_curl()) # equivalent cURL command
```
Pass `debug=True` to `request_with_retries` to log the cURL preview and request flow.
`preview()` is synchronous and never touches the backend session — it works before any
event loop is running (e.g. building a preview at import/setup time), not just inside
`asyncio.run()`. Its default cookie lookup only reads cookies from an already-built
session; if the session hasn't been built yet there are no cookies to read, so it
returns none by default (pass `cookies=` explicitly to preview cookies for a session
that hasn't sent a request yet). `as_curl()` renders `params` (merged into the url's
query string) and `timeout` (as `--max-time`) as well as headers/body/proxy, so the
emitted command is faithful to what `request()` actually sends. Pass `debug=True` to
`request_with_retries` to log the cURL preview and request flow.
## Custom backends
@@ -132,6 +150,123 @@ Two changes can't be shimmed without re-introducing the bugs they fix:
## Changelog
### v1.1.0
- **Hang-guard on `request()` — a wedged backend can no longer park the loop forever.**
`request()` now wraps the backend call (`_raw_request`) in an outer `asyncio.wait_for`
deadline sized at the effective timeout **+ 5s slack** (`_HANG_GUARD_SLACK`). The
backend's own timeout still fires first in every healthy failure (slack, not
replacement); the envelope only trips when the backend's timer is dead — e.g. a
`curl_cffi` transfer that loses both its wakeup and its own enforcement in the same
failure — and its cancellation is what unwedges the orphaned transfer. The guard's
deadline is the per-call numeric `timeout` when given, else the session timeout
(`Session(timeout=...)`, default 10s); it composes cleanly with `request_with_retries`
(a wedge becomes one logged `ServerTimeoutError`, the next attempt proceeds, and an
exhausted retry returns a `FailureResponse` rather than hanging). **Opt-out:** a
session deliberately constructed unbounded (`Session(timeout=None)`) stays unguarded,
for genuinely long-lived calls (long-poll / streaming). Additive; no new knob.
### v0.1.13
- **Session-default timeout no longer poisons pooled keep-alive connections.**
The default `ClientTimeout` set `sock_read=timeout/2` alongside `total`. Under
`aiohttp>=3.14`, that read timer re-arms on every request dispatched over a
pooled protocol, including idle connections between requests; when it fires
it permanently poisons the pooled connection (`SocketTimeoutError` on the next
use, instantly, without contacting the server) — a real error from the server
(e.g. a 503) could come back as a client-side `FailureResponse(status=0,
reason='timeout')` instead. `sock_read` is now dropped from the session
default; `total` (and `connect`/`sock_connect`) still bound every request, and
the per-call `timeout=N` path (`ClientTimeout(total=N)`, no `sock_read`) was
already unaffected.
### v0.1.12
- **Docstring-only.** Restored one-line docstrings on `FailureResponse`'s
`redirect_chain`, `text()`, `json()`, and `raise_for_status()` — a prior
de-bloat pass stripped them below the public tier while the `Response`
twins kept theirs. No behavior change.
### v0.1.11
- **`preview()` no longer requires a running event loop.** Its default cookie
lookup (`cookies=` not passed) previously called `_cookies_for_url()`, which
reached the backend session and built it if absent — under aiohttp>=3.14 that
needs a running loop, so `preview()` raised `RuntimeError('no running event
loop')` when called before the loop starts, defeating its own pre-loop
build/inspect use case (v0.1.7). Now the default cookie lookup is skipped
entirely when the session hasn't been built yet (nothing could have been set
on a session that doesn't exist); `cookies={}` and in-loop calls are unaffected.
- **`set_cookie(domain=None)` now honors `path`.** The `domain=None` branch
returned right after `update_cookies()`, before the line that sets the
morsel's `path` — so a shared cookie (`set_cookie(name, value, path="/api")`,
no `domain=`) always stored the `SimpleCookie` default `path="/"` instead. The
morsel's `path` is now set before the `domain=None` early return. The
domain-bound branch was already correct and is unchanged.
### v0.1.10
- Docs-only pass: compressed module/method docstrings and comments that restated
README/CLAUDE prose, replaced em-dashes with hyphens. No behavior change.
### v0.1.9
- **`_get_proxy()`/`request()` proxy resolution now checks `is None`, not
truthiness.** A per-call `proxies={}` previously fell back to the session's
configured proxies instead of disabling them for that call. `request()` also no
longer clobbers a native `proxy=` kwarg with the resolved session/`proxies=`
value (a real IP-unmasking leak) — passing both now raises `ValueError` instead
of silently picking one.
- **`Response.text(encoding=...)` no longer returns a stale cached decode.** A
second call with an explicit `encoding=` previously still returned the first
(possibly differently-encoded) cached decode; an explicit encoding now bypasses
the cache.
- **`preview()` now rewrites the url before resolving cookies**, matching what
`request()` actually sends — previously cookies were resolved against the
pre-rewrite host, which could miss or misattribute host-bound cookies.
- **`request_with_retries(attempts=0)` now floors to 1 attempt**, not silently
`DEFAULT_ATTEMPTS` (3). `attempts` is checked with `is None`, not truthiness.
### v0.1.8
- **`request()` no longer flattens `aiohttp.ClientError` subtypes.** Every failure
(connect errors, proxy errors, `raise_for_status()`-style response errors,
redirect limits, ...) was re-raised as a bare `aiohttp.ClientError`, losing the
real subtype and its attributes (`.os_error`, `.status`, `.headers`, ...) — a
direct caller doing `except ClientProxyConnectionError:` or `if e.status == 401`
would silently never match. Now the original exception is re-raised as-is (its
subtype, attributes, and `__cause__` all preserved). `request_with_retries`
still catches the base `aiohttp.ClientError` across attempts, so its behavior
(and its `FailureResponse` return on exhaustion) is unchanged.
- **`as_curl()` now renders `params` and `timeout`.** Previously a preview built
with `params=` silently omitted the query string (and a `timeout=` omitted
`--max-time`), so a `debug=True` cURL replay of a params-driven request hit a
different URL than the one actually sent. `params` are now merged into the url's
query string (via `yarl`) and `timeout` is emitted as `--max-time`.
### v0.1.7
- **`get_cookies()` now returns real cookies.** Previously called `filter_cookies()`
with no URL, which only ever returns domain-less shared cookies — every normal
domain-bound cookie (including ones set by a real `Set-Cookie` response) was
silently omitted. Now iterates the jar directly.
- **`set_cookie()` no longer leaks a shared cookie to every host.** A bare hostname
(`domain="example.com"`) built a schemeless URL, which aiohttp's jar treats as a
domain-less "shared" cookie sent with every request the session makes, including
unrelated hosts. A scheme is now added when missing so the cookie is scoped to
that host.
- **`set_cookie()` now honors `path`** (previously ignored — the cookie always
landed at `path="/"`). `domain=None` is unchanged in meaning but now stores a
truly shared cookie (sent to every host) instead of one silently bound to
`localhost` only, which made `set_cookie(name, value)` (no domain) a silent
no-op for any real request.
- **The backend session is built lazily**, not in `__init__`. Under aiohttp 3.14,
constructing `aiohttp.ClientSession` requires a running event loop; eager
construction crashed the common host pattern of attaching a session before the
loop starts (e.g. `bot.http = ExtendedSession(...)` in `Bot.__init__`). The
session (and any subclass's `_create_session` override) now builds on first
access instead.
### v0.1.2
- Pinned `commons` to v0.2.1 (retry `attempts` floor fix).
+2 -2
View File
@@ -4,13 +4,13 @@ build-backend = "hatchling.build"
[project]
name = "aioweb"
version = "0.1.6"
version = "1.1.0"
description = "Async HTTP session wrapper over aiohttp — proxies, header overwrites, retries, previews. Config-free, installable."
requires-python = ">=3.10"
dependencies = [
"aiohttp>=3.9",
"yarl>=1.9",
"commons @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v0.2.1",
"commons @ git+https://git.rethinkstudios.io/rethink-public/commons.git@v1.0.0",
]
[tool.hatch.metadata]
+9 -8
View File
@@ -1,7 +1,14 @@
from importlib.metadata import version, PackageNotFoundError
from .session import ExtendedSession, DEFAULT_ATTEMPTS, RETRY_STATUSES
from .responses import Response, FailureResponse, AiowebError, aiowebResponse
from .preview import RequestPreview
try:
__version__ = version("aioweb")
except PackageNotFoundError:
__version__ = "0.0.0+unknown"
__all__ = [
"ExtendedSession",
"Response",
@@ -17,17 +24,11 @@ __all__ = [
async def request_retries(session, method, url, **kwargs):
"""back-compat wrapper for session.request_with_retries(...)
prefer calling session.request_with_retries(...) directly.
"""
"""back-compat wrapper for session.request_with_retries(...); prefer calling that directly"""
return await session.request_with_retries(method, url, **kwargs)
async def test_proxies(session, url="https://api.ipify.org?format=json"):
"""fetch the public IP via the session (verifies proxy config)
url defaults to ipify; pass another IP-echo endpoint to point elsewhere.
"""
"""fetch the public IP via the session (verifies proxy config); url defaults to ipify"""
response = await session.request("GET", url)
return response.json()
+18 -11
View File
@@ -1,9 +1,12 @@
"""
request preview for aioweb format or export a request without sending it
request preview for aioweb - format or export a request without sending it
"""
import json as _json
import shlex
from urllib.parse import urlencode
from yarl import URL
class RequestPreview:
@@ -26,22 +29,26 @@ class RequestPreview:
return "\n".join(f"{key}: {value}" for key, value in self.details.items())
def as_curl(self):
"""equivalent cURL command for the request
every interpolated value is shell-quoted with shlex.quote, so headers,
body, url, or proxy containing quotes/spaces/metacharacters produce a
valid, non-injectable command rather than a broken or unsafe one.
"""
"""equivalent cURL command for the request (values shlex.quote'd; matches what request() sends)"""
parts = [f"curl -X {shlex.quote(self.details['method'])}"]
for header, value in (self.details["headers"] or {}).items():
parts.append(f"-H {shlex.quote(f'{header}: {value}')}")
if self.details["data"] is not None:
parts.append(f"--data {shlex.quote(str(self.details['data']))}")
data = self.details["data"]
# aiohttp form-encodes a dict data= body (application/x-www-form-urlencoded);
# render that wire form, not the python repr, so the curl replays identically.
# doseq=True mirrors aiohttp's FormData: a list value becomes repeated k=v pairs
rendered = urlencode(data, doseq=True) if isinstance(data, dict) else str(data)
parts.append(f"--data {shlex.quote(rendered)}")
elif self.details["json"] is not None:
# is-not-None, not truthiness: an empty-but-valid body ({} / []) must still
# render rather than being dropped as falsy
# is-not-None: an empty-but-valid body ({} / []) must still render
parts.append(f"--data {shlex.quote(_json.dumps(self.details['json']))}")
parts.append(shlex.quote(str(self.details["url"])))
url = self.details["url"]
if self.details["params"]:
url = str(URL(url).update_query(self.details["params"]))
parts.append(shlex.quote(str(url)))
if self.details["proxy"]:
parts.append(f"--proxy {shlex.quote(str(self.details['proxy']))}")
if self.details["timeout"] is not None:
parts.append(f"--max-time {shlex.quote(str(self.details['timeout']))}")
return " \\\n ".join(parts)
+13 -19
View File
@@ -1,10 +1,7 @@
"""
backend-agnostic response objects for aioweb
Response is built from primitives (status, headers, content, url, history) rather
than holding a raw aiohttp object, so any backend can produce one. FailureResponse
mirrors the same surface so callers can branch uniformly — every status/predicate is
a property on both.
backend-agnostic response objects for aioweb - Response and FailureResponse share
the same surface (every status/predicate a property on both) so callers can branch
uniformly regardless of which one they got.
"""
import json as _json
@@ -34,32 +31,26 @@ class Response:
@property
def status_code(self) -> int:
"""HTTP status code"""
return self._status_code
@property
def headers(self):
"""response headers"""
return self._headers
@property
def url(self) -> str:
"""final URL after redirects"""
return self._url
@property
def reason(self):
"""reason phrase for the status code"""
return self._reason
@property
def cookies(self):
"""cookies set in the response"""
return self._cookies
@property
def history(self):
"""redirect history (list of (status, url) tuples)"""
return self._history
@property
@@ -69,23 +60,22 @@ class Response:
@property
def is_redirect(self) -> bool:
"""whether the status is a redirect"""
return self._status_code in (301, 302, 303, 307, 308)
@property
def is_success(self) -> bool:
"""whether the status indicates success (2xx)"""
return 200 <= self._status_code < 300
@property
def content(self) -> bytes:
"""raw response bytes"""
return self._content
def text(self, encoding: Optional[str] = None) -> str:
"""decoded text content (cached)"""
"""decoded text content, cached only for the default encoding"""
if encoding is not None:
return self._content.decode(encoding)
if self._text is None:
self._text = self._content.decode(encoding or self._encoding)
self._text = self._content.decode(self._encoding)
return self._text
def json(self):
@@ -94,7 +84,7 @@ class Response:
return _json.loads(self.text())
except (_json.JSONDecodeError, UnicodeDecodeError):
# text() decodes the body and can raise UnicodeDecodeError on a non-UTF-8
# payload that's a "not valid JSON" outcome, not an error to propagate
# payload - that's a "not valid JSON" outcome, not an error to propagate
return None
def raise_for_status(self):
@@ -143,6 +133,7 @@ class FailureResponse:
@property
def redirect_chain(self):
"""list of (status, url) tuples for all redirects"""
return []
@property
@@ -158,12 +149,15 @@ class FailureResponse:
return None
def text(self, encoding=None):
"""always None"""
return None
def json(self):
"""always None"""
return None
def raise_for_status(self):
"""raise AiowebError(reason)"""
raise AiowebError(self._reason)
def __bool__(self) -> bool:
@@ -173,5 +167,5 @@ class FailureResponse:
return f"<FailureResponse [{self._status_code}] {self._reason}>"
# back-compat alias the response class was renamed Response
# back-compat alias - the response class was renamed Response
aiowebResponse = Response
+238 -91
View File
@@ -1,25 +1,18 @@
"""
async HTTP session wrapper over aiohttp
ExtendedSession adds session-level proxies, header overwrites, ephemeral
(per-request generated) headers, domain rewriting, request previews, and
retry/backoff on top of aiohttp. The actual byte-sending is isolated in
_raw_request() so a different backend (e.g. a TLS-fingerprinting client) can
subclass and override just that one method, inheriting everything else.
async HTTP session wrapper over aiohttp - proxies, header overwrites, ephemeral
headers, domain rewriting, previews, retry/backoff. See README for usage and contract.
async with ExtendedSession(proxies={"https": "http://..."}) as s:
resp = await s.request_with_retries("GET", url)
if resp: # FailureResponse is falsy
data = resp.json()
config-free: proxies/headers/timeouts are passed at construction or per call.
sessions must be closed explicitly (async with, or await s.close()); there is no
__del__ auto-close (that pattern is unsafe for async resources).
"""
import asyncio
import ipaddress
import logging
import warnings
from http.cookies import SimpleCookie
import aiohttp
from yarl import URL
@@ -30,23 +23,35 @@ from .responses import Response, FailureResponse
def _route_body(data):
"""split a body into (data=, json=) kwargs
dict OR list bodies are valid JSON and route to json=; everything else
(str/bytes/form) routes to data=. previously only dicts went to json=, so a
JSON list was wrongly form-encoded.
"""
"""split a body into (data=, json=) kwargs; dict OR list routes to json=, else data="""
if isinstance(data, (dict, list)):
return None, data
return data, None
class _RetryStatus(Exception):
"""internal signal: a retryable HTTP status; carries the real Response
def _is_ip_host(host) -> bool:
"""whether host is a literal IPv4/IPv6 address rather than a hostname"""
if not host:
return False
try:
ipaddress.ip_address(host)
except ValueError:
return False
return True
raised inside an attempt so commons.aretry drives the backoff + cap; the caller
catches the final one to return the REAL last response, not a synthetic failure.
"""
def _is_ipv6_literal(host) -> bool:
"""whether host is a bare IPv6 literal (must be a whole-host swap, never a substring splice)"""
if not host:
return False
try:
return isinstance(ipaddress.ip_address(host), ipaddress.IPv6Address)
except ValueError:
return False
class _RetryStatus(Exception):
"""internal signal: a retryable HTTP status; carries the real Response through aretry"""
def __init__(self, response):
super().__init__(f"retryable status {response.status_code}")
@@ -57,8 +62,8 @@ log = logging.getLogger(__name__)
DEFAULT_ATTEMPTS = 3
DEFAULT_BACKOFF_BASE = 2.0
# statuses worth retrying: rate limit + transient server errors
RETRY_STATUSES = frozenset({429, 500, 502, 503, 504})
_HANG_GUARD_SLACK = 5
class ExtendedSession:
@@ -84,31 +89,43 @@ class ExtendedSession:
self.domain_overwrites = domain_overwrites or {}
self.ephemeral_headers = {}
self.proxies = proxies or {}
# track our own default headers instead of touching aiohttp privates
# own header layer, not aiohttp privates, so update_headers/clear_headers work
self._default_headers = dict(headers or {})
self.session = self._create_session(self._default_headers, timeout, **kwargs)
self._session_timeout = timeout
self._session_kwargs = kwargs
# aiohttp>=3.14 needs a running loop to build ClientSession; built lazily instead
self._session = None
@property
def session(self):
"""the backend session, built lazily on first access (needs a running loop)"""
self._ensure_session()
return self._session
def _ensure_session(self):
"""build the backend session on first use - idempotent"""
if self._session is None:
self._session = self._create_session(
self._default_headers, self._session_timeout, **self._session_kwargs,
)
def _create_session(self, headers, timeout, **kwargs):
"""create the backend HTTP session override to use a different client
"""create the backend HTTP session - override to swap HTTP clients
a subclass swapping the HTTP backend (e.g. a TLS-fingerprinting client)
overrides this to return its own session object. the overwrite/domain/
proxy/retry/preview logic in this class never touches the session object
directly (only _raw_request, the cookie methods, and close do), so those
features work unchanged on any backend.
`headers` isn't baked in here - aiohttp would copy it into an immutable map
update_headers/clear_headers can't touch; `_default_headers` is the mutable
layer request()/preview() merge per call instead.
`headers` is the session-default header set; the default aiohttp backend
does NOT bake it into the ClientSession (which would copy it into an
immutable per-session map that update_headers/clear_headers can't touch).
instead `_default_headers` is our own mutable layer that request() and
preview() merge per call, so the mutable session-header API actually works.
a backend that needs the defaults baked at construction may use `headers`.
no `sock_read` here - aiohttp re-arms that timer on every request dispatched
over a pooled protocol, including idle keep-alive connections, and firing it
calls `set_exception(SocketTimeoutError)` on the protocol, permanently
poisoning that pooled connection; the next request on it fails instantly
without contacting the server. `total` still bounds every request overall.
"""
return aiohttp.ClientSession(
timeout=aiohttp.ClientTimeout(
total=timeout,
connect=timeout / 2,
sock_read=timeout / 2,
sock_connect=timeout / 2,
),
**kwargs,
@@ -124,9 +141,8 @@ class ExtendedSession:
def _apply_overwrites(self, request_headers):
"""apply static overwrites and ephemeral headers to a request's headers"""
request_headers = dict(request_headers or {})
# snapshot the shared override dicts so a concurrent mutation (e.g. a command
# editing ephemerals on a shared session) can't raise "dict changed size during
# iteration" — the loop body is sync, but the snapshot is cheap insurance
# list()-snapshot so a concurrent mutation of the shared dicts can't raise
# "dict changed size during iteration"
for header, value in list(self.header_overwrites.items()):
if self.inject or header in request_headers:
request_headers[header] = value
@@ -181,16 +197,47 @@ class ExtendedSession:
# domain overwrites
def overwrite_domain(self, target, replacement):
"""register a host-substring rewrite (target -> replacement)"""
"""register a host rewrite (target -> replacement)
raises ValueError if the replacement is not a valid host for yarl's with_host()
(e.g. a 'host:port' string or a pre-bracketed '[::1]') - caught here rather than
deep in yarl at request time. a bare IPv6 literal ('::1', '2001:db8::1') is valid
and accepted: yarl brackets it itself.
a hostname/IPv4 replacement rewrites any host CONTAINING target (substring splice,
e.g. 'example.com' -> 'internal.example.com'). an IPv6 replacement can only ever be
a WHOLE-HOST swap - splicing '::1' into the middle of a name yields an invalid host
('internal::1') - so at request time an IPv6 replacement applies ONLY when target
equals the whole request host; a request whose host merely CONTAINS the target (a
substring) is left unrewritten instead of composing an invalid host and exploding
deep in yarl. see _apply_domain_overwrites.
"""
try:
URL("http://placeholder").with_host(replacement)
except ValueError as exc:
raise ValueError(
f"overwrite_domain replacement {replacement!r} is not a valid host: {exc}; "
"pass a bare hostname or IP (a bare IPv6 literal is fine), not a host:port or "
"bracketed form - rewrite a port via a full URL override, not domain_overwrites"
) from exc
self.domain_overwrites[target] = replacement
def _apply_domain_overwrites(self, url: str) -> str:
"""apply any host-substring rewrites to a url"""
"""apply any host rewrites to a url
a hostname/IPv4 replacement splices on a substring match; an IPv6 replacement only
applies on a whole-host (exact) match, since it can't be spliced mid-host - this pairs
with overwrite_domain's registration guard so an IPv6 rewrite never composes an
invalid host.
"""
parsed = URL(url)
if not parsed.host:
return url
for target, replacement in self.domain_overwrites.items():
if target in parsed.host:
if _is_ipv6_literal(replacement):
if target == parsed.host:
return str(parsed.with_host(replacement))
elif target in parsed.host:
return str(parsed.with_host(parsed.host.replace(target, replacement)))
return url
@@ -208,8 +255,13 @@ class ExtendedSession:
self.proxies.clear()
def _get_proxy(self, url, proxies=None):
"""resolve the proxy for a url's scheme"""
proxies = proxies or self.proxies
"""resolve the proxy for a url's scheme
proxies is checked with `is None`, not truthiness, so an explicit
proxies={} disables session proxies for that one call instead of
falling back to them.
"""
proxies = self.proxies if proxies is None else proxies
scheme = url.split("://")[0]
return proxies.get(scheme)
@@ -217,31 +269,89 @@ class ExtendedSession:
# cookies
def get_cookies(self):
"""cookies stored in the session jar"""
return self.session.cookie_jar.filter_cookies()
"""all cookies in the session jar, regardless of domain binding
iterates the jar directly rather than filter_cookies() (which needs a url and,
given none, returns only domain-less shared cookies - {} for any normal one).
returns {} off-loop when no session was built yet (nothing could be set), mirroring
preview() - reaching self.session would build it and need a running loop.
"""
if self._session is None:
return {}
return {c.key: c.value for c in self.session.cookie_jar}
def set_cookie(self, name, value, domain=None, path="/"):
"""set a cookie in the session jar"""
response_url = URL(domain or "http://localhost")
self.session.cookie_jar.update_cookies({name: value}, response_url=response_url)
"""set a cookie in the session jar
domain=None stores a truly shared cookie sent to every host. domain='example.com'
(scheme optional, defaulted to http://) scopes it to that host - a bare hostname
is normalized into a URL so the jar binds by host instead of silently storing
another domain-less shared cookie. `path` is honored via the morsel itself.
raises:
ValueError: if domain is a bare IP host (e.g. '127.0.0.1') and the jar is
the default aiohttp.CookieJar(unsafe=False), which drops cookies bound
to IP hosts with no store, no send, and no log - construct the session
with cookie_jar=aiohttp.CookieJar(unsafe=True) to allow IP-host cookies
"""
if self._session is None:
try:
self._ensure_session()
except RuntimeError as error:
raise RuntimeError(
"set_cookie needs the backend cookie jar, which builds on first use and "
"requires a running event loop; call it inside an async context (unlike "
"get_cookies/clear_cookies, which no-op off-loop)"
) from error
cookie = SimpleCookie()
cookie[name] = value
cookie[name]["path"] = path
if domain is None:
self.session.cookie_jar.update_cookies(cookie)
return
if "://" not in domain:
domain = "http://" + domain
url = URL(domain)
jar = self.session.cookie_jar
if _is_ip_host(url.raw_host) and not getattr(jar, "unsafe", False):
raise ValueError(
f"set_cookie domain {domain!r} resolves to a bare IP host, but "
"aiohttp's cookie jar rejects IP-address domains under unsafe=False "
"(the default) - the cookie would be silently dropped; construct "
"ExtendedSession with cookie_jar=aiohttp.CookieJar(unsafe=True) to "
"allow IP-host cookies"
)
jar.update_cookies(cookie, response_url=url)
def clear_cookies(self):
"""clear the session cookie jar"""
"""clear the session cookie jar
a no-op off-loop when no session was built yet (an unbuilt jar is already empty),
so callers can clear before the loop starts without a spurious session build.
"""
if self._session is None:
return
self.session.cookie_jar.clear()
def _cookies_for_url(self, url):
"""dict of cookies the backend would send for url override per backend
used by preview(). non-aiohttp backends override this (or return {}); the
rest of preview (domain rewrites, header overwrites) is backend-agnostic.
"""
"""dict of cookies the backend would send for url - override per backend (used by preview())"""
return {k: v.value for k, v in self.session.cookie_jar.filter_cookies(URL(url)).items()}
# -------------------------------------------------------------------------
# preview
def preview(self, method, url, **kwargs):
"""build a RequestPreview for a request without sending it"""
"""build a RequestPreview for a request without sending it - synchronous, no loop required
url is domain-rewritten before cookies are resolved, matching the real
request - cookie binding is host-based, so resolving against the pre-rewrite
host could miss or misattribute cookies. the default cookie lookup never
touches the backend session: building it needs a running loop (aiohttp>=3.14),
which would defeat preview()'s own pre-loop inspection use case, and an unbuilt
session has no cookies to report anyway - pass cookies= explicitly to preview
cookies that would come from a not-yet-built session.
"""
url = self._apply_domain_overwrites(url)
proxy = self._get_proxy(url, kwargs.pop("proxies", None))
merged = {**self._default_headers, **(kwargs.pop("headers", None) or {})}
headers = self._apply_overwrites(merged)
@@ -250,7 +360,7 @@ class ExtendedSession:
timeout_total = timeout if isinstance(timeout, (int, float)) else None
cookies = kwargs.pop("cookies", None)
if cookies is None:
if cookies is None and self._session is not None:
cookies = self._cookies_for_url(url)
if cookies:
cookie_header = "; ".join(f"{k}={v}" for k, v in cookies.items())
@@ -260,7 +370,7 @@ class ExtendedSession:
return RequestPreview(
method=method,
url=self._apply_domain_overwrites(url),
url=url,
headers=headers,
proxy=proxy,
data=kwargs.get("data"),
@@ -275,10 +385,9 @@ class ExtendedSession:
async def _raw_request(self, method, url, **kwargs) -> Response:
"""send one request with aiohttp and adapt it into a Response
this is the backend seam: a subclass can override only this method (using a
different HTTP client, e.g. a TLS-fingerprinting one), building a Response
from that backend's primitives. everything else (overwrites, retries,
preview) is inherited.
the backend seam: a subclass overrides only this to swap HTTP clients,
building a Response from that backend's primitives; everything else
(overwrites, retries, preview) is inherited.
"""
response = await self.session.request(method, url, **kwargs)
async with response:
@@ -295,8 +404,21 @@ class ExtendedSession:
)
async def request(self, method, url, **kwargs) -> Response:
"""make a request, applying overwrites/domain rewrites/proxy resolution"""
kwargs["proxy"] = self._get_proxy(url, kwargs.pop("proxies", None))
"""make a request, applying overwrites/domain rewrites/proxy resolution
raises the real exception subtype on failure (ServerTimeoutError on a total
timeout; any other aiohttp.ClientError re-raised as-is, not flattened, so
callers can branch by type or read subtype attributes like .status/.headers).
a native proxy= is never silently clobbered by resolved proxies= (that would
unmask the caller's real IP) - pass only one of them.
"""
resolved_proxy = self._get_proxy(url, kwargs.pop("proxies", None))
if kwargs.get("proxy") is not None and resolved_proxy is not None:
raise ValueError("pass only one of proxy= or proxies= (session/per-call), not both")
# a bare proxy=None must not shadow a resolved proxies= (that would send direct and
# unmask the caller's IP); setdefault can't fix it because the None key already exists
if kwargs.get("proxy") is None:
kwargs["proxy"] = resolved_proxy
debug = kwargs.pop("debug", False)
merged = {**self._default_headers, **(kwargs.get("headers") or {})}
@@ -304,13 +426,23 @@ class ExtendedSession:
kwargs["headers"] = {str(k): str(v) for k, v in kwargs["headers"].items()}
timeout = kwargs.get("timeout")
# the hang-guard deadline, captured as a bare number BEFORE the ClientTimeout
# conversion below consumes it: a per-call numeric timeout, else the session
# default. a timeout=None (omitted, or the explicit None request_with_retries
# passes) means "use the session bound" - so it guards on self._session_timeout,
# NOT unguarded. the ONLY unguarded case is a session deliberately constructed
# unbounded (Session(timeout=None) -> self._session_timeout is None): the
# documented opt-out for a genuinely unbounded call (long-poll/streaming).
if isinstance(timeout, (int, float)):
guard_secs = timeout
else:
guard_secs = self._session_timeout
if isinstance(timeout, (int, float)):
kwargs["timeout"] = aiohttp.ClientTimeout(total=timeout)
elif timeout is None and "timeout" in kwargs:
# an explicit timeout=None reaches aiohttp as ClientTimeout(total=None),
# which DISABLES the timeout and overrides the session default; drop it so
# the session-level timeout applies (matters for request_with_retries, whose
# timeout kwarg defaults to None)
# explicit timeout=None would reach aiohttp as ClientTimeout(total=None) -
# infinite, disabling the session default - so drop it instead (matters for
# request_with_retries, whose timeout kwarg defaults to None)
del kwargs["timeout"]
url = self._apply_domain_overwrites(url)
@@ -318,18 +450,31 @@ class ExtendedSession:
log.info("sending request to: %s", url)
try:
result = await self._raw_request(method, url, **kwargs)
# hang-guard: an outer asyncio deadline (session/curl timers can die in the
# same failure that parks the transfer - a backend that loses both wakeup and
# its own timeout enforcement would otherwise leave this await parked forever).
# the backend's own timeout still fires first in every healthy failure (slack,
# not replacement); wait_for only trips when the backend's enforcement is dead,
# and its cancellation is what unwedges the orphaned transfer.
if guard_secs is not None:
result = await asyncio.wait_for(
self._raw_request(method, url, **kwargs), guard_secs + _HANG_GUARD_SLACK
)
else:
result = await self._raw_request(method, url, **kwargs)
if debug and result.redirect_chain:
log.info("redirect chain: %s", result.redirect_chain)
return result
except asyncio.TimeoutError as error:
# a total ClientTimeout raises a bare asyncio.TimeoutError, which is NOT an
# aiohttp.ClientError subclass — wrap it as ServerTimeoutError (which IS both
# a ClientError AND a TimeoutError) so direct callers get a typed failure and
# request_with_retries can still label it a timeout
# not an aiohttp.ClientError subclass - wrap as ServerTimeoutError so callers
# get a typed failure and request_with_retries can label it a timeout. now also
# catches the hang-guard firing (asyncio.wait_for raises asyncio.TimeoutError).
raise aiohttp.ServerTimeoutError(f"timeout for {url}: {error}") from error
except aiohttp.ClientError as error:
raise aiohttp.ClientError(f"client error for {url}: {error}") from error
# re-raise the original subtype (not flattened) - request_with_retries still
# catches the base aiohttp.ClientError below and is unaffected
log.error("client error for %s: %s", url, error)
raise
async def request_with_retries(
self, method, url, *, data=None, proxies=None, timeout=None, attempts=None,
@@ -340,13 +485,11 @@ class ExtendedSession:
returns a Response on success (or non-retryable status), or a falsy
FailureResponse if every attempt fails. backoff is exponential
(backoff_base ** attempt).
timeout defaults to None, which falls back to the session-level timeout set
at construction (aioweb pops a None timeout so it does not reach aiohttp as an
infinite ClientTimeout); pass a number to override per call.
(backoff_base ** attempt). timeout=None falls back to the session-level
timeout (see request()'s note on why explicit None is dropped, not passed
through). attempts=0 floors to 1 (via commons.aretry), not DEFAULT_ATTEMPTS.
"""
attempts = attempts or DEFAULT_ATTEMPTS
attempts = DEFAULT_ATTEMPTS if attempts is None else attempts
body_data, body_json = _route_body(data)
if debug:
@@ -378,9 +521,7 @@ class ExtendedSession:
attempts, url, exhausted.response.status_code)
return exhausted.response
except asyncio.TimeoutError:
# request() wraps a total timeout as ServerTimeoutError (a ClientError AND a
# TimeoutError); catch the timeout case first so it's labeled a timeout rather
# than falling into the generic client-error branch below
# catch before ClientError so a timeout is labeled as such, not generic
log.error("all %d attempts timed out for %s", attempts, url)
return FailureResponse(reason="timeout", url=url)
except aiohttp.ClientError as error:
@@ -394,12 +535,18 @@ class ExtendedSession:
# lifecycle
async def close(self):
"""close the backend session override if the backend's close differs"""
await self.session.close()
"""close the backend session - override if the backend's close differs; no-op if never built"""
if self._session is not None:
await self._session.close()
def _is_closed(self) -> bool:
"""whether the backend session is closed override for non-aiohttp backends"""
return self.session.closed
"""whether the backend session is closed - override for non-aiohttp backends
an unbuilt session counts as closed: nothing was opened, nothing to leak.
"""
if self._session is None:
return True
return self._session.closed
async def __aenter__(self):
return self
@@ -408,8 +555,8 @@ class ExtendedSession:
await self.close()
def __del__(self):
# do NOT attempt async cleanup here spinning event loops in a finalizer
# is unsafe. just warn so the leak is visible; callers must close explicitly.
# no async cleanup here - spinning event loops in a finalizer is unsafe;
# just warn so the leak is visible, callers must close explicitly
try:
closed = self._is_closed()
except Exception: