23 Commits
Author SHA1 Message Date
dsql 6633d6cba3 fix: pin inter-lib dependencies to their v1.0.0 tags
the v1.0.0 release still pinned pre-1.0.0 sibling tags, so a fresh install dragged in
stale transitive deps. update the pin(s) to the current v1.0.x release and bump this lib
to 1.0.1 so the corrected dependency chain ships under a new tag (v1.0.0 left intact).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-17 17:46:10 -04:00
dsql 1deb05ce4a release: 1.0.0
first stable release. pre-1.0.0 verification complete: all surviving MED regressions and
gaps resolved and independently re-fired, tree audited clean across the suite.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-09 18:53:15 -04:00
dsql 45d8830833 fix: as_curl form-encodes a dict data= body with doseq=True
urlencode(data) without doseq rendered a list-valued form field as its urlencoded python
repr (x=%5B%271%27...) instead of aiohttp FormData's repeated-pair wire form (x=1&x=2), so
a debug curl for a list-valued body replayed a different request than request() sent. add
doseq=True to match the wire. preview/debug-only; scalar values and non-dict bodies are
byte-unchanged.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 21:16:08 -04:00
dsql eed607b08c fix: an IPv6 domain overwrite applies only on a whole-host match, never a substring splice
overwrite_domain accepted a bare IPv6 replacement, but _apply_domain_overwrites spliced it
as a substring into any host CONTAINING the target, composing an invalid host ('internal::1'
from '.local'->'::1' on internal.local) that exploded deep in yarl at request time. An IPv6
replacement now applies ONLY when the target equals the whole request host; a substring match
is left unrewritten instead of building an invalid host. Hostname/IPv4 substring splices are
unchanged. (Registration can't distinguish whole-host from substring for a hostname target
without the request host, so the guard lives at compose time where the host is known.)

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:56:52 -04:00
dsql ed773e2c1b docs+fix: as_curl form-encodes a dict data= body; README notes the commons dependency
as_curl() rendered a dict data= body as its Python repr ({'a': 1}), but aiohttp
form-encodes a dict data= as application/x-www-form-urlencoded (a=1&b=two), so the emitted
curl replayed a different body - now urlencode()s a dict (str bodies unchanged). README
install section omitted the hard sibling commons dependency (a private git+ssh package).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:37:09 -04:00
dsql 85735023f5 fix: proxy=None no longer drops an explicit proxies= (IP unmask); cookie views work off-loop
request(proxy=None, proxies={...}) silently sent direct because setdefault('proxy', ...)
is a no-op when the None key already exists, unmasking the caller's real IP - now a bare
proxy=None is replaced by the resolved proxies= (the both-set guard still rejects two real
proxies). get_cookies/clear_cookies return {}/no-op when no session was built yet (mirroring
preview) instead of forcing a build that needs a running loop; set_cookie, which genuinely
needs the jar, now raises a clear actionable error off-loop instead of aiohttp's opaque one.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:23:31 -04:00
dsql 51c421c9a2 fix: overwrite_domain accepts a bare IPv6 replacement instead of rejecting any ':'
the guard rejected any replacement containing ':', which over-rejected a valid bare IPv6
literal host ('::1', '2001:db8::1') that yarl's URL.with_host() accepts and brackets. it now
delegates the check to yarl (try with_host, catch ValueError), so a bare IPv6 is accepted
(rewrites to http://[::1]/...) while a genuine host:port or pre-bracketed form still raises.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 16:47:35 -04:00
dsql 59eadabe16 fix: fail loud on unreachable domain overwrites and jar-dropped IP cookies
overwrite_domain() now rejects a port-bearing replacement immediately at
registration time with a clear ValueError, instead of only surfacing a
deep yarl error from inside with_host() the first time a matching request
is made, far from the misconfiguration site.

set_cookie() now raises ValueError when domain resolves to a bare IP host
and the jar is the default CookieJar(unsafe=False), which silently drops
IP-bound cookies with no store, no send, and no log. Non-IP domains and a
caller-supplied unsafe=True jar are unaffected.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 00:09:37 -04:00
dsql 78dfba0962 fix: drop session-default sock_read that poisons pooled keep-alive connections
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 19:05:51 -04:00
dsql a7302a6356 refactor: derive __version__ from package metadata (single source)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 17:00:19 -04:00
dsql 397d74efe3 docs: restore FailureResponse public-method docstrings stripped in de-bloat
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:46:03 -04:00
dsql 7f73f21c93 fix: preview() works without a running loop; set_cookie honors path when domain=None
preview()'s default cookie lookup reached the backend session and could force-
build it, raising RuntimeError('no running event loop') off-loop and defeating
the v0.1.7 lazy-session pre-loop use case; it's now skipped when the session
isn't built yet. set_cookie(domain=None) returned before setting the morsel's
path, so a shared cookie always stored path='/' regardless of the path= arg;
the morsel path is now set before that early return.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:17:46 -04:00
dsql 27f0e49341 docs: compress prose/module docstrings, em-dash->hyphen (de-bloat wave 1)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 00:15:24 -04:00
dsql ca23099e06 fix: proxy/attempts truthiness bugs, stale text() cache, preview cookie order (v0.1.9)
_get_proxy() and request() checked proxies/proxy with truthiness instead of
is None, so a per-call proxies={} silently fell back to session proxies, and a
native proxy= kwarg was clobbered by resolved session proxies (unmasking the
caller's real IP) instead of raising on conflicting sources. Response.text()
cached the first decode regardless of a later explicit encoding= argument.
preview() computed the Cookie header from the pre-rewrite url instead of the
rewritten one actually used by request(). request_with_retries(attempts=0)
became DEFAULT_ATTEMPTS via truthiness instead of flooring to 1.

Also compresses docstrings/comments across the module (no behavior change).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 23:25:43 -04:00
dsql 0768d643b1 fix: preserve exception subtype in request(), render params/timeout in as_curl()
request() re-raised every aiohttp.ClientError subclass as the bare base class,
losing ClientConnectorError/ClientProxyConnectionError/ClientResponseError
(.status/.headers)/TooManyRedirects and their attributes; direct callers
branching by type never matched. Now the original exception is bare-raised,
preserving subtype, attributes, and __cause__. request_with_retries (which
catches the base ClientError) is unaffected.

as_curl() silently dropped params and timeout, so a debug=True replay of a
params-driven request hit a different URL than the one actually sent. params
are now merged into the URL via yarl before quoting, and timeout renders as
--max-time.

Bumps 0.1.7 -> 0.1.8.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 16:57:04 -04:00
dsql e1ab5d38a0 fix: cookie methods actually work; session builds lazily (v0.1.7)
get_cookies() called filter_cookies() with no URL and always returned {}
for domain-bound cookies; now iterates the jar directly. set_cookie()
ignored path and leaked a shared cookie to every host when given a bare
domain string; scheme is now normalized and path honored, with
domain=None made an intentionally shared cookie instead of a silent
localhost-only no-op.

ExtendedSession also built its aiohttp.ClientSession synchronously in
__init__, which requires a running event loop under aiohttp>=3.14 and
crashed the common construct-before-the-loop-starts host pattern. The
session now builds lazily on first access, preserving the
_create_session subclass override seam used by aioweb_tls.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 16:42:30 -04:00
dsql b8cd184c64 fix: request_with_retries honors session timeout (timeout=None no longer disables it)
request() forwarded an explicit timeout=None to aiohttp as ClientTimeout(total=None),
which disables the timeout and overrides the session default. request_with_retries
defaults timeout=None, so its flagship path had zero timeout protection: a hung server
stalled the coroutine forever, pinning connector-pool slots. Pop a None timeout in
request() so the session-level timeout applies; numeric per-call timeouts still wrap.

aioweb-1 (v0.1.6).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 16:33:28 -04:00
dsql 74ed83cf73 chore: ignore .claude/ dir (CLAUDE.md now lives under .claude/)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:55:13 -04:00
dsql 14a3ee1456 fix: AW-2 json() returns None on a non-UTF-8 body instead of raising
responses.json() catches UnicodeDecodeError alongside JSONDecodeError — text() can raise
it on a non-UTF-8 payload, which is a 'not valid JSON' outcome per the docstring, not an
error to propagate.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:34:37 -04:00
dsql 3737af0cf5 fix: total-timeout labeled 'timeout' in request_with_retries (dead branch live) (v0.1.5)
AW-1: request() wraps a total ClientTimeout's bare asyncio.TimeoutError before
request_with_retries sees it, so the dedicated 'timeout' branch was dead and its comment
lied. wrap it as aiohttp.ServerTimeoutError (which IS both a ClientError AND a
TimeoutError) so direct request() callers still get a typed failure (M1 preserved) while
request_with_retries catches the timeout case first and labels it 'timeout'.

verified by execution: request() raises ServerTimeoutError (typed, M1 intact);
request_with_retries returns reason='timeout'; control confirms a real client error still
labels 'client error'. sibling-grep: aioweb_tls/aiowebhooks catch ClientError/TimeoutError,
both of which ServerTimeoutError satisfies — no consumer break.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 20:47:55 -04:00
dsql d3f2bed7fe docs: pin install line to release, note unpinned-latest option
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 18:13:34 -04:00
dsql 849200985c docs: show unpinned install line; note tag-pinning for reproducibility
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 18:07:19 -04:00
dsql 7da06443c8 fix: label timeouts, render empty-but-valid preview bodies, snapshot override dicts (v0.1.4)
- request_with_retries labels an exhausted total-timeout as 'timeout' instead of the
  generic 'unexpected error' catch-all (nit)
- as_curl() renders an empty-but-valid json body ({} / []) via is-not-None instead of
  dropping it as falsy (nit)
- _apply_overwrites snapshots the shared override dicts before iterating, so a
  concurrent mutation can't raise 'dict changed size during iteration' (nit).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 17:57:54 -04:00
7 changed files with 396 additions and 124 deletions
+1 -1
View File
@@ -1,5 +1,5 @@
# claude # claude
CLAUDE.md .claude/
# python # python
__pycache__/ __pycache__/
+126 -5
View File
@@ -11,16 +11,19 @@ and swap the HTTP client while inheriting everything else.
`requirements.txt`: `requirements.txt`:
``` ```
aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.3 aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v1.0.1
``` ```
Direct: Direct:
```bash ```bash
pip install "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.3" pip install "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v1.0.1"
``` ```
Requires `aiohttp` and `yarl` (pulled transitively). Requires `aiohttp` and `yarl` (pulled transitively), plus the sibling `commons` library
(a private `git+ssh` package — the install needs access to the `rethink-public` org).
Drop the `@v1.0.1` suffix from the line above to install the latest unpinned.
## Usage ## Usage
@@ -63,6 +66,15 @@ resp = await s.request_with_retries(
Returns a `FailureResponse` (falsy) if every attempt fails. Returns a `FailureResponse` (falsy) if every attempt fails.
`request()` (the non-retrying call) raises on failure: a total timeout raises
`aiohttp.ServerTimeoutError`, and any other network/protocol failure raises its real
`aiohttp.ClientError` subtype as-is (`ClientConnectorError`, `ClientProxyConnectionError`,
`ClientResponseError` with `.status`/`.headers`, `TooManyRedirects`, ...) — it is not
flattened into the base `ClientError`, so direct callers can branch by type or read
subtype attributes. `request_with_retries` catches the base `aiohttp.ClientError` (and
`asyncio.TimeoutError`) across all attempts and returns a falsy `FailureResponse` instead
of raising.
## Header overwrites & ephemeral headers ## Header overwrites & ephemeral headers
```python ```python
@@ -86,7 +98,15 @@ s.overwrite_domain("internal.local", "127.0.0.1") # host-substring rewrite
print(s.preview("POST", url, json={"a": 1}).as_curl()) # equivalent cURL command print(s.preview("POST", url, json={"a": 1}).as_curl()) # equivalent cURL command
``` ```
Pass `debug=True` to `request_with_retries` to log the cURL preview and request flow. `preview()` is synchronous and never touches the backend session — it works before any
event loop is running (e.g. building a preview at import/setup time), not just inside
`asyncio.run()`. Its default cookie lookup only reads cookies from an already-built
session; if the session hasn't been built yet there are no cookies to read, so it
returns none by default (pass `cookies=` explicitly to preview cookies for a session
that hasn't sent a request yet). `as_curl()` renders `params` (merged into the url's
query string) and `timeout` (as `--max-time`) as well as headers/body/proxy, so the
emitted command is faithful to what `request()` actually sends. Pass `debug=True` to
`request_with_retries` to log the cURL preview and request flow.
## Custom backends ## Custom backends
@@ -130,6 +150,107 @@ Two changes can't be shimmed without re-introducing the bugs they fix:
## Changelog ## Changelog
### v0.1.13
- **Session-default timeout no longer poisons pooled keep-alive connections.**
The default `ClientTimeout` set `sock_read=timeout/2` alongside `total`. Under
`aiohttp>=3.14`, that read timer re-arms on every request dispatched over a
pooled protocol, including idle connections between requests; when it fires
it permanently poisons the pooled connection (`SocketTimeoutError` on the next
use, instantly, without contacting the server) — a real error from the server
(e.g. a 503) could come back as a client-side `FailureResponse(status=0,
reason='timeout')` instead. `sock_read` is now dropped from the session
default; `total` (and `connect`/`sock_connect`) still bound every request, and
the per-call `timeout=N` path (`ClientTimeout(total=N)`, no `sock_read`) was
already unaffected.
### v0.1.12
- **Docstring-only.** Restored one-line docstrings on `FailureResponse`'s
`redirect_chain`, `text()`, `json()`, and `raise_for_status()` — a prior
de-bloat pass stripped them below the public tier while the `Response`
twins kept theirs. No behavior change.
### v0.1.11
- **`preview()` no longer requires a running event loop.** Its default cookie
lookup (`cookies=` not passed) previously called `_cookies_for_url()`, which
reached the backend session and built it if absent — under aiohttp>=3.14 that
needs a running loop, so `preview()` raised `RuntimeError('no running event
loop')` when called before the loop starts, defeating its own pre-loop
build/inspect use case (v0.1.7). Now the default cookie lookup is skipped
entirely when the session hasn't been built yet (nothing could have been set
on a session that doesn't exist); `cookies={}` and in-loop calls are unaffected.
- **`set_cookie(domain=None)` now honors `path`.** The `domain=None` branch
returned right after `update_cookies()`, before the line that sets the
morsel's `path` — so a shared cookie (`set_cookie(name, value, path="/api")`,
no `domain=`) always stored the `SimpleCookie` default `path="/"` instead. The
morsel's `path` is now set before the `domain=None` early return. The
domain-bound branch was already correct and is unchanged.
### v0.1.10
- Docs-only pass: compressed module/method docstrings and comments that restated
README/CLAUDE prose, replaced em-dashes with hyphens. No behavior change.
### v0.1.9
- **`_get_proxy()`/`request()` proxy resolution now checks `is None`, not
truthiness.** A per-call `proxies={}` previously fell back to the session's
configured proxies instead of disabling them for that call. `request()` also no
longer clobbers a native `proxy=` kwarg with the resolved session/`proxies=`
value (a real IP-unmasking leak) — passing both now raises `ValueError` instead
of silently picking one.
- **`Response.text(encoding=...)` no longer returns a stale cached decode.** A
second call with an explicit `encoding=` previously still returned the first
(possibly differently-encoded) cached decode; an explicit encoding now bypasses
the cache.
- **`preview()` now rewrites the url before resolving cookies**, matching what
`request()` actually sends — previously cookies were resolved against the
pre-rewrite host, which could miss or misattribute host-bound cookies.
- **`request_with_retries(attempts=0)` now floors to 1 attempt**, not silently
`DEFAULT_ATTEMPTS` (3). `attempts` is checked with `is None`, not truthiness.
### v0.1.8
- **`request()` no longer flattens `aiohttp.ClientError` subtypes.** Every failure
(connect errors, proxy errors, `raise_for_status()`-style response errors,
redirect limits, ...) was re-raised as a bare `aiohttp.ClientError`, losing the
real subtype and its attributes (`.os_error`, `.status`, `.headers`, ...) — a
direct caller doing `except ClientProxyConnectionError:` or `if e.status == 401`
would silently never match. Now the original exception is re-raised as-is (its
subtype, attributes, and `__cause__` all preserved). `request_with_retries`
still catches the base `aiohttp.ClientError` across attempts, so its behavior
(and its `FailureResponse` return on exhaustion) is unchanged.
- **`as_curl()` now renders `params` and `timeout`.** Previously a preview built
with `params=` silently omitted the query string (and a `timeout=` omitted
`--max-time`), so a `debug=True` cURL replay of a params-driven request hit a
different URL than the one actually sent. `params` are now merged into the url's
query string (via `yarl`) and `timeout` is emitted as `--max-time`.
### v0.1.7
- **`get_cookies()` now returns real cookies.** Previously called `filter_cookies()`
with no URL, which only ever returns domain-less shared cookies — every normal
domain-bound cookie (including ones set by a real `Set-Cookie` response) was
silently omitted. Now iterates the jar directly.
- **`set_cookie()` no longer leaks a shared cookie to every host.** A bare hostname
(`domain="example.com"`) built a schemeless URL, which aiohttp's jar treats as a
domain-less "shared" cookie sent with every request the session makes, including
unrelated hosts. A scheme is now added when missing so the cookie is scoped to
that host.
- **`set_cookie()` now honors `path`** (previously ignored — the cookie always
landed at `path="/"`). `domain=None` is unchanged in meaning but now stores a
truly shared cookie (sent to every host) instead of one silently bound to
`localhost` only, which made `set_cookie(name, value)` (no domain) a silent
no-op for any real request.
- **The backend session is built lazily**, not in `__init__`. Under aiohttp 3.14,
constructing `aiohttp.ClientSession` requires a running event loop; eager
construction crashed the common host pattern of attaching a session before the
loop starts (e.g. `bot.http = ExtendedSession(...)` in `Bot.__init__`). The
session (and any subclass's `_create_session` override) now builds on first
access instead.
### v0.1.2 ### v0.1.2
- Pinned `commons` to v0.2.1 (retry `attempts` floor fix). - Pinned `commons` to v0.2.1 (retry `attempts` floor fix).
@@ -147,4 +268,4 @@ Two changes can't be shimmed without re-introducing the bugs they fix:
## Versioning ## Versioning
Tagged `vX.Y.Z`. Pin the tag in `requirements.txt`. Releases are tagged `vX.Y.Z`. The install line above pins a release; drop the `@vX.Y.Z` suffix to install the latest unpinned. Pin deliberately for reproducible installs.
+2 -2
View File
@@ -4,13 +4,13 @@ build-backend = "hatchling.build"
[project] [project]
name = "aioweb" name = "aioweb"
version = "0.1.3" version = "1.0.1"
description = "Async HTTP session wrapper over aiohttp — proxies, header overwrites, retries, previews. Config-free, installable." description = "Async HTTP session wrapper over aiohttp — proxies, header overwrites, retries, previews. Config-free, installable."
requires-python = ">=3.10" requires-python = ">=3.10"
dependencies = [ dependencies = [
"aiohttp>=3.9", "aiohttp>=3.9",
"yarl>=1.9", "yarl>=1.9",
"commons @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v0.2.1", "commons @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v1.0.0",
] ]
[tool.hatch.metadata] [tool.hatch.metadata]
+9 -8
View File
@@ -1,7 +1,14 @@
from importlib.metadata import version, PackageNotFoundError
from .session import ExtendedSession, DEFAULT_ATTEMPTS, RETRY_STATUSES from .session import ExtendedSession, DEFAULT_ATTEMPTS, RETRY_STATUSES
from .responses import Response, FailureResponse, AiowebError, aiowebResponse from .responses import Response, FailureResponse, AiowebError, aiowebResponse
from .preview import RequestPreview from .preview import RequestPreview
try:
__version__ = version("aioweb")
except PackageNotFoundError:
__version__ = "0.0.0+unknown"
__all__ = [ __all__ = [
"ExtendedSession", "ExtendedSession",
"Response", "Response",
@@ -17,17 +24,11 @@ __all__ = [
async def request_retries(session, method, url, **kwargs): async def request_retries(session, method, url, **kwargs):
"""back-compat wrapper for session.request_with_retries(...) """back-compat wrapper for session.request_with_retries(...); prefer calling that directly"""
prefer calling session.request_with_retries(...) directly.
"""
return await session.request_with_retries(method, url, **kwargs) return await session.request_with_retries(method, url, **kwargs)
async def test_proxies(session, url="https://api.ipify.org?format=json"): async def test_proxies(session, url="https://api.ipify.org?format=json"):
"""fetch the public IP via the session (verifies proxy config) """fetch the public IP via the session (verifies proxy config); url defaults to ipify"""
url defaults to ipify; pass another IP-echo endpoint to point elsewhere.
"""
response = await session.request("GET", url) response = await session.request("GET", url)
return response.json() return response.json()
+20 -11
View File
@@ -1,9 +1,12 @@
""" """
request preview for aioweb format or export a request without sending it request preview for aioweb - format or export a request without sending it
""" """
import json as _json import json as _json
import shlex import shlex
from urllib.parse import urlencode
from yarl import URL
class RequestPreview: class RequestPreview:
@@ -26,20 +29,26 @@ class RequestPreview:
return "\n".join(f"{key}: {value}" for key, value in self.details.items()) return "\n".join(f"{key}: {value}" for key, value in self.details.items())
def as_curl(self): def as_curl(self):
"""equivalent cURL command for the request """equivalent cURL command for the request (values shlex.quote'd; matches what request() sends)"""
every interpolated value is shell-quoted with shlex.quote, so headers,
body, url, or proxy containing quotes/spaces/metacharacters produce a
valid, non-injectable command rather than a broken or unsafe one.
"""
parts = [f"curl -X {shlex.quote(self.details['method'])}"] parts = [f"curl -X {shlex.quote(self.details['method'])}"]
for header, value in (self.details["headers"] or {}).items(): for header, value in (self.details["headers"] or {}).items():
parts.append(f"-H {shlex.quote(f'{header}: {value}')}") parts.append(f"-H {shlex.quote(f'{header}: {value}')}")
if self.details["data"]: if self.details["data"] is not None:
parts.append(f"--data {shlex.quote(str(self.details['data']))}") data = self.details["data"]
elif self.details["json"]: # aiohttp form-encodes a dict data= body (application/x-www-form-urlencoded);
# render that wire form, not the python repr, so the curl replays identically.
# doseq=True mirrors aiohttp's FormData: a list value becomes repeated k=v pairs
rendered = urlencode(data, doseq=True) if isinstance(data, dict) else str(data)
parts.append(f"--data {shlex.quote(rendered)}")
elif self.details["json"] is not None:
# is-not-None: an empty-but-valid body ({} / []) must still render
parts.append(f"--data {shlex.quote(_json.dumps(self.details['json']))}") parts.append(f"--data {shlex.quote(_json.dumps(self.details['json']))}")
parts.append(shlex.quote(str(self.details["url"]))) url = self.details["url"]
if self.details["params"]:
url = str(URL(url).update_query(self.details["params"]))
parts.append(shlex.quote(str(url)))
if self.details["proxy"]: if self.details["proxy"]:
parts.append(f"--proxy {shlex.quote(str(self.details['proxy']))}") parts.append(f"--proxy {shlex.quote(str(self.details['proxy']))}")
if self.details["timeout"] is not None:
parts.append(f"--max-time {shlex.quote(str(self.details['timeout']))}")
return " \\\n ".join(parts) return " \\\n ".join(parts)
+15 -19
View File
@@ -1,10 +1,7 @@
""" """
backend-agnostic response objects for aioweb backend-agnostic response objects for aioweb - Response and FailureResponse share
the same surface (every status/predicate a property on both) so callers can branch
Response is built from primitives (status, headers, content, url, history) rather uniformly regardless of which one they got.
than holding a raw aiohttp object, so any backend can produce one. FailureResponse
mirrors the same surface so callers can branch uniformly — every status/predicate is
a property on both.
""" """
import json as _json import json as _json
@@ -34,32 +31,26 @@ class Response:
@property @property
def status_code(self) -> int: def status_code(self) -> int:
"""HTTP status code"""
return self._status_code return self._status_code
@property @property
def headers(self): def headers(self):
"""response headers"""
return self._headers return self._headers
@property @property
def url(self) -> str: def url(self) -> str:
"""final URL after redirects"""
return self._url return self._url
@property @property
def reason(self): def reason(self):
"""reason phrase for the status code"""
return self._reason return self._reason
@property @property
def cookies(self): def cookies(self):
"""cookies set in the response"""
return self._cookies return self._cookies
@property @property
def history(self): def history(self):
"""redirect history (list of (status, url) tuples)"""
return self._history return self._history
@property @property
@@ -69,30 +60,31 @@ class Response:
@property @property
def is_redirect(self) -> bool: def is_redirect(self) -> bool:
"""whether the status is a redirect"""
return self._status_code in (301, 302, 303, 307, 308) return self._status_code in (301, 302, 303, 307, 308)
@property @property
def is_success(self) -> bool: def is_success(self) -> bool:
"""whether the status indicates success (2xx)"""
return 200 <= self._status_code < 300 return 200 <= self._status_code < 300
@property @property
def content(self) -> bytes: def content(self) -> bytes:
"""raw response bytes"""
return self._content return self._content
def text(self, encoding: Optional[str] = None) -> str: def text(self, encoding: Optional[str] = None) -> str:
"""decoded text content (cached)""" """decoded text content, cached only for the default encoding"""
if encoding is not None:
return self._content.decode(encoding)
if self._text is None: if self._text is None:
self._text = self._content.decode(encoding or self._encoding) self._text = self._content.decode(self._encoding)
return self._text return self._text
def json(self): def json(self):
"""parsed JSON content, or None if not valid JSON""" """parsed JSON content, or None if not valid JSON"""
try: try:
return _json.loads(self.text()) return _json.loads(self.text())
except _json.JSONDecodeError: except (_json.JSONDecodeError, UnicodeDecodeError):
# text() decodes the body and can raise UnicodeDecodeError on a non-UTF-8
# payload - that's a "not valid JSON" outcome, not an error to propagate
return None return None
def raise_for_status(self): def raise_for_status(self):
@@ -141,6 +133,7 @@ class FailureResponse:
@property @property
def redirect_chain(self): def redirect_chain(self):
"""list of (status, url) tuples for all redirects"""
return [] return []
@property @property
@@ -156,12 +149,15 @@ class FailureResponse:
return None return None
def text(self, encoding=None): def text(self, encoding=None):
"""always None"""
return None return None
def json(self): def json(self):
"""always None"""
return None return None
def raise_for_status(self): def raise_for_status(self):
"""raise AiowebError(reason)"""
raise AiowebError(self._reason) raise AiowebError(self._reason)
def __bool__(self) -> bool: def __bool__(self) -> bool:
@@ -171,5 +167,5 @@ class FailureResponse:
return f"<FailureResponse [{self._status_code}] {self._reason}>" return f"<FailureResponse [{self._status_code}] {self._reason}>"
# back-compat alias the response class was renamed Response # back-compat alias - the response class was renamed Response
aiowebResponse = Response aiowebResponse = Response
+223 -78
View File
@@ -1,25 +1,18 @@
""" """
async HTTP session wrapper over aiohttp async HTTP session wrapper over aiohttp - proxies, header overwrites, ephemeral
headers, domain rewriting, previews, retry/backoff. See README for usage and contract.
ExtendedSession adds session-level proxies, header overwrites, ephemeral
(per-request generated) headers, domain rewriting, request previews, and
retry/backoff on top of aiohttp. The actual byte-sending is isolated in
_raw_request() so a different backend (e.g. a TLS-fingerprinting client) can
subclass and override just that one method, inheriting everything else.
async with ExtendedSession(proxies={"https": "http://..."}) as s: async with ExtendedSession(proxies={"https": "http://..."}) as s:
resp = await s.request_with_retries("GET", url) resp = await s.request_with_retries("GET", url)
if resp: # FailureResponse is falsy if resp: # FailureResponse is falsy
data = resp.json() data = resp.json()
config-free: proxies/headers/timeouts are passed at construction or per call.
sessions must be closed explicitly (async with, or await s.close()); there is no
__del__ auto-close (that pattern is unsafe for async resources).
""" """
import asyncio import asyncio
import ipaddress
import logging import logging
import warnings import warnings
from http.cookies import SimpleCookie
import aiohttp import aiohttp
from yarl import URL from yarl import URL
@@ -30,23 +23,35 @@ from .responses import Response, FailureResponse
def _route_body(data): def _route_body(data):
"""split a body into (data=, json=) kwargs """split a body into (data=, json=) kwargs; dict OR list routes to json=, else data="""
dict OR list bodies are valid JSON and route to json=; everything else
(str/bytes/form) routes to data=. previously only dicts went to json=, so a
JSON list was wrongly form-encoded.
"""
if isinstance(data, (dict, list)): if isinstance(data, (dict, list)):
return None, data return None, data
return data, None return data, None
class _RetryStatus(Exception): def _is_ip_host(host) -> bool:
"""internal signal: a retryable HTTP status; carries the real Response """whether host is a literal IPv4/IPv6 address rather than a hostname"""
if not host:
return False
try:
ipaddress.ip_address(host)
except ValueError:
return False
return True
raised inside an attempt so commons.aretry drives the backoff + cap; the caller
catches the final one to return the REAL last response, not a synthetic failure. def _is_ipv6_literal(host) -> bool:
""" """whether host is a bare IPv6 literal (must be a whole-host swap, never a substring splice)"""
if not host:
return False
try:
return isinstance(ipaddress.ip_address(host), ipaddress.IPv6Address)
except ValueError:
return False
class _RetryStatus(Exception):
"""internal signal: a retryable HTTP status; carries the real Response through aretry"""
def __init__(self, response): def __init__(self, response):
super().__init__(f"retryable status {response.status_code}") super().__init__(f"retryable status {response.status_code}")
@@ -57,7 +62,6 @@ log = logging.getLogger(__name__)
DEFAULT_ATTEMPTS = 3 DEFAULT_ATTEMPTS = 3
DEFAULT_BACKOFF_BASE = 2.0 DEFAULT_BACKOFF_BASE = 2.0
# statuses worth retrying: rate limit + transient server errors
RETRY_STATUSES = frozenset({429, 500, 502, 503, 504}) RETRY_STATUSES = frozenset({429, 500, 502, 503, 504})
@@ -84,31 +88,43 @@ class ExtendedSession:
self.domain_overwrites = domain_overwrites or {} self.domain_overwrites = domain_overwrites or {}
self.ephemeral_headers = {} self.ephemeral_headers = {}
self.proxies = proxies or {} self.proxies = proxies or {}
# track our own default headers instead of touching aiohttp privates # own header layer, not aiohttp privates, so update_headers/clear_headers work
self._default_headers = dict(headers or {}) self._default_headers = dict(headers or {})
self.session = self._create_session(self._default_headers, timeout, **kwargs) self._session_timeout = timeout
self._session_kwargs = kwargs
# aiohttp>=3.14 needs a running loop to build ClientSession; built lazily instead
self._session = None
@property
def session(self):
"""the backend session, built lazily on first access (needs a running loop)"""
self._ensure_session()
return self._session
def _ensure_session(self):
"""build the backend session on first use - idempotent"""
if self._session is None:
self._session = self._create_session(
self._default_headers, self._session_timeout, **self._session_kwargs,
)
def _create_session(self, headers, timeout, **kwargs): def _create_session(self, headers, timeout, **kwargs):
"""create the backend HTTP session override to use a different client """create the backend HTTP session - override to swap HTTP clients
a subclass swapping the HTTP backend (e.g. a TLS-fingerprinting client) `headers` isn't baked in here - aiohttp would copy it into an immutable map
overrides this to return its own session object. the overwrite/domain/ update_headers/clear_headers can't touch; `_default_headers` is the mutable
proxy/retry/preview logic in this class never touches the session object layer request()/preview() merge per call instead.
directly (only _raw_request, the cookie methods, and close do), so those
features work unchanged on any backend.
`headers` is the session-default header set; the default aiohttp backend no `sock_read` here - aiohttp re-arms that timer on every request dispatched
does NOT bake it into the ClientSession (which would copy it into an over a pooled protocol, including idle keep-alive connections, and firing it
immutable per-session map that update_headers/clear_headers can't touch). calls `set_exception(SocketTimeoutError)` on the protocol, permanently
instead `_default_headers` is our own mutable layer that request() and poisoning that pooled connection; the next request on it fails instantly
preview() merge per call, so the mutable session-header API actually works. without contacting the server. `total` still bounds every request overall.
a backend that needs the defaults baked at construction may use `headers`.
""" """
return aiohttp.ClientSession( return aiohttp.ClientSession(
timeout=aiohttp.ClientTimeout( timeout=aiohttp.ClientTimeout(
total=timeout, total=timeout,
connect=timeout / 2, connect=timeout / 2,
sock_read=timeout / 2,
sock_connect=timeout / 2, sock_connect=timeout / 2,
), ),
**kwargs, **kwargs,
@@ -124,10 +140,12 @@ class ExtendedSession:
def _apply_overwrites(self, request_headers): def _apply_overwrites(self, request_headers):
"""apply static overwrites and ephemeral headers to a request's headers""" """apply static overwrites and ephemeral headers to a request's headers"""
request_headers = dict(request_headers or {}) request_headers = dict(request_headers or {})
for header, value in self.header_overwrites.items(): # list()-snapshot so a concurrent mutation of the shared dicts can't raise
# "dict changed size during iteration"
for header, value in list(self.header_overwrites.items()):
if self.inject or header in request_headers: if self.inject or header in request_headers:
request_headers[header] = value request_headers[header] = value
for header, value_callable in self.ephemeral_headers.items(): for header, value_callable in list(self.ephemeral_headers.items()):
if self.inject or header in request_headers: if self.inject or header in request_headers:
value = value_callable() value = value_callable()
if isinstance(value, dict): if isinstance(value, dict):
@@ -178,16 +196,47 @@ class ExtendedSession:
# domain overwrites # domain overwrites
def overwrite_domain(self, target, replacement): def overwrite_domain(self, target, replacement):
"""register a host-substring rewrite (target -> replacement)""" """register a host rewrite (target -> replacement)
raises ValueError if the replacement is not a valid host for yarl's with_host()
(e.g. a 'host:port' string or a pre-bracketed '[::1]') - caught here rather than
deep in yarl at request time. a bare IPv6 literal ('::1', '2001:db8::1') is valid
and accepted: yarl brackets it itself.
a hostname/IPv4 replacement rewrites any host CONTAINING target (substring splice,
e.g. 'example.com' -> 'internal.example.com'). an IPv6 replacement can only ever be
a WHOLE-HOST swap - splicing '::1' into the middle of a name yields an invalid host
('internal::1') - so at request time an IPv6 replacement applies ONLY when target
equals the whole request host; a request whose host merely CONTAINS the target (a
substring) is left unrewritten instead of composing an invalid host and exploding
deep in yarl. see _apply_domain_overwrites.
"""
try:
URL("http://placeholder").with_host(replacement)
except ValueError as exc:
raise ValueError(
f"overwrite_domain replacement {replacement!r} is not a valid host: {exc}; "
"pass a bare hostname or IP (a bare IPv6 literal is fine), not a host:port or "
"bracketed form - rewrite a port via a full URL override, not domain_overwrites"
) from exc
self.domain_overwrites[target] = replacement self.domain_overwrites[target] = replacement
def _apply_domain_overwrites(self, url: str) -> str: def _apply_domain_overwrites(self, url: str) -> str:
"""apply any host-substring rewrites to a url""" """apply any host rewrites to a url
a hostname/IPv4 replacement splices on a substring match; an IPv6 replacement only
applies on a whole-host (exact) match, since it can't be spliced mid-host - this pairs
with overwrite_domain's registration guard so an IPv6 rewrite never composes an
invalid host.
"""
parsed = URL(url) parsed = URL(url)
if not parsed.host: if not parsed.host:
return url return url
for target, replacement in self.domain_overwrites.items(): for target, replacement in self.domain_overwrites.items():
if target in parsed.host: if _is_ipv6_literal(replacement):
if target == parsed.host:
return str(parsed.with_host(replacement))
elif target in parsed.host:
return str(parsed.with_host(parsed.host.replace(target, replacement))) return str(parsed.with_host(parsed.host.replace(target, replacement)))
return url return url
@@ -205,8 +254,13 @@ class ExtendedSession:
self.proxies.clear() self.proxies.clear()
def _get_proxy(self, url, proxies=None): def _get_proxy(self, url, proxies=None):
"""resolve the proxy for a url's scheme""" """resolve the proxy for a url's scheme
proxies = proxies or self.proxies
proxies is checked with `is None`, not truthiness, so an explicit
proxies={} disables session proxies for that one call instead of
falling back to them.
"""
proxies = self.proxies if proxies is None else proxies
scheme = url.split("://")[0] scheme = url.split("://")[0]
return proxies.get(scheme) return proxies.get(scheme)
@@ -214,31 +268,89 @@ class ExtendedSession:
# cookies # cookies
def get_cookies(self): def get_cookies(self):
"""cookies stored in the session jar""" """all cookies in the session jar, regardless of domain binding
return self.session.cookie_jar.filter_cookies()
iterates the jar directly rather than filter_cookies() (which needs a url and,
given none, returns only domain-less shared cookies - {} for any normal one).
returns {} off-loop when no session was built yet (nothing could be set), mirroring
preview() - reaching self.session would build it and need a running loop.
"""
if self._session is None:
return {}
return {c.key: c.value for c in self.session.cookie_jar}
def set_cookie(self, name, value, domain=None, path="/"): def set_cookie(self, name, value, domain=None, path="/"):
"""set a cookie in the session jar""" """set a cookie in the session jar
response_url = URL(domain or "http://localhost")
self.session.cookie_jar.update_cookies({name: value}, response_url=response_url) domain=None stores a truly shared cookie sent to every host. domain='example.com'
(scheme optional, defaulted to http://) scopes it to that host - a bare hostname
is normalized into a URL so the jar binds by host instead of silently storing
another domain-less shared cookie. `path` is honored via the morsel itself.
raises:
ValueError: if domain is a bare IP host (e.g. '127.0.0.1') and the jar is
the default aiohttp.CookieJar(unsafe=False), which drops cookies bound
to IP hosts with no store, no send, and no log - construct the session
with cookie_jar=aiohttp.CookieJar(unsafe=True) to allow IP-host cookies
"""
if self._session is None:
try:
self._ensure_session()
except RuntimeError as error:
raise RuntimeError(
"set_cookie needs the backend cookie jar, which builds on first use and "
"requires a running event loop; call it inside an async context (unlike "
"get_cookies/clear_cookies, which no-op off-loop)"
) from error
cookie = SimpleCookie()
cookie[name] = value
cookie[name]["path"] = path
if domain is None:
self.session.cookie_jar.update_cookies(cookie)
return
if "://" not in domain:
domain = "http://" + domain
url = URL(domain)
jar = self.session.cookie_jar
if _is_ip_host(url.raw_host) and not getattr(jar, "unsafe", False):
raise ValueError(
f"set_cookie domain {domain!r} resolves to a bare IP host, but "
"aiohttp's cookie jar rejects IP-address domains under unsafe=False "
"(the default) - the cookie would be silently dropped; construct "
"ExtendedSession with cookie_jar=aiohttp.CookieJar(unsafe=True) to "
"allow IP-host cookies"
)
jar.update_cookies(cookie, response_url=url)
def clear_cookies(self): def clear_cookies(self):
"""clear the session cookie jar""" """clear the session cookie jar
a no-op off-loop when no session was built yet (an unbuilt jar is already empty),
so callers can clear before the loop starts without a spurious session build.
"""
if self._session is None:
return
self.session.cookie_jar.clear() self.session.cookie_jar.clear()
def _cookies_for_url(self, url): def _cookies_for_url(self, url):
"""dict of cookies the backend would send for url override per backend """dict of cookies the backend would send for url - override per backend (used by preview())"""
used by preview(). non-aiohttp backends override this (or return {}); the
rest of preview (domain rewrites, header overwrites) is backend-agnostic.
"""
return {k: v.value for k, v in self.session.cookie_jar.filter_cookies(URL(url)).items()} return {k: v.value for k, v in self.session.cookie_jar.filter_cookies(URL(url)).items()}
# ------------------------------------------------------------------------- # -------------------------------------------------------------------------
# preview # preview
def preview(self, method, url, **kwargs): def preview(self, method, url, **kwargs):
"""build a RequestPreview for a request without sending it""" """build a RequestPreview for a request without sending it - synchronous, no loop required
url is domain-rewritten before cookies are resolved, matching the real
request - cookie binding is host-based, so resolving against the pre-rewrite
host could miss or misattribute cookies. the default cookie lookup never
touches the backend session: building it needs a running loop (aiohttp>=3.14),
which would defeat preview()'s own pre-loop inspection use case, and an unbuilt
session has no cookies to report anyway - pass cookies= explicitly to preview
cookies that would come from a not-yet-built session.
"""
url = self._apply_domain_overwrites(url)
proxy = self._get_proxy(url, kwargs.pop("proxies", None)) proxy = self._get_proxy(url, kwargs.pop("proxies", None))
merged = {**self._default_headers, **(kwargs.pop("headers", None) or {})} merged = {**self._default_headers, **(kwargs.pop("headers", None) or {})}
headers = self._apply_overwrites(merged) headers = self._apply_overwrites(merged)
@@ -247,7 +359,7 @@ class ExtendedSession:
timeout_total = timeout if isinstance(timeout, (int, float)) else None timeout_total = timeout if isinstance(timeout, (int, float)) else None
cookies = kwargs.pop("cookies", None) cookies = kwargs.pop("cookies", None)
if cookies is None: if cookies is None and self._session is not None:
cookies = self._cookies_for_url(url) cookies = self._cookies_for_url(url)
if cookies: if cookies:
cookie_header = "; ".join(f"{k}={v}" for k, v in cookies.items()) cookie_header = "; ".join(f"{k}={v}" for k, v in cookies.items())
@@ -257,7 +369,7 @@ class ExtendedSession:
return RequestPreview( return RequestPreview(
method=method, method=method,
url=self._apply_domain_overwrites(url), url=url,
headers=headers, headers=headers,
proxy=proxy, proxy=proxy,
data=kwargs.get("data"), data=kwargs.get("data"),
@@ -272,10 +384,9 @@ class ExtendedSession:
async def _raw_request(self, method, url, **kwargs) -> Response: async def _raw_request(self, method, url, **kwargs) -> Response:
"""send one request with aiohttp and adapt it into a Response """send one request with aiohttp and adapt it into a Response
this is the backend seam: a subclass can override only this method (using a the backend seam: a subclass overrides only this to swap HTTP clients,
different HTTP client, e.g. a TLS-fingerprinting one), building a Response building a Response from that backend's primitives; everything else
from that backend's primitives. everything else (overwrites, retries, (overwrites, retries, preview) is inherited.
preview) is inherited.
""" """
response = await self.session.request(method, url, **kwargs) response = await self.session.request(method, url, **kwargs)
async with response: async with response:
@@ -292,8 +403,21 @@ class ExtendedSession:
) )
async def request(self, method, url, **kwargs) -> Response: async def request(self, method, url, **kwargs) -> Response:
"""make a request, applying overwrites/domain rewrites/proxy resolution""" """make a request, applying overwrites/domain rewrites/proxy resolution
kwargs["proxy"] = self._get_proxy(url, kwargs.pop("proxies", None))
raises the real exception subtype on failure (ServerTimeoutError on a total
timeout; any other aiohttp.ClientError re-raised as-is, not flattened, so
callers can branch by type or read subtype attributes like .status/.headers).
a native proxy= is never silently clobbered by resolved proxies= (that would
unmask the caller's real IP) - pass only one of them.
"""
resolved_proxy = self._get_proxy(url, kwargs.pop("proxies", None))
if kwargs.get("proxy") is not None and resolved_proxy is not None:
raise ValueError("pass only one of proxy= or proxies= (session/per-call), not both")
# a bare proxy=None must not shadow a resolved proxies= (that would send direct and
# unmask the caller's IP); setdefault can't fix it because the None key already exists
if kwargs.get("proxy") is None:
kwargs["proxy"] = resolved_proxy
debug = kwargs.pop("debug", False) debug = kwargs.pop("debug", False)
merged = {**self._default_headers, **(kwargs.get("headers") or {})} merged = {**self._default_headers, **(kwargs.get("headers") or {})}
@@ -303,6 +427,11 @@ class ExtendedSession:
timeout = kwargs.get("timeout") timeout = kwargs.get("timeout")
if isinstance(timeout, (int, float)): if isinstance(timeout, (int, float)):
kwargs["timeout"] = aiohttp.ClientTimeout(total=timeout) kwargs["timeout"] = aiohttp.ClientTimeout(total=timeout)
elif timeout is None and "timeout" in kwargs:
# explicit timeout=None would reach aiohttp as ClientTimeout(total=None) -
# infinite, disabling the session default - so drop it instead (matters for
# request_with_retries, whose timeout kwarg defaults to None)
del kwargs["timeout"]
url = self._apply_domain_overwrites(url) url = self._apply_domain_overwrites(url)
if debug: if debug:
@@ -313,11 +442,15 @@ class ExtendedSession:
if debug and result.redirect_chain: if debug and result.redirect_chain:
log.info("redirect chain: %s", result.redirect_chain) log.info("redirect chain: %s", result.redirect_chain)
return result return result
except (aiohttp.ClientError, asyncio.TimeoutError) as error: except asyncio.TimeoutError as error:
# a total ClientTimeout raises a bare asyncio.TimeoutError, which is NOT an # not an aiohttp.ClientError subclass - wrap as ServerTimeoutError so
# aiohttp.ClientError subclass — wrap it into the same typed path so direct # callers get a typed failure and request_with_retries can label it a timeout
# callers get a consistent failure instead of a raw timeout raise aiohttp.ServerTimeoutError(f"timeout for {url}: {error}") from error
raise aiohttp.ClientError(f"client error for {url}: {error}") from error except aiohttp.ClientError as error:
# re-raise the original subtype (not flattened) - request_with_retries still
# catches the base aiohttp.ClientError below and is unaffected
log.error("client error for %s: %s", url, error)
raise
async def request_with_retries( async def request_with_retries(
self, method, url, *, data=None, proxies=None, timeout=None, attempts=None, self, method, url, *, data=None, proxies=None, timeout=None, attempts=None,
@@ -328,9 +461,11 @@ class ExtendedSession:
returns a Response on success (or non-retryable status), or a falsy returns a Response on success (or non-retryable status), or a falsy
FailureResponse if every attempt fails. backoff is exponential FailureResponse if every attempt fails. backoff is exponential
(backoff_base ** attempt). (backoff_base ** attempt). timeout=None falls back to the session-level
timeout (see request()'s note on why explicit None is dropped, not passed
through). attempts=0 floors to 1 (via commons.aretry), not DEFAULT_ATTEMPTS.
""" """
attempts = attempts or DEFAULT_ATTEMPTS attempts = DEFAULT_ATTEMPTS if attempts is None else attempts
body_data, body_json = _route_body(data) body_data, body_json = _route_body(data)
if debug: if debug:
@@ -361,6 +496,10 @@ class ExtendedSession:
log.error("all %d attempts failed for %s (last status %s)", log.error("all %d attempts failed for %s (last status %s)",
attempts, url, exhausted.response.status_code) attempts, url, exhausted.response.status_code)
return exhausted.response return exhausted.response
except asyncio.TimeoutError:
# catch before ClientError so a timeout is labeled as such, not generic
log.error("all %d attempts timed out for %s", attempts, url)
return FailureResponse(reason="timeout", url=url)
except aiohttp.ClientError as error: except aiohttp.ClientError as error:
log.error("all %d attempts failed for %s (client error: %s)", attempts, url, error) log.error("all %d attempts failed for %s (client error: %s)", attempts, url, error)
return FailureResponse(reason=f"client error: {error}", url=url) return FailureResponse(reason=f"client error: {error}", url=url)
@@ -372,12 +511,18 @@ class ExtendedSession:
# lifecycle # lifecycle
async def close(self): async def close(self):
"""close the backend session override if the backend's close differs""" """close the backend session - override if the backend's close differs; no-op if never built"""
await self.session.close() if self._session is not None:
await self._session.close()
def _is_closed(self) -> bool: def _is_closed(self) -> bool:
"""whether the backend session is closed override for non-aiohttp backends""" """whether the backend session is closed - override for non-aiohttp backends
return self.session.closed
an unbuilt session counts as closed: nothing was opened, nothing to leak.
"""
if self._session is None:
return True
return self._session.closed
async def __aenter__(self): async def __aenter__(self):
return self return self
@@ -386,8 +531,8 @@ class ExtendedSession:
await self.close() await self.close()
def __del__(self): def __del__(self):
# do NOT attempt async cleanup here spinning event loops in a finalizer # no async cleanup here - spinning event loops in a finalizer is unsafe;
# is unsafe. just warn so the leak is visible; callers must close explicitly. # just warn so the leak is visible, callers must close explicitly
try: try:
closed = self._is_closed() closed = self._is_closed()
except Exception: except Exception: