11 Commits
Author SHA1 Message Date
dsql ca23099e06 fix: proxy/attempts truthiness bugs, stale text() cache, preview cookie order (v0.1.9)
_get_proxy() and request() checked proxies/proxy with truthiness instead of
is None, so a per-call proxies={} silently fell back to session proxies, and a
native proxy= kwarg was clobbered by resolved session proxies (unmasking the
caller's real IP) instead of raising on conflicting sources. Response.text()
cached the first decode regardless of a later explicit encoding= argument.
preview() computed the Cookie header from the pre-rewrite url instead of the
rewritten one actually used by request(). request_with_retries(attempts=0)
became DEFAULT_ATTEMPTS via truthiness instead of flooring to 1.

Also compresses docstrings/comments across the module (no behavior change).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 23:25:43 -04:00
dsql 0768d643b1 fix: preserve exception subtype in request(), render params/timeout in as_curl()
request() re-raised every aiohttp.ClientError subclass as the bare base class,
losing ClientConnectorError/ClientProxyConnectionError/ClientResponseError
(.status/.headers)/TooManyRedirects and their attributes; direct callers
branching by type never matched. Now the original exception is bare-raised,
preserving subtype, attributes, and __cause__. request_with_retries (which
catches the base ClientError) is unaffected.

as_curl() silently dropped params and timeout, so a debug=True replay of a
params-driven request hit a different URL than the one actually sent. params
are now merged into the URL via yarl before quoting, and timeout renders as
--max-time.

Bumps 0.1.7 -> 0.1.8.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 16:57:04 -04:00
dsql e1ab5d38a0 fix: cookie methods actually work; session builds lazily (v0.1.7)
get_cookies() called filter_cookies() with no URL and always returned {}
for domain-bound cookies; now iterates the jar directly. set_cookie()
ignored path and leaked a shared cookie to every host when given a bare
domain string; scheme is now normalized and path honored, with
domain=None made an intentionally shared cookie instead of a silent
localhost-only no-op.

ExtendedSession also built its aiohttp.ClientSession synchronously in
__init__, which requires a running event loop under aiohttp>=3.14 and
crashed the common construct-before-the-loop-starts host pattern. The
session now builds lazily on first access, preserving the
_create_session subclass override seam used by aioweb_tls.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 16:42:30 -04:00
dsql b8cd184c64 fix: request_with_retries honors session timeout (timeout=None no longer disables it)
request() forwarded an explicit timeout=None to aiohttp as ClientTimeout(total=None),
which disables the timeout and overrides the session default. request_with_retries
defaults timeout=None, so its flagship path had zero timeout protection: a hung server
stalled the coroutine forever, pinning connector-pool slots. Pop a None timeout in
request() so the session-level timeout applies; numeric per-call timeouts still wrap.

aioweb-1 (v0.1.6).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 16:33:28 -04:00
dsql 74ed83cf73 chore: ignore .claude/ dir (CLAUDE.md now lives under .claude/)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:55:13 -04:00
dsql 14a3ee1456 fix: AW-2 json() returns None on a non-UTF-8 body instead of raising
responses.json() catches UnicodeDecodeError alongside JSONDecodeError — text() can raise
it on a non-UTF-8 payload, which is a 'not valid JSON' outcome per the docstring, not an
error to propagate.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:34:37 -04:00
dsql 3737af0cf5 fix: total-timeout labeled 'timeout' in request_with_retries (dead branch live) (v0.1.5)
AW-1: request() wraps a total ClientTimeout's bare asyncio.TimeoutError before
request_with_retries sees it, so the dedicated 'timeout' branch was dead and its comment
lied. wrap it as aiohttp.ServerTimeoutError (which IS both a ClientError AND a
TimeoutError) so direct request() callers still get a typed failure (M1 preserved) while
request_with_retries catches the timeout case first and labels it 'timeout'.

verified by execution: request() raises ServerTimeoutError (typed, M1 intact);
request_with_retries returns reason='timeout'; control confirms a real client error still
labels 'client error'. sibling-grep: aioweb_tls/aiowebhooks catch ClientError/TimeoutError,
both of which ServerTimeoutError satisfies — no consumer break.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 20:47:55 -04:00
dsql d3f2bed7fe docs: pin install line to release, note unpinned-latest option
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 18:13:34 -04:00
dsql 849200985c docs: show unpinned install line; note tag-pinning for reproducibility
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 18:07:19 -04:00
dsql 7da06443c8 fix: label timeouts, render empty-but-valid preview bodies, snapshot override dicts (v0.1.4)
- request_with_retries labels an exhausted total-timeout as 'timeout' instead of the
  generic 'unexpected error' catch-all (nit)
- as_curl() renders an empty-but-valid json body ({} / []) via is-not-None instead of
  dropping it as falsy (nit)
- _apply_overwrites snapshots the shared override dicts before iterating, so a
  concurrent mutation can't raise 'dict changed size during iteration' (nit).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 17:57:54 -04:00
dsql 382b8aa632 fix: request() wraps bare asyncio.TimeoutError on total-timeout (v0.1.3)
broaden the except to (aiohttp.ClientError, asyncio.TimeoutError) and re-wrap into
the same typed aiohttp.ClientError path. a total ClientTimeout raises a bare
asyncio.TimeoutError, which is NOT an aiohttp.ClientError subclass, so it previously
leaked raw out of request()/test_proxies(). add the missing asyncio import.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 17:09:20 -04:00
6 changed files with 234 additions and 83 deletions
+1 -1
View File
@@ -1,5 +1,5 @@
# claude
CLAUDE.md
.claude/
# python
__pycache__/
+76 -4
View File
@@ -11,17 +11,19 @@ and swap the HTTP client while inheriting everything else.
`requirements.txt`:
```
aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.2
aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.9
```
Direct:
```bash
pip install "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.2"
pip install "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.9"
```
Requires `aiohttp` and `yarl` (pulled transitively).
Drop the `@v0.1.9` suffix from the line above to install the latest unpinned.
## Usage
```python
@@ -63,6 +65,15 @@ resp = await s.request_with_retries(
Returns a `FailureResponse` (falsy) if every attempt fails.
`request()` (the non-retrying call) raises on failure: a total timeout raises
`aiohttp.ServerTimeoutError`, and any other network/protocol failure raises its real
`aiohttp.ClientError` subtype as-is (`ClientConnectorError`, `ClientProxyConnectionError`,
`ClientResponseError` with `.status`/`.headers`, `TooManyRedirects`, ...) — it is not
flattened into the base `ClientError`, so direct callers can branch by type or read
subtype attributes. `request_with_retries` catches the base `aiohttp.ClientError` (and
`asyncio.TimeoutError`) across all attempts and returns a falsy `FailureResponse` instead
of raising.
## Header overwrites & ephemeral headers
```python
@@ -86,7 +97,10 @@ s.overwrite_domain("internal.local", "127.0.0.1") # host-substring rewrite
print(s.preview("POST", url, json={"a": 1}).as_curl()) # equivalent cURL command
```
Pass `debug=True` to `request_with_retries` to log the cURL preview and request flow.
`as_curl()` renders `params` (merged into the url's query string) and `timeout` (as
`--max-time`) as well as headers/body/proxy, so the emitted command is faithful to what
`request()` actually sends. Pass `debug=True` to `request_with_retries` to log the cURL
preview and request flow.
## Custom backends
@@ -130,6 +144,64 @@ Two changes can't be shimmed without re-introducing the bugs they fix:
## Changelog
### v0.1.9
- **`_get_proxy()`/`request()` proxy resolution now checks `is None`, not
truthiness.** A per-call `proxies={}` previously fell back to the session's
configured proxies instead of disabling them for that call. `request()` also no
longer clobbers a native `proxy=` kwarg with the resolved session/`proxies=`
value (a real IP-unmasking leak) — passing both now raises `ValueError` instead
of silently picking one.
- **`Response.text(encoding=...)` no longer returns a stale cached decode.** A
second call with an explicit `encoding=` previously still returned the first
(possibly differently-encoded) cached decode; an explicit encoding now bypasses
the cache.
- **`preview()` now rewrites the url before resolving cookies**, matching what
`request()` actually sends — previously cookies were resolved against the
pre-rewrite host, which could miss or misattribute host-bound cookies.
- **`request_with_retries(attempts=0)` now floors to 1 attempt**, not silently
`DEFAULT_ATTEMPTS` (3). `attempts` is checked with `is None`, not truthiness.
### v0.1.8
- **`request()` no longer flattens `aiohttp.ClientError` subtypes.** Every failure
(connect errors, proxy errors, `raise_for_status()`-style response errors,
redirect limits, ...) was re-raised as a bare `aiohttp.ClientError`, losing the
real subtype and its attributes (`.os_error`, `.status`, `.headers`, ...) — a
direct caller doing `except ClientProxyConnectionError:` or `if e.status == 401`
would silently never match. Now the original exception is re-raised as-is (its
subtype, attributes, and `__cause__` all preserved). `request_with_retries`
still catches the base `aiohttp.ClientError` across attempts, so its behavior
(and its `FailureResponse` return on exhaustion) is unchanged.
- **`as_curl()` now renders `params` and `timeout`.** Previously a preview built
with `params=` silently omitted the query string (and a `timeout=` omitted
`--max-time`), so a `debug=True` cURL replay of a params-driven request hit a
different URL than the one actually sent. `params` are now merged into the url's
query string (via `yarl`) and `timeout` is emitted as `--max-time`.
### v0.1.7
- **`get_cookies()` now returns real cookies.** Previously called `filter_cookies()`
with no URL, which only ever returns domain-less shared cookies — every normal
domain-bound cookie (including ones set by a real `Set-Cookie` response) was
silently omitted. Now iterates the jar directly.
- **`set_cookie()` no longer leaks a shared cookie to every host.** A bare hostname
(`domain="example.com"`) built a schemeless URL, which aiohttp's jar treats as a
domain-less "shared" cookie sent with every request the session makes, including
unrelated hosts. A scheme is now added when missing so the cookie is scoped to
that host.
- **`set_cookie()` now honors `path`** (previously ignored — the cookie always
landed at `path="/"`). `domain=None` is unchanged in meaning but now stores a
truly shared cookie (sent to every host) instead of one silently bound to
`localhost` only, which made `set_cookie(name, value)` (no domain) a silent
no-op for any real request.
- **The backend session is built lazily**, not in `__init__`. Under aiohttp 3.14,
constructing `aiohttp.ClientSession` requires a running event loop; eager
construction crashed the common host pattern of attaching a session before the
loop starts (e.g. `bot.http = ExtendedSession(...)` in `Bot.__init__`). The
session (and any subclass's `_create_session` override) now builds on first
access instead.
### v0.1.2
- Pinned `commons` to v0.2.1 (retry `attempts` floor fix).
@@ -147,4 +219,4 @@ Two changes can't be shimmed without re-introducing the bugs they fix:
## Versioning
Tagged `vX.Y.Z`. Pin the tag in `requirements.txt`.
Releases are tagged `vX.Y.Z`. The install line above pins a release; drop the `@vX.Y.Z` suffix to install the latest unpinned. Pin deliberately for reproducible installs.
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "aioweb"
version = "0.1.2"
version = "0.1.9"
description = "Async HTTP session wrapper over aiohttp — proxies, header overwrites, retries, previews. Config-free, installable."
requires-python = ">=3.10"
dependencies = [
+15 -6
View File
@@ -5,6 +5,8 @@ request preview for aioweb — format or export a request without sending it
import json as _json
import shlex
from yarl import URL
class RequestPreview:
"""a formatted preview of an HTTP request (does not send)"""
@@ -28,18 +30,25 @@ class RequestPreview:
def as_curl(self):
"""equivalent cURL command for the request
every interpolated value is shell-quoted with shlex.quote, so headers,
body, url, or proxy containing quotes/spaces/metacharacters produce a
valid, non-injectable command rather than a broken or unsafe one.
every interpolated value is shlex.quote'd (non-injectable even with quotes/
spaces/metacharacters). `params` is merged into the url's query string and
`timeout` rendered as `--max-time`, so the command matches what request()
actually sends.
"""
parts = [f"curl -X {shlex.quote(self.details['method'])}"]
for header, value in (self.details["headers"] or {}).items():
parts.append(f"-H {shlex.quote(f'{header}: {value}')}")
if self.details["data"]:
if self.details["data"] is not None:
parts.append(f"--data {shlex.quote(str(self.details['data']))}")
elif self.details["json"]:
elif self.details["json"] is not None:
# is-not-None: an empty-but-valid body ({} / []) must still render
parts.append(f"--data {shlex.quote(_json.dumps(self.details['json']))}")
parts.append(shlex.quote(str(self.details["url"])))
url = self.details["url"]
if self.details["params"]:
url = str(URL(url).update_query(self.details["params"]))
parts.append(shlex.quote(str(url)))
if self.details["proxy"]:
parts.append(f"--proxy {shlex.quote(str(self.details['proxy']))}")
if self.details["timeout"] is not None:
parts.append(f"--max-time {shlex.quote(str(self.details['timeout']))}")
return " \\\n ".join(parts)
+11 -9
View File
@@ -1,10 +1,8 @@
"""
backend-agnostic response objects for aioweb
Response is built from primitives (status, headers, content, url, history) rather
than holding a raw aiohttp object, so any backend can produce one. FailureResponse
mirrors the same surface so callers can branch uniformly — every status/predicate is
a property on both.
backend-agnostic response objects for aioweb — Response is built from primitives
(status, headers, content, url, history), not a raw aiohttp object, so any backend can
produce one; FailureResponse mirrors the same surface (every status/predicate a
property on both) so callers can branch uniformly.
"""
import json as _json
@@ -83,16 +81,20 @@ class Response:
return self._content
def text(self, encoding: Optional[str] = None) -> str:
"""decoded text content (cached)"""
"""decoded text content, cached only for the default encoding"""
if encoding is not None:
return self._content.decode(encoding)
if self._text is None:
self._text = self._content.decode(encoding or self._encoding)
self._text = self._content.decode(self._encoding)
return self._text
def json(self):
"""parsed JSON content, or None if not valid JSON"""
try:
return _json.loads(self.text())
except _json.JSONDecodeError:
except (_json.JSONDecodeError, UnicodeDecodeError):
# text() decodes the body and can raise UnicodeDecodeError on a non-UTF-8
# payload — that's a "not valid JSON" outcome, not an error to propagate
return None
def raise_for_status(self):
+130 -62
View File
@@ -1,24 +1,24 @@
"""
async HTTP session wrapper over aiohttp
ExtendedSession adds session-level proxies, header overwrites, ephemeral
(per-request generated) headers, domain rewriting, request previews, and
retry/backoff on top of aiohttp. The actual byte-sending is isolated in
_raw_request() so a different backend (e.g. a TLS-fingerprinting client) can
subclass and override just that one method, inheriting everything else.
async HTTP session wrapper over aiohttp — proxies, header overwrites, ephemeral
headers, domain rewriting, previews, retry/backoff; byte-sending is isolated in
_raw_request() so a subclass can swap backends and inherit everything else.
async with ExtendedSession(proxies={"https": "http://..."}) as s:
resp = await s.request_with_retries("GET", url)
if resp: # FailureResponse is falsy
data = resp.json()
config-free: proxies/headers/timeouts are passed at construction or per call.
sessions must be closed explicitly (async with, or await s.close()); there is no
__del__ auto-close (that pattern is unsafe for async resources).
config-free (proxies/headers/timeouts passed at construction or per call). the
backend session is built lazily on first use, not in __init__, so construction is
safe before an event loop is running (e.g. bot.http = ExtendedSession(...)).
sessions must be closed explicitly (async with, or await s.close()) — no __del__
auto-close (unsafe for async resources).
"""
import asyncio
import logging
import warnings
from http.cookies import SimpleCookie
import aiohttp
from yarl import URL
@@ -29,12 +29,7 @@ from .responses import Response, FailureResponse
def _route_body(data):
"""split a body into (data=, json=) kwargs
dict OR list bodies are valid JSON and route to json=; everything else
(str/bytes/form) routes to data=. previously only dicts went to json=, so a
JSON list was wrongly form-encoded.
"""
"""split a body into (data=, json=) kwargs; dict OR list routes to json=, else data="""
if isinstance(data, (dict, list)):
return None, data
return data, None
@@ -83,25 +78,36 @@ class ExtendedSession:
self.domain_overwrites = domain_overwrites or {}
self.ephemeral_headers = {}
self.proxies = proxies or {}
# track our own default headers instead of touching aiohttp privates
# own header layer, not aiohttp privates, so update_headers/clear_headers work
self._default_headers = dict(headers or {})
self.session = self._create_session(self._default_headers, timeout, **kwargs)
self._session_timeout = timeout
self._session_kwargs = kwargs
# aiohttp>=3.14 requires a running loop to build ClientSession; build lazily
# (via `session` / _ensure_session()) so construction before the loop starts
# (e.g. bot.http = ExtendedSession(...) in Bot.__init__) doesn't crash
self._session = None
@property
def session(self):
"""the backend session, built lazily on first access (needs a running loop)"""
self._ensure_session()
return self._session
def _ensure_session(self):
"""build the backend session on first use — idempotent"""
if self._session is None:
self._session = self._create_session(
self._default_headers, self._session_timeout, **self._session_kwargs,
)
def _create_session(self, headers, timeout, **kwargs):
"""create the backend HTTP session — override to use a different client
"""create the backend HTTP session — override to swap HTTP clients
a subclass swapping the HTTP backend (e.g. a TLS-fingerprinting client)
overrides this to return its own session object. the overwrite/domain/
proxy/retry/preview logic in this class never touches the session object
directly (only _raw_request, the cookie methods, and close do), so those
features work unchanged on any backend.
`headers` is the session-default header set; the default aiohttp backend
does NOT bake it into the ClientSession (which would copy it into an
immutable per-session map that update_headers/clear_headers can't touch).
instead `_default_headers` is our own mutable layer that request() and
preview() merge per call, so the mutable session-header API actually works.
a backend that needs the defaults baked at construction may use `headers`.
overwrite/domain/proxy/retry/preview logic never touches the session object
directly (only _raw_request, cookies, close do), so those work unchanged on
any backend. `headers` isn't baked in here — aiohttp would copy it into an
immutable map update_headers/clear_headers can't touch; `_default_headers`
is the mutable layer request()/preview() merge per call instead.
"""
return aiohttp.ClientSession(
timeout=aiohttp.ClientTimeout(
@@ -123,10 +129,12 @@ class ExtendedSession:
def _apply_overwrites(self, request_headers):
"""apply static overwrites and ephemeral headers to a request's headers"""
request_headers = dict(request_headers or {})
for header, value in self.header_overwrites.items():
# list()-snapshot so a concurrent mutation of the shared dicts can't raise
# "dict changed size during iteration"
for header, value in list(self.header_overwrites.items()):
if self.inject or header in request_headers:
request_headers[header] = value
for header, value_callable in self.ephemeral_headers.items():
for header, value_callable in list(self.ephemeral_headers.items()):
if self.inject or header in request_headers:
value = value_callable()
if isinstance(value, dict):
@@ -204,8 +212,13 @@ class ExtendedSession:
self.proxies.clear()
def _get_proxy(self, url, proxies=None):
"""resolve the proxy for a url's scheme"""
proxies = proxies or self.proxies
"""resolve the proxy for a url's scheme
proxies is checked with `is None`, not truthiness, so an explicit
proxies={} disables session proxies for that one call instead of
falling back to them.
"""
proxies = self.proxies if proxies is None else proxies
scheme = url.split("://")[0]
return proxies.get(scheme)
@@ -213,31 +226,50 @@ class ExtendedSession:
# cookies
def get_cookies(self):
"""cookies stored in the session jar"""
return self.session.cookie_jar.filter_cookies()
"""all cookies in the session jar, regardless of domain binding
iterates the jar directly rather than filter_cookies() (which needs a url and,
given none, returns only domain-less shared cookies — {} for any normal one).
"""
return {c.key: c.value for c in self.session.cookie_jar}
def set_cookie(self, name, value, domain=None, path="/"):
"""set a cookie in the session jar"""
response_url = URL(domain or "http://localhost")
self.session.cookie_jar.update_cookies({name: value}, response_url=response_url)
"""set a cookie in the session jar
domain=None stores a truly shared cookie sent to every host. domain='example.com'
(scheme optional, defaulted to http://) scopes it to that host — a bare hostname
is normalized into a URL so the jar binds by host instead of silently storing
another domain-less shared cookie. `path` is honored via the morsel itself.
"""
cookie = SimpleCookie()
cookie[name] = value
if domain is None:
self.session.cookie_jar.update_cookies(cookie)
return
if "://" not in domain:
domain = "http://" + domain
cookie[name]["path"] = path
self.session.cookie_jar.update_cookies(cookie, response_url=URL(domain))
def clear_cookies(self):
"""clear the session cookie jar"""
self.session.cookie_jar.clear()
def _cookies_for_url(self, url):
"""dict of cookies the backend would send for url — override per backend
used by preview(). non-aiohttp backends override this (or return {}); the
rest of preview (domain rewrites, header overwrites) is backend-agnostic.
"""
"""dict of cookies the backend would send for url — override per backend (used by preview())"""
return {k: v.value for k, v in self.session.cookie_jar.filter_cookies(URL(url)).items()}
# -------------------------------------------------------------------------
# preview
def preview(self, method, url, **kwargs):
"""build a RequestPreview for a request without sending it"""
"""build a RequestPreview for a request without sending it
url is domain-rewritten before cookies are resolved, matching the real
request — cookie binding is host-based, so resolving against the pre-rewrite
host could miss or misattribute cookies.
"""
url = self._apply_domain_overwrites(url)
proxy = self._get_proxy(url, kwargs.pop("proxies", None))
merged = {**self._default_headers, **(kwargs.pop("headers", None) or {})}
headers = self._apply_overwrites(merged)
@@ -256,7 +288,7 @@ class ExtendedSession:
return RequestPreview(
method=method,
url=self._apply_domain_overwrites(url),
url=url,
headers=headers,
proxy=proxy,
data=kwargs.get("data"),
@@ -271,10 +303,9 @@ class ExtendedSession:
async def _raw_request(self, method, url, **kwargs) -> Response:
"""send one request with aiohttp and adapt it into a Response
this is the backend seam: a subclass can override only this method (using a
different HTTP client, e.g. a TLS-fingerprinting one), building a Response
from that backend's primitives. everything else (overwrites, retries,
preview) is inherited.
the backend seam: a subclass overrides only this to swap HTTP clients,
building a Response from that backend's primitives; everything else
(overwrites, retries, preview) is inherited.
"""
response = await self.session.request(method, url, **kwargs)
async with response:
@@ -291,8 +322,18 @@ class ExtendedSession:
)
async def request(self, method, url, **kwargs) -> Response:
"""make a request, applying overwrites/domain rewrites/proxy resolution"""
kwargs["proxy"] = self._get_proxy(url, kwargs.pop("proxies", None))
"""make a request, applying overwrites/domain rewrites/proxy resolution
raises the real exception subtype on failure (ServerTimeoutError on a total
timeout; any other aiohttp.ClientError re-raised as-is, not flattened, so
callers can branch by type or read subtype attributes like .status/.headers).
a native proxy= is never silently clobbered by resolved proxies= (that would
unmask the caller's real IP) — pass only one of them.
"""
resolved_proxy = self._get_proxy(url, kwargs.pop("proxies", None))
if "proxy" in kwargs and kwargs["proxy"] is not None and resolved_proxy is not None:
raise ValueError("pass only one of proxy= or proxies= (session/per-call), not both")
kwargs.setdefault("proxy", resolved_proxy)
debug = kwargs.pop("debug", False)
merged = {**self._default_headers, **(kwargs.get("headers") or {})}
@@ -302,6 +343,11 @@ class ExtendedSession:
timeout = kwargs.get("timeout")
if isinstance(timeout, (int, float)):
kwargs["timeout"] = aiohttp.ClientTimeout(total=timeout)
elif timeout is None and "timeout" in kwargs:
# explicit timeout=None would reach aiohttp as ClientTimeout(total=None) —
# infinite, disabling the session default — so drop it instead (matters for
# request_with_retries, whose timeout kwarg defaults to None)
del kwargs["timeout"]
url = self._apply_domain_overwrites(url)
if debug:
@@ -312,8 +358,16 @@ class ExtendedSession:
if debug and result.redirect_chain:
log.info("redirect chain: %s", result.redirect_chain)
return result
except asyncio.TimeoutError as error:
# a bare asyncio.TimeoutError isn't an aiohttp.ClientError subclass — wrap it
# as ServerTimeoutError (both a ClientError AND a TimeoutError) so callers get
# a typed failure and request_with_retries can still label it a timeout
raise aiohttp.ServerTimeoutError(f"timeout for {url}: {error}") from error
except aiohttp.ClientError as error:
raise aiohttp.ClientError(f"client error for {url}: {error}") from error
# re-raise the original subtype (not flattened) — request_with_retries still
# catches the base aiohttp.ClientError below and is unaffected
log.error("client error for %s: %s", url, error)
raise
async def request_with_retries(
self, method, url, *, data=None, proxies=None, timeout=None, attempts=None,
@@ -324,9 +378,11 @@ class ExtendedSession:
returns a Response on success (or non-retryable status), or a falsy
FailureResponse if every attempt fails. backoff is exponential
(backoff_base ** attempt).
(backoff_base ** attempt). timeout=None falls back to the session-level
timeout (see request()'s note on why explicit None is dropped, not passed
through). attempts=0 floors to 1 (via commons.aretry), not DEFAULT_ATTEMPTS.
"""
attempts = attempts or DEFAULT_ATTEMPTS
attempts = DEFAULT_ATTEMPTS if attempts is None else attempts
body_data, body_json = _route_body(data)
if debug:
@@ -357,6 +413,11 @@ class ExtendedSession:
log.error("all %d attempts failed for %s (last status %s)",
attempts, url, exhausted.response.status_code)
return exhausted.response
except asyncio.TimeoutError:
# request() wraps timeouts as ServerTimeoutError (ClientError + TimeoutError);
# catch it first so it's labeled a timeout, not a generic client error below
log.error("all %d attempts timed out for %s", attempts, url)
return FailureResponse(reason="timeout", url=url)
except aiohttp.ClientError as error:
log.error("all %d attempts failed for %s (client error: %s)", attempts, url, error)
return FailureResponse(reason=f"client error: {error}", url=url)
@@ -368,12 +429,19 @@ class ExtendedSession:
# lifecycle
async def close(self):
"""close the backend session — override if the backend's close differs"""
await self.session.close()
"""close the backend session — override if the backend's close differs; no-op if never built"""
if self._session is not None:
await self._session.close()
def _is_closed(self) -> bool:
"""whether the backend session is closed — override for non-aiohttp backends"""
return self.session.closed
"""whether the backend session is closed — override for non-aiohttp backends
an unbuilt session counts as closed: nothing was opened, nothing to leak,
__del__ should not warn.
"""
if self._session is None:
return True
return self._session.closed
async def __aenter__(self):
return self
@@ -382,8 +450,8 @@ class ExtendedSession:
await self.close()
def __del__(self):
# do NOT attempt async cleanup here — spinning event loops in a finalizer
# is unsafe. just warn so the leak is visible; callers must close explicitly.
# no async cleanup here — spinning event loops in a finalizer is unsafe;
# just warn so the leak is visible, callers must close explicitly
try:
closed = self._is_closed()
except Exception: