19 Commits
Author SHA1 Message Date
dsql 987c4bdf9c chore: bump to 1.3.0
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-10-16 04:00:40 -04:00
dsql 250c9cf2eb fix: copy config dicts, case-insensitive scheme/headers, guard honors ClientTimeout
T1-4: proxies/overwrites/domain_overwrites were aliased, so clear_proxies()
on one session emptied a sibling built from the same config dict, and the
proxy lookup used the raw scheme so HTTP:// missed the http key. both sent
the request direct. copy the dicts; resolve the scheme via yarl (lowercase).

T2-7: the hang-guard ignored a per-call ClientTimeout and killed the call at
the session timeout + slack. read .total off it.

T2-8: header names merged and overwritten case-sensitively, so curl_cffi
sent both User-Agent and user-agent (400 on strict servers). merge through
CIMultiDict and match overwrites case-insensitively.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-10-16 04:00:33 -04:00
dsql b009c0cf50 chore: bump to 1.2.0 (logging-discipline audit)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-08-10 23:00:50 -04:00
dsql e6eddf5bc3 fix: logging discipline in request()/request_with_retries
- delete the per-attempt log.error before raise in request()'s ClientError branch: the
  path re-raises (raise XOR log), commons.aretry already emits the per-attempt WARNING
  with the attempt count, and the terminal branch logs on exhaustion - the line was pure
  duplicate reporting of a retried failure.
- demote the retryable-status per-attempt line to debug for the same reason (it raises
  _RetryStatus; aretry owns the retry WARNING).
- level the four terminal branches of request_with_retries from ERROR to WARNING: they
  SWALLOW exhaustion into a falsy FailureResponse the caller branches on (recovered
  cleanly per the swallow contract), so the lib does not claim ERROR on the caller's
  behalf; the caller escalates on the falsy result. all four stay one uniform level.

verified: the terminal branch still fires on genuine exhaustion, request_with_retries
still returns a falsy FailureResponse, and the deleted per-attempt ERROR no longer emits.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-08-09 02:08:58 -04:00
dsql 952fe8a00f feat: hang-guard request() with an outer asyncio deadline
a backend can lose both its transfer wakeup and its own timeout enforcement in one
failure (observed: a curl_cffi request that parked an aioweb consumer's event loop for
24h+ - loop idle, nothing in flight, the await never completing). wrap _raw_request in
asyncio.wait_for at the effective timeout + 5s slack: the backend's own timeout still
fires first in every healthy failure, the envelope only trips when the backend's timer
is dead, and its cancellation unwedges the orphaned transfer. deadline = per-call numeric
timeout else the session timeout (default 10); timeout=None from request_with_retries
guards on the session bound (that was the wedge path). explicit Session(timeout=None)
stays unguarded - the documented opt-out for genuinely unbounded calls. additive, no new
knob. fired: a never-completing backend hangs forever without this, dies at deadline+slack
with it, and request_with_retries returns a FailureResponse instead of hanging.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-27 11:33:35 -04:00
dsql 6ec132f97c build: use git+https for inter-lib deps (docker ssh limitation)
docker builds can't use git+ssh (no ssh key / agent in the build), so the inter-lib
dependency references move to git+https (repos are public, anonymous clone). pins are
unchanged in target; bump to 1.0.2 so the https dependency spec ships under a new tag.
README install lines intentionally keep the ssh form for local/dev use.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-20 22:25:06 -04:00
dsql 6633d6cba3 fix: pin inter-lib dependencies to their v1.0.0 tags
the v1.0.0 release still pinned pre-1.0.0 sibling tags, so a fresh install dragged in
stale transitive deps. update the pin(s) to the current v1.0.x release and bump this lib
to 1.0.1 so the corrected dependency chain ships under a new tag (v1.0.0 left intact).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-17 17:46:10 -04:00
dsql 1deb05ce4a release: 1.0.0
first stable release. pre-1.0.0 verification complete: all surviving MED regressions and
gaps resolved and independently re-fired, tree audited clean across the suite.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-09 18:53:15 -04:00
dsql 45d8830833 fix: as_curl form-encodes a dict data= body with doseq=True
urlencode(data) without doseq rendered a list-valued form field as its urlencoded python
repr (x=%5B%271%27...) instead of aiohttp FormData's repeated-pair wire form (x=1&x=2), so
a debug curl for a list-valued body replayed a different request than request() sent. add
doseq=True to match the wire. preview/debug-only; scalar values and non-dict bodies are
byte-unchanged.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 21:16:08 -04:00
dsql eed607b08c fix: an IPv6 domain overwrite applies only on a whole-host match, never a substring splice
overwrite_domain accepted a bare IPv6 replacement, but _apply_domain_overwrites spliced it
as a substring into any host CONTAINING the target, composing an invalid host ('internal::1'
from '.local'->'::1' on internal.local) that exploded deep in yarl at request time. An IPv6
replacement now applies ONLY when the target equals the whole request host; a substring match
is left unrewritten instead of building an invalid host. Hostname/IPv4 substring splices are
unchanged. (Registration can't distinguish whole-host from substring for a hostname target
without the request host, so the guard lives at compose time where the host is known.)

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:56:52 -04:00
dsql ed773e2c1b docs+fix: as_curl form-encodes a dict data= body; README notes the commons dependency
as_curl() rendered a dict data= body as its Python repr ({'a': 1}), but aiohttp
form-encodes a dict data= as application/x-www-form-urlencoded (a=1&b=two), so the emitted
curl replayed a different body - now urlencode()s a dict (str bodies unchanged). README
install section omitted the hard sibling commons dependency (a private git+ssh package).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:37:09 -04:00
dsql 85735023f5 fix: proxy=None no longer drops an explicit proxies= (IP unmask); cookie views work off-loop
request(proxy=None, proxies={...}) silently sent direct because setdefault('proxy', ...)
is a no-op when the None key already exists, unmasking the caller's real IP - now a bare
proxy=None is replaced by the resolved proxies= (the both-set guard still rejects two real
proxies). get_cookies/clear_cookies return {}/no-op when no session was built yet (mirroring
preview) instead of forcing a build that needs a running loop; set_cookie, which genuinely
needs the jar, now raises a clear actionable error off-loop instead of aiohttp's opaque one.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:23:31 -04:00
dsql 51c421c9a2 fix: overwrite_domain accepts a bare IPv6 replacement instead of rejecting any ':'
the guard rejected any replacement containing ':', which over-rejected a valid bare IPv6
literal host ('::1', '2001:db8::1') that yarl's URL.with_host() accepts and brackets. it now
delegates the check to yarl (try with_host, catch ValueError), so a bare IPv6 is accepted
(rewrites to http://[::1]/...) while a genuine host:port or pre-bracketed form still raises.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 16:47:35 -04:00
dsql 59eadabe16 fix: fail loud on unreachable domain overwrites and jar-dropped IP cookies
overwrite_domain() now rejects a port-bearing replacement immediately at
registration time with a clear ValueError, instead of only surfacing a
deep yarl error from inside with_host() the first time a matching request
is made, far from the misconfiguration site.

set_cookie() now raises ValueError when domain resolves to a bare IP host
and the jar is the default CookieJar(unsafe=False), which silently drops
IP-bound cookies with no store, no send, and no log. Non-IP domains and a
caller-supplied unsafe=True jar are unaffected.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 00:09:37 -04:00
dsql 78dfba0962 fix: drop session-default sock_read that poisons pooled keep-alive connections
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 19:05:51 -04:00
dsql a7302a6356 refactor: derive __version__ from package metadata (single source)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 17:00:19 -04:00
dsql 397d74efe3 docs: restore FailureResponse public-method docstrings stripped in de-bloat
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:46:03 -04:00
dsql 7f73f21c93 fix: preview() works without a running loop; set_cookie honors path when domain=None
preview()'s default cookie lookup reached the backend session and could force-
build it, raising RuntimeError('no running event loop') off-loop and defeating
the v0.1.7 lazy-session pre-loop use case; it's now skipped when the session
isn't built yet. set_cookie(domain=None) returned before setting the morsel's
path, so a shared cookie always stored path='/' regardless of the path= arg;
the morsel path is now set before that early return.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:17:46 -04:00
dsql 27f0e49341 docs: compress prose/module docstrings, em-dash->hyphen (de-bloat wave 1)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 00:15:24 -04:00
6 changed files with 299 additions and 116 deletions

No files matched your search

+73 -8
View File
@@ -11,18 +11,19 @@ and swap the HTTP client while inheriting everything else.
`requirements.txt`:
```
aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.9
aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v1.3.0
```
Direct:
```bash
pip install "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.9"
pip install "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v1.3.0"
```
Requires `aiohttp` and `yarl` (pulled transitively).
Requires `aiohttp` and `yarl` (pulled transitively), plus the sibling `commons` library
(a private `git+ssh` package — the install needs access to the `rethink-public` org).
Drop the `@v0.1.9` suffix from the line above to install the latest unpinned.
Drop the `@v1.3.0` suffix from the line above to install the latest unpinned.
## Usage
@@ -97,10 +98,15 @@ s.overwrite_domain("internal.local", "127.0.0.1") # host-substring rewrite
print(s.preview("POST", url, json={"a": 1}).as_curl()) # equivalent cURL command
```
`as_curl()` renders `params` (merged into the url's query string) and `timeout` (as
`--max-time`) as well as headers/body/proxy, so the emitted command is faithful to what
`request()` actually sends. Pass `debug=True` to `request_with_retries` to log the cURL
preview and request flow.
`preview()` is synchronous and never touches the backend session — it works before any
event loop is running (e.g. building a preview at import/setup time), not just inside
`asyncio.run()`. Its default cookie lookup only reads cookies from an already-built
session; if the session hasn't been built yet there are no cookies to read, so it
returns none by default (pass `cookies=` explicitly to preview cookies for a session
that hasn't sent a request yet). `as_curl()` renders `params` (merged into the url's
query string) and `timeout` (as `--max-time`) as well as headers/body/proxy, so the
emitted command is faithful to what `request()` actually sends. Pass `debug=True` to
`request_with_retries` to log the cURL preview and request flow.
## Custom backends
@@ -144,6 +150,65 @@ Two changes can't be shimmed without re-introducing the bugs they fix:
## Changelog
### v1.1.0
- **Hang-guard on `request()` — a wedged backend can no longer park the loop forever.**
`request()` now wraps the backend call (`_raw_request`) in an outer `asyncio.wait_for`
deadline sized at the effective timeout **+ 5s slack** (`_HANG_GUARD_SLACK`). The
backend's own timeout still fires first in every healthy failure (slack, not
replacement); the envelope only trips when the backend's timer is dead — e.g. a
`curl_cffi` transfer that loses both its wakeup and its own enforcement in the same
failure — and its cancellation is what unwedges the orphaned transfer. The guard's
deadline is the per-call numeric `timeout` when given, else the session timeout
(`Session(timeout=...)`, default 10s); it composes cleanly with `request_with_retries`
(a wedge becomes one logged `ServerTimeoutError`, the next attempt proceeds, and an
exhausted retry returns a `FailureResponse` rather than hanging). **Opt-out:** a
session deliberately constructed unbounded (`Session(timeout=None)`) stays unguarded,
for genuinely long-lived calls (long-poll / streaming). Additive; no new knob.
### v0.1.13
- **Session-default timeout no longer poisons pooled keep-alive connections.**
The default `ClientTimeout` set `sock_read=timeout/2` alongside `total`. Under
`aiohttp>=3.14`, that read timer re-arms on every request dispatched over a
pooled protocol, including idle connections between requests; when it fires
it permanently poisons the pooled connection (`SocketTimeoutError` on the next
use, instantly, without contacting the server) — a real error from the server
(e.g. a 503) could come back as a client-side `FailureResponse(status=0,
reason='timeout')` instead. `sock_read` is now dropped from the session
default; `total` (and `connect`/`sock_connect`) still bound every request, and
the per-call `timeout=N` path (`ClientTimeout(total=N)`, no `sock_read`) was
already unaffected.
### v0.1.12
- **Docstring-only.** Restored one-line docstrings on `FailureResponse`'s
`redirect_chain`, `text()`, `json()`, and `raise_for_status()` — a prior
de-bloat pass stripped them below the public tier while the `Response`
twins kept theirs. No behavior change.
### v0.1.11
- **`preview()` no longer requires a running event loop.** Its default cookie
lookup (`cookies=` not passed) previously called `_cookies_for_url()`, which
reached the backend session and built it if absent — under aiohttp>=3.14 that
needs a running loop, so `preview()` raised `RuntimeError('no running event
loop')` when called before the loop starts, defeating its own pre-loop
build/inspect use case (v0.1.7). Now the default cookie lookup is skipped
entirely when the session hasn't been built yet (nothing could have been set
on a session that doesn't exist); `cookies={}` and in-loop calls are unaffected.
- **`set_cookie(domain=None)` now honors `path`.** The `domain=None` branch
returned right after `update_cookies()`, before the line that sets the
morsel's `path` — so a shared cookie (`set_cookie(name, value, path="/api")`,
no `domain=`) always stored the `SimpleCookie` default `path="/"` instead. The
morsel's `path` is now set before the `domain=None` early return. The
domain-bound branch was already correct and is unchanged.
### v0.1.10
- Docs-only pass: compressed module/method docstrings and comments that restated
README/CLAUDE prose, replaced em-dashes with hyphens. No behavior change.
### v0.1.9
- **`_get_proxy()`/`request()` proxy resolution now checks `is None`, not
+2 -2
View File
@@ -4,13 +4,13 @@ build-backend = "hatchling.build"
[project]
name = "aioweb"
version = "0.1.9"
version = "1.3.0"
description = "Async HTTP session wrapper over aiohttp — proxies, header overwrites, retries, previews. Config-free, installable."
requires-python = ">=3.10"
dependencies = [
"aiohttp>=3.9",
"yarl>=1.9",
"commons @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v0.2.1",
"commons @ git+https://git.rethinkstudios.io/rethink-public/commons.git@v1.0.0",
]
[tool.hatch.metadata]
+9 -8
View File
@@ -1,7 +1,14 @@
from importlib.metadata import version, PackageNotFoundError
from .session import ExtendedSession, DEFAULT_ATTEMPTS, RETRY_STATUSES
from .responses import Response, FailureResponse, AiowebError, aiowebResponse
from .preview import RequestPreview
try:
__version__ = version("aioweb")
except PackageNotFoundError:
__version__ = "0.0.0+unknown"
__all__ = [
"ExtendedSession",
"Response",
@@ -17,17 +24,11 @@ __all__ = [
async def request_retries(session, method, url, **kwargs):
"""back-compat wrapper for session.request_with_retries(...)
prefer calling session.request_with_retries(...) directly.
"""
"""back-compat wrapper for session.request_with_retries(...); prefer calling that directly"""
return await session.request_with_retries(method, url, **kwargs)
async def test_proxies(session, url="https://api.ipify.org?format=json"):
"""fetch the public IP via the session (verifies proxy config)
url defaults to ipify; pass another IP-echo endpoint to point elsewhere.
"""
"""fetch the public IP via the session (verifies proxy config); url defaults to ipify"""
response = await session.request("GET", url)
return response.json()
+9 -9
View File
@@ -1,9 +1,10 @@
"""
request preview for aioweb — format or export a request without sending it
request preview for aioweb - format or export a request without sending it
"""
import json as _json
import shlex
from urllib.parse import urlencode
from yarl import URL
@@ -28,18 +29,17 @@ class RequestPreview:
return "\n".join(f"{key}: {value}" for key, value in self.details.items())
def as_curl(self):
"""equivalent cURL command for the request
every interpolated value is shlex.quote'd (non-injectable even with quotes/
spaces/metacharacters). `params` is merged into the url's query string and
`timeout` rendered as `--max-time`, so the command matches what request()
actually sends.
"""
"""equivalent cURL command for the request (values shlex.quote'd; matches what request() sends)"""
parts = [f"curl -X {shlex.quote(self.details['method'])}"]
for header, value in (self.details["headers"] or {}).items():
parts.append(f"-H {shlex.quote(f'{header}: {value}')}")
if self.details["data"] is not None:
parts.append(f"--data {shlex.quote(str(self.details['data']))}")
data = self.details["data"]
# aiohttp form-encodes a dict data= body (application/x-www-form-urlencoded);
# render that wire form, not the python repr, so the curl replays identically.
# doseq=True mirrors aiohttp's FormData: a list value becomes repeated k=v pairs
rendered = urlencode(data, doseq=True) if isinstance(data, dict) else str(data)
parts.append(f"--data {shlex.quote(rendered)}")
elif self.details["json"] is not None:
# is-not-None: an empty-but-valid body ({} / []) must still render
parts.append(f"--data {shlex.quote(_json.dumps(self.details['json']))}")
+9 -15
View File
@@ -1,8 +1,7 @@
"""
backend-agnostic response objects for aioweb — Response is built from primitives
(status, headers, content, url, history), not a raw aiohttp object, so any backend can
produce one; FailureResponse mirrors the same surface (every status/predicate a
property on both) so callers can branch uniformly.
backend-agnostic response objects for aioweb - Response and FailureResponse share
the same surface (every status/predicate a property on both) so callers can branch
uniformly regardless of which one they got.
"""
import json as _json
@@ -32,32 +31,26 @@ class Response:
@property
def status_code(self) -> int:
"""HTTP status code"""
return self._status_code
@property
def headers(self):
"""response headers"""
return self._headers
@property
def url(self) -> str:
"""final URL after redirects"""
return self._url
@property
def reason(self):
"""reason phrase for the status code"""
return self._reason
@property
def cookies(self):
"""cookies set in the response"""
return self._cookies
@property
def history(self):
"""redirect history (list of (status, url) tuples)"""
return self._history
@property
@@ -67,17 +60,14 @@ class Response:
@property
def is_redirect(self) -> bool:
"""whether the status is a redirect"""
return self._status_code in (301, 302, 303, 307, 308)
@property
def is_success(self) -> bool:
"""whether the status indicates success (2xx)"""
return 200 <= self._status_code < 300
@property
def content(self) -> bytes:
"""raw response bytes"""
return self._content
def text(self, encoding: Optional[str] = None) -> str:
@@ -94,7 +84,7 @@ class Response:
return _json.loads(self.text())
except (_json.JSONDecodeError, UnicodeDecodeError):
# text() decodes the body and can raise UnicodeDecodeError on a non-UTF-8
# payload — that's a "not valid JSON" outcome, not an error to propagate
# payload - that's a "not valid JSON" outcome, not an error to propagate
return None
def raise_for_status(self):
@@ -143,6 +133,7 @@ class FailureResponse:
@property
def redirect_chain(self):
"""list of (status, url) tuples for all redirects"""
return []
@property
@@ -158,12 +149,15 @@ class FailureResponse:
return None
def text(self, encoding=None):
"""always None"""
return None
def json(self):
"""always None"""
return None
def raise_for_status(self):
"""raise AiowebError(reason)"""
raise AiowebError(self._reason)
def __bool__(self) -> bool:
@@ -173,5 +167,5 @@ class FailureResponse:
return f"<FailureResponse [{self._status_code}] {self._reason}>"
# back-compat alias — the response class was renamed Response
# back-compat alias - the response class was renamed Response
aiowebResponse = Response
+197 -74
View File
@@ -1,26 +1,21 @@
"""
async HTTP session wrapper over aiohttp — proxies, header overwrites, ephemeral
headers, domain rewriting, previews, retry/backoff; byte-sending is isolated in
_raw_request() so a subclass can swap backends and inherit everything else.
async HTTP session wrapper over aiohttp - proxies, header overwrites, ephemeral
headers, domain rewriting, previews, retry/backoff. See README for usage and contract.
async with ExtendedSession(proxies={"https": "http://..."}) as s:
resp = await s.request_with_retries("GET", url)
if resp: # FailureResponse is falsy
data = resp.json()
config-free (proxies/headers/timeouts passed at construction or per call). the
backend session is built lazily on first use, not in __init__, so construction is
safe before an event loop is running (e.g. bot.http = ExtendedSession(...)).
sessions must be closed explicitly (async with, or await s.close()) — no __del__
auto-close (unsafe for async resources).
"""
import asyncio
import ipaddress
import logging
import warnings
from http.cookies import SimpleCookie
import aiohttp
from multidict import CIMultiDict
from yarl import URL
from commons import aretry
@@ -35,12 +30,29 @@ def _route_body(data):
return data, None
class _RetryStatus(Exception):
"""internal signal: a retryable HTTP status; carries the real Response
def _is_ip_host(host) -> bool:
"""whether host is a literal IPv4/IPv6 address rather than a hostname"""
if not host:
return False
try:
ipaddress.ip_address(host)
except ValueError:
return False
return True
raised inside an attempt so commons.aretry drives the backoff + cap; the caller
catches the final one to return the REAL last response, not a synthetic failure.
"""
def _is_ipv6_literal(host) -> bool:
"""whether host is a bare IPv6 literal (must be a whole-host swap, never a substring splice)"""
if not host:
return False
try:
return isinstance(ipaddress.ip_address(host), ipaddress.IPv6Address)
except ValueError:
return False
class _RetryStatus(Exception):
"""internal signal: a retryable HTTP status; carries the real Response through aretry"""
def __init__(self, response):
super().__init__(f"retryable status {response.status_code}")
@@ -51,8 +63,8 @@ log = logging.getLogger(__name__)
DEFAULT_ATTEMPTS = 3
DEFAULT_BACKOFF_BASE = 2.0
# statuses worth retrying: rate limit + transient server errors
RETRY_STATUSES = frozenset({429, 500, 502, 503, 504})
_HANG_GUARD_SLACK = 5
class ExtendedSession:
@@ -74,17 +86,18 @@ class ExtendedSession:
kwargs: passed through to aiohttp.ClientSession
"""
self.inject = inject
self.header_overwrites = overwrites or {}
self.domain_overwrites = domain_overwrites or {}
# copied, not aliased: a shared config dict must not be mutated by (or leak a
# clear_proxies() into) a sibling session
self.header_overwrites = dict(overwrites or {})
self.domain_overwrites = dict(domain_overwrites or {})
self.ephemeral_headers = {}
self.proxies = proxies or {}
# own header layer, not aiohttp privates, so update_headers/clear_headers work
self._default_headers = dict(headers or {})
self.proxies = dict(proxies or {})
# own header layer, not aiohttp privates, so update_headers/clear_headers work;
# case-insensitive like HTTP header names
self._default_headers = CIMultiDict(headers or {})
self._session_timeout = timeout
self._session_kwargs = kwargs
# aiohttp>=3.14 requires a running loop to build ClientSession; build lazily
# (via `session` / _ensure_session()) so construction before the loop starts
# (e.g. bot.http = ExtendedSession(...) in Bot.__init__) doesn't crash
# aiohttp>=3.14 needs a running loop to build ClientSession; built lazily instead
self._session = None
@property
@@ -94,26 +107,29 @@ class ExtendedSession:
return self._session
def _ensure_session(self):
"""build the backend session on first use — idempotent"""
"""build the backend session on first use - idempotent"""
if self._session is None:
self._session = self._create_session(
self._default_headers, self._session_timeout, **self._session_kwargs,
)
def _create_session(self, headers, timeout, **kwargs):
"""create the backend HTTP session — override to swap HTTP clients
"""create the backend HTTP session - override to swap HTTP clients
overwrite/domain/proxy/retry/preview logic never touches the session object
directly (only _raw_request, cookies, close do), so those work unchanged on
any backend. `headers` isn't baked in here — aiohttp would copy it into an
immutable map update_headers/clear_headers can't touch; `_default_headers`
is the mutable layer request()/preview() merge per call instead.
`headers` isn't baked in here - aiohttp would copy it into an immutable map
update_headers/clear_headers can't touch; `_default_headers` is the mutable
layer request()/preview() merge per call instead.
no `sock_read` here - aiohttp re-arms that timer on every request dispatched
over a pooled protocol, including idle keep-alive connections, and firing it
calls `set_exception(SocketTimeoutError)` on the protocol, permanently
poisoning that pooled connection; the next request on it fails instantly
without contacting the server. `total` still bounds every request overall.
"""
return aiohttp.ClientSession(
timeout=aiohttp.ClientTimeout(
total=timeout,
connect=timeout / 2,
sock_read=timeout / 2,
sock_connect=timeout / 2,
),
**kwargs,
@@ -127,8 +143,12 @@ class ExtendedSession:
self.header_overwrites[header] = value
def _apply_overwrites(self, request_headers):
"""apply static overwrites and ephemeral headers to a request's headers"""
request_headers = dict(request_headers or {})
"""apply static overwrites and ephemeral headers to a request's headers
header names match case-insensitively, so an overwrite replaces a `user-agent`
rather than sending it alongside a second `User-Agent`
"""
request_headers = CIMultiDict(request_headers or {})
# list()-snapshot so a concurrent mutation of the shared dicts can't raise
# "dict changed size during iteration"
for header, value in list(self.header_overwrites.items()):
@@ -141,7 +161,7 @@ class ExtendedSession:
request_headers.update(value)
else:
request_headers[header] = value
return request_headers
return dict(request_headers)
def set_ephemeral(self, header, value_callable):
"""register a header generated fresh (via callable) for each request"""
@@ -185,16 +205,47 @@ class ExtendedSession:
# domain overwrites
def overwrite_domain(self, target, replacement):
"""register a host-substring rewrite (target -> replacement)"""
"""register a host rewrite (target -> replacement)
raises ValueError if the replacement is not a valid host for yarl's with_host()
(e.g. a 'host:port' string or a pre-bracketed '[::1]') - caught here rather than
deep in yarl at request time. a bare IPv6 literal ('::1', '2001:db8::1') is valid
and accepted: yarl brackets it itself.
a hostname/IPv4 replacement rewrites any host CONTAINING target (substring splice,
e.g. 'example.com' -> 'internal.example.com'). an IPv6 replacement can only ever be
a WHOLE-HOST swap - splicing '::1' into the middle of a name yields an invalid host
('internal::1') - so at request time an IPv6 replacement applies ONLY when target
equals the whole request host; a request whose host merely CONTAINS the target (a
substring) is left unrewritten instead of composing an invalid host and exploding
deep in yarl. see _apply_domain_overwrites.
"""
try:
URL("http://placeholder").with_host(replacement)
except ValueError as exc:
raise ValueError(
f"overwrite_domain replacement {replacement!r} is not a valid host: {exc}; "
"pass a bare hostname or IP (a bare IPv6 literal is fine), not a host:port or "
"bracketed form - rewrite a port via a full URL override, not domain_overwrites"
) from exc
self.domain_overwrites[target] = replacement
def _apply_domain_overwrites(self, url: str) -> str:
"""apply any host-substring rewrites to a url"""
"""apply any host rewrites to a url
a hostname/IPv4 replacement splices on a substring match; an IPv6 replacement only
applies on a whole-host (exact) match, since it can't be spliced mid-host - this pairs
with overwrite_domain's registration guard so an IPv6 rewrite never composes an
invalid host.
"""
parsed = URL(url)
if not parsed.host:
return url
for target, replacement in self.domain_overwrites.items():
if target in parsed.host:
if _is_ipv6_literal(replacement):
if target == parsed.host:
return str(parsed.with_host(replacement))
elif target in parsed.host:
return str(parsed.with_host(parsed.host.replace(target, replacement)))
return url
@@ -219,7 +270,7 @@ class ExtendedSession:
falling back to them.
"""
proxies = self.proxies if proxies is None else proxies
scheme = url.split("://")[0]
scheme = URL(url).scheme
return proxies.get(scheme)
# -------------------------------------------------------------------------
@@ -229,56 +280,96 @@ class ExtendedSession:
"""all cookies in the session jar, regardless of domain binding
iterates the jar directly rather than filter_cookies() (which needs a url and,
given none, returns only domain-less shared cookies — {} for any normal one).
given none, returns only domain-less shared cookies - {} for any normal one).
returns {} off-loop when no session was built yet (nothing could be set), mirroring
preview() - reaching self.session would build it and need a running loop.
"""
if self._session is None:
return {}
return {c.key: c.value for c in self.session.cookie_jar}
def set_cookie(self, name, value, domain=None, path="/"):
"""set a cookie in the session jar
domain=None stores a truly shared cookie sent to every host. domain='example.com'
(scheme optional, defaulted to http://) scopes it to that host — a bare hostname
(scheme optional, defaulted to http://) scopes it to that host - a bare hostname
is normalized into a URL so the jar binds by host instead of silently storing
another domain-less shared cookie. `path` is honored via the morsel itself.
raises:
ValueError: if domain is a bare IP host (e.g. '127.0.0.1') and the jar is
the default aiohttp.CookieJar(unsafe=False), which drops cookies bound
to IP hosts with no store, no send, and no log - construct the session
with cookie_jar=aiohttp.CookieJar(unsafe=True) to allow IP-host cookies
"""
if self._session is None:
try:
self._ensure_session()
except RuntimeError as error:
raise RuntimeError(
"set_cookie needs the backend cookie jar, which builds on first use and "
"requires a running event loop; call it inside an async context (unlike "
"get_cookies/clear_cookies, which no-op off-loop)"
) from error
cookie = SimpleCookie()
cookie[name] = value
cookie[name]["path"] = path
if domain is None:
self.session.cookie_jar.update_cookies(cookie)
return
if "://" not in domain:
domain = "http://" + domain
cookie[name]["path"] = path
self.session.cookie_jar.update_cookies(cookie, response_url=URL(domain))
url = URL(domain)
jar = self.session.cookie_jar
if _is_ip_host(url.raw_host) and not getattr(jar, "unsafe", False):
raise ValueError(
f"set_cookie domain {domain!r} resolves to a bare IP host, but "
"aiohttp's cookie jar rejects IP-address domains under unsafe=False "
"(the default) - the cookie would be silently dropped; construct "
"ExtendedSession with cookie_jar=aiohttp.CookieJar(unsafe=True) to "
"allow IP-host cookies"
)
jar.update_cookies(cookie, response_url=url)
def clear_cookies(self):
"""clear the session cookie jar"""
"""clear the session cookie jar
a no-op off-loop when no session was built yet (an unbuilt jar is already empty),
so callers can clear before the loop starts without a spurious session build.
"""
if self._session is None:
return
self.session.cookie_jar.clear()
def _cookies_for_url(self, url):
"""dict of cookies the backend would send for url — override per backend (used by preview())"""
"""dict of cookies the backend would send for url - override per backend (used by preview())"""
return {k: v.value for k, v in self.session.cookie_jar.filter_cookies(URL(url)).items()}
# -------------------------------------------------------------------------
# preview
def preview(self, method, url, **kwargs):
"""build a RequestPreview for a request without sending it
"""build a RequestPreview for a request without sending it - synchronous, no loop required
url is domain-rewritten before cookies are resolved, matching the real
request — cookie binding is host-based, so resolving against the pre-rewrite
host could miss or misattribute cookies.
request - cookie binding is host-based, so resolving against the pre-rewrite
host could miss or misattribute cookies. the default cookie lookup never
touches the backend session: building it needs a running loop (aiohttp>=3.14),
which would defeat preview()'s own pre-loop inspection use case, and an unbuilt
session has no cookies to report anyway - pass cookies= explicitly to preview
cookies that would come from a not-yet-built session.
"""
url = self._apply_domain_overwrites(url)
proxy = self._get_proxy(url, kwargs.pop("proxies", None))
merged = {**self._default_headers, **(kwargs.pop("headers", None) or {})}
merged = CIMultiDict(self._default_headers)
merged.update(kwargs.pop("headers", None) or {})
headers = self._apply_overwrites(merged)
timeout = kwargs.get("timeout")
timeout_total = timeout if isinstance(timeout, (int, float)) else None
cookies = kwargs.pop("cookies", None)
if cookies is None:
if cookies is None and self._session is not None:
cookies = self._cookies_for_url(url)
if cookies:
cookie_header = "; ".join(f"{k}={v}" for k, v in cookies.items())
@@ -328,24 +419,35 @@ class ExtendedSession:
timeout; any other aiohttp.ClientError re-raised as-is, not flattened, so
callers can branch by type or read subtype attributes like .status/.headers).
a native proxy= is never silently clobbered by resolved proxies= (that would
unmask the caller's real IP) — pass only one of them.
unmask the caller's real IP) - pass only one of them.
"""
resolved_proxy = self._get_proxy(url, kwargs.pop("proxies", None))
if "proxy" in kwargs and kwargs["proxy"] is not None and resolved_proxy is not None:
if kwargs.get("proxy") is not None and resolved_proxy is not None:
raise ValueError("pass only one of proxy= or proxies= (session/per-call), not both")
kwargs.setdefault("proxy", resolved_proxy)
# a bare proxy=None must not shadow a resolved proxies= (that would send direct and
# unmask the caller's IP); setdefault can't fix it because the None key already exists
if kwargs.get("proxy") is None:
kwargs["proxy"] = resolved_proxy
debug = kwargs.pop("debug", False)
merged = {**self._default_headers, **(kwargs.get("headers") or {})}
merged = CIMultiDict(self._default_headers)
merged.update(kwargs.get("headers") or {})
kwargs["headers"] = self._apply_overwrites(merged)
kwargs["headers"] = {str(k): str(v) for k, v in kwargs["headers"].items()}
timeout = kwargs.get("timeout")
# the hang-guard deadline: a per-call timeout (a number, or a ClientTimeout's
# .total), else the session default. a timeout=None (omitted, or the explicit None
# request_with_retries passes) means "use the session bound" - so it guards on
# self._session_timeout, NOT unguarded. unguarded only when the caller asked for no
# bound: a session constructed with timeout=None (the documented long-poll/streaming
# opt-out), or a per-call ClientTimeout(total=None).
guard_secs = self._session_timeout if timeout is None else getattr(timeout, "total", timeout)
if isinstance(timeout, (int, float)):
kwargs["timeout"] = aiohttp.ClientTimeout(total=timeout)
elif timeout is None and "timeout" in kwargs:
# explicit timeout=None would reach aiohttp as ClientTimeout(total=None) —
# infinite, disabling the session default — so drop it instead (matters for
# explicit timeout=None would reach aiohttp as ClientTimeout(total=None) -
# infinite, disabling the session default - so drop it instead (matters for
# request_with_retries, whose timeout kwarg defaults to None)
del kwargs["timeout"]
@@ -354,19 +456,32 @@ class ExtendedSession:
log.info("sending request to: %s", url)
try:
# hang-guard: an outer asyncio deadline (session/curl timers can die in the
# same failure that parks the transfer - a backend that loses both wakeup and
# its own timeout enforcement would otherwise leave this await parked forever).
# the backend's own timeout still fires first in every healthy failure (slack,
# not replacement); wait_for only trips when the backend's enforcement is dead,
# and its cancellation is what unwedges the orphaned transfer.
if guard_secs is not None:
result = await asyncio.wait_for(
self._raw_request(method, url, **kwargs), guard_secs + _HANG_GUARD_SLACK
)
else:
result = await self._raw_request(method, url, **kwargs)
if debug and result.redirect_chain:
log.info("redirect chain: %s", result.redirect_chain)
return result
except asyncio.TimeoutError as error:
# a bare asyncio.TimeoutError isn't an aiohttp.ClientError subclass — wrap it
# as ServerTimeoutError (both a ClientError AND a TimeoutError) so callers get
# a typed failure and request_with_retries can still label it a timeout
# not an aiohttp.ClientError subclass - wrap as ServerTimeoutError so callers
# get a typed failure and request_with_retries can label it a timeout. now also
# catches the hang-guard firing (asyncio.wait_for raises asyncio.TimeoutError).
raise aiohttp.ServerTimeoutError(f"timeout for {url}: {error}") from error
except aiohttp.ClientError as error:
# re-raise the original subtype (not flattened) — request_with_retries still
# catches the base aiohttp.ClientError below and is unaffected
log.error("client error for %s: %s", url, error)
except aiohttp.ClientError:
# re-raise the original subtype (not flattened) - request_with_retries still
# catches the base aiohttp.ClientError below and is unaffected. no log here:
# this path RAISES, so the exception carries the failure (raise XOR log); on the
# retrying path commons.aretry emits the per-attempt WARNING and the terminal
# branch logs on exhaustion - logging here would double-report the same failure.
raise
async def request_with_retries(
@@ -400,7 +515,11 @@ class ExtendedSession:
headers=headers, proxies=proxies, timeout=timeout, debug=debug,
)
if response.status_code in retry_statuses:
log.warning("retryable status %s for %s", response.status_code, url)
# debug, not warning: this RAISES _RetryStatus to trigger a retry, so it must
# not also warn/error the same failure (raise XOR log). commons.aretry emits
# the per-attempt WARNING with the attempt count when it catches and retries;
# the terminal branch below logs on exhaustion.
log.debug("retryable status %s for %s", response.status_code, url)
raise _RetryStatus(response)
return response
@@ -409,35 +528,39 @@ class ExtendedSession:
attempt, attempts=attempts, backoff=1.0, factor=backoff_base,
jitter=False, on=(Exception,),
)
# terminal path: retries are exhausted and the failure is SWALLOWED into a falsy
# FailureResponse the caller branches on (the documented contract). rule 2 mandates a
# log on a swallow; level is WARNING because the lib recovered cleanly into a
# branchable value - the failing op is the caller's to escalate once it sees the falsy
# result, so the lib does not claim ERROR on the caller's behalf. all four branches are
# the same event class (exhausted retries) and stay at one uniform level.
except _RetryStatus as exhausted:
log.error("all %d attempts failed for %s (last status %s)",
log.warning("all %d attempts failed for %s (last status %s)",
attempts, url, exhausted.response.status_code)
return exhausted.response
except asyncio.TimeoutError:
# request() wraps timeouts as ServerTimeoutError (ClientError + TimeoutError);
# catch it first so it's labeled a timeout, not a generic client error below
log.error("all %d attempts timed out for %s", attempts, url)
# catch before ClientError so a timeout is labeled as such, not generic
log.warning("all %d attempts timed out for %s", attempts, url)
return FailureResponse(reason="timeout", url=url)
except aiohttp.ClientError as error:
log.error("all %d attempts failed for %s (client error: %s)", attempts, url, error)
log.warning("all %d attempts failed for %s (client error: %s)", attempts, url, error)
return FailureResponse(reason=f"client error: {error}", url=url)
except Exception as error:
log.error("all %d attempts failed for %s (unexpected: %s)", attempts, url, error)
log.warning("all %d attempts failed for %s (unexpected: %s)", attempts, url, error)
return FailureResponse(reason=f"unexpected error: {error}", url=url)
# -------------------------------------------------------------------------
# lifecycle
async def close(self):
"""close the backend session — override if the backend's close differs; no-op if never built"""
"""close the backend session - override if the backend's close differs; no-op if never built"""
if self._session is not None:
await self._session.close()
def _is_closed(self) -> bool:
"""whether the backend session is closed — override for non-aiohttp backends
"""whether the backend session is closed - override for non-aiohttp backends
an unbuilt session counts as closed: nothing was opened, nothing to leak,
__del__ should not warn.
an unbuilt session counts as closed: nothing was opened, nothing to leak.
"""
if self._session is None:
return True
@@ -450,7 +573,7 @@ class ExtendedSession:
await self.close()
def __del__(self):
# no async cleanup here — spinning event loops in a finalizer is unsafe;
# no async cleanup here - spinning event loops in a finalizer is unsafe;
# just warn so the leak is visible, callers must close explicitly
try:
closed = self._is_closed()