15 Commits
Author SHA1 Message Date
dsql 6633d6cba3 fix: pin inter-lib dependencies to their v1.0.0 tags
the v1.0.0 release still pinned pre-1.0.0 sibling tags, so a fresh install dragged in
stale transitive deps. update the pin(s) to the current v1.0.x release and bump this lib
to 1.0.1 so the corrected dependency chain ships under a new tag (v1.0.0 left intact).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-17 17:46:10 -04:00
dsql 1deb05ce4a release: 1.0.0
first stable release. pre-1.0.0 verification complete: all surviving MED regressions and
gaps resolved and independently re-fired, tree audited clean across the suite.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-09 18:53:15 -04:00
dsql 45d8830833 fix: as_curl form-encodes a dict data= body with doseq=True
urlencode(data) without doseq rendered a list-valued form field as its urlencoded python
repr (x=%5B%271%27...) instead of aiohttp FormData's repeated-pair wire form (x=1&x=2), so
a debug curl for a list-valued body replayed a different request than request() sent. add
doseq=True to match the wire. preview/debug-only; scalar values and non-dict bodies are
byte-unchanged.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 21:16:08 -04:00
dsql eed607b08c fix: an IPv6 domain overwrite applies only on a whole-host match, never a substring splice
overwrite_domain accepted a bare IPv6 replacement, but _apply_domain_overwrites spliced it
as a substring into any host CONTAINING the target, composing an invalid host ('internal::1'
from '.local'->'::1' on internal.local) that exploded deep in yarl at request time. An IPv6
replacement now applies ONLY when the target equals the whole request host; a substring match
is left unrewritten instead of building an invalid host. Hostname/IPv4 substring splices are
unchanged. (Registration can't distinguish whole-host from substring for a hostname target
without the request host, so the guard lives at compose time where the host is known.)

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:56:52 -04:00
dsql ed773e2c1b docs+fix: as_curl form-encodes a dict data= body; README notes the commons dependency
as_curl() rendered a dict data= body as its Python repr ({'a': 1}), but aiohttp
form-encodes a dict data= as application/x-www-form-urlencoded (a=1&b=two), so the emitted
curl replayed a different body - now urlencode()s a dict (str bodies unchanged). README
install section omitted the hard sibling commons dependency (a private git+ssh package).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:37:09 -04:00
dsql 85735023f5 fix: proxy=None no longer drops an explicit proxies= (IP unmask); cookie views work off-loop
request(proxy=None, proxies={...}) silently sent direct because setdefault('proxy', ...)
is a no-op when the None key already exists, unmasking the caller's real IP - now a bare
proxy=None is replaced by the resolved proxies= (the both-set guard still rejects two real
proxies). get_cookies/clear_cookies return {}/no-op when no session was built yet (mirroring
preview) instead of forcing a build that needs a running loop; set_cookie, which genuinely
needs the jar, now raises a clear actionable error off-loop instead of aiohttp's opaque one.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:23:31 -04:00
dsql 51c421c9a2 fix: overwrite_domain accepts a bare IPv6 replacement instead of rejecting any ':'
the guard rejected any replacement containing ':', which over-rejected a valid bare IPv6
literal host ('::1', '2001:db8::1') that yarl's URL.with_host() accepts and brackets. it now
delegates the check to yarl (try with_host, catch ValueError), so a bare IPv6 is accepted
(rewrites to http://[::1]/...) while a genuine host:port or pre-bracketed form still raises.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 16:47:35 -04:00
dsql 59eadabe16 fix: fail loud on unreachable domain overwrites and jar-dropped IP cookies
overwrite_domain() now rejects a port-bearing replacement immediately at
registration time with a clear ValueError, instead of only surfacing a
deep yarl error from inside with_host() the first time a matching request
is made, far from the misconfiguration site.

set_cookie() now raises ValueError when domain resolves to a bare IP host
and the jar is the default CookieJar(unsafe=False), which silently drops
IP-bound cookies with no store, no send, and no log. Non-IP domains and a
caller-supplied unsafe=True jar are unaffected.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 00:09:37 -04:00
dsql 78dfba0962 fix: drop session-default sock_read that poisons pooled keep-alive connections
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 19:05:51 -04:00
dsql a7302a6356 refactor: derive __version__ from package metadata (single source)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 17:00:19 -04:00
dsql 397d74efe3 docs: restore FailureResponse public-method docstrings stripped in de-bloat
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:46:03 -04:00
dsql 7f73f21c93 fix: preview() works without a running loop; set_cookie honors path when domain=None
preview()'s default cookie lookup reached the backend session and could force-
build it, raising RuntimeError('no running event loop') off-loop and defeating
the v0.1.7 lazy-session pre-loop use case; it's now skipped when the session
isn't built yet. set_cookie(domain=None) returned before setting the morsel's
path, so a shared cookie always stored path='/' regardless of the path= arg;
the morsel path is now set before that early return.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:17:46 -04:00
dsql 27f0e49341 docs: compress prose/module docstrings, em-dash->hyphen (de-bloat wave 1)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 00:15:24 -04:00
dsql ca23099e06 fix: proxy/attempts truthiness bugs, stale text() cache, preview cookie order (v0.1.9)
_get_proxy() and request() checked proxies/proxy with truthiness instead of
is None, so a per-call proxies={} silently fell back to session proxies, and a
native proxy= kwarg was clobbered by resolved session proxies (unmasking the
caller's real IP) instead of raising on conflicting sources. Response.text()
cached the first decode regardless of a later explicit encoding= argument.
preview() computed the Cookie header from the pre-rewrite url instead of the
rewritten one actually used by request(). request_with_retries(attempts=0)
became DEFAULT_ATTEMPTS via truthiness instead of flooring to 1.

Also compresses docstrings/comments across the module (no behavior change).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 23:25:43 -04:00
dsql 0768d643b1 fix: preserve exception subtype in request(), render params/timeout in as_curl()
request() re-raised every aiohttp.ClientError subclass as the bare base class,
losing ClientConnectorError/ClientProxyConnectionError/ClientResponseError
(.status/.headers)/TooManyRedirects and their attributes; direct callers
branching by type never matched. Now the original exception is bare-raised,
preserving subtype, attributes, and __cause__. request_with_retries (which
catches the base ClientError) is unaffected.

as_curl() silently dropped params and timeout, so a debug=True replay of a
params-driven request hit a different URL than the one actually sent. params
are now merged into the URL via yarl before quoting, and timeout renders as
--max-time.

Bumps 0.1.7 -> 0.1.8.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 16:57:04 -04:00
6 changed files with 320 additions and 157 deletions
+101 -5
View File
@@ -11,18 +11,19 @@ and swap the HTTP client while inheriting everything else.
`requirements.txt`: `requirements.txt`:
``` ```
aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.7 aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v1.0.1
``` ```
Direct: Direct:
```bash ```bash
pip install "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.7" pip install "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v1.0.1"
``` ```
Requires `aiohttp` and `yarl` (pulled transitively). Requires `aiohttp` and `yarl` (pulled transitively), plus the sibling `commons` library
(a private `git+ssh` package — the install needs access to the `rethink-public` org).
Drop the `@v0.1.7` suffix from the line above to install the latest unpinned. Drop the `@v1.0.1` suffix from the line above to install the latest unpinned.
## Usage ## Usage
@@ -65,6 +66,15 @@ resp = await s.request_with_retries(
Returns a `FailureResponse` (falsy) if every attempt fails. Returns a `FailureResponse` (falsy) if every attempt fails.
`request()` (the non-retrying call) raises on failure: a total timeout raises
`aiohttp.ServerTimeoutError`, and any other network/protocol failure raises its real
`aiohttp.ClientError` subtype as-is (`ClientConnectorError`, `ClientProxyConnectionError`,
`ClientResponseError` with `.status`/`.headers`, `TooManyRedirects`, ...) — it is not
flattened into the base `ClientError`, so direct callers can branch by type or read
subtype attributes. `request_with_retries` catches the base `aiohttp.ClientError` (and
`asyncio.TimeoutError`) across all attempts and returns a falsy `FailureResponse` instead
of raising.
## Header overwrites & ephemeral headers ## Header overwrites & ephemeral headers
```python ```python
@@ -88,7 +98,15 @@ s.overwrite_domain("internal.local", "127.0.0.1") # host-substring rewrite
print(s.preview("POST", url, json={"a": 1}).as_curl()) # equivalent cURL command print(s.preview("POST", url, json={"a": 1}).as_curl()) # equivalent cURL command
``` ```
Pass `debug=True` to `request_with_retries` to log the cURL preview and request flow. `preview()` is synchronous and never touches the backend session — it works before any
event loop is running (e.g. building a preview at import/setup time), not just inside
`asyncio.run()`. Its default cookie lookup only reads cookies from an already-built
session; if the session hasn't been built yet there are no cookies to read, so it
returns none by default (pass `cookies=` explicitly to preview cookies for a session
that hasn't sent a request yet). `as_curl()` renders `params` (merged into the url's
query string) and `timeout` (as `--max-time`) as well as headers/body/proxy, so the
emitted command is faithful to what `request()` actually sends. Pass `debug=True` to
`request_with_retries` to log the cURL preview and request flow.
## Custom backends ## Custom backends
@@ -132,6 +150,84 @@ Two changes can't be shimmed without re-introducing the bugs they fix:
## Changelog ## Changelog
### v0.1.13
- **Session-default timeout no longer poisons pooled keep-alive connections.**
The default `ClientTimeout` set `sock_read=timeout/2` alongside `total`. Under
`aiohttp>=3.14`, that read timer re-arms on every request dispatched over a
pooled protocol, including idle connections between requests; when it fires
it permanently poisons the pooled connection (`SocketTimeoutError` on the next
use, instantly, without contacting the server) — a real error from the server
(e.g. a 503) could come back as a client-side `FailureResponse(status=0,
reason='timeout')` instead. `sock_read` is now dropped from the session
default; `total` (and `connect`/`sock_connect`) still bound every request, and
the per-call `timeout=N` path (`ClientTimeout(total=N)`, no `sock_read`) was
already unaffected.
### v0.1.12
- **Docstring-only.** Restored one-line docstrings on `FailureResponse`'s
`redirect_chain`, `text()`, `json()`, and `raise_for_status()` — a prior
de-bloat pass stripped them below the public tier while the `Response`
twins kept theirs. No behavior change.
### v0.1.11
- **`preview()` no longer requires a running event loop.** Its default cookie
lookup (`cookies=` not passed) previously called `_cookies_for_url()`, which
reached the backend session and built it if absent — under aiohttp>=3.14 that
needs a running loop, so `preview()` raised `RuntimeError('no running event
loop')` when called before the loop starts, defeating its own pre-loop
build/inspect use case (v0.1.7). Now the default cookie lookup is skipped
entirely when the session hasn't been built yet (nothing could have been set
on a session that doesn't exist); `cookies={}` and in-loop calls are unaffected.
- **`set_cookie(domain=None)` now honors `path`.** The `domain=None` branch
returned right after `update_cookies()`, before the line that sets the
morsel's `path` — so a shared cookie (`set_cookie(name, value, path="/api")`,
no `domain=`) always stored the `SimpleCookie` default `path="/"` instead. The
morsel's `path` is now set before the `domain=None` early return. The
domain-bound branch was already correct and is unchanged.
### v0.1.10
- Docs-only pass: compressed module/method docstrings and comments that restated
README/CLAUDE prose, replaced em-dashes with hyphens. No behavior change.
### v0.1.9
- **`_get_proxy()`/`request()` proxy resolution now checks `is None`, not
truthiness.** A per-call `proxies={}` previously fell back to the session's
configured proxies instead of disabling them for that call. `request()` also no
longer clobbers a native `proxy=` kwarg with the resolved session/`proxies=`
value (a real IP-unmasking leak) — passing both now raises `ValueError` instead
of silently picking one.
- **`Response.text(encoding=...)` no longer returns a stale cached decode.** A
second call with an explicit `encoding=` previously still returned the first
(possibly differently-encoded) cached decode; an explicit encoding now bypasses
the cache.
- **`preview()` now rewrites the url before resolving cookies**, matching what
`request()` actually sends — previously cookies were resolved against the
pre-rewrite host, which could miss or misattribute host-bound cookies.
- **`request_with_retries(attempts=0)` now floors to 1 attempt**, not silently
`DEFAULT_ATTEMPTS` (3). `attempts` is checked with `is None`, not truthiness.
### v0.1.8
- **`request()` no longer flattens `aiohttp.ClientError` subtypes.** Every failure
(connect errors, proxy errors, `raise_for_status()`-style response errors,
redirect limits, ...) was re-raised as a bare `aiohttp.ClientError`, losing the
real subtype and its attributes (`.os_error`, `.status`, `.headers`, ...) — a
direct caller doing `except ClientProxyConnectionError:` or `if e.status == 401`
would silently never match. Now the original exception is re-raised as-is (its
subtype, attributes, and `__cause__` all preserved). `request_with_retries`
still catches the base `aiohttp.ClientError` across attempts, so its behavior
(and its `FailureResponse` return on exhaustion) is unchanged.
- **`as_curl()` now renders `params` and `timeout`.** Previously a preview built
with `params=` silently omitted the query string (and a `timeout=` omitted
`--max-time`), so a `debug=True` cURL replay of a params-driven request hit a
different URL than the one actually sent. `params` are now merged into the url's
query string (via `yarl`) and `timeout` is emitted as `--max-time`.
### v0.1.7 ### v0.1.7
- **`get_cookies()` now returns real cookies.** Previously called `filter_cookies()` - **`get_cookies()` now returns real cookies.** Previously called `filter_cookies()`
+2 -2
View File
@@ -4,13 +4,13 @@ build-backend = "hatchling.build"
[project] [project]
name = "aioweb" name = "aioweb"
version = "0.1.7" version = "1.0.1"
description = "Async HTTP session wrapper over aiohttp — proxies, header overwrites, retries, previews. Config-free, installable." description = "Async HTTP session wrapper over aiohttp — proxies, header overwrites, retries, previews. Config-free, installable."
requires-python = ">=3.10" requires-python = ">=3.10"
dependencies = [ dependencies = [
"aiohttp>=3.9", "aiohttp>=3.9",
"yarl>=1.9", "yarl>=1.9",
"commons @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v0.2.1", "commons @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v1.0.0",
] ]
[tool.hatch.metadata] [tool.hatch.metadata]
+9 -8
View File
@@ -1,7 +1,14 @@
from importlib.metadata import version, PackageNotFoundError
from .session import ExtendedSession, DEFAULT_ATTEMPTS, RETRY_STATUSES from .session import ExtendedSession, DEFAULT_ATTEMPTS, RETRY_STATUSES
from .responses import Response, FailureResponse, AiowebError, aiowebResponse from .responses import Response, FailureResponse, AiowebError, aiowebResponse
from .preview import RequestPreview from .preview import RequestPreview
try:
__version__ = version("aioweb")
except PackageNotFoundError:
__version__ = "0.0.0+unknown"
__all__ = [ __all__ = [
"ExtendedSession", "ExtendedSession",
"Response", "Response",
@@ -17,17 +24,11 @@ __all__ = [
async def request_retries(session, method, url, **kwargs): async def request_retries(session, method, url, **kwargs):
"""back-compat wrapper for session.request_with_retries(...) """back-compat wrapper for session.request_with_retries(...); prefer calling that directly"""
prefer calling session.request_with_retries(...) directly.
"""
return await session.request_with_retries(method, url, **kwargs) return await session.request_with_retries(method, url, **kwargs)
async def test_proxies(session, url="https://api.ipify.org?format=json"): async def test_proxies(session, url="https://api.ipify.org?format=json"):
"""fetch the public IP via the session (verifies proxy config) """fetch the public IP via the session (verifies proxy config); url defaults to ipify"""
url defaults to ipify; pass another IP-echo endpoint to point elsewhere.
"""
response = await session.request("GET", url) response = await session.request("GET", url)
return response.json() return response.json()
+18 -11
View File
@@ -1,9 +1,12 @@
""" """
request preview for aioweb format or export a request without sending it request preview for aioweb - format or export a request without sending it
""" """
import json as _json import json as _json
import shlex import shlex
from urllib.parse import urlencode
from yarl import URL
class RequestPreview: class RequestPreview:
@@ -26,22 +29,26 @@ class RequestPreview:
return "\n".join(f"{key}: {value}" for key, value in self.details.items()) return "\n".join(f"{key}: {value}" for key, value in self.details.items())
def as_curl(self): def as_curl(self):
"""equivalent cURL command for the request """equivalent cURL command for the request (values shlex.quote'd; matches what request() sends)"""
every interpolated value is shell-quoted with shlex.quote, so headers,
body, url, or proxy containing quotes/spaces/metacharacters produce a
valid, non-injectable command rather than a broken or unsafe one.
"""
parts = [f"curl -X {shlex.quote(self.details['method'])}"] parts = [f"curl -X {shlex.quote(self.details['method'])}"]
for header, value in (self.details["headers"] or {}).items(): for header, value in (self.details["headers"] or {}).items():
parts.append(f"-H {shlex.quote(f'{header}: {value}')}") parts.append(f"-H {shlex.quote(f'{header}: {value}')}")
if self.details["data"] is not None: if self.details["data"] is not None:
parts.append(f"--data {shlex.quote(str(self.details['data']))}") data = self.details["data"]
# aiohttp form-encodes a dict data= body (application/x-www-form-urlencoded);
# render that wire form, not the python repr, so the curl replays identically.
# doseq=True mirrors aiohttp's FormData: a list value becomes repeated k=v pairs
rendered = urlencode(data, doseq=True) if isinstance(data, dict) else str(data)
parts.append(f"--data {shlex.quote(rendered)}")
elif self.details["json"] is not None: elif self.details["json"] is not None:
# is-not-None, not truthiness: an empty-but-valid body ({} / []) must still # is-not-None: an empty-but-valid body ({} / []) must still render
# render rather than being dropped as falsy
parts.append(f"--data {shlex.quote(_json.dumps(self.details['json']))}") parts.append(f"--data {shlex.quote(_json.dumps(self.details['json']))}")
parts.append(shlex.quote(str(self.details["url"]))) url = self.details["url"]
if self.details["params"]:
url = str(URL(url).update_query(self.details["params"]))
parts.append(shlex.quote(str(url)))
if self.details["proxy"]: if self.details["proxy"]:
parts.append(f"--proxy {shlex.quote(str(self.details['proxy']))}") parts.append(f"--proxy {shlex.quote(str(self.details['proxy']))}")
if self.details["timeout"] is not None:
parts.append(f"--max-time {shlex.quote(str(self.details['timeout']))}")
return " \\\n ".join(parts) return " \\\n ".join(parts)
+13 -19
View File
@@ -1,10 +1,7 @@
""" """
backend-agnostic response objects for aioweb backend-agnostic response objects for aioweb - Response and FailureResponse share
the same surface (every status/predicate a property on both) so callers can branch
Response is built from primitives (status, headers, content, url, history) rather uniformly regardless of which one they got.
than holding a raw aiohttp object, so any backend can produce one. FailureResponse
mirrors the same surface so callers can branch uniformly — every status/predicate is
a property on both.
""" """
import json as _json import json as _json
@@ -34,32 +31,26 @@ class Response:
@property @property
def status_code(self) -> int: def status_code(self) -> int:
"""HTTP status code"""
return self._status_code return self._status_code
@property @property
def headers(self): def headers(self):
"""response headers"""
return self._headers return self._headers
@property @property
def url(self) -> str: def url(self) -> str:
"""final URL after redirects"""
return self._url return self._url
@property @property
def reason(self): def reason(self):
"""reason phrase for the status code"""
return self._reason return self._reason
@property @property
def cookies(self): def cookies(self):
"""cookies set in the response"""
return self._cookies return self._cookies
@property @property
def history(self): def history(self):
"""redirect history (list of (status, url) tuples)"""
return self._history return self._history
@property @property
@@ -69,23 +60,22 @@ class Response:
@property @property
def is_redirect(self) -> bool: def is_redirect(self) -> bool:
"""whether the status is a redirect"""
return self._status_code in (301, 302, 303, 307, 308) return self._status_code in (301, 302, 303, 307, 308)
@property @property
def is_success(self) -> bool: def is_success(self) -> bool:
"""whether the status indicates success (2xx)"""
return 200 <= self._status_code < 300 return 200 <= self._status_code < 300
@property @property
def content(self) -> bytes: def content(self) -> bytes:
"""raw response bytes"""
return self._content return self._content
def text(self, encoding: Optional[str] = None) -> str: def text(self, encoding: Optional[str] = None) -> str:
"""decoded text content (cached)""" """decoded text content, cached only for the default encoding"""
if encoding is not None:
return self._content.decode(encoding)
if self._text is None: if self._text is None:
self._text = self._content.decode(encoding or self._encoding) self._text = self._content.decode(self._encoding)
return self._text return self._text
def json(self): def json(self):
@@ -94,7 +84,7 @@ class Response:
return _json.loads(self.text()) return _json.loads(self.text())
except (_json.JSONDecodeError, UnicodeDecodeError): except (_json.JSONDecodeError, UnicodeDecodeError):
# text() decodes the body and can raise UnicodeDecodeError on a non-UTF-8 # text() decodes the body and can raise UnicodeDecodeError on a non-UTF-8
# payload that's a "not valid JSON" outcome, not an error to propagate # payload - that's a "not valid JSON" outcome, not an error to propagate
return None return None
def raise_for_status(self): def raise_for_status(self):
@@ -143,6 +133,7 @@ class FailureResponse:
@property @property
def redirect_chain(self): def redirect_chain(self):
"""list of (status, url) tuples for all redirects"""
return [] return []
@property @property
@@ -158,12 +149,15 @@ class FailureResponse:
return None return None
def text(self, encoding=None): def text(self, encoding=None):
"""always None"""
return None return None
def json(self): def json(self):
"""always None"""
return None return None
def raise_for_status(self): def raise_for_status(self):
"""raise AiowebError(reason)"""
raise AiowebError(self._reason) raise AiowebError(self._reason)
def __bool__(self) -> bool: def __bool__(self) -> bool:
@@ -173,5 +167,5 @@ class FailureResponse:
return f"<FailureResponse [{self._status_code}] {self._reason}>" return f"<FailureResponse [{self._status_code}] {self._reason}>"
# back-compat alias the response class was renamed Response # back-compat alias - the response class was renamed Response
aiowebResponse = Response aiowebResponse = Response
+177 -112
View File
@@ -1,26 +1,15 @@
""" """
async HTTP session wrapper over aiohttp async HTTP session wrapper over aiohttp - proxies, header overwrites, ephemeral
headers, domain rewriting, previews, retry/backoff. See README for usage and contract.
ExtendedSession adds session-level proxies, header overwrites, ephemeral
(per-request generated) headers, domain rewriting, request previews, and
retry/backoff on top of aiohttp. The actual byte-sending is isolated in
_raw_request() so a different backend (e.g. a TLS-fingerprinting client) can
subclass and override just that one method, inheriting everything else.
async with ExtendedSession(proxies={"https": "http://..."}) as s: async with ExtendedSession(proxies={"https": "http://..."}) as s:
resp = await s.request_with_retries("GET", url) resp = await s.request_with_retries("GET", url)
if resp: # FailureResponse is falsy if resp: # FailureResponse is falsy
data = resp.json() data = resp.json()
config-free: proxies/headers/timeouts are passed at construction or per call. the
backend HTTP session is built lazily on first use (request/cookie access/close), not
in __init__, so construction is safe before an event loop is running (e.g. attaching
to a host object at process startup).
sessions must be closed explicitly (async with, or await s.close()); there is no
__del__ auto-close (that pattern is unsafe for async resources).
""" """
import asyncio import asyncio
import ipaddress
import logging import logging
import warnings import warnings
from http.cookies import SimpleCookie from http.cookies import SimpleCookie
@@ -34,23 +23,35 @@ from .responses import Response, FailureResponse
def _route_body(data): def _route_body(data):
"""split a body into (data=, json=) kwargs """split a body into (data=, json=) kwargs; dict OR list routes to json=, else data="""
dict OR list bodies are valid JSON and route to json=; everything else
(str/bytes/form) routes to data=. previously only dicts went to json=, so a
JSON list was wrongly form-encoded.
"""
if isinstance(data, (dict, list)): if isinstance(data, (dict, list)):
return None, data return None, data
return data, None return data, None
class _RetryStatus(Exception): def _is_ip_host(host) -> bool:
"""internal signal: a retryable HTTP status; carries the real Response """whether host is a literal IPv4/IPv6 address rather than a hostname"""
if not host:
return False
try:
ipaddress.ip_address(host)
except ValueError:
return False
return True
raised inside an attempt so commons.aretry drives the backoff + cap; the caller
catches the final one to return the REAL last response, not a synthetic failure. def _is_ipv6_literal(host) -> bool:
""" """whether host is a bare IPv6 literal (must be a whole-host swap, never a substring splice)"""
if not host:
return False
try:
return isinstance(ipaddress.ip_address(host), ipaddress.IPv6Address)
except ValueError:
return False
class _RetryStatus(Exception):
"""internal signal: a retryable HTTP status; carries the real Response through aretry"""
def __init__(self, response): def __init__(self, response):
super().__init__(f"retryable status {response.status_code}") super().__init__(f"retryable status {response.status_code}")
@@ -61,7 +62,6 @@ log = logging.getLogger(__name__)
DEFAULT_ATTEMPTS = 3 DEFAULT_ATTEMPTS = 3
DEFAULT_BACKOFF_BASE = 2.0 DEFAULT_BACKOFF_BASE = 2.0
# statuses worth retrying: rate limit + transient server errors
RETRY_STATUSES = frozenset({429, 500, 502, 503, 504}) RETRY_STATUSES = frozenset({429, 500, 502, 503, 504})
@@ -88,15 +88,11 @@ class ExtendedSession:
self.domain_overwrites = domain_overwrites or {} self.domain_overwrites = domain_overwrites or {}
self.ephemeral_headers = {} self.ephemeral_headers = {}
self.proxies = proxies or {} self.proxies = proxies or {}
# track our own default headers instead of touching aiohttp privates # own header layer, not aiohttp privates, so update_headers/clear_headers work
self._default_headers = dict(headers or {}) self._default_headers = dict(headers or {})
self._session_timeout = timeout self._session_timeout = timeout
self._session_kwargs = kwargs self._session_kwargs = kwargs
# aiohttp.ClientSession (via _create_session) requires a running event loop # aiohttp>=3.14 needs a running loop to build ClientSession; built lazily instead
# (aiohttp >= 3.14 raises RuntimeError otherwise); building it here would
# break the common host pattern of constructing before the loop starts
# (e.g. bot.http = ExtendedSession(...) in Bot.__init__). build lazily on
# first access instead, via the `session` property / _ensure_session().
self._session = None self._session = None
@property @property
@@ -106,33 +102,29 @@ class ExtendedSession:
return self._session return self._session
def _ensure_session(self): def _ensure_session(self):
"""build the backend session on first use idempotent""" """build the backend session on first use - idempotent"""
if self._session is None: if self._session is None:
self._session = self._create_session( self._session = self._create_session(
self._default_headers, self._session_timeout, **self._session_kwargs, self._default_headers, self._session_timeout, **self._session_kwargs,
) )
def _create_session(self, headers, timeout, **kwargs): def _create_session(self, headers, timeout, **kwargs):
"""create the backend HTTP session override to use a different client """create the backend HTTP session - override to swap HTTP clients
a subclass swapping the HTTP backend (e.g. a TLS-fingerprinting client) `headers` isn't baked in here - aiohttp would copy it into an immutable map
overrides this to return its own session object. the overwrite/domain/ update_headers/clear_headers can't touch; `_default_headers` is the mutable
proxy/retry/preview logic in this class never touches the session object layer request()/preview() merge per call instead.
directly (only _raw_request, the cookie methods, and close do), so those
features work unchanged on any backend.
`headers` is the session-default header set; the default aiohttp backend no `sock_read` here - aiohttp re-arms that timer on every request dispatched
does NOT bake it into the ClientSession (which would copy it into an over a pooled protocol, including idle keep-alive connections, and firing it
immutable per-session map that update_headers/clear_headers can't touch). calls `set_exception(SocketTimeoutError)` on the protocol, permanently
instead `_default_headers` is our own mutable layer that request() and poisoning that pooled connection; the next request on it fails instantly
preview() merge per call, so the mutable session-header API actually works. without contacting the server. `total` still bounds every request overall.
a backend that needs the defaults baked at construction may use `headers`.
""" """
return aiohttp.ClientSession( return aiohttp.ClientSession(
timeout=aiohttp.ClientTimeout( timeout=aiohttp.ClientTimeout(
total=timeout, total=timeout,
connect=timeout / 2, connect=timeout / 2,
sock_read=timeout / 2,
sock_connect=timeout / 2, sock_connect=timeout / 2,
), ),
**kwargs, **kwargs,
@@ -148,9 +140,8 @@ class ExtendedSession:
def _apply_overwrites(self, request_headers): def _apply_overwrites(self, request_headers):
"""apply static overwrites and ephemeral headers to a request's headers""" """apply static overwrites and ephemeral headers to a request's headers"""
request_headers = dict(request_headers or {}) request_headers = dict(request_headers or {})
# snapshot the shared override dicts so a concurrent mutation (e.g. a command # list()-snapshot so a concurrent mutation of the shared dicts can't raise
# editing ephemerals on a shared session) can't raise "dict changed size during # "dict changed size during iteration"
# iteration" — the loop body is sync, but the snapshot is cheap insurance
for header, value in list(self.header_overwrites.items()): for header, value in list(self.header_overwrites.items()):
if self.inject or header in request_headers: if self.inject or header in request_headers:
request_headers[header] = value request_headers[header] = value
@@ -205,16 +196,47 @@ class ExtendedSession:
# domain overwrites # domain overwrites
def overwrite_domain(self, target, replacement): def overwrite_domain(self, target, replacement):
"""register a host-substring rewrite (target -> replacement)""" """register a host rewrite (target -> replacement)
raises ValueError if the replacement is not a valid host for yarl's with_host()
(e.g. a 'host:port' string or a pre-bracketed '[::1]') - caught here rather than
deep in yarl at request time. a bare IPv6 literal ('::1', '2001:db8::1') is valid
and accepted: yarl brackets it itself.
a hostname/IPv4 replacement rewrites any host CONTAINING target (substring splice,
e.g. 'example.com' -> 'internal.example.com'). an IPv6 replacement can only ever be
a WHOLE-HOST swap - splicing '::1' into the middle of a name yields an invalid host
('internal::1') - so at request time an IPv6 replacement applies ONLY when target
equals the whole request host; a request whose host merely CONTAINS the target (a
substring) is left unrewritten instead of composing an invalid host and exploding
deep in yarl. see _apply_domain_overwrites.
"""
try:
URL("http://placeholder").with_host(replacement)
except ValueError as exc:
raise ValueError(
f"overwrite_domain replacement {replacement!r} is not a valid host: {exc}; "
"pass a bare hostname or IP (a bare IPv6 literal is fine), not a host:port or "
"bracketed form - rewrite a port via a full URL override, not domain_overwrites"
) from exc
self.domain_overwrites[target] = replacement self.domain_overwrites[target] = replacement
def _apply_domain_overwrites(self, url: str) -> str: def _apply_domain_overwrites(self, url: str) -> str:
"""apply any host-substring rewrites to a url""" """apply any host rewrites to a url
a hostname/IPv4 replacement splices on a substring match; an IPv6 replacement only
applies on a whole-host (exact) match, since it can't be spliced mid-host - this pairs
with overwrite_domain's registration guard so an IPv6 rewrite never composes an
invalid host.
"""
parsed = URL(url) parsed = URL(url)
if not parsed.host: if not parsed.host:
return url return url
for target, replacement in self.domain_overwrites.items(): for target, replacement in self.domain_overwrites.items():
if target in parsed.host: if _is_ipv6_literal(replacement):
if target == parsed.host:
return str(parsed.with_host(replacement))
elif target in parsed.host:
return str(parsed.with_host(parsed.host.replace(target, replacement))) return str(parsed.with_host(parsed.host.replace(target, replacement)))
return url return url
@@ -232,8 +254,13 @@ class ExtendedSession:
self.proxies.clear() self.proxies.clear()
def _get_proxy(self, url, proxies=None): def _get_proxy(self, url, proxies=None):
"""resolve the proxy for a url's scheme""" """resolve the proxy for a url's scheme
proxies = proxies or self.proxies
proxies is checked with `is None`, not truthiness, so an explicit
proxies={} disables session proxies for that one call instead of
falling back to them.
"""
proxies = self.proxies if proxies is None else proxies
scheme = url.split("://")[0] scheme = url.split("://")[0]
return proxies.get(scheme) return proxies.get(scheme)
@@ -241,54 +268,89 @@ class ExtendedSession:
# cookies # cookies
def get_cookies(self): def get_cookies(self):
"""all cookies stored in the session jar, regardless of domain binding """all cookies in the session jar, regardless of domain binding
iterates the jar directly rather than filter_cookies() (which needs a url iterates the jar directly rather than filter_cookies() (which needs a url and,
and, given none, returns only domain-less shared cookies — i.e. {} for any given none, returns only domain-less shared cookies - {} for any normal one).
normal domain-bound cookie). returns {} off-loop when no session was built yet (nothing could be set), mirroring
preview() - reaching self.session would build it and need a running loop.
""" """
if self._session is None:
return {}
return {c.key: c.value for c in self.session.cookie_jar} return {c.key: c.value for c in self.session.cookie_jar}
def set_cookie(self, name, value, domain=None, path="/"): def set_cookie(self, name, value, domain=None, path="/"):
"""set a cookie in the session jar """set a cookie in the session jar
domain=None (the default) stores a truly shared cookie sent with every domain=None stores a truly shared cookie sent to every host. domain='example.com'
request regardless of host — the jar's own "no response_url" behavior. (scheme optional, defaulted to http://) scopes it to that host - a bare hostname
pass domain='example.com' (a scheme is optional and defaulted to http://) is normalized into a URL so the jar binds by host instead of silently storing
to scope the cookie to one host; a bare hostname like 'example.com' is another domain-less shared cookie. `path` is honored via the morsel itself.
normalized into a URL so the jar binds it by host instead of silently
storing another domain-less shared cookie (a schemeless domain has no raises:
raw_host, so the jar can't tell it apart from the shared case). ValueError: if domain is a bare IP host (e.g. '127.0.0.1') and the jar is
`path` is honored via the morsel itself, since the jar only derives a path the default aiohttp.CookieJar(unsafe=False), which drops cookies bound
from response_url when the morsel doesn't already carry one. to IP hosts with no store, no send, and no log - construct the session
with cookie_jar=aiohttp.CookieJar(unsafe=True) to allow IP-host cookies
""" """
if self._session is None:
try:
self._ensure_session()
except RuntimeError as error:
raise RuntimeError(
"set_cookie needs the backend cookie jar, which builds on first use and "
"requires a running event loop; call it inside an async context (unlike "
"get_cookies/clear_cookies, which no-op off-loop)"
) from error
cookie = SimpleCookie() cookie = SimpleCookie()
cookie[name] = value cookie[name] = value
cookie[name]["path"] = path
if domain is None: if domain is None:
self.session.cookie_jar.update_cookies(cookie) self.session.cookie_jar.update_cookies(cookie)
return return
if "://" not in domain: if "://" not in domain:
domain = "http://" + domain domain = "http://" + domain
cookie[name]["path"] = path url = URL(domain)
self.session.cookie_jar.update_cookies(cookie, response_url=URL(domain)) jar = self.session.cookie_jar
if _is_ip_host(url.raw_host) and not getattr(jar, "unsafe", False):
raise ValueError(
f"set_cookie domain {domain!r} resolves to a bare IP host, but "
"aiohttp's cookie jar rejects IP-address domains under unsafe=False "
"(the default) - the cookie would be silently dropped; construct "
"ExtendedSession with cookie_jar=aiohttp.CookieJar(unsafe=True) to "
"allow IP-host cookies"
)
jar.update_cookies(cookie, response_url=url)
def clear_cookies(self): def clear_cookies(self):
"""clear the session cookie jar""" """clear the session cookie jar
a no-op off-loop when no session was built yet (an unbuilt jar is already empty),
so callers can clear before the loop starts without a spurious session build.
"""
if self._session is None:
return
self.session.cookie_jar.clear() self.session.cookie_jar.clear()
def _cookies_for_url(self, url): def _cookies_for_url(self, url):
"""dict of cookies the backend would send for url override per backend """dict of cookies the backend would send for url - override per backend (used by preview())"""
used by preview(). non-aiohttp backends override this (or return {}); the
rest of preview (domain rewrites, header overwrites) is backend-agnostic.
"""
return {k: v.value for k, v in self.session.cookie_jar.filter_cookies(URL(url)).items()} return {k: v.value for k, v in self.session.cookie_jar.filter_cookies(URL(url)).items()}
# ------------------------------------------------------------------------- # -------------------------------------------------------------------------
# preview # preview
def preview(self, method, url, **kwargs): def preview(self, method, url, **kwargs):
"""build a RequestPreview for a request without sending it""" """build a RequestPreview for a request without sending it - synchronous, no loop required
url is domain-rewritten before cookies are resolved, matching the real
request - cookie binding is host-based, so resolving against the pre-rewrite
host could miss or misattribute cookies. the default cookie lookup never
touches the backend session: building it needs a running loop (aiohttp>=3.14),
which would defeat preview()'s own pre-loop inspection use case, and an unbuilt
session has no cookies to report anyway - pass cookies= explicitly to preview
cookies that would come from a not-yet-built session.
"""
url = self._apply_domain_overwrites(url)
proxy = self._get_proxy(url, kwargs.pop("proxies", None)) proxy = self._get_proxy(url, kwargs.pop("proxies", None))
merged = {**self._default_headers, **(kwargs.pop("headers", None) or {})} merged = {**self._default_headers, **(kwargs.pop("headers", None) or {})}
headers = self._apply_overwrites(merged) headers = self._apply_overwrites(merged)
@@ -297,7 +359,7 @@ class ExtendedSession:
timeout_total = timeout if isinstance(timeout, (int, float)) else None timeout_total = timeout if isinstance(timeout, (int, float)) else None
cookies = kwargs.pop("cookies", None) cookies = kwargs.pop("cookies", None)
if cookies is None: if cookies is None and self._session is not None:
cookies = self._cookies_for_url(url) cookies = self._cookies_for_url(url)
if cookies: if cookies:
cookie_header = "; ".join(f"{k}={v}" for k, v in cookies.items()) cookie_header = "; ".join(f"{k}={v}" for k, v in cookies.items())
@@ -307,7 +369,7 @@ class ExtendedSession:
return RequestPreview( return RequestPreview(
method=method, method=method,
url=self._apply_domain_overwrites(url), url=url,
headers=headers, headers=headers,
proxy=proxy, proxy=proxy,
data=kwargs.get("data"), data=kwargs.get("data"),
@@ -322,10 +384,9 @@ class ExtendedSession:
async def _raw_request(self, method, url, **kwargs) -> Response: async def _raw_request(self, method, url, **kwargs) -> Response:
"""send one request with aiohttp and adapt it into a Response """send one request with aiohttp and adapt it into a Response
this is the backend seam: a subclass can override only this method (using a the backend seam: a subclass overrides only this to swap HTTP clients,
different HTTP client, e.g. a TLS-fingerprinting one), building a Response building a Response from that backend's primitives; everything else
from that backend's primitives. everything else (overwrites, retries, (overwrites, retries, preview) is inherited.
preview) is inherited.
""" """
response = await self.session.request(method, url, **kwargs) response = await self.session.request(method, url, **kwargs)
async with response: async with response:
@@ -342,8 +403,21 @@ class ExtendedSession:
) )
async def request(self, method, url, **kwargs) -> Response: async def request(self, method, url, **kwargs) -> Response:
"""make a request, applying overwrites/domain rewrites/proxy resolution""" """make a request, applying overwrites/domain rewrites/proxy resolution
kwargs["proxy"] = self._get_proxy(url, kwargs.pop("proxies", None))
raises the real exception subtype on failure (ServerTimeoutError on a total
timeout; any other aiohttp.ClientError re-raised as-is, not flattened, so
callers can branch by type or read subtype attributes like .status/.headers).
a native proxy= is never silently clobbered by resolved proxies= (that would
unmask the caller's real IP) - pass only one of them.
"""
resolved_proxy = self._get_proxy(url, kwargs.pop("proxies", None))
if kwargs.get("proxy") is not None and resolved_proxy is not None:
raise ValueError("pass only one of proxy= or proxies= (session/per-call), not both")
# a bare proxy=None must not shadow a resolved proxies= (that would send direct and
# unmask the caller's IP); setdefault can't fix it because the None key already exists
if kwargs.get("proxy") is None:
kwargs["proxy"] = resolved_proxy
debug = kwargs.pop("debug", False) debug = kwargs.pop("debug", False)
merged = {**self._default_headers, **(kwargs.get("headers") or {})} merged = {**self._default_headers, **(kwargs.get("headers") or {})}
@@ -354,10 +428,9 @@ class ExtendedSession:
if isinstance(timeout, (int, float)): if isinstance(timeout, (int, float)):
kwargs["timeout"] = aiohttp.ClientTimeout(total=timeout) kwargs["timeout"] = aiohttp.ClientTimeout(total=timeout)
elif timeout is None and "timeout" in kwargs: elif timeout is None and "timeout" in kwargs:
# an explicit timeout=None reaches aiohttp as ClientTimeout(total=None), # explicit timeout=None would reach aiohttp as ClientTimeout(total=None) -
# which DISABLES the timeout and overrides the session default; drop it so # infinite, disabling the session default - so drop it instead (matters for
# the session-level timeout applies (matters for request_with_retries, whose # request_with_retries, whose timeout kwarg defaults to None)
# timeout kwarg defaults to None)
del kwargs["timeout"] del kwargs["timeout"]
url = self._apply_domain_overwrites(url) url = self._apply_domain_overwrites(url)
@@ -370,13 +443,14 @@ class ExtendedSession:
log.info("redirect chain: %s", result.redirect_chain) log.info("redirect chain: %s", result.redirect_chain)
return result return result
except asyncio.TimeoutError as error: except asyncio.TimeoutError as error:
# a total ClientTimeout raises a bare asyncio.TimeoutError, which is NOT an # not an aiohttp.ClientError subclass - wrap as ServerTimeoutError so
# aiohttp.ClientError subclass — wrap it as ServerTimeoutError (which IS both # callers get a typed failure and request_with_retries can label it a timeout
# a ClientError AND a TimeoutError) so direct callers get a typed failure and
# request_with_retries can still label it a timeout
raise aiohttp.ServerTimeoutError(f"timeout for {url}: {error}") from error raise aiohttp.ServerTimeoutError(f"timeout for {url}: {error}") from error
except aiohttp.ClientError as error: except aiohttp.ClientError as error:
raise aiohttp.ClientError(f"client error for {url}: {error}") from error # re-raise the original subtype (not flattened) - request_with_retries still
# catches the base aiohttp.ClientError below and is unaffected
log.error("client error for %s: %s", url, error)
raise
async def request_with_retries( async def request_with_retries(
self, method, url, *, data=None, proxies=None, timeout=None, attempts=None, self, method, url, *, data=None, proxies=None, timeout=None, attempts=None,
@@ -387,13 +461,11 @@ class ExtendedSession:
returns a Response on success (or non-retryable status), or a falsy returns a Response on success (or non-retryable status), or a falsy
FailureResponse if every attempt fails. backoff is exponential FailureResponse if every attempt fails. backoff is exponential
(backoff_base ** attempt). (backoff_base ** attempt). timeout=None falls back to the session-level
timeout (see request()'s note on why explicit None is dropped, not passed
timeout defaults to None, which falls back to the session-level timeout set through). attempts=0 floors to 1 (via commons.aretry), not DEFAULT_ATTEMPTS.
at construction (aioweb pops a None timeout so it does not reach aiohttp as an
infinite ClientTimeout); pass a number to override per call.
""" """
attempts = attempts or DEFAULT_ATTEMPTS attempts = DEFAULT_ATTEMPTS if attempts is None else attempts
body_data, body_json = _route_body(data) body_data, body_json = _route_body(data)
if debug: if debug:
@@ -425,9 +497,7 @@ class ExtendedSession:
attempts, url, exhausted.response.status_code) attempts, url, exhausted.response.status_code)
return exhausted.response return exhausted.response
except asyncio.TimeoutError: except asyncio.TimeoutError:
# request() wraps a total timeout as ServerTimeoutError (a ClientError AND a # catch before ClientError so a timeout is labeled as such, not generic
# TimeoutError); catch the timeout case first so it's labeled a timeout rather
# than falling into the generic client-error branch below
log.error("all %d attempts timed out for %s", attempts, url) log.error("all %d attempts timed out for %s", attempts, url)
return FailureResponse(reason="timeout", url=url) return FailureResponse(reason="timeout", url=url)
except aiohttp.ClientError as error: except aiohttp.ClientError as error:
@@ -441,19 +511,14 @@ class ExtendedSession:
# lifecycle # lifecycle
async def close(self): async def close(self):
"""close the backend session override if the backend's close differs """close the backend session - override if the backend's close differs; no-op if never built"""
a no-op if the session was never built (lazy construction means a session
that made no request and was never otherwise touched has nothing to close).
"""
if self._session is not None: if self._session is not None:
await self._session.close() await self._session.close()
def _is_closed(self) -> bool: def _is_closed(self) -> bool:
"""whether the backend session is closed override for non-aiohttp backends """whether the backend session is closed - override for non-aiohttp backends
an unbuilt (never-lazily-created) session counts as closed: nothing was an unbuilt session counts as closed: nothing was opened, nothing to leak.
opened, so there is nothing to leak and __del__ should not warn.
""" """
if self._session is None: if self._session is None:
return True return True
@@ -466,8 +531,8 @@ class ExtendedSession:
await self.close() await self.close()
def __del__(self): def __del__(self):
# do NOT attempt async cleanup here spinning event loops in a finalizer # no async cleanup here - spinning event loops in a finalizer is unsafe;
# is unsafe. just warn so the leak is visible; callers must close explicitly. # just warn so the leak is visible, callers must close explicitly
try: try:
closed = self._is_closed() closed = self._is_closed()
except Exception: except Exception: