7 Commits
Author SHA1 Message Date
dsql 987c4bdf9c chore: bump to 1.3.0
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-10-16 04:00:40 -04:00
dsql 250c9cf2eb fix: copy config dicts, case-insensitive scheme/headers, guard honors ClientTimeout
T1-4: proxies/overwrites/domain_overwrites were aliased, so clear_proxies()
on one session emptied a sibling built from the same config dict, and the
proxy lookup used the raw scheme so HTTP:// missed the http key. both sent
the request direct. copy the dicts; resolve the scheme via yarl (lowercase).

T2-7: the hang-guard ignored a per-call ClientTimeout and killed the call at
the session timeout + slack. read .total off it.

T2-8: header names merged and overwritten case-sensitively, so curl_cffi
sent both User-Agent and user-agent (400 on strict servers). merge through
CIMultiDict and match overwrites case-insensitively.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-10-16 04:00:33 -04:00
dsql b009c0cf50 chore: bump to 1.2.0 (logging-discipline audit)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-08-10 23:00:50 -04:00
dsql e6eddf5bc3 fix: logging discipline in request()/request_with_retries
- delete the per-attempt log.error before raise in request()'s ClientError branch: the
  path re-raises (raise XOR log), commons.aretry already emits the per-attempt WARNING
  with the attempt count, and the terminal branch logs on exhaustion - the line was pure
  duplicate reporting of a retried failure.
- demote the retryable-status per-attempt line to debug for the same reason (it raises
  _RetryStatus; aretry owns the retry WARNING).
- level the four terminal branches of request_with_retries from ERROR to WARNING: they
  SWALLOW exhaustion into a falsy FailureResponse the caller branches on (recovered
  cleanly per the swallow contract), so the lib does not claim ERROR on the caller's
  behalf; the caller escalates on the falsy result. all four stay one uniform level.

verified: the terminal branch still fires on genuine exhaustion, request_with_retries
still returns a falsy FailureResponse, and the deleted per-attempt ERROR no longer emits.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-08-09 02:08:58 -04:00
dsql 952fe8a00f feat: hang-guard request() with an outer asyncio deadline
a backend can lose both its transfer wakeup and its own timeout enforcement in one
failure (observed: a curl_cffi request that parked an aioweb consumer's event loop for
24h+ - loop idle, nothing in flight, the await never completing). wrap _raw_request in
asyncio.wait_for at the effective timeout + 5s slack: the backend's own timeout still
fires first in every healthy failure, the envelope only trips when the backend's timer
is dead, and its cancellation unwedges the orphaned transfer. deadline = per-call numeric
timeout else the session timeout (default 10); timeout=None from request_with_retries
guards on the session bound (that was the wedge path). explicit Session(timeout=None)
stays unguarded - the documented opt-out for genuinely unbounded calls. additive, no new
knob. fired: a never-completing backend hangs forever without this, dies at deadline+slack
with it, and request_with_retries returns a FailureResponse instead of hanging.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-27 11:33:35 -04:00
dsql 6ec132f97c build: use git+https for inter-lib deps (docker ssh limitation)
docker builds can't use git+ssh (no ssh key / agent in the build), so the inter-lib
dependency references move to git+https (repos are public, anonymous clone). pins are
unchanged in target; bump to 1.0.2 so the https dependency spec ships under a new tag.
README install lines intentionally keep the ssh form for local/dev use.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-20 22:25:06 -04:00
dsql 6633d6cba3 fix: pin inter-lib dependencies to their v1.0.0 tags
the v1.0.0 release still pinned pre-1.0.0 sibling tags, so a fresh install dragged in
stale transitive deps. update the pin(s) to the current v1.0.x release and bump this lib
to 1.0.1 so the corrected dependency chain ships under a new tag (v1.0.0 left intact).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-17 17:46:10 -04:00
3 changed files with 86 additions and 28 deletions

No files matched your search

+19 -3
View File
@@ -11,19 +11,19 @@ and swap the HTTP client while inheriting everything else.
`requirements.txt`: `requirements.txt`:
``` ```
aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v1.0.0 aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v1.3.0
``` ```
Direct: Direct:
```bash ```bash
pip install "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v1.0.0" pip install "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v1.3.0"
``` ```
Requires `aiohttp` and `yarl` (pulled transitively), plus the sibling `commons` library Requires `aiohttp` and `yarl` (pulled transitively), plus the sibling `commons` library
(a private `git+ssh` package — the install needs access to the `rethink-public` org). (a private `git+ssh` package — the install needs access to the `rethink-public` org).
Drop the `@v1.0.0` suffix from the line above to install the latest unpinned. Drop the `@v1.3.0` suffix from the line above to install the latest unpinned.
## Usage ## Usage
@@ -150,6 +150,22 @@ Two changes can't be shimmed without re-introducing the bugs they fix:
## Changelog ## Changelog
### v1.1.0
- **Hang-guard on `request()` — a wedged backend can no longer park the loop forever.**
`request()` now wraps the backend call (`_raw_request`) in an outer `asyncio.wait_for`
deadline sized at the effective timeout **+ 5s slack** (`_HANG_GUARD_SLACK`). The
backend's own timeout still fires first in every healthy failure (slack, not
replacement); the envelope only trips when the backend's timer is dead — e.g. a
`curl_cffi` transfer that loses both its wakeup and its own enforcement in the same
failure — and its cancellation is what unwedges the orphaned transfer. The guard's
deadline is the per-call numeric `timeout` when given, else the session timeout
(`Session(timeout=...)`, default 10s); it composes cleanly with `request_with_retries`
(a wedge becomes one logged `ServerTimeoutError`, the next attempt proceeds, and an
exhausted retry returns a `FailureResponse` rather than hanging). **Opt-out:** a
session deliberately constructed unbounded (`Session(timeout=None)`) stays unguarded,
for genuinely long-lived calls (long-poll / streaming). Additive; no new knob.
### v0.1.13 ### v0.1.13
- **Session-default timeout no longer poisons pooled keep-alive connections.** - **Session-default timeout no longer poisons pooled keep-alive connections.**
+2 -2
View File
@@ -4,13 +4,13 @@ build-backend = "hatchling.build"
[project] [project]
name = "aioweb" name = "aioweb"
version = "1.0.0" version = "1.3.0"
description = "Async HTTP session wrapper over aiohttp — proxies, header overwrites, retries, previews. Config-free, installable." description = "Async HTTP session wrapper over aiohttp — proxies, header overwrites, retries, previews. Config-free, installable."
requires-python = ">=3.10" requires-python = ">=3.10"
dependencies = [ dependencies = [
"aiohttp>=3.9", "aiohttp>=3.9",
"yarl>=1.9", "yarl>=1.9",
"commons @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v0.2.1", "commons @ git+https://git.rethinkstudios.io/rethink-public/commons.git@v1.0.0",
] ]
[tool.hatch.metadata] [tool.hatch.metadata]
+65 -23
View File
@@ -15,6 +15,7 @@ import warnings
from http.cookies import SimpleCookie from http.cookies import SimpleCookie
import aiohttp import aiohttp
from multidict import CIMultiDict
from yarl import URL from yarl import URL
from commons import aretry from commons import aretry
@@ -63,6 +64,7 @@ log = logging.getLogger(__name__)
DEFAULT_ATTEMPTS = 3 DEFAULT_ATTEMPTS = 3
DEFAULT_BACKOFF_BASE = 2.0 DEFAULT_BACKOFF_BASE = 2.0
RETRY_STATUSES = frozenset({429, 500, 502, 503, 504}) RETRY_STATUSES = frozenset({429, 500, 502, 503, 504})
_HANG_GUARD_SLACK = 5
class ExtendedSession: class ExtendedSession:
@@ -84,12 +86,15 @@ class ExtendedSession:
kwargs: passed through to aiohttp.ClientSession kwargs: passed through to aiohttp.ClientSession
""" """
self.inject = inject self.inject = inject
self.header_overwrites = overwrites or {} # copied, not aliased: a shared config dict must not be mutated by (or leak a
self.domain_overwrites = domain_overwrites or {} # clear_proxies() into) a sibling session
self.header_overwrites = dict(overwrites or {})
self.domain_overwrites = dict(domain_overwrites or {})
self.ephemeral_headers = {} self.ephemeral_headers = {}
self.proxies = proxies or {} self.proxies = dict(proxies or {})
# own header layer, not aiohttp privates, so update_headers/clear_headers work # own header layer, not aiohttp privates, so update_headers/clear_headers work;
self._default_headers = dict(headers or {}) # case-insensitive like HTTP header names
self._default_headers = CIMultiDict(headers or {})
self._session_timeout = timeout self._session_timeout = timeout
self._session_kwargs = kwargs self._session_kwargs = kwargs
# aiohttp>=3.14 needs a running loop to build ClientSession; built lazily instead # aiohttp>=3.14 needs a running loop to build ClientSession; built lazily instead
@@ -138,8 +143,12 @@ class ExtendedSession:
self.header_overwrites[header] = value self.header_overwrites[header] = value
def _apply_overwrites(self, request_headers): def _apply_overwrites(self, request_headers):
"""apply static overwrites and ephemeral headers to a request's headers""" """apply static overwrites and ephemeral headers to a request's headers
request_headers = dict(request_headers or {})
header names match case-insensitively, so an overwrite replaces a `user-agent`
rather than sending it alongside a second `User-Agent`
"""
request_headers = CIMultiDict(request_headers or {})
# list()-snapshot so a concurrent mutation of the shared dicts can't raise # list()-snapshot so a concurrent mutation of the shared dicts can't raise
# "dict changed size during iteration" # "dict changed size during iteration"
for header, value in list(self.header_overwrites.items()): for header, value in list(self.header_overwrites.items()):
@@ -152,7 +161,7 @@ class ExtendedSession:
request_headers.update(value) request_headers.update(value)
else: else:
request_headers[header] = value request_headers[header] = value
return request_headers return dict(request_headers)
def set_ephemeral(self, header, value_callable): def set_ephemeral(self, header, value_callable):
"""register a header generated fresh (via callable) for each request""" """register a header generated fresh (via callable) for each request"""
@@ -261,7 +270,7 @@ class ExtendedSession:
falling back to them. falling back to them.
""" """
proxies = self.proxies if proxies is None else proxies proxies = self.proxies if proxies is None else proxies
scheme = url.split("://")[0] scheme = URL(url).scheme
return proxies.get(scheme) return proxies.get(scheme)
# ------------------------------------------------------------------------- # -------------------------------------------------------------------------
@@ -352,7 +361,8 @@ class ExtendedSession:
""" """
url = self._apply_domain_overwrites(url) url = self._apply_domain_overwrites(url)
proxy = self._get_proxy(url, kwargs.pop("proxies", None)) proxy = self._get_proxy(url, kwargs.pop("proxies", None))
merged = {**self._default_headers, **(kwargs.pop("headers", None) or {})} merged = CIMultiDict(self._default_headers)
merged.update(kwargs.pop("headers", None) or {})
headers = self._apply_overwrites(merged) headers = self._apply_overwrites(merged)
timeout = kwargs.get("timeout") timeout = kwargs.get("timeout")
@@ -420,11 +430,19 @@ class ExtendedSession:
kwargs["proxy"] = resolved_proxy kwargs["proxy"] = resolved_proxy
debug = kwargs.pop("debug", False) debug = kwargs.pop("debug", False)
merged = {**self._default_headers, **(kwargs.get("headers") or {})} merged = CIMultiDict(self._default_headers)
merged.update(kwargs.get("headers") or {})
kwargs["headers"] = self._apply_overwrites(merged) kwargs["headers"] = self._apply_overwrites(merged)
kwargs["headers"] = {str(k): str(v) for k, v in kwargs["headers"].items()} kwargs["headers"] = {str(k): str(v) for k, v in kwargs["headers"].items()}
timeout = kwargs.get("timeout") timeout = kwargs.get("timeout")
# the hang-guard deadline: a per-call timeout (a number, or a ClientTimeout's
# .total), else the session default. a timeout=None (omitted, or the explicit None
# request_with_retries passes) means "use the session bound" - so it guards on
# self._session_timeout, NOT unguarded. unguarded only when the caller asked for no
# bound: a session constructed with timeout=None (the documented long-poll/streaming
# opt-out), or a per-call ClientTimeout(total=None).
guard_secs = self._session_timeout if timeout is None else getattr(timeout, "total", timeout)
if isinstance(timeout, (int, float)): if isinstance(timeout, (int, float)):
kwargs["timeout"] = aiohttp.ClientTimeout(total=timeout) kwargs["timeout"] = aiohttp.ClientTimeout(total=timeout)
elif timeout is None and "timeout" in kwargs: elif timeout is None and "timeout" in kwargs:
@@ -438,18 +456,32 @@ class ExtendedSession:
log.info("sending request to: %s", url) log.info("sending request to: %s", url)
try: try:
result = await self._raw_request(method, url, **kwargs) # hang-guard: an outer asyncio deadline (session/curl timers can die in the
# same failure that parks the transfer - a backend that loses both wakeup and
# its own timeout enforcement would otherwise leave this await parked forever).
# the backend's own timeout still fires first in every healthy failure (slack,
# not replacement); wait_for only trips when the backend's enforcement is dead,
# and its cancellation is what unwedges the orphaned transfer.
if guard_secs is not None:
result = await asyncio.wait_for(
self._raw_request(method, url, **kwargs), guard_secs + _HANG_GUARD_SLACK
)
else:
result = await self._raw_request(method, url, **kwargs)
if debug and result.redirect_chain: if debug and result.redirect_chain:
log.info("redirect chain: %s", result.redirect_chain) log.info("redirect chain: %s", result.redirect_chain)
return result return result
except asyncio.TimeoutError as error: except asyncio.TimeoutError as error:
# not an aiohttp.ClientError subclass - wrap as ServerTimeoutError so # not an aiohttp.ClientError subclass - wrap as ServerTimeoutError so callers
# callers get a typed failure and request_with_retries can label it a timeout # get a typed failure and request_with_retries can label it a timeout. now also
# catches the hang-guard firing (asyncio.wait_for raises asyncio.TimeoutError).
raise aiohttp.ServerTimeoutError(f"timeout for {url}: {error}") from error raise aiohttp.ServerTimeoutError(f"timeout for {url}: {error}") from error
except aiohttp.ClientError as error: except aiohttp.ClientError:
# re-raise the original subtype (not flattened) - request_with_retries still # re-raise the original subtype (not flattened) - request_with_retries still
# catches the base aiohttp.ClientError below and is unaffected # catches the base aiohttp.ClientError below and is unaffected. no log here:
log.error("client error for %s: %s", url, error) # this path RAISES, so the exception carries the failure (raise XOR log); on the
# retrying path commons.aretry emits the per-attempt WARNING and the terminal
# branch logs on exhaustion - logging here would double-report the same failure.
raise raise
async def request_with_retries( async def request_with_retries(
@@ -483,7 +515,11 @@ class ExtendedSession:
headers=headers, proxies=proxies, timeout=timeout, debug=debug, headers=headers, proxies=proxies, timeout=timeout, debug=debug,
) )
if response.status_code in retry_statuses: if response.status_code in retry_statuses:
log.warning("retryable status %s for %s", response.status_code, url) # debug, not warning: this RAISES _RetryStatus to trigger a retry, so it must
# not also warn/error the same failure (raise XOR log). commons.aretry emits
# the per-attempt WARNING with the attempt count when it catches and retries;
# the terminal branch below logs on exhaustion.
log.debug("retryable status %s for %s", response.status_code, url)
raise _RetryStatus(response) raise _RetryStatus(response)
return response return response
@@ -492,19 +528,25 @@ class ExtendedSession:
attempt, attempts=attempts, backoff=1.0, factor=backoff_base, attempt, attempts=attempts, backoff=1.0, factor=backoff_base,
jitter=False, on=(Exception,), jitter=False, on=(Exception,),
) )
# terminal path: retries are exhausted and the failure is SWALLOWED into a falsy
# FailureResponse the caller branches on (the documented contract). rule 2 mandates a
# log on a swallow; level is WARNING because the lib recovered cleanly into a
# branchable value - the failing op is the caller's to escalate once it sees the falsy
# result, so the lib does not claim ERROR on the caller's behalf. all four branches are
# the same event class (exhausted retries) and stay at one uniform level.
except _RetryStatus as exhausted: except _RetryStatus as exhausted:
log.error("all %d attempts failed for %s (last status %s)", log.warning("all %d attempts failed for %s (last status %s)",
attempts, url, exhausted.response.status_code) attempts, url, exhausted.response.status_code)
return exhausted.response return exhausted.response
except asyncio.TimeoutError: except asyncio.TimeoutError:
# catch before ClientError so a timeout is labeled as such, not generic # catch before ClientError so a timeout is labeled as such, not generic
log.error("all %d attempts timed out for %s", attempts, url) log.warning("all %d attempts timed out for %s", attempts, url)
return FailureResponse(reason="timeout", url=url) return FailureResponse(reason="timeout", url=url)
except aiohttp.ClientError as error: except aiohttp.ClientError as error:
log.error("all %d attempts failed for %s (client error: %s)", attempts, url, error) log.warning("all %d attempts failed for %s (client error: %s)", attempts, url, error)
return FailureResponse(reason=f"client error: {error}", url=url) return FailureResponse(reason=f"client error: {error}", url=url)
except Exception as error: except Exception as error:
log.error("all %d attempts failed for %s (unexpected: %s)", attempts, url, error) log.warning("all %d attempts failed for %s (unexpected: %s)", attempts, url, error)
return FailureResponse(reason=f"unexpected error: {error}", url=url) return FailureResponse(reason=f"unexpected error: {error}", url=url)
# ------------------------------------------------------------------------- # -------------------------------------------------------------------------