aioweb

Async HTTP session wrapper over aiohttp. Adds session-level proxies, header overwrites, ephemeral (per-request generated) headers, domain rewriting, request previews / cURL export, and retry-with-backoff. The byte-sending is isolated behind one overridable method (_raw_request), so a TLS-fingerprinting backend can subclass and swap the HTTP client while inheriting everything else.

Install

requirements.txt:

aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v1.1.0

Direct:

pip install "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v1.1.0"

Requires aiohttp and yarl (pulled transitively), plus the sibling commons library (a private git+ssh package — the install needs access to the rethink-public org).

Drop the @v1.1.0 suffix from the line above to install the latest unpinned.

Usage

from aioweb import ExtendedSession

async with ExtendedSession(proxies={"https": "http://user:pass@host:port"}, timeout=15) as s:
    resp = await s.request_with_retries("GET", "https://example.com")
    if resp:                          # FailureResponse is falsy
        data = resp.json()            # or resp.text(), resp.content

Sessions must be closed explicitly — use async with or await s.close(). There is no __del__ auto-close (unsafe for async resources); leaking a session emits a ResourceWarning.

Responses

request/request_with_retries return a Response (success or non-retryable status) or a falsy FailureResponse (all retries failed). Both expose the same surface as properties, so callers branch uniformly:

• status_code, headers, url, reason, cookies, history, redirect_chain
• is_success (2xx), is_redirect
• content (bytes), text(encoding=None), json() (None if not JSON)
• raise_for_status() raises AiowebError on non-2xx
• bool(resp) / if resp: is is_success

Retries

request_with_retries retries on exceptions and retryable statuses (429, 500, 502, 503, 504 by default), with exponential backoff (backoff_base ** attempt).

resp = await s.request_with_retries(
    "GET", url, attempts=5, backoff_base=2.0,
    retry_statuses={429, 503},        # override which statuses retry
)

Returns a FailureResponse (falsy) if every attempt fails.

request() (the non-retrying call) raises on failure: a total timeout raises aiohttp.ServerTimeoutError, and any other network/protocol failure raises its real aiohttp.ClientError subtype as-is (ClientConnectorError, ClientProxyConnectionError, ClientResponseError with .status/.headers, TooManyRedirects, ...) — it is not flattened into the base ClientError, so direct callers can branch by type or read subtype attributes. request_with_retries catches the base aiohttp.ClientError (and asyncio.TimeoutError) across all attempts and returns a falsy FailureResponse instead of raising.

Header overwrites & ephemeral headers

s.overwrite_header("User-Agent", "custom")     # replace per request
s.overwrite_inject(True)                        # also add when absent
s.set_ephemeral("X-Time", lambda: str(time.time()))  # generated fresh each request

With inject=False (default) overwrites only replace headers already present in a request; with inject=True they're added regardless.

Domain rewriting

s.overwrite_domain("internal.local", "127.0.0.1")   # host-substring rewrite

Preview / debug

print(s.preview("POST", url, json={"a": 1}).as_curl())   # equivalent cURL command

preview() is synchronous and never touches the backend session — it works before any event loop is running (e.g. building a preview at import/setup time), not just inside asyncio.run(). Its default cookie lookup only reads cookies from an already-built session; if the session hasn't been built yet there are no cookies to read, so it returns none by default (pass cookies= explicitly to preview cookies for a session that hasn't sent a request yet). as_curl() renders params (merged into the url's query string) and timeout (as --max-time) as well as headers/body/proxy, so the emitted command is faithful to what request() actually sends. Pass debug=True to request_with_retries to log the cURL preview and request flow.

Custom backends

The _raw_request(method, url, **kwargs) -> Response method is the only place that touches the HTTP client. To use a different backend (e.g. a TLS-fingerprinting client like curl_cffi), subclass ExtendedSession and override just _raw_request, building a Response from that backend's primitives:

class MySession(ExtendedSession):
    async def _raw_request(self, method, url, **kwargs):
        r = await my_client.request(method, url, **kwargs)
        return Response(
            status_code=r.status, headers=r.headers, content=await r.read(),
            url=str(r.url), reason=r.reason,
        )

Everything else — header overwrites, ephemeral headers, domain rewriting, proxy resolution, retries, previews — is inherited. Response is built from primitives (status, headers, content, url, history) precisely so any backend can produce one.

Migrating from the original

Back-compat shims are in place for the common path:

• aiowebResponse is aliased to Response (the class was renamed) — old imports work.
• request_retries(session, ...) and test_proxies(session) remain as module functions.
• raise_for_status now raises AiowebError (a subclass of Exception), so except Exception still catches it.

Two changes can't be shimmed without re-introducing the bugs they fix:

• is_success is now a property on FailureResponse, not a method. Code that called failure.is_success() must drop the parens to failure.is_success. (Code that wrote if failure.is_success was previously always-truthy — a bug — and now behaves correctly.)
• No __del__ auto-close. Sessions must be closed via async with or await s.close(); a leaked session emits a ResourceWarning. The old finalizer-based auto-close was unsafe and was removed.

Changelog

v1.1.0

• Hang-guard on request() — a wedged backend can no longer park the loop forever. request() now wraps the backend call (_raw_request) in an outer asyncio.wait_for deadline sized at the effective timeout + 5s slack (_HANG_GUARD_SLACK). The backend's own timeout still fires first in every healthy failure (slack, not replacement); the envelope only trips when the backend's timer is dead — e.g. a curl_cffi transfer that loses both its wakeup and its own enforcement in the same failure — and its cancellation is what unwedges the orphaned transfer. The guard's deadline is the per-call numeric timeout when given, else the session timeout (Session(timeout=...), default 10s); it composes cleanly with request_with_retries (a wedge becomes one logged ServerTimeoutError, the next attempt proceeds, and an exhausted retry returns a FailureResponse rather than hanging). Opt-out: a session deliberately constructed unbounded (Session(timeout=None)) stays unguarded, for genuinely long-lived calls (long-poll / streaming). Additive; no new knob.

v0.1.13

• Session-default timeout no longer poisons pooled keep-alive connections. The default ClientTimeout set sock_read=timeout/2 alongside total. Under aiohttp>=3.14, that read timer re-arms on every request dispatched over a pooled protocol, including idle connections between requests; when it fires it permanently poisons the pooled connection (SocketTimeoutError on the next use, instantly, without contacting the server) — a real error from the server (e.g. a 503) could come back as a client-side FailureResponse(status=0, reason='timeout') instead. sock_read is now dropped from the session default; total (and connect/sock_connect) still bound every request, and the per-call timeout=N path (ClientTimeout(total=N), no sock_read) was already unaffected.

v0.1.12

• Docstring-only. Restored one-line docstrings on FailureResponse's redirect_chain, text(), json(), and raise_for_status() — a prior de-bloat pass stripped them below the public tier while the Response twins kept theirs. No behavior change.

v0.1.11

• preview() no longer requires a running event loop. Its default cookie lookup (cookies= not passed) previously called _cookies_for_url(), which reached the backend session and built it if absent — under aiohttp>=3.14 that needs a running loop, so preview() raised RuntimeError('no running event loop') when called before the loop starts, defeating its own pre-loop build/inspect use case (v0.1.7). Now the default cookie lookup is skipped entirely when the session hasn't been built yet (nothing could have been set on a session that doesn't exist); cookies={} and in-loop calls are unaffected.
• set_cookie(domain=None) now honors path. The domain=None branch returned right after update_cookies(), before the line that sets the morsel's path — so a shared cookie (set_cookie(name, value, path="/api"), no domain=) always stored the SimpleCookie default path="/" instead. The morsel's path is now set before the domain=None early return. The domain-bound branch was already correct and is unchanged.

v0.1.10

• Docs-only pass: compressed module/method docstrings and comments that restated README/CLAUDE prose, replaced em-dashes with hyphens. No behavior change.

v0.1.9

• _get_proxy()/request() proxy resolution now checks is None, not truthiness. A per-call proxies={} previously fell back to the session's configured proxies instead of disabling them for that call. request() also no longer clobbers a native proxy= kwarg with the resolved session/proxies= value (a real IP-unmasking leak) — passing both now raises ValueError instead of silently picking one.
• Response.text(encoding=...) no longer returns a stale cached decode. A second call with an explicit encoding= previously still returned the first (possibly differently-encoded) cached decode; an explicit encoding now bypasses the cache.
• preview() now rewrites the url before resolving cookies, matching what request() actually sends — previously cookies were resolved against the pre-rewrite host, which could miss or misattribute host-bound cookies.
• request_with_retries(attempts=0) now floors to 1 attempt, not silently DEFAULT_ATTEMPTS (3). attempts is checked with is None, not truthiness.

v0.1.8

• request() no longer flattens aiohttp.ClientError subtypes. Every failure (connect errors, proxy errors, raise_for_status()-style response errors, redirect limits, ...) was re-raised as a bare aiohttp.ClientError, losing the real subtype and its attributes (.os_error, .status, .headers, ...) — a direct caller doing except ClientProxyConnectionError: or if e.status == 401 would silently never match. Now the original exception is re-raised as-is (its subtype, attributes, and __cause__ all preserved). request_with_retries still catches the base aiohttp.ClientError across attempts, so its behavior (and its FailureResponse return on exhaustion) is unchanged.
• as_curl() now renders params and timeout. Previously a preview built with params= silently omitted the query string (and a timeout= omitted --max-time), so a debug=True cURL replay of a params-driven request hit a different URL than the one actually sent. params are now merged into the url's query string (via yarl) and timeout is emitted as --max-time.

v0.1.7

• get_cookies() now returns real cookies. Previously called filter_cookies() with no URL, which only ever returns domain-less shared cookies — every normal domain-bound cookie (including ones set by a real Set-Cookie response) was silently omitted. Now iterates the jar directly.
• set_cookie() no longer leaks a shared cookie to every host. A bare hostname (domain="example.com") built a schemeless URL, which aiohttp's jar treats as a domain-less "shared" cookie sent with every request the session makes, including unrelated hosts. A scheme is now added when missing so the cookie is scoped to that host.
• set_cookie() now honors path (previously ignored — the cookie always landed at path="/"). domain=None is unchanged in meaning but now stores a truly shared cookie (sent to every host) instead of one silently bound to localhost only, which made set_cookie(name, value) (no domain) a silent no-op for any real request.
• The backend session is built lazily, not in __init__. Under aiohttp 3.14, constructing aiohttp.ClientSession requires a running event loop; eager construction crashed the common host pattern of attaching a session before the loop starts (e.g. bot.http = ExtendedSession(...) in Bot.__init__). The session (and any subclass's _create_session override) now builds on first access instead.

v0.1.2

• Pinned commons to v0.2.1 (retry attempts floor fix).

v0.1.1

• JSON list bodies now route to json= (were wrongly form-encoded via data= — only dicts went to json= before).
• Exhausted retries return the real last response. When every attempt hit a retryable status (429/5xx), the loop discarded it and returned a synthetic FailureResponse (status 0); now the real last 4xx/5xx Response is returned (only a pure-exception failure yields FailureResponse).
• Retry/backoff moved onto commons.aretry (shared engine); backoff schedule unchanged. Adds a commons dependency.

Versioning

Releases are tagged vX.Y.Z. The install line above pins a release; drop the @vX.Y.Z suffix to install the latest unpinned. Pin deliberately for reproducible installs.