aioweb

Async HTTP session wrapper over aiohttp. Adds session-level proxies, header overwrites, ephemeral (per-request generated) headers, domain rewriting, request previews / cURL export, and retry-with-backoff. The byte-sending is isolated behind one overridable method (_raw_request), so a TLS-fingerprinting backend can subclass and swap the HTTP client while inheriting everything else.

Install

requirements.txt:

aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.9

Direct:

pip install "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.9"

Requires aiohttp and yarl (pulled transitively).

Drop the @v0.1.9 suffix from the line above to install the latest unpinned.

Usage

from aioweb import ExtendedSession

async with ExtendedSession(proxies={"https": "http://user:pass@host:port"}, timeout=15) as s:
    resp = await s.request_with_retries("GET", "https://example.com")
    if resp:                          # FailureResponse is falsy
        data = resp.json()            # or resp.text(), resp.content

Sessions must be closed explicitly — use async with or await s.close(). There is no __del__ auto-close (unsafe for async resources); leaking a session emits a ResourceWarning.

Responses

request/request_with_retries return a Response (success or non-retryable status) or a falsy FailureResponse (all retries failed). Both expose the same surface as properties, so callers branch uniformly:

• status_code, headers, url, reason, cookies, history, redirect_chain
• is_success (2xx), is_redirect
• content (bytes), text(encoding=None), json() (None if not JSON)
• raise_for_status() raises AiowebError on non-2xx
• bool(resp) / if resp: is is_success

Retries

request_with_retries retries on exceptions and retryable statuses (429, 500, 502, 503, 504 by default), with exponential backoff (backoff_base ** attempt).

resp = await s.request_with_retries(
    "GET", url, attempts=5, backoff_base=2.0,
    retry_statuses={429, 503},        # override which statuses retry
)

Returns a FailureResponse (falsy) if every attempt fails.

request() (the non-retrying call) raises on failure: a total timeout raises aiohttp.ServerTimeoutError, and any other network/protocol failure raises its real aiohttp.ClientError subtype as-is (ClientConnectorError, ClientProxyConnectionError, ClientResponseError with .status/.headers, TooManyRedirects, ...) — it is not flattened into the base ClientError, so direct callers can branch by type or read subtype attributes. request_with_retries catches the base aiohttp.ClientError (and asyncio.TimeoutError) across all attempts and returns a falsy FailureResponse instead of raising.

Header overwrites & ephemeral headers

s.overwrite_header("User-Agent", "custom")     # replace per request
s.overwrite_inject(True)                        # also add when absent
s.set_ephemeral("X-Time", lambda: str(time.time()))  # generated fresh each request

With inject=False (default) overwrites only replace headers already present in a request; with inject=True they're added regardless.

Domain rewriting

s.overwrite_domain("internal.local", "127.0.0.1")   # host-substring rewrite

Preview / debug

print(s.preview("POST", url, json={"a": 1}).as_curl())   # equivalent cURL command

as_curl() renders params (merged into the url's query string) and timeout (as --max-time) as well as headers/body/proxy, so the emitted command is faithful to what request() actually sends. Pass debug=True to request_with_retries to log the cURL preview and request flow.

Custom backends

The _raw_request(method, url, **kwargs) -> Response method is the only place that touches the HTTP client. To use a different backend (e.g. a TLS-fingerprinting client like curl_cffi), subclass ExtendedSession and override just _raw_request, building a Response from that backend's primitives:

class MySession(ExtendedSession):
    async def _raw_request(self, method, url, **kwargs):
        r = await my_client.request(method, url, **kwargs)
        return Response(
            status_code=r.status, headers=r.headers, content=await r.read(),
            url=str(r.url), reason=r.reason,
        )

Everything else — header overwrites, ephemeral headers, domain rewriting, proxy resolution, retries, previews — is inherited. Response is built from primitives (status, headers, content, url, history) precisely so any backend can produce one.

Migrating from the original

Back-compat shims are in place for the common path:

• aiowebResponse is aliased to Response (the class was renamed) — old imports work.
• request_retries(session, ...) and test_proxies(session) remain as module functions.
• raise_for_status now raises AiowebError (a subclass of Exception), so except Exception still catches it.

Two changes can't be shimmed without re-introducing the bugs they fix:

• is_success is now a property on FailureResponse, not a method. Code that called failure.is_success() must drop the parens to failure.is_success. (Code that wrote if failure.is_success was previously always-truthy — a bug — and now behaves correctly.)
• No __del__ auto-close. Sessions must be closed via async with or await s.close(); a leaked session emits a ResourceWarning. The old finalizer-based auto-close was unsafe and was removed.

Changelog

v0.1.9

• _get_proxy()/request() proxy resolution now checks is None, not truthiness. A per-call proxies={} previously fell back to the session's configured proxies instead of disabling them for that call. request() also no longer clobbers a native proxy= kwarg with the resolved session/proxies= value (a real IP-unmasking leak) — passing both now raises ValueError instead of silently picking one.
• Response.text(encoding=...) no longer returns a stale cached decode. A second call with an explicit encoding= previously still returned the first (possibly differently-encoded) cached decode; an explicit encoding now bypasses the cache.
• preview() now rewrites the url before resolving cookies, matching what request() actually sends — previously cookies were resolved against the pre-rewrite host, which could miss or misattribute host-bound cookies.
• request_with_retries(attempts=0) now floors to 1 attempt, not silently DEFAULT_ATTEMPTS (3). attempts is checked with is None, not truthiness.

v0.1.8

• request() no longer flattens aiohttp.ClientError subtypes. Every failure (connect errors, proxy errors, raise_for_status()-style response errors, redirect limits, ...) was re-raised as a bare aiohttp.ClientError, losing the real subtype and its attributes (.os_error, .status, .headers, ...) — a direct caller doing except ClientProxyConnectionError: or if e.status == 401 would silently never match. Now the original exception is re-raised as-is (its subtype, attributes, and __cause__ all preserved). request_with_retries still catches the base aiohttp.ClientError across attempts, so its behavior (and its FailureResponse return on exhaustion) is unchanged.
• as_curl() now renders params and timeout. Previously a preview built with params= silently omitted the query string (and a timeout= omitted --max-time), so a debug=True cURL replay of a params-driven request hit a different URL than the one actually sent. params are now merged into the url's query string (via yarl) and timeout is emitted as --max-time.

v0.1.7

• get_cookies() now returns real cookies. Previously called filter_cookies() with no URL, which only ever returns domain-less shared cookies — every normal domain-bound cookie (including ones set by a real Set-Cookie response) was silently omitted. Now iterates the jar directly.
• set_cookie() no longer leaks a shared cookie to every host. A bare hostname (domain="example.com") built a schemeless URL, which aiohttp's jar treats as a domain-less "shared" cookie sent with every request the session makes, including unrelated hosts. A scheme is now added when missing so the cookie is scoped to that host.
• set_cookie() now honors path (previously ignored — the cookie always landed at path="/"). domain=None is unchanged in meaning but now stores a truly shared cookie (sent to every host) instead of one silently bound to localhost only, which made set_cookie(name, value) (no domain) a silent no-op for any real request.
• The backend session is built lazily, not in __init__. Under aiohttp 3.14, constructing aiohttp.ClientSession requires a running event loop; eager construction crashed the common host pattern of attaching a session before the loop starts (e.g. bot.http = ExtendedSession(...) in Bot.__init__). The session (and any subclass's _create_session override) now builds on first access instead.

v0.1.2

• Pinned commons to v0.2.1 (retry attempts floor fix).

v0.1.1

• JSON list bodies now route to json= (were wrongly form-encoded via data= — only dicts went to json= before).
• Exhausted retries return the real last response. When every attempt hit a retryable status (429/5xx), the loop discarded it and returned a synthetic FailureResponse (status 0); now the real last 4xx/5xx Response is returned (only a pure-exception failure yields FailureResponse).
• Retry/backoff moved onto commons.aretry (shared engine); backoff schedule unchanged. Adds a commons dependency.

Versioning

Releases are tagged vX.Y.Z. The install line above pins a release; drop the @vX.Y.Z suffix to install the latest unpinned. Pin deliberately for reproducible installs.