aioweb/README.md

# aioweb

Async HTTP session wrapper over `aiohttp`. Adds session-level proxies, header
overwrites, ephemeral (per-request generated) headers, domain rewriting, request
previews / cURL export, and retry-with-backoff. The byte-sending is isolated behind
one overridable method (`_raw_request`), so a TLS-fingerprinting backend can subclass
and swap the HTTP client while inheriting everything else.

## Install

`requirements.txt`:

```
aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.13
```

Direct:

```bash
pip install "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.13"
```

Requires `aiohttp` and `yarl` (pulled transitively).

Drop the `@v0.1.13` suffix from the line above to install the latest unpinned.

## Usage

```python
from aioweb import ExtendedSession

async with ExtendedSession(proxies={"https": "http://user:pass@host:port"}, timeout=15) as s:
    resp = await s.request_with_retries("GET", "https://example.com")
    if resp:                          # FailureResponse is falsy
        data = resp.json()            # or resp.text(), resp.content
```

Sessions must be closed explicitly — use `async with` or `await s.close()`. There is
no `__del__` auto-close (unsafe for async resources); leaking a session emits a
`ResourceWarning`.

## Responses

`request`/`request_with_retries` return a `Response` (success or non-retryable status)
or a falsy `FailureResponse` (all retries failed). Both expose the same surface as
**properties**, so callers branch uniformly:

- `status_code`, `headers`, `url`, `reason`, `cookies`, `history`, `redirect_chain`
- `is_success` (2xx), `is_redirect`
- `content` (bytes), `text(encoding=None)`, `json()` (None if not JSON)
- `raise_for_status()` raises `AiowebError` on non-2xx
- `bool(resp)` / `if resp:` is `is_success`

## Retries

`request_with_retries` retries on exceptions **and** retryable statuses (429, 500,
502, 503, 504 by default), with exponential backoff (`backoff_base ** attempt`).

```python
resp = await s.request_with_retries(
    "GET", url, attempts=5, backoff_base=2.0,
    retry_statuses={429, 503},        # override which statuses retry
)
```

Returns a `FailureResponse` (falsy) if every attempt fails.

`request()` (the non-retrying call) raises on failure: a total timeout raises
`aiohttp.ServerTimeoutError`, and any other network/protocol failure raises its real
`aiohttp.ClientError` subtype as-is (`ClientConnectorError`, `ClientProxyConnectionError`,
`ClientResponseError` with `.status`/`.headers`, `TooManyRedirects`, ...) — it is not
flattened into the base `ClientError`, so direct callers can branch by type or read
subtype attributes. `request_with_retries` catches the base `aiohttp.ClientError` (and
`asyncio.TimeoutError`) across all attempts and returns a falsy `FailureResponse` instead
of raising.

## Header overwrites & ephemeral headers

```python
s.overwrite_header("User-Agent", "custom")     # replace per request
s.overwrite_inject(True)                        # also add when absent
s.set_ephemeral("X-Time", lambda: str(time.time()))  # generated fresh each request
```

With `inject=False` (default) overwrites only replace headers already present in a
request; with `inject=True` they're added regardless.

## Domain rewriting

```python
s.overwrite_domain("internal.local", "127.0.0.1")   # host-substring rewrite
```

## Preview / debug

```python
print(s.preview("POST", url, json={"a": 1}).as_curl())   # equivalent cURL command
```

`preview()` is synchronous and never touches the backend session — it works before any
event loop is running (e.g. building a preview at import/setup time), not just inside
`asyncio.run()`. Its default cookie lookup only reads cookies from an already-built
session; if the session hasn't been built yet there are no cookies to read, so it
returns none by default (pass `cookies=` explicitly to preview cookies for a session
that hasn't sent a request yet). `as_curl()` renders `params` (merged into the url's
query string) and `timeout` (as `--max-time`) as well as headers/body/proxy, so the
emitted command is faithful to what `request()` actually sends. Pass `debug=True` to
`request_with_retries` to log the cURL preview and request flow.

## Custom backends

The `_raw_request(method, url, **kwargs) -> Response` method is the only place that
touches the HTTP client. To use a different backend (e.g. a TLS-fingerprinting client
like curl_cffi), subclass `ExtendedSession` and override just `_raw_request`, building
a `Response` from that backend's primitives:

```python
class MySession(ExtendedSession):
    async def _raw_request(self, method, url, **kwargs):
        r = await my_client.request(method, url, **kwargs)
        return Response(
            status_code=r.status, headers=r.headers, content=await r.read(),
            url=str(r.url), reason=r.reason,
        )
```

Everything else — header overwrites, ephemeral headers, domain rewriting, proxy
resolution, retries, previews — is inherited. `Response` is built from primitives
(status, headers, content, url, history) precisely so any backend can produce one.

## Migrating from the original

Back-compat shims are in place for the common path:

- `aiowebResponse` is aliased to `Response` (the class was renamed) — old imports work.
- `request_retries(session, ...)` and `test_proxies(session)` remain as module functions.
- `raise_for_status` now raises `AiowebError` (a subclass of `Exception`), so
  `except Exception` still catches it.

Two changes can't be shimmed without re-introducing the bugs they fix:

- **`is_success` is now a property on `FailureResponse`, not a method.** Code that
  *called* `failure.is_success()` must drop the parens to `failure.is_success`. (Code
  that wrote `if failure.is_success` was previously always-truthy — a bug — and now
  behaves correctly.)
- **No `__del__` auto-close.** Sessions must be closed via `async with` or
  `await s.close()`; a leaked session emits a `ResourceWarning`. The old finalizer-based
  auto-close was unsafe and was removed.

## Changelog

### v0.1.13

- **Session-default timeout no longer poisons pooled keep-alive connections.**
  The default `ClientTimeout` set `sock_read=timeout/2` alongside `total`. Under
  `aiohttp>=3.14`, that read timer re-arms on every request dispatched over a
  pooled protocol, including idle connections between requests; when it fires
  it permanently poisons the pooled connection (`SocketTimeoutError` on the next
  use, instantly, without contacting the server) — a real error from the server
  (e.g. a 503) could come back as a client-side `FailureResponse(status=0,
  reason='timeout')` instead. `sock_read` is now dropped from the session
  default; `total` (and `connect`/`sock_connect`) still bound every request, and
  the per-call `timeout=N` path (`ClientTimeout(total=N)`, no `sock_read`) was
  already unaffected.

### v0.1.12

- **Docstring-only.** Restored one-line docstrings on `FailureResponse`'s
  `redirect_chain`, `text()`, `json()`, and `raise_for_status()` — a prior
  de-bloat pass stripped them below the public tier while the `Response`
  twins kept theirs. No behavior change.

### v0.1.11

- **`preview()` no longer requires a running event loop.** Its default cookie
  lookup (`cookies=` not passed) previously called `_cookies_for_url()`, which
  reached the backend session and built it if absent — under aiohttp>=3.14 that
  needs a running loop, so `preview()` raised `RuntimeError('no running event
  loop')` when called before the loop starts, defeating its own pre-loop
  build/inspect use case (v0.1.7). Now the default cookie lookup is skipped
  entirely when the session hasn't been built yet (nothing could have been set
  on a session that doesn't exist); `cookies={}` and in-loop calls are unaffected.
- **`set_cookie(domain=None)` now honors `path`.** The `domain=None` branch
  returned right after `update_cookies()`, before the line that sets the
  morsel's `path` — so a shared cookie (`set_cookie(name, value, path="/api")`,
  no `domain=`) always stored the `SimpleCookie` default `path="/"` instead. The
  morsel's `path` is now set before the `domain=None` early return. The
  domain-bound branch was already correct and is unchanged.

### v0.1.10

- Docs-only pass: compressed module/method docstrings and comments that restated
  README/CLAUDE prose, replaced em-dashes with hyphens. No behavior change.

### v0.1.9

- **`_get_proxy()`/`request()` proxy resolution now checks `is None`, not
  truthiness.** A per-call `proxies={}` previously fell back to the session's
  configured proxies instead of disabling them for that call. `request()` also no
  longer clobbers a native `proxy=` kwarg with the resolved session/`proxies=`
  value (a real IP-unmasking leak) — passing both now raises `ValueError` instead
  of silently picking one.
- **`Response.text(encoding=...)` no longer returns a stale cached decode.** A
  second call with an explicit `encoding=` previously still returned the first
  (possibly differently-encoded) cached decode; an explicit encoding now bypasses
  the cache.
- **`preview()` now rewrites the url before resolving cookies**, matching what
  `request()` actually sends — previously cookies were resolved against the
  pre-rewrite host, which could miss or misattribute host-bound cookies.
- **`request_with_retries(attempts=0)` now floors to 1 attempt**, not silently
  `DEFAULT_ATTEMPTS` (3). `attempts` is checked with `is None`, not truthiness.

### v0.1.8

- **`request()` no longer flattens `aiohttp.ClientError` subtypes.** Every failure
  (connect errors, proxy errors, `raise_for_status()`-style response errors,
  redirect limits, ...) was re-raised as a bare `aiohttp.ClientError`, losing the
  real subtype and its attributes (`.os_error`, `.status`, `.headers`, ...) — a
  direct caller doing `except ClientProxyConnectionError:` or `if e.status == 401`
  would silently never match. Now the original exception is re-raised as-is (its
  subtype, attributes, and `__cause__` all preserved). `request_with_retries`
  still catches the base `aiohttp.ClientError` across attempts, so its behavior
  (and its `FailureResponse` return on exhaustion) is unchanged.
- **`as_curl()` now renders `params` and `timeout`.** Previously a preview built
  with `params=` silently omitted the query string (and a `timeout=` omitted
  `--max-time`), so a `debug=True` cURL replay of a params-driven request hit a
  different URL than the one actually sent. `params` are now merged into the url's
  query string (via `yarl`) and `timeout` is emitted as `--max-time`.

### v0.1.7

- **`get_cookies()` now returns real cookies.** Previously called `filter_cookies()`
  with no URL, which only ever returns domain-less shared cookies — every normal
  domain-bound cookie (including ones set by a real `Set-Cookie` response) was
  silently omitted. Now iterates the jar directly.
- **`set_cookie()` no longer leaks a shared cookie to every host.** A bare hostname
  (`domain="example.com"`) built a schemeless URL, which aiohttp's jar treats as a
  domain-less "shared" cookie sent with every request the session makes, including
  unrelated hosts. A scheme is now added when missing so the cookie is scoped to
  that host.
- **`set_cookie()` now honors `path`** (previously ignored — the cookie always
  landed at `path="/"`). `domain=None` is unchanged in meaning but now stores a
  truly shared cookie (sent to every host) instead of one silently bound to
  `localhost` only, which made `set_cookie(name, value)` (no domain) a silent
  no-op for any real request.
- **The backend session is built lazily**, not in `__init__`. Under aiohttp 3.14,
  constructing `aiohttp.ClientSession` requires a running event loop; eager
  construction crashed the common host pattern of attaching a session before the
  loop starts (e.g. `bot.http = ExtendedSession(...)` in `Bot.__init__`). The
  session (and any subclass's `_create_session` override) now builds on first
  access instead.

### v0.1.2

- Pinned `commons` to v0.2.1 (retry `attempts` floor fix).

### v0.1.1

- **JSON list bodies** now route to `json=` (were wrongly form-encoded via `data=` —
  only dicts went to `json=` before).
- **Exhausted retries return the real last response.** When every attempt hit a
  retryable status (429/5xx), the loop discarded it and returned a synthetic
  `FailureResponse` (status 0); now the real last 4xx/5xx `Response` is returned (only a
  pure-exception failure yields `FailureResponse`).
- Retry/backoff moved onto `commons.aretry` (shared engine); backoff schedule unchanged.
  Adds a `commons` dependency.

## Versioning

Releases are tagged `vX.Y.Z`. The install line above pins a release; drop the `@vX.Y.Z` suffix to install the latest unpinned. Pin deliberately for reproducible installs.