# aioweb Async HTTP session wrapper over `aiohttp`. Adds session-level proxies, header overwrites, ephemeral (per-request generated) headers, domain rewriting, request previews / cURL export, and retry-with-backoff. The byte-sending is isolated behind one overridable method (`_raw_request`), so a TLS-fingerprinting backend can subclass and swap the HTTP client while inheriting everything else. ## Install `requirements.txt`: ``` aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.5 ``` Direct: ```bash pip install "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.5" ``` Requires `aiohttp` and `yarl` (pulled transitively). Drop the `@v0.1.5` suffix from the line above to install the latest unpinned. ## Usage ```python from aioweb import ExtendedSession async with ExtendedSession(proxies={"https": "http://user:pass@host:port"}, timeout=15) as s: resp = await s.request_with_retries("GET", "https://example.com") if resp: # FailureResponse is falsy data = resp.json() # or resp.text(), resp.content ``` Sessions must be closed explicitly — use `async with` or `await s.close()`. There is no `__del__` auto-close (unsafe for async resources); leaking a session emits a `ResourceWarning`. ## Responses `request`/`request_with_retries` return a `Response` (success or non-retryable status) or a falsy `FailureResponse` (all retries failed). Both expose the same surface as **properties**, so callers branch uniformly: - `status_code`, `headers`, `url`, `reason`, `cookies`, `history`, `redirect_chain` - `is_success` (2xx), `is_redirect` - `content` (bytes), `text(encoding=None)`, `json()` (None if not JSON) - `raise_for_status()` raises `AiowebError` on non-2xx - `bool(resp)` / `if resp:` is `is_success` ## Retries `request_with_retries` retries on exceptions **and** retryable statuses (429, 500, 502, 503, 504 by default), with exponential backoff (`backoff_base ** attempt`). ```python resp = await s.request_with_retries( "GET", url, attempts=5, backoff_base=2.0, retry_statuses={429, 503}, # override which statuses retry ) ``` Returns a `FailureResponse` (falsy) if every attempt fails. ## Header overwrites & ephemeral headers ```python s.overwrite_header("User-Agent", "custom") # replace per request s.overwrite_inject(True) # also add when absent s.set_ephemeral("X-Time", lambda: str(time.time())) # generated fresh each request ``` With `inject=False` (default) overwrites only replace headers already present in a request; with `inject=True` they're added regardless. ## Domain rewriting ```python s.overwrite_domain("internal.local", "127.0.0.1") # host-substring rewrite ``` ## Preview / debug ```python print(s.preview("POST", url, json={"a": 1}).as_curl()) # equivalent cURL command ``` Pass `debug=True` to `request_with_retries` to log the cURL preview and request flow. ## Custom backends The `_raw_request(method, url, **kwargs) -> Response` method is the only place that touches the HTTP client. To use a different backend (e.g. a TLS-fingerprinting client like curl_cffi), subclass `ExtendedSession` and override just `_raw_request`, building a `Response` from that backend's primitives: ```python class MySession(ExtendedSession): async def _raw_request(self, method, url, **kwargs): r = await my_client.request(method, url, **kwargs) return Response( status_code=r.status, headers=r.headers, content=await r.read(), url=str(r.url), reason=r.reason, ) ``` Everything else — header overwrites, ephemeral headers, domain rewriting, proxy resolution, retries, previews — is inherited. `Response` is built from primitives (status, headers, content, url, history) precisely so any backend can produce one. ## Migrating from the original Back-compat shims are in place for the common path: - `aiowebResponse` is aliased to `Response` (the class was renamed) — old imports work. - `request_retries(session, ...)` and `test_proxies(session)` remain as module functions. - `raise_for_status` now raises `AiowebError` (a subclass of `Exception`), so `except Exception` still catches it. Two changes can't be shimmed without re-introducing the bugs they fix: - **`is_success` is now a property on `FailureResponse`, not a method.** Code that *called* `failure.is_success()` must drop the parens to `failure.is_success`. (Code that wrote `if failure.is_success` was previously always-truthy — a bug — and now behaves correctly.) - **No `__del__` auto-close.** Sessions must be closed via `async with` or `await s.close()`; a leaked session emits a `ResourceWarning`. The old finalizer-based auto-close was unsafe and was removed. ## Changelog ### v0.1.2 - Pinned `commons` to v0.2.1 (retry `attempts` floor fix). ### v0.1.1 - **JSON list bodies** now route to `json=` (were wrongly form-encoded via `data=` — only dicts went to `json=` before). - **Exhausted retries return the real last response.** When every attempt hit a retryable status (429/5xx), the loop discarded it and returned a synthetic `FailureResponse` (status 0); now the real last 4xx/5xx `Response` is returned (only a pure-exception failure yields `FailureResponse`). - Retry/backoff moved onto `commons.aretry` (shared engine); backoff schedule unchanged. Adds a `commons` dependency. ## Versioning Releases are tagged `vX.Y.Z`. The install line above pins a release; drop the `@vX.Y.Z` suffix to install the latest unpinned. Pin deliberately for reproducible installs.