aioweb_tls/README.md

# aioweb_tls

TLS-fingerprinting backends for [aioweb](https://git.rethinkstudios.io/rethink-public/aioweb).
One session class — `TLSSession` — takes an **injected backend** that swaps the HTTP
client (and thus the TLS/HTTP fingerprint), while inheriting every aioweb feature —
header overwrites, domain rewriting, ephemeral headers, proxies, retries, previews —
unchanged. The backend swaps the wire, not the session.

```python
from aioweb_tls import TLSSession, CurlCffi, Noble   # the two bundled backends

TLSSession(backend=CurlCffi(impersonate="chrome"))
TLSSession(backend=Noble(client="chrome_133"))
TLSSession(backend=MyBackend(...))   # or one you write — see "Writing your own backend"
```

## Install

Depends on `aioweb`. The TLS clients are **optional extras** — install the backend(s)
you want; importing the package never fails because an extra is missing.

`requirements.txt` (pick the extra you need):

```
aioweb_tls[curl]  @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.8
aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.8
aioweb_tls[all]   @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.8
```

Direct:

```bash
pip install "aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.8"
pip install "aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.8"
pip install "aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.8"
```

- `[curl]` → curl_cffi backend · `[noble]` → noble_tls backend · `[all]` → both.
- **No extra** (`pip install aioweb_tls`) is also valid: you get `TLSSession` and the
  `TLSBackend` protocol — the framework and the bring-your-own-backend path — but no
  bundled client. `CurlCffi()` / `Noble()` then raise a clear `RuntimeError` naming the
  extra to install. Use a bare install when you only need a custom backend of your own.

Constructing a backend whose client isn't installed raises that `RuntimeError` at
construction, never at import.

Drop the `@v0.1.8` suffix from the line above to install the latest unpinned.

## curl_cffi backend

```python
from aioweb_tls import TLSSession, CurlCffi

async with TLSSession(backend=CurlCffi(impersonate="chrome"), proxies={"https": "http://..."}) as s:
    resp = await s.request_with_retries("GET", "https://tls.peet.ws/api/all")
    if resp:                          # FailureResponse is falsy
        print(resp.json()["tls"]["ja3"])
```

- `CurlCffi(impersonate="chrome")` sets the forged profile; override it per call by
  passing `impersonate=` to the low-level `request()` (which forwards `**kwargs` to the
  backend). `request_with_retries` has a fixed signature and does **not** accept extra
  backend kwargs — passing `impersonate=` there raises `TypeError`; set the profile on
  the `CurlCffi` instance for the retrying path.
- curl_cffi forges JA3/JA4 + HTTP/2 fingerprints via the bundled curl-impersonate binary.
- A wire-level timeout raises `aiohttp.ServerTimeoutError` (matching aioweb's own
  contract), not a generic `aiohttp.ClientError` — both backends detect their
  native timeout (curl_cffi's `Timeout` type, noble_tls's Go-side timeout text) and
  re-wrap it before the fallback client-error path (v0.1.5).

## noble backend

```python
from aioweb_tls import TLSSession, Noble

async with TLSSession(backend=Noble(client="chrome_133")) as s:
    await s.setup()                   # fetch noble's Go shared lib once (network)
    resp = await s.request_with_retries("GET", "https://tls.peet.ws/api/all")
    if resp:
        print(resp.json()["tls"]["ja3"])
```

- `Noble(client="chrome_133")` — accepts a `noble_tls.Client` enum or a string name;
  an unknown string raises `ValueError` listing the valid profile names (v0.1.5).
- noble_tls downloads a Go shared library on first use. `await s.setup()` fetches it
  once at startup; if you skip it, the first request fetches it lazily. The fetch is
  guarded by a lock, so even concurrent first requests download it exactly once.
- Binary bodies (images, zips, PDFs, protobuf) round-trip as true bytes: noble_tls
  returns response bodies as a UTF-8 JSON string by default, which mangles non-UTF-8
  bytes (`U+FFFD` replacement, wrong length) even on a 200 response — the Noble
  backend always requests `is_byte_response=True` and decodes the resulting
  base64 data-URI back into raw bytes, so `resp.content` is never lossy.
- Multi-valued response headers (e.g. two `Set-Cookie` lines) arrive from noble_tls's
  Go side as a Python list — the Noble backend flattens them to a single
  comma-joined string per RFC 7230, so `resp.headers[...]` is always a plain string
  (v0.1.5).

## Writing your own backend (the `TLSBackend` protocol)

**aioweb_tls ships exactly two backends — `CurlCffi` and `Noble` — plus the
`TLSBackend` protocol for writing your own.** There is no Go, remote, or other
bundled backend; anything beyond curl/noble is something you implement against the
protocol. `TLSSession` itself is backend-agnostic, so a backend you write injects
exactly like the built-ins (`TLSSession(backend=YourBackend(...))`) and inherits
every aioweb feature for free.

To add a backend, write a small object satisfying `TLSBackend` (see `protocol.py`
for the authoritative contract):

| member | required? | signature | purpose |
|---|---|---|---|
| `create_session` | **required** | `(headers, timeout, **kwargs) -> session` | build the client object `TLSSession` stores as `self.session` |
| `raw_request` | **required** | `async (session, method, url, **kwargs) -> aioweb.Response` | send one request; adapt the client's response into an `aioweb.Response` |
| `is_closed` | **required** | `(session) -> bool` | whether the session is closed |
| `cookies_for_url` | optional | `(session, url) -> dict` | cookies for `preview()`; defaults to `{}` |
| `set_cookie` | optional | `(session, name, value, domain=None, path="/") -> None` | backs `TLSSession.set_cookie()`; raises `NotImplementedError` if absent |
| `get_cookies` | optional | `(session) -> dict` | backs `TLSSession.get_cookies()`; raises `NotImplementedError` if absent |
| `clear_cookies` | optional | `(session) -> None` | backs `TLSSession.clear_cookies()`; raises `NotImplementedError` if absent |
| `setup` | optional | `async () -> None` | one-time prep (e.g. fetch a native lib); idempotent. `TLSSession` never calls this automatically - a backend needing lazy setup must self-invoke it from its own `raw_request`, as `Noble` does, or the caller must run `await session.setup()` explicitly |
| `close` | optional | `async (session) -> None` | close the session; defaults to `await session.close()` |

`raw_request` receives aioweb-shaped kwargs: the proxy is already resolved into
`kwargs["proxy"]`, headers are merged into `kwargs["headers"]`, and a numeric timeout
is wrapped in an `aiohttp.ClientTimeout` (unwrap `.total`).

### Example — a backend you would write (not included)

The class below is **illustrative template code you implement yourself**, shown to
make the protocol concrete. It is **not** part of aioweb_tls — there is no
`from aioweb_tls import GoTLSBackend`. It sketches one case (talking to a local Go
TLS service over HTTP) so the shape of a conformant backend is clear:

```python
# example: a backend YOU write to drive a local Go TLS service.
# NOT shipped by aioweb_tls — this is a TLSBackend-conformance template.
from aioweb import Response
from aioweb_tls import TLSSession

class GoTLSBackend:
    """example user-written backend: drives a local Go tls-server over HTTP"""

    def __init__(self, endpoint: str):
        self.endpoint = endpoint

    def create_session(self, headers, timeout, **kwargs):
        # build/return whatever client object raw_request will use
        import aiohttp
        return aiohttp.ClientSession(headers=headers)

    async def raw_request(self, session, method, url, **kwargs) -> Response:
        # kwargs arrive aioweb-shaped: proxy in kwargs["proxy"], headers merged,
        # numeric timeout wrapped in an aiohttp.ClientTimeout (unwrap .total).
        payload = {"method": method, "url": url, "headers": kwargs.get("headers", {})}
        async with session.post(self.endpoint, json=payload) as r:
            body = await r.json()
            return Response(
                status_code=body["status"], headers=body["headers"],
                content=body["body"].encode(), url=url, reason=body.get("reason"),
            )

    def is_closed(self, session) -> bool:
        return session.closed

    async def close(self, session) -> None:
        await session.close()

# inject your backend — same shape as CurlCffi / Noble, all aioweb features inherited
async with TLSSession(backend=GoTLSBackend("http://localhost:8080")) as s:
    s.overwrite_domain("internal.local", "127.0.0.1")
    resp = await s.request_with_retries("GET", "https://internal.local/x")
```

## Inherited features work unchanged

aioweb's overwrite/domain/ephemeral/proxy/retry/preview logic operates on plain dicts
and never touches the HTTP backend — only the seams do. Header overwrites, domain
rewriting, ephemeral headers, proxies, retries, and previews behave identically on
any backend:

```python
async with TLSSession(backend=CurlCffi(impersonate="chrome")) as s:
    s.overwrite_domain("internal.local", "127.0.0.1")
    s.set_ephemeral("X-Time", lambda: str(time.time()))
    s.overwrite_inject(True)
    print(s.preview("GET", "https://internal.local/x").as_curl())   # reflects all of the above
```

Session-default headers are never baked into the underlying client (neither
`CurlCffi` nor `Noble` passes `headers=` to their client's constructor) — they flow
through aioweb's own per-request `_default_headers` merge instead. That keeps
`update_headers()` / `clear_headers()` accurate for both backends: what
`get_headers()` and `preview()` report is what actually goes out on the wire, with
no stale, already-baked value resurfacing after a clear.

The mutable cookie API — `set_cookie()` / `get_cookies()` / `clear_cookies()` — is
also backend-aware: `CurlCffi` and `Noble` each route it through their own client's
cookie store (both expose a `requests`-style `session.cookies` with `set()` /
`items()` / `clear()`), so these calls work the same way they do on the base
`aioweb.ExtendedSession`, not just `_cookies_for_url()` (used by `preview()`).

`resp.history` / `resp.redirect_chain` (`[(status, url), ...]`) and `resp.is_redirect`
are also threaded through on both backends: `CurlCffi.raw_request` maps curl_cffi's
`Response.history` (`list[dict]`) and `Noble.raw_request` maps noble_tls's
`list[Response]` into aioweb's `(status, url)` tuple shape, so `debug=True`'s "redirect
chain:" log line and any caller reading `resp.history` after a redirecting request see
the real hops on either backend, matching the base `aioweb.ExtendedSession` behavior.

## Honesty note

TLS fingerprinting changes one layer — the TLS/HTTP fingerprint. It does **not** by
itself defeat modern bot protection: behavioral analysis, captchas, and JS challenges
are separate signals. Use this as one component, not a complete anti-bot solution.

## Changelog

### v0.1.8

- Compressed 4 residual internal-helper docstrings (`_is_timeout_error`,
  `_noble_timeout_seconds`, `_noble_content`, `_jar_to_dict`) to one-liners.
  Cosmetic, zero behavior change.

### v0.1.7

- **Both backends now thread redirect history.** `CurlCffi.raw_request` and
  `Noble.raw_request` built their `Response` without `history=`, so
  `resp.history`/`resp.redirect_chain`/`resp.is_redirect`-after-follow were
  always empty and aioweb's own `debug=True` "redirect chain:" log line was
  permanently dead on both TLS backends, despite the "inherits every aioweb
  feature unchanged" claim. Each backend's native history (curl_cffi
  `list[dict]`, noble_tls `list[Response]`) is now mapped into aioweb's
  `(status, url)` tuple shape and passed through.

## Versioning

Releases are tagged `vX.Y.Z`. The install line above pins a release; drop the `@vX.Y.Z` suffix to install the latest unpinned. Pin deliberately for reproducible installs.