fix: seam mutable cookie api, byte-safe noble bodies, drop header baking, coerce noble session timeout (v0.1.4)

TLSSession.set_cookie/get_cookies/clear_cookies crashed with AttributeError on
both backends (they reached into self.session.cookie_jar, which curl_cffi and
noble_tls sessions don't have); both backends now route the mutable cookie api
through their own requests-style session.cookies store.

Noble.raw_request now requests is_byte_response=True and decodes the resulting
base64 data-URI body, since noble_tls's default text response silently corrupts
any binary payload (image/zip/pdf) via lossy UTF-8 decoding on the Go side.

CurlCffi/Noble.create_session no longer bake session-default headers into the
underlying client; baking caused clear_headers()/get_headers() to lie about
what's actually still on the wire (a credential-leak divergence from the
aiohttp base, which never bakes). Headers flow through aioweb's per-request
merge only, matching the base's documented contract.

Noble.create_session now applies the same max(1, ceil()) timeout coercion
raw_request already had (extracted into a shared _noble_timeout_seconds
helper) — without it, a sub-second/float session-default timeout made every
request fail Go-side JSON unmarshal.

README corrected: dropped the 'every aioweb feature behaves identically'
overclaim re: cookies, documented the binary-body handling and the
no-header-baking rationale, bumped install pins to v0.1.4.

Signed-off-by: disqualifier <dev@disqualifier.me>

.gitignore

@@ -1,5 +1,5 @@
 # claude
-CLAUDE.md
+.claude/
 
 # python
 __pycache__/

README.md

@@ -22,17 +22,17 @@ you want; importing the package never fails because an extra is missing.
 `requirements.txt` (pick the extra you need):
 
 ```
-aioweb_tls[curl]  @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.1
-aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.1
-aioweb_tls[all]   @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.1
+aioweb_tls[curl]  @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.4
+aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.4
+aioweb_tls[all]   @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.4
 ```
 
 Direct:
 
 ```bash
-pip install "aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.1"
-pip install "aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.1"
-pip install "aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.1"
+pip install "aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.4"
+pip install "aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.4"
+pip install "aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.4"
 ```
 
 - `[curl]` → curl_cffi backend · `[noble]` → noble_tls backend · `[all]` → both.
@@ -44,6 +44,8 @@ pip install "aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-publi
 Constructing a backend whose client isn't installed raises that `RuntimeError` at
 construction, never at import.
 
+Drop the `@v0.1.4` suffix from the line above to install the latest unpinned.
+
 ## curl_cffi backend
 
 ```python
@@ -55,8 +57,11 @@ async with TLSSession(backend=CurlCffi(impersonate="chrome"), proxies={"https":
         print(resp.json()["tls"]["ja3"])
 ```
 
-- `CurlCffi(impersonate="chrome")` sets the forged profile; override per call by
-  passing `impersonate=` to any request method.
+- `CurlCffi(impersonate="chrome")` sets the forged profile; override it per call by
+  passing `impersonate=` to the low-level `request()` (which forwards `**kwargs` to the
+  backend). `request_with_retries` has a fixed signature and does **not** accept extra
+  backend kwargs — passing `impersonate=` there raises `TypeError`; set the profile on
+  the `CurlCffi` instance for the retrying path.
 - curl_cffi forges JA3/JA4 + HTTP/2 fingerprints via the bundled curl-impersonate binary.
 
 ## noble backend
@@ -75,6 +80,11 @@ async with TLSSession(backend=Noble(client="chrome_133")) as s:
 - noble_tls downloads a Go shared library on first use. `await s.setup()` fetches it
   once at startup; if you skip it, the first request fetches it lazily. The fetch is
   guarded by a lock, so even concurrent first requests download it exactly once.
+- Binary bodies (images, zips, PDFs, protobuf) round-trip as true bytes: noble_tls
+  returns response bodies as a UTF-8 JSON string by default, which mangles non-UTF-8
+  bytes (`U+FFFD` replacement, wrong length) even on a 200 response — the Noble
+  backend always requests `is_byte_response=True` and decodes the resulting
+  base64 data-URI back into raw bytes, so `resp.content` is never lossy.
 
 ## Writing your own backend (the `TLSBackend` protocol)
 
@@ -94,6 +104,9 @@ for the authoritative contract):
 | `raw_request` | **required** | `async (session, method, url, **kwargs) -> aioweb.Response` | send one request; adapt the client's response into an `aioweb.Response` |
 | `is_closed` | **required** | `(session) -> bool` | whether the session is closed |
 | `cookies_for_url` | optional | `(session, url) -> dict` | cookies for `preview()`; defaults to `{}` |
+| `set_cookie` | optional | `(session, name, value, domain=None, path="/") -> None` | backs `TLSSession.set_cookie()`; raises `NotImplementedError` if absent |
+| `get_cookies` | optional | `(session) -> dict` | backs `TLSSession.get_cookies()`; raises `NotImplementedError` if absent |
+| `clear_cookies` | optional | `(session) -> None` | backs `TLSSession.clear_cookies()`; raises `NotImplementedError` if absent |
 | `setup` | optional | `async () -> None` | one-time prep (e.g. fetch a native lib); idempotent |
 | `close` | optional | `async (session) -> None` | close the session; defaults to `await session.close()` |
 
@@ -151,8 +164,9 @@ async with TLSSession(backend=GoTLSBackend("http://localhost:8080")) as s:
 ## Inherited features work unchanged
 
 aioweb's overwrite/domain/ephemeral/proxy/retry/preview logic operates on plain dicts
-and never touches the HTTP backend — only the seams do. Every aioweb feature behaves
-identically on any backend:
+and never touches the HTTP backend — only the seams do. Header overwrites, domain
+rewriting, ephemeral headers, proxies, retries, and previews behave identically on
+any backend:
 
 ```python
 async with TLSSession(backend=CurlCffi(impersonate="chrome")) as s:
@@ -162,6 +176,19 @@ async with TLSSession(backend=CurlCffi(impersonate="chrome")) as s:
     print(s.preview("GET", "https://internal.local/x").as_curl())   # reflects all of the above
 ```
 
+Session-default headers are never baked into the underlying client (neither
+`CurlCffi` nor `Noble` passes `headers=` to their client's constructor) — they flow
+through aioweb's own per-request `_default_headers` merge instead. That keeps
+`update_headers()` / `clear_headers()` accurate for both backends: what
+`get_headers()` and `preview()` report is what actually goes out on the wire, with
+no stale, already-baked value resurfacing after a clear.
+
+The mutable cookie API — `set_cookie()` / `get_cookies()` / `clear_cookies()` — is
+also backend-aware: `CurlCffi` and `Noble` each route it through their own client's
+cookie store (both expose a `requests`-style `session.cookies` with `set()` /
+`items()` / `clear()`), so these calls work the same way they do on the base
+`aioweb.ExtendedSession`, not just `_cookies_for_url()` (used by `preview()`).
+
 ## Honesty note
 
 TLS fingerprinting changes one layer — the TLS/HTTP fingerprint. It does **not** by
@@ -170,4 +197,4 @@ are separate signals. Use this as one component, not a complete anti-bot solutio
 
 ## Versioning
 
-Tagged `vX.Y.Z`. Pin the tag in `requirements.txt`.
+Releases are tagged `vX.Y.Z`. The install line above pins a release; drop the `@vX.Y.Z` suffix to install the latest unpinned. Pin deliberately for reproducible installs.

pyproject.toml

@@ -4,11 +4,11 @@ build-backend = "hatchling.build"
 
 [project]
 name = "aioweb_tls"
-version = "0.1.1"
-description = "TLS-fingerprinting backends for aioweb — curl_cffi / noble_tls ExtendedSession subclasses, config-free, installable."
+version = "0.1.4"
+description = "TLS-fingerprinting backends (curl_cffi / noble_tls) for aioweb via one injectable TLSSession, config-free, installable."
 requires-python = ">=3.10"
 dependencies = [
-    "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.0",
+    "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.5",
 ]
 
 [project.optional-dependencies]

src/aioweb_tls/backends.py

@@ -12,13 +12,27 @@ this module never fails because an extra is missing.
 """
 
 import asyncio
+import base64
 import logging
 import math
 
+import aiohttp
 from aioweb import Response
 
 log = logging.getLogger(__name__)
 
+
+def _as_client_error(error: Exception, backend: str) -> aiohttp.ClientError:
+    """wrap a backend-native network exception as an aiohttp.ClientError
+
+    aioweb's request() only re-wraps aiohttp.ClientError; curl_cffi raises
+    RequestException(OSError) and noble_tls raises TLSClientException(IOError), neither
+    of which is an aiohttp.ClientError. translating here gives TLS backends the same
+    typed failure contract as the aiohttp path on the bare request() route.
+    """
+    return aiohttp.ClientError(f"{backend} request failed: {error}")
+
+
 try:
     from curl_cffi import AsyncSession as _CurlAsyncSession
     _CURL_ERROR = None
@@ -46,8 +60,46 @@ def _coerce_timeout(value):
     return total if isinstance(total, (int, float)) else None
 
 
+def _noble_timeout_seconds(value):
+    """coerce a raw or wrapped timeout into whole seconds for noble's Go int field
+
+    single source of truth for both of Noble's timeout paths (create_session's
+    session-default and raw_request's per-call override): noble's Go field
+    timeoutSeconds is an int, so a sub-second float (e.g. 0.5) truncates to 0 — which
+    Go reads as no/instant timeout — and any non-integer value fails Go-side JSON
+    unmarshal outright. rounds UP so a sub-second timeout still waits at least 1s
+    instead of truncating away. returns None when there is no timeout to coerce.
+    """
+    timeout = _coerce_timeout(value)
+    if timeout is None:
+        return None
+    return max(1, math.ceil(timeout))
+
+
+def _noble_content(response) -> bytes:
+    """extract true response bytes from a noble_tls response fetched with is_byte_response
+
+    with is_byte_response=True, noble_tls's Go side returns the body as a
+    `data:<mime>;base64,<payload>` URI string in response.text (the whole Go response is
+    UTF-8-decoded before JSON parsing, so raw bytes have to travel as base64 to survive
+    that round trip) — response.content is useless here since it is just
+    `self.text.encode()`, re-encoding the data-URI string itself rather than decoding it.
+    falls back to a plain utf-8 encode if the body isn't a data-URI (e.g. an error body).
+    """
+    text = getattr(response, "text", "") or ""
+    if text.startswith("data:") and ";base64," in text:
+        _, _, payload = text.partition(";base64,")
+        return base64.b64decode(payload)
+    return text.encode()
+
+
 def _jar_to_dict(session):
-    """best-effort map of a requests-style cookie jar on session to a plain dict"""
+    """best-effort map of a requests-style cookie jar on session to a plain dict
+
+    intentionally broad: this feeds preview() only, the two backends expose differently-
+    shaped jars, and a cookie read must never crash a request — so any jar that doesn't
+    iterate cleanly degrades to {} rather than raising.
+    """
     jar = getattr(session, "cookies", None)
     if not jar:
         return {}
@@ -62,7 +114,10 @@ class CurlCffi:
 
     config:
         impersonate: browser profile to forge (default "chrome"); override per call
-            by passing impersonate= to any request method.
+            by passing impersonate= to the low-level request()/_raw_request path,
+            which forwards **kwargs to the backend. NOT request_with_retries — its
+            signature is fixed (no **kwargs) and would raise TypeError. for a
+            per-call profile under retries, set it on the CurlCffi instance instead.
 
     requires the [curl] extra (pip install "aioweb_tls[curl]").
     """
@@ -76,8 +131,17 @@ class CurlCffi:
         self.impersonate = impersonate
 
     def create_session(self, headers, timeout, **kwargs):
-        """build the curl_cffi AsyncSession"""
-        return _CurlAsyncSession(headers=headers, timeout=timeout, **kwargs)
+        """build the curl_cffi AsyncSession
+
+        deliberately does NOT pass `headers` to AsyncSession: curl_cffi bakes a
+        constructor `headers=` into the client and re-merges it under whatever
+        per-request headers() sends, so update_headers()/clear_headers() would stop
+        matching what's actually on the wire (aioweb's session-default headers are
+        already applied per request by the base's _default_headers merge — see
+        aioweb.ExtendedSession._create_session's docstring for why baking breaks the
+        mutable header api).
+        """
+        return _CurlAsyncSession(timeout=timeout, **kwargs)
 
     async def raw_request(self, session, method, url, **kwargs) -> Response:
         """send via curl_cffi and adapt the result into an aioweb.Response"""
@@ -91,10 +155,18 @@ class CurlCffi:
         if proxy:
             kwargs["proxy"] = proxy
 
-        response = await session.request(method, url, impersonate=impersonate, **kwargs)
-        content = response.content
-        if content is None:
-            content = response.text.encode() if response.text else b""
+        try:
+            response = await session.request(method, url, impersonate=impersonate, **kwargs)
+        except aiohttp.ClientError:
+            raise
+        except asyncio.TimeoutError:
+            raise
+        except OSError as error:
+            # curl_cffi's RequestException subclasses OSError; translate the native
+            # network error into aiohttp.ClientError. narrowed from a bare Exception so a
+            # real bug (AttributeError/TypeError) isn't laundered into 'client error'
+            raise _as_client_error(error, "curl_cffi") from error
+        content = response.content if response.content is not None else b""
         return Response(
             status_code=response.status_code,
             headers=dict(response.headers),
@@ -121,6 +193,18 @@ class CurlCffi:
         """cookies curl_cffi would send for url (best-effort)"""
         return _jar_to_dict(session)
 
+    def set_cookie(self, session, name, value, domain=None, path="/") -> None:
+        """set a cookie in curl_cffi's own cookie store"""
+        session.cookies.set(name, value, domain=domain or "", path=path)
+
+    def get_cookies(self, session) -> dict:
+        """all cookies stored in curl_cffi's cookie store"""
+        return dict(session.cookies.items())
+
+    def clear_cookies(self, session) -> None:
+        """clear curl_cffi's cookie store"""
+        session.cookies.clear()
+
     async def close(self, session) -> None:
         """close the curl_cffi session"""
         await session.close()
@@ -158,9 +242,9 @@ class Noble:
     async def setup(self) -> None:
         """fetch the noble_tls Go shared library once; idempotent and concurrency-safe
 
-        download_if_necessary handles the first-time fetch (no lib present);
-        update_if_necessary refreshes an existing one. try download first so a
-        clean environment works, falling back to update.
+        uses noble_tls.download_if_necessary (the current API: it fetches the asset on
+        first use and no-ops when it already exists). older noble_tls without that name
+        is handled via update_if_necessary as a fallback.
 
         guarded by an asyncio.Lock with a check-lock-recheck so concurrent first
         requests don't both run the fetch: the fast path returns once _updated is
@@ -174,44 +258,62 @@ class Noble:
             download = getattr(noble_tls, "download_if_necessary", None)
             if download is not None:
                 await download()
-            else:
+            elif hasattr(noble_tls, "update_if_necessary"):
                 await noble_tls.update_if_necessary()
             self._updated = True
 
     def create_session(self, headers, timeout, **kwargs):
-        """build the noble_tls Session, honoring session-default headers + timeout
+        """build the noble_tls Session, honoring the session-default timeout
 
-        noble_tls.Session takes neither headers nor timeout in its constructor, so
-        aioweb's session-default headers (and the coerced timeout) are applied after
-        construction — matching CurlCffi, which passes both through. without this a
-        TLSSession(backend=Noble(...), headers=...) would silently drop the headers.
+        noble_tls.Session takes neither headers nor timeout in its constructor.
+        deliberately does NOT bake `headers` into session.headers: aioweb's
+        session-default headers are already applied per request by the base's
+        _default_headers merge, and baking them here would make
+        update_headers()/clear_headers() stop matching what's actually on the wire
+        (see aioweb.ExtendedSession._create_session's docstring, and CurlCffi.create_session
+        above for the same rationale) — the per-request merge means headers are never
+        silently dropped, contrary to what this docstring used to claim.
+
+        the coerced timeout IS applied here (matching raw_request's per-call path): noble's
+        Go field timeoutSeconds is an int, so a raw sub-second/float session-default (e.g.
+        timeout=7.5) would fail Go-side JSON unmarshal on every request that doesn't
+        override it per call. max(1, ceil()) mirrors the guard raw_request already has.
         """
         session = noble_tls.Session(client=self.client, **kwargs)
-        if headers:
-            session.headers.update(headers)
-        if timeout is not None:
-            session.timeout_seconds = timeout
+        timeout_seconds = _noble_timeout_seconds(timeout)
+        if timeout_seconds is not None:
+            session.timeout_seconds = timeout_seconds
         return session
 
     async def raw_request(self, session, method, url, **kwargs) -> Response:
         """send via noble_tls and adapt the result into an aioweb.Response"""
         await self.setup()
 
-        timeout = _coerce_timeout(kwargs.pop("timeout", None))
-        if timeout is not None:
-            # noble takes whole seconds; round UP so a sub-second timeout (e.g. 0.5)
-            # doesn't truncate to 0 (which would mean no/instant timeout)
-            kwargs["timeout_seconds"] = max(1, math.ceil(timeout))
+        timeout_seconds = _noble_timeout_seconds(kwargs.pop("timeout", None))
+        if timeout_seconds is not None:
+            kwargs["timeout_seconds"] = timeout_seconds
 
         proxy = kwargs.pop("proxy", None)
         if proxy:
             kwargs["proxy"] = proxy
 
-        response = await session.execute_request(method=method.upper(), url=url, **kwargs)
-        content = getattr(response, "content", None)
-        if content is None:
-            text = getattr(response, "text", "") or ""
-            content = text.encode()
+        # force byte-safe transport: without this, noble_tls's Go side returns the body
+        # as a plain UTF-8 JSON string, replacing invalid bytes with U+FFFD — silently
+        # corrupting any binary payload (image/zip/pdf) even though the request succeeds
+        # with status 200. see _noble_content() for how the byte-safe body is decoded back.
+        kwargs.setdefault("is_byte_response", True)
+
+        try:
+            response = await session.execute_request(method=method.upper(), url=url, **kwargs)
+        except aiohttp.ClientError:
+            raise
+        except asyncio.TimeoutError:
+            raise
+        except OSError as error:
+            # noble_tls's TLSClientException subclasses IOError (== OSError); translate
+            # the native network error, narrowed from bare Exception so a real bug surfaces
+            raise _as_client_error(error, "noble_tls") from error
+        content = _noble_content(response)
         return Response(
             status_code=response.status_code,
             headers=dict(getattr(response, "headers", {}) or {}),
@@ -229,6 +331,18 @@ class Noble:
         """cookies noble_tls would send for url (best-effort)"""
         return _jar_to_dict(session)
 
+    def set_cookie(self, session, name, value, domain=None, path="/") -> None:
+        """set a cookie in noble_tls's own cookie jar"""
+        session.cookies.set(name, value, domain=domain or "", path=path)
+
+    def get_cookies(self, session) -> dict:
+        """all cookies stored in noble_tls's cookie jar"""
+        return dict(session.cookies.items())
+
+    def clear_cookies(self, session) -> None:
+        """clear noble_tls's cookie jar"""
+        session.cookies.clear()
+
     async def close(self, session) -> None:
         """close the noble_tls session if it exposes a close"""
         close = getattr(session, "close", None)

src/aioweb_tls/protocol.py

@@ -30,6 +30,15 @@ optional:
         cookies the client would send for url, for preview(). default {} (used when
         the backend has no introspectable jar).
 
+    set_cookie(session, name, value, domain=None, path="/") -> None
+    get_cookies(session) -> dict
+    clear_cookies(session) -> None
+        the mutable cookie api TLSSession.set_cookie/get_cookies/clear_cookies
+        delegate to. the aiohttp-only base implementations reach into
+        session.cookie_jar, which TLS backends don't have, so a backend without
+        these raises NotImplementedError from TLSSession rather than crashing with
+        an AttributeError on a private aiohttp attribute.
+
     async setup() -> None
         one-time async preparation (e.g. fetch a native lib). called once via
         TLSSession.setup() and lazily before the first request; make it idempotent.

src/aioweb_tls/session.py

@@ -81,6 +81,38 @@ class TLSSession(ExtendedSession):
             return True
         return self.backend.is_closed(self.session)
 
+    # -------------------------------------------------------------------------
+    #   mutable cookie api — the base's set_cookie/get_cookies/clear_cookies reach
+    #   into self.session.cookie_jar (aiohttp-only), so TLS backends override them
+    #   to route through the backend instead of crashing with AttributeError
+
+    def set_cookie(self, name, value, domain=None, path="/"):
+        """set a cookie via the backend's own cookie store"""
+        set_cookie = getattr(self.backend, "set_cookie", None)
+        if set_cookie is None:
+            raise NotImplementedError(
+                f"{type(self.backend).__name__} does not support the mutable cookie api"
+            )
+        set_cookie(self.session, name, value, domain=domain, path=path)
+
+    def get_cookies(self) -> dict:
+        """all cookies stored in the backend's cookie store"""
+        get_cookies = getattr(self.backend, "get_cookies", None)
+        if get_cookies is None:
+            raise NotImplementedError(
+                f"{type(self.backend).__name__} does not support the mutable cookie api"
+            )
+        return get_cookies(self.session)
+
+    def clear_cookies(self) -> None:
+        """clear the backend's cookie store"""
+        clear_cookies = getattr(self.backend, "clear_cookies", None)
+        if clear_cookies is None:
+            raise NotImplementedError(
+                f"{type(self.backend).__name__} does not support the mutable cookie api"
+            )
+        clear_cookies(self.session)
+
     # -------------------------------------------------------------------------
     #   lifecycle — backends close differently, so route through the backend