12 Commits
Author SHA1 Message Date
dsql 7ee6cc9b88 release: 1.0.0
first stable release. pre-1.0.0 verification complete: all surviving MED regressions and
gaps resolved and independently re-fired, tree audited clean across the suite.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-09 18:53:15 -04:00
dsql b5cc6b9374 fix: _jar_to_dict prefers get_dict() to survive cross-domain cookie conflicts
the preview cookie map still used jar.items(), which raises curl_cffi CookieConflict when
the same cookie name lives on two domains - the blanket except then returned {}, so preview()
silently showed zero cookies in exactly the state ec1a20a adopted get_dict() to survive on
get_cookies. prefer get_dict() when the jar exposes it (hasattr-guarded), fall back to items()
for a plain mapping jar. preview-only; the items() path is unchanged for jars without get_dict.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 21:16:15 -04:00
dsql cc5e4a9414 docs: correct the CurlCffi redirect-history claim
curl_cffi never populates Response.history (it follows redirects in the native curl layer
and surfaces only the final URL/status), so the v0.1.7 claim that resp.history shows 'the
real hops on either backend' was false for CurlCffi - resp.history/redirect_chain are []
there even after a redirect. README + changelog now document this curl_cffi limitation
instead. Noble threads whatever noble_tls records. No code change (the adapter is correct;
curl_cffi just has no data to map).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:57:59 -04:00
dsql ec1a20a3f6 fix: CurlCffi preserves duplicate Set-Cookie and get_cookies survives cross-domain names
CIMultiDict(response.headers) consumed curl_cffi Headers.items(), which comma-joins
duplicate header keys, collapsing multiple Set-Cookie lines into one corrupted value -
now uses multi_items() so duplicates stay separate. get_cookies used dict(cookies.items()),
which raises curl_cffi CookieConflict when the same name exists on two domains - now uses
get_dict() (flattens, no raise). Noble.get_cookies prefers get_dict() where the jar exposes
it (unverified live-gap: the noble extra isn't installed here).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:28:40 -04:00
dsql 8ed97a185f docs: bump stale aioweb dependency pin; note setup() is not auto-invoked
aioweb's git+ssh pin was stale at v0.1.5 against aioweb's actual latest tag,
v0.1.10 - bumped the pin, no change to aioweb_tls's own version. Also updates the
README's backend-protocol table to state that TLSSession never auto-invokes a
backend's setup(), matching the session.py docstring fix.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 00:10:27 -04:00
dsql df48d1cea3 fix: TLS backends return case-insensitive response headers
CurlCffi.raw_request and Noble's _flatten_headers built Response.headers as a plain
case-sensitive dict, while aioweb's aiohttp-backed path returns a CIMultiDict -
resp.headers.get('content-type') silently returned None on TLS backends when the
server sent 'Content-Type', contradicting the "backends behave identically" claim.
Both paths now build a multidict.CIMultiDict instead.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 00:10:20 -04:00
dsql b23e1d399e fix: TLSSession guards unbuilt session in _is_closed/close; document setup() is not auto-invoked
_is_closed() and close() called self.session unconditionally, routing through the
lazy session property and building a real backend client even when the session was
never used - including during __del__ on GC of a constructed-but-unused TLSSession,
silently building (and leaking) a backend client nothing ever closes. Mirrors
aioweb.ExtendedSession's own _session is None guards. Also corrects setup()'s
docstring, which claimed TLSSession invokes a backend's setup() lazily before the
first request - it never does; only Noble self-invokes it from its own raw_request.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 00:10:11 -04:00
dsql d40be6928a refactor: derive __version__ from package metadata (single source)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 17:01:01 -04:00
dsql 76c3024ccc docs: compress residual internal helper docstrings
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:46:55 -04:00
dsql 92ddd5dc39 fix: thread redirect history through both TLS backends
CurlCffi.raw_request and Noble.raw_request built their Response without
history=, so resp.history/redirect_chain were always empty after a real
redirect despite inheriting aioweb's feature set unchanged. A shared
_history_entries() now maps each client's native history shape (curl_cffi
list[dict], noble_tls list[Response]) into aioweb's (status, url) tuples.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:19:46 -04:00
dsql 226f273695 docs: compress prose/module docstrings, em-dash->hyphen (de-bloat wave 1)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 00:16:00 -04:00
dsql ce240b0757 fix: label TLS-backend timeouts ServerTimeoutError, flatten noble multi headers, ValueError on unknown profile (v0.1.5)
Both backends' asyncio.TimeoutError except-branch was dead code (neither curl_cffi
nor noble_tls raises it), so a real wire timeout fell through to the generic OSError
branch and got mislabeled a plain aiohttp.ClientError instead of aioweb's own
ServerTimeoutError contract; now detected via curl_cffi's Timeout type or Go-side
timeout text and re-wrapped correctly. Noble's multi-valued response headers (e.g.
two Set-Cookie lines) arrived as Python lists instead of strings, breaking any
downstream .split()/.lower() call; now comma-joined per RFC 7230. An unknown Noble
client profile string raised a raw AttributeError from the enum lookup; now a
ValueError listing the valid profile names. Also compresses essay-length docstrings
and narrating comments across the module with no behavior change.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 23:28:42 -04:00
6 changed files with 203 additions and 202 deletions
+49 -9
View File
@@ -22,17 +22,17 @@ you want; importing the package never fails because an extra is missing.
`requirements.txt` (pick the extra you need): `requirements.txt` (pick the extra you need):
``` ```
aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.4 aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v1.0.0
aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.4 aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v1.0.0
aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.4 aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v1.0.0
``` ```
Direct: Direct:
```bash ```bash
pip install "aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.4" pip install "aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v1.0.0"
pip install "aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.4" pip install "aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v1.0.0"
pip install "aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.4" pip install "aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v1.0.0"
``` ```
- `[curl]` → curl_cffi backend · `[noble]` → noble_tls backend · `[all]` → both. - `[curl]` → curl_cffi backend · `[noble]` → noble_tls backend · `[all]` → both.
@@ -44,7 +44,7 @@ pip install "aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-publi
Constructing a backend whose client isn't installed raises that `RuntimeError` at Constructing a backend whose client isn't installed raises that `RuntimeError` at
construction, never at import. construction, never at import.
Drop the `@v0.1.4` suffix from the line above to install the latest unpinned. Drop the `@v1.0.0` suffix from the line above to install the latest unpinned.
## curl_cffi backend ## curl_cffi backend
@@ -63,6 +63,10 @@ async with TLSSession(backend=CurlCffi(impersonate="chrome"), proxies={"https":
backend kwargs — passing `impersonate=` there raises `TypeError`; set the profile on backend kwargs — passing `impersonate=` there raises `TypeError`; set the profile on
the `CurlCffi` instance for the retrying path. the `CurlCffi` instance for the retrying path.
- curl_cffi forges JA3/JA4 + HTTP/2 fingerprints via the bundled curl-impersonate binary. - curl_cffi forges JA3/JA4 + HTTP/2 fingerprints via the bundled curl-impersonate binary.
- A wire-level timeout raises `aiohttp.ServerTimeoutError` (matching aioweb's own
contract), not a generic `aiohttp.ClientError` — both backends detect their
native timeout (curl_cffi's `Timeout` type, noble_tls's Go-side timeout text) and
re-wrap it before the fallback client-error path (v0.1.5).
## noble backend ## noble backend
@@ -76,7 +80,8 @@ async with TLSSession(backend=Noble(client="chrome_133")) as s:
print(resp.json()["tls"]["ja3"]) print(resp.json()["tls"]["ja3"])
``` ```
- `Noble(client="chrome_133")` — accepts a `noble_tls.Client` enum or a string name. - `Noble(client="chrome_133")` — accepts a `noble_tls.Client` enum or a string name;
an unknown string raises `ValueError` listing the valid profile names (v0.1.5).
- noble_tls downloads a Go shared library on first use. `await s.setup()` fetches it - noble_tls downloads a Go shared library on first use. `await s.setup()` fetches it
once at startup; if you skip it, the first request fetches it lazily. The fetch is once at startup; if you skip it, the first request fetches it lazily. The fetch is
guarded by a lock, so even concurrent first requests download it exactly once. guarded by a lock, so even concurrent first requests download it exactly once.
@@ -85,6 +90,10 @@ async with TLSSession(backend=Noble(client="chrome_133")) as s:
bytes (`U+FFFD` replacement, wrong length) even on a 200 response — the Noble bytes (`U+FFFD` replacement, wrong length) even on a 200 response — the Noble
backend always requests `is_byte_response=True` and decodes the resulting backend always requests `is_byte_response=True` and decodes the resulting
base64 data-URI back into raw bytes, so `resp.content` is never lossy. base64 data-URI back into raw bytes, so `resp.content` is never lossy.
- Multi-valued response headers (e.g. two `Set-Cookie` lines) arrive from noble_tls's
Go side as a Python list — the Noble backend flattens them to a single
comma-joined string per RFC 7230, so `resp.headers[...]` is always a plain string
(v0.1.5).
## Writing your own backend (the `TLSBackend` protocol) ## Writing your own backend (the `TLSBackend` protocol)
@@ -107,7 +116,7 @@ for the authoritative contract):
| `set_cookie` | optional | `(session, name, value, domain=None, path="/") -> None` | backs `TLSSession.set_cookie()`; raises `NotImplementedError` if absent | | `set_cookie` | optional | `(session, name, value, domain=None, path="/") -> None` | backs `TLSSession.set_cookie()`; raises `NotImplementedError` if absent |
| `get_cookies` | optional | `(session) -> dict` | backs `TLSSession.get_cookies()`; raises `NotImplementedError` if absent | | `get_cookies` | optional | `(session) -> dict` | backs `TLSSession.get_cookies()`; raises `NotImplementedError` if absent |
| `clear_cookies` | optional | `(session) -> None` | backs `TLSSession.clear_cookies()`; raises `NotImplementedError` if absent | | `clear_cookies` | optional | `(session) -> None` | backs `TLSSession.clear_cookies()`; raises `NotImplementedError` if absent |
| `setup` | optional | `async () -> None` | one-time prep (e.g. fetch a native lib); idempotent | | `setup` | optional | `async () -> None` | one-time prep (e.g. fetch a native lib); idempotent. `TLSSession` never calls this automatically - a backend needing lazy setup must self-invoke it from its own `raw_request`, as `Noble` does, or the caller must run `await session.setup()` explicitly |
| `close` | optional | `async (session) -> None` | close the session; defaults to `await session.close()` | | `close` | optional | `async (session) -> None` | close the session; defaults to `await session.close()` |
`raw_request` receives aioweb-shaped kwargs: the proxy is already resolved into `raw_request` receives aioweb-shaped kwargs: the proxy is already resolved into
@@ -189,12 +198,43 @@ cookie store (both expose a `requests`-style `session.cookies` with `set()` /
`items()` / `clear()`), so these calls work the same way they do on the base `items()` / `clear()`), so these calls work the same way they do on the base
`aioweb.ExtendedSession`, not just `_cookies_for_url()` (used by `preview()`). `aioweb.ExtendedSession`, not just `_cookies_for_url()` (used by `preview()`).
`resp.history` / `resp.redirect_chain` (`[(status, url), ...]`) and `resp.is_redirect`
are threaded through from whatever redirect history the backend exposes. **Caveat — the
CurlCffi backend has no per-hop history:** `curl_cffi` follows redirects internally in the
native curl layer and surfaces only the final URL/status, leaving `Response.history` an
empty list (it never populates it). So on `CurlCffi`, `resp.history`/`resp.redirect_chain`
are `[]` and `resp.is_redirect` reflects only the final response, even after a redirect —
this is a `curl_cffi` limitation, not a bug here, and it differs from the base aiohttp
`ExtendedSession` (which does record the hops). The `Noble` backend threads whatever
`noble_tls` exposes as its per-response history. If you need the redirect chain, use the
base backend or read the final URL.
## Honesty note ## Honesty note
TLS fingerprinting changes one layer — the TLS/HTTP fingerprint. It does **not** by TLS fingerprinting changes one layer — the TLS/HTTP fingerprint. It does **not** by
itself defeat modern bot protection: behavioral analysis, captchas, and JS challenges itself defeat modern bot protection: behavioral analysis, captchas, and JS challenges
are separate signals. Use this as one component, not a complete anti-bot solution. are separate signals. Use this as one component, not a complete anti-bot solution.
## Changelog
### v0.1.8
- Compressed 4 residual internal-helper docstrings (`_is_timeout_error`,
`_noble_timeout_seconds`, `_noble_content`, `_jar_to_dict`) to one-liners.
Cosmetic, zero behavior change.
### v0.1.7
- **Backends thread whatever redirect history the client exposes.** `CurlCffi.raw_request`
and `Noble.raw_request` built their `Response` without `history=`, so
`resp.history`/`resp.redirect_chain`/`resp.is_redirect`-after-follow were always empty
and aioweb's own `debug=True` "redirect chain:" log line was dead on both TLS backends.
Each backend now maps its client's history into aioweb's `(status, url)` tuple shape.
**Caveat:** `curl_cffi` never populates `Response.history` (it follows redirects in the
native curl layer and surfaces only the final URL/status), so on the `CurlCffi` backend
`resp.history` is `[]` even after a redirect — a `curl_cffi` limitation, not addressable
here. `Noble` passes through whatever `noble_tls` records.
## Versioning ## Versioning
Releases are tagged `vX.Y.Z`. The install line above pins a release; drop the `@vX.Y.Z` suffix to install the latest unpinned. Pin deliberately for reproducible installs. Releases are tagged `vX.Y.Z`. The install line above pins a release; drop the `@vX.Y.Z` suffix to install the latest unpinned. Pin deliberately for reproducible installs.
+2 -2
View File
@@ -4,11 +4,11 @@ build-backend = "hatchling.build"
[project] [project]
name = "aioweb_tls" name = "aioweb_tls"
version = "0.1.4" version = "1.0.0"
description = "TLS-fingerprinting backends (curl_cffi / noble_tls) for aioweb via one injectable TLSSession, config-free, installable." description = "TLS-fingerprinting backends (curl_cffi / noble_tls) for aioweb via one injectable TLSSession, config-free, installable."
requires-python = ">=3.10" requires-python = ">=3.10"
dependencies = [ dependencies = [
"aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.5", "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.10",
] ]
[project.optional-dependencies] [project.optional-dependencies]
+8 -15
View File
@@ -1,28 +1,21 @@
""" """
tls-fingerprinting backends for aioweb tls-fingerprinting backends for aioweb - see README for usage and the extras contract
one session class, TLSSession, takes an injected backend that swaps the HTTP client
(and thus the TLS/HTTP fingerprint) while inheriting every aioweb feature — header
overwrites, domain rewriting, ephemeral headers, proxies, retries, previews —
unchanged.
from aioweb_tls import TLSSession, CurlCffi, Noble from aioweb_tls import TLSSession, CurlCffi, Noble
async with TLSSession(backend=CurlCffi(impersonate="chrome")) as s: # [curl] extra async with TLSSession(backend=CurlCffi(impersonate="chrome")) as s: # [curl] extra
resp = await s.request_with_retries("GET", url) resp = await s.request_with_retries("GET", url)
async with TLSSession(backend=Noble(client="chrome_133")) as s: # [noble] extra
await s.setup() # fetch Go lib once
resp = await s.request_with_retries("GET", url)
the tls clients are optional extras, not base deps. importing this package never
fails because an extra is missing; the matching RuntimeError is raised only when you
construct a backend whose client isn't installed. custom backends implement the
TLSBackend protocol and inject the same way.
""" """
from importlib.metadata import version, PackageNotFoundError
from .session import TLSSession from .session import TLSSession
from .backends import CurlCffi, Noble from .backends import CurlCffi, Noble
from .protocol import TLSBackend from .protocol import TLSBackend
try:
__version__ = version("aioweb_tls")
except PackageNotFoundError:
__version__ = "0.0.0+unknown"
__all__ = ["TLSSession", "CurlCffi", "Noble", "TLSBackend"] __all__ = ["TLSSession", "CurlCffi", "Noble", "TLSBackend"]
+119 -100
View File
@@ -1,14 +1,6 @@
""" """
tls backends for TLSSession tls backends for TLSSession - stateless config+behavior objects implementing the
TLSBackend protocol (see protocol.py); see README for the extras contract
each backend is a stateless config+behavior object implementing the TLSBackend
protocol (see protocol.py). it owns its own config vocabulary — CurlCffi takes
impersonate=, Noble takes client= — so there is no shared kwarg-soup. TLSSession
owns the live session object and passes it into every method here.
the underlying tls clients are optional extras: constructing a backend whose client
is not installed raises a clear RuntimeError naming the extra to install. importing
this module never fails because an extra is missing.
""" """
import asyncio import asyncio
@@ -17,22 +9,33 @@ import logging
import math import math
import aiohttp import aiohttp
from multidict import CIMultiDict
from aioweb import Response from aioweb import Response
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
def _as_client_error(error: Exception, backend: str) -> aiohttp.ClientError: def _as_client_error(error: Exception, backend: str) -> aiohttp.ClientError:
"""wrap a backend-native network exception as an aiohttp.ClientError """wrap a backend-native network exception (neither is an aiohttp.ClientError) as one"""
aioweb's request() only re-wraps aiohttp.ClientError; curl_cffi raises
RequestException(OSError) and noble_tls raises TLSClientException(IOError), neither
of which is an aiohttp.ClientError. translating here gives TLS backends the same
typed failure contract as the aiohttp path on the bare request() route.
"""
return aiohttp.ClientError(f"{backend} request failed: {error}") return aiohttp.ClientError(f"{backend} request failed: {error}")
_TIMEOUT_TEXT = ("timeout", "timed out", "deadline exceeded")
def _is_timeout_error(error: Exception) -> bool:
"""whether a backend-native OSError is a timeout, by type name (curl_cffi) or Go error text (noble_tls)"""
timeout_type = getattr(error, "__class__", None)
if timeout_type is not None and any(base.__name__ == "Timeout" for base in timeout_type.__mro__):
return True
return any(needle in str(error).lower() for needle in _TIMEOUT_TEXT)
def _as_timeout_error(error: Exception, backend: str) -> aiohttp.ServerTimeoutError:
"""wrap a backend-native timeout as aiohttp.ServerTimeoutError, matching aioweb's own contract"""
return aiohttp.ServerTimeoutError(f"{backend} request timed out: {error}")
try: try:
from curl_cffi import AsyncSession as _CurlAsyncSession from curl_cffi import AsyncSession as _CurlAsyncSession
_CURL_ERROR = None _CURL_ERROR = None
@@ -51,25 +54,14 @@ except ImportError as error:
def _coerce_timeout(value): def _coerce_timeout(value):
"""turn aioweb's aiohttp ClientTimeout (or a number) into a plain number """unwrap aioweb's aiohttp.ClientTimeout (or a plain number) to a bare number for the tls clients"""
aioweb.request() wraps a numeric timeout in an aiohttp.ClientTimeout before the
seam sees it; the tls clients want a number, so unwrap .total when present.
"""
total = getattr(value, "total", value) total = getattr(value, "total", value)
return total if isinstance(total, (int, float)) else None return total if isinstance(total, (int, float)) else None
def _noble_timeout_seconds(value): def _noble_timeout_seconds(value):
"""coerce a raw or wrapped timeout into whole seconds for noble's Go int field """coerce a raw or wrapped timeout into whole seconds (min 1), rounding up so noble's Go int
field never truncates to 0/fails unmarshal; None if uncoercible; shared by both Noble timeout paths"""
single source of truth for both of Noble's timeout paths (create_session's
session-default and raw_request's per-call override): noble's Go field
timeoutSeconds is an int, so a sub-second float (e.g. 0.5) truncates to 0 — which
Go reads as no/instant timeout — and any non-integer value fails Go-side JSON
unmarshal outright. rounds UP so a sub-second timeout still waits at least 1s
instead of truncating away. returns None when there is no timeout to coerce.
"""
timeout = _coerce_timeout(value) timeout = _coerce_timeout(value)
if timeout is None: if timeout is None:
return None return None
@@ -77,15 +69,8 @@ def _noble_timeout_seconds(value):
def _noble_content(response) -> bytes: def _noble_content(response) -> bytes:
"""extract true response bytes from a noble_tls response fetched with is_byte_response """extract true response bytes from a noble_tls response fetched with is_byte_response, decoding
the base64 data-URI in response.text (falls back to a plain utf-8 encode if not a data-URI)"""
with is_byte_response=True, noble_tls's Go side returns the body as a
`data:<mime>;base64,<payload>` URI string in response.text (the whole Go response is
UTF-8-decoded before JSON parsing, so raw bytes have to travel as base64 to survive
that round trip) — response.content is useless here since it is just
`self.text.encode()`, re-encoding the data-URI string itself rather than decoding it.
falls back to a plain utf-8 encode if the body isn't a data-URI (e.g. an error body).
"""
text = getattr(response, "text", "") or "" text = getattr(response, "text", "") or ""
if text.startswith("data:") and ";base64," in text: if text.startswith("data:") and ";base64," in text:
_, _, payload = text.partition(";base64,") _, _, payload = text.partition(";base64,")
@@ -93,17 +78,50 @@ def _noble_content(response) -> bytes:
return text.encode() return text.encode()
def _jar_to_dict(session): def _flatten_headers(headers) -> CIMultiDict:
"""best-effort map of a requests-style cookie jar on session to a plain dict """flatten a Go-style map[str][]str header dict (multi-valued headers as a list)
into plain str values, joined with ", " per RFC 7230, into a case-insensitive
mapping matching aioweb's aiohttp-backed Response.headers"""
return CIMultiDict(
(key, ", ".join(value) if isinstance(value, list) else value)
for key, value in headers.items()
)
intentionally broad: this feeds preview() only, the two backends expose differently-
shaped jars, and a cookie read must never crash a request — so any jar that doesn't def _history_entries(history) -> list:
iterate cleanly degrades to {} rather than raising. """map a backend-native redirect history into aioweb's `Response(history=...)` shape
aioweb's own `_raw_request` threads `[(status, url), ...]`; curl_cffi's
`Response.history` is `list[dict]` (dict-shaped hop records) and noble_tls's is
`list[Response]` (object-shaped, same accessors as the top-level response) - this
reads a hop's status/url either way. an unparseable hop is skipped rather than
raising, so a redirect-history quirk never breaks the response it's attached to.
""" """
entries = []
for hop in history or []:
if isinstance(hop, dict):
status = hop.get("status_code", hop.get("status"))
url = hop.get("url")
else:
status = getattr(hop, "status_code", getattr(hop, "status", None))
url = getattr(hop, "url", None)
if status is None or url is None:
continue
entries.append((status, str(url)))
return entries
def _jar_to_dict(session):
"""best-effort map of a requests-style cookie jar on session to a plain dict, feeding preview()
only; a jar that fails to iterate degrades to {} rather than raising"""
jar = getattr(session, "cookies", None) jar = getattr(session, "cookies", None)
if not jar: if not jar:
return {} return {}
try: try:
# prefer get_dict() where the jar exposes it: items() raises curl_cffi CookieConflict
# when the same name lives on two domains, get_dict() flattens instead (mirrors get_cookies)
if hasattr(jar, "get_dict"):
return jar.get_dict()
return {k: v for k, v in jar.items()} return {k: v for k, v in jar.items()}
except Exception: except Exception:
return {} return {}
@@ -114,10 +132,9 @@ class CurlCffi:
config: config:
impersonate: browser profile to forge (default "chrome"); override per call impersonate: browser profile to forge (default "chrome"); override per call
by passing impersonate= to the low-level request()/_raw_request path, via request()/_raw_request (forwards **kwargs) - NOT request_with_retries,
which forwards **kwargs to the backend. NOT request_with_retries — its whose fixed signature raises TypeError on it. for the retrying path, set
signature is fixed (no **kwargs) and would raise TypeError. for a the profile on the CurlCffi instance instead.
per-call profile under retries, set it on the CurlCffi instance instead.
requires the [curl] extra (pip install "aioweb_tls[curl]"). requires the [curl] extra (pip install "aioweb_tls[curl]").
""" """
@@ -133,13 +150,11 @@ class CurlCffi:
def create_session(self, headers, timeout, **kwargs): def create_session(self, headers, timeout, **kwargs):
"""build the curl_cffi AsyncSession """build the curl_cffi AsyncSession
deliberately does NOT pass `headers` to AsyncSession: curl_cffi bakes a deliberately does NOT pass `headers`: curl_cffi bakes a constructor
constructor `headers=` into the client and re-merges it under whatever `headers=` into the client and re-merges it under per-request headers,
per-request headers() sends, so update_headers()/clear_headers() would stop which would desync update_headers()/clear_headers() from what's actually on
matching what's actually on the wire (aioweb's session-default headers are the wire - aioweb's session-default headers already apply per request via
already applied per request by the base's _default_headers merge — see the base's _default_headers merge (see ExtendedSession._create_session).
aioweb.ExtendedSession._create_session's docstring for why baking breaks the
mutable header api).
""" """
return _CurlAsyncSession(timeout=timeout, **kwargs) return _CurlAsyncSession(timeout=timeout, **kwargs)
@@ -162,27 +177,26 @@ class CurlCffi:
except asyncio.TimeoutError: except asyncio.TimeoutError:
raise raise
except OSError as error: except OSError as error:
# curl_cffi's RequestException subclasses OSError; translate the native if _is_timeout_error(error):
# network error into aiohttp.ClientError. narrowed from a bare Exception so a raise _as_timeout_error(error, "curl_cffi") from error
# real bug (AttributeError/TypeError) isn't laundered into 'client error'
raise _as_client_error(error, "curl_cffi") from error raise _as_client_error(error, "curl_cffi") from error
content = response.content if response.content is not None else b"" content = response.content if response.content is not None else b""
return Response( return Response(
status_code=response.status_code, status_code=response.status_code,
headers=dict(response.headers), headers=CIMultiDict(response.headers.multi_items()),
content=content, content=content,
url=str(response.url), url=str(response.url),
reason=getattr(response, "reason", None), reason=getattr(response, "reason", None),
history=_history_entries(getattr(response, "history", None)),
cookies=getattr(response, "cookies", None), cookies=getattr(response, "cookies", None),
) )
def is_closed(self, session) -> bool: def is_closed(self, session) -> bool:
"""whether the curl_cffi session is closed """whether the curl_cffi session is closed
curl_cffi tracks closed state in the private `_closed` (no public `closed` curl_cffi tracks this in the private `_closed` (no public `closed` today);
property), so read that; fall back to a public `closed` if a future version falls back to a public `closed` if a future version adds one. best-effort -
adds one. TLSSession's own `_closed` flag is the primary signal — this is a TLSSession's own `_closed` flag is the primary signal.
best-effort backend check for out-of-band closes.
""" """
closed = getattr(session, "_closed", None) closed = getattr(session, "_closed", None)
if closed is None: if closed is None:
@@ -198,8 +212,13 @@ class CurlCffi:
session.cookies.set(name, value, domain=domain or "", path=path) session.cookies.set(name, value, domain=domain or "", path=path)
def get_cookies(self, session) -> dict: def get_cookies(self, session) -> dict:
"""all cookies stored in curl_cffi's cookie store""" """all cookies stored in curl_cffi's cookie store
return dict(session.cookies.items())
uses get_dict() rather than dict(cookies.items()): items() raises curl_cffi
CookieConflict when the same name exists on two domains, get_dict() flattens
(last value wins) without raising.
"""
return session.cookies.get_dict()
def clear_cookies(self, session) -> None: def clear_cookies(self, session) -> None:
"""clear curl_cffi's cookie store""" """clear curl_cffi's cookie store"""
@@ -216,8 +235,8 @@ class Noble:
config: config:
client: noble_tls Client profile (enum or string, default "chrome_133"). client: noble_tls Client profile (enum or string, default "chrome_133").
noble_tls downloads a Go shared library on first use; setup() fetches it once downloads a Go shared library on first use; setup() fetches it once (via
(run via TLSSession.setup() or lazily before the first request). TLSSession.setup() or lazily before the first request).
requires the [noble] extra (pip install "aioweb_tls[noble]"). requires the [noble] extra (pip install "aioweb_tls[noble]").
""" """
@@ -234,21 +253,23 @@ class Noble:
@staticmethod @staticmethod
def _resolve_client(client): def _resolve_client(client):
"""turn a string or Client enum into a noble_tls Client value""" """turn a string or Client enum into a noble_tls Client value, raising ValueError
if isinstance(client, str): naming the valid profiles for an unknown string"""
return getattr(_NobleClient, client.upper()) if not isinstance(client, str):
return client return client
name = client.upper()
resolved = getattr(_NobleClient, name, None)
if resolved is None:
valid = ", ".join(member.name for member in _NobleClient)
raise ValueError(f"unknown noble_tls client profile {client!r}; valid profiles: {valid}")
return resolved
async def setup(self) -> None: async def setup(self) -> None:
"""fetch the noble_tls Go shared library once; idempotent and concurrency-safe """fetch the noble_tls Go shared library once; idempotent and concurrency-safe
uses noble_tls.download_if_necessary (the current API: it fetches the asset on uses download_if_necessary (falls back to update_if_necessary on older
first use and no-ops when it already exists). older noble_tls without that name noble_tls). guarded by an asyncio.Lock with a check-lock-recheck so concurrent
is handled via update_if_necessary as a fallback. first requests don't both run the fetch.
guarded by an asyncio.Lock with a check-lock-recheck so concurrent first
requests don't both run the fetch: the fast path returns once _updated is
set, and only the first caller through the lock does the work.
""" """
if self._updated: if self._updated:
return return
@@ -265,19 +286,13 @@ class Noble:
def create_session(self, headers, timeout, **kwargs): def create_session(self, headers, timeout, **kwargs):
"""build the noble_tls Session, honoring the session-default timeout """build the noble_tls Session, honoring the session-default timeout
noble_tls.Session takes neither headers nor timeout in its constructor. deliberately does NOT bake `headers` into session.headers - same header-baking
deliberately does NOT bake `headers` into session.headers: aioweb's rationale as CurlCffi.create_session (see its docstring).
session-default headers are already applied per request by the base's
_default_headers merge, and baking them here would make
update_headers()/clear_headers() stop matching what's actually on the wire
(see aioweb.ExtendedSession._create_session's docstring, and CurlCffi.create_session
above for the same rationale) — the per-request merge means headers are never
silently dropped, contrary to what this docstring used to claim.
the coerced timeout IS applied here (matching raw_request's per-call path): noble's the coerced timeout IS applied here (matching raw_request's per-call path):
Go field timeoutSeconds is an int, so a raw sub-second/float session-default (e.g. noble's Go field timeoutSeconds is an int, so an uncoerced sub-second/float
timeout=7.5) would fail Go-side JSON unmarshal on every request that doesn't session-default would fail Go-side JSON unmarshal on any request that doesn't
override it per call. max(1, ceil()) mirrors the guard raw_request already has. override it per call; max(1, ceil()) mirrors raw_request's own guard.
""" """
session = noble_tls.Session(client=self.client, **kwargs) session = noble_tls.Session(client=self.client, **kwargs)
timeout_seconds = _noble_timeout_seconds(timeout) timeout_seconds = _noble_timeout_seconds(timeout)
@@ -297,10 +312,6 @@ class Noble:
if proxy: if proxy:
kwargs["proxy"] = proxy kwargs["proxy"] = proxy
# force byte-safe transport: without this, noble_tls's Go side returns the body
# as a plain UTF-8 JSON string, replacing invalid bytes with U+FFFD — silently
# corrupting any binary payload (image/zip/pdf) even though the request succeeds
# with status 200. see _noble_content() for how the byte-safe body is decoded back.
kwargs.setdefault("is_byte_response", True) kwargs.setdefault("is_byte_response", True)
try: try:
@@ -310,16 +321,17 @@ class Noble:
except asyncio.TimeoutError: except asyncio.TimeoutError:
raise raise
except OSError as error: except OSError as error:
# noble_tls's TLSClientException subclasses IOError (== OSError); translate if _is_timeout_error(error):
# the native network error, narrowed from bare Exception so a real bug surfaces raise _as_timeout_error(error, "noble_tls") from error
raise _as_client_error(error, "noble_tls") from error raise _as_client_error(error, "noble_tls") from error
content = _noble_content(response) content = _noble_content(response)
return Response( return Response(
status_code=response.status_code, status_code=response.status_code,
headers=dict(getattr(response, "headers", {}) or {}), headers=_flatten_headers(getattr(response, "headers", {}) or {}),
content=content, content=content,
url=str(getattr(response, "url", url)), url=str(getattr(response, "url", url)),
reason=getattr(response, "reason", None), reason=getattr(response, "reason", None),
history=_history_entries(getattr(response, "history", None)),
cookies=getattr(response, "cookies", None), cookies=getattr(response, "cookies", None),
) )
@@ -336,8 +348,15 @@ class Noble:
session.cookies.set(name, value, domain=domain or "", path=path) session.cookies.set(name, value, domain=domain or "", path=path)
def get_cookies(self, session) -> dict: def get_cookies(self, session) -> dict:
"""all cookies stored in noble_tls's cookie jar""" """all cookies stored in noble_tls's cookie jar
return dict(session.cookies.items())
prefers get_dict() when the jar exposes it (flattens cross-domain duplicate
names without raising, like curl_cffi); falls back to items() otherwise
"""
jar = session.cookies
if hasattr(jar, "get_dict"):
return jar.get_dict()
return dict(jar.items())
def clear_cookies(self, session) -> None: def clear_cookies(self, session) -> None:
"""clear noble_tls's cookie jar""" """clear noble_tls's cookie jar"""
+1 -48
View File
@@ -1,51 +1,4 @@
""" """the tls backend protocol - see TLSBackend below and README"""
the tls backend protocol
a backend is a stateless config+behavior object that teaches TLSSession how to talk
to one HTTP client. TLSSession owns the live session object (built by create_session)
and passes it into every backend call, so backends hold no per-request state.
implement this protocol to add a custom backend (e.g. a local Go TLS server); inject
it via TLSSession(backend=MyBackend(...)) and it inherits all of aioweb's domain /
header / ephemeral / proxy / retry / preview logic unchanged — those operate on plain
dicts and never touch the backend.
required:
create_session(headers: dict, timeout, **kwargs) -> session
build and return the live client session object. TLSSession stores it as
self.session and hands it back to every other method below.
async raw_request(session, method, url, **kwargs) -> aioweb.Response
send one request with `session` and adapt the client's response into an
aioweb.Response built from primitives (status_code, headers, content bytes,
url, reason). kwargs arrive aioweb-shaped — the base has already resolved the
proxy into kwargs["proxy"] and merged headers into kwargs["headers"], and a
numeric timeout is wrapped in an aiohttp.ClientTimeout (unwrap .total).
is_closed(session) -> bool
whether `session` is closed.
optional:
cookies_for_url(session, url) -> dict
cookies the client would send for url, for preview(). default {} (used when
the backend has no introspectable jar).
set_cookie(session, name, value, domain=None, path="/") -> None
get_cookies(session) -> dict
clear_cookies(session) -> None
the mutable cookie api TLSSession.set_cookie/get_cookies/clear_cookies
delegate to. the aiohttp-only base implementations reach into
session.cookie_jar, which TLS backends don't have, so a backend without
these raises NotImplementedError from TLSSession rather than crashing with
an AttributeError on a private aiohttp attribute.
async setup() -> None
one-time async preparation (e.g. fetch a native lib). called once via
TLSSession.setup() and lazily before the first request; make it idempotent.
async close(session) -> None
close `session`. default awaits session.close() if present.
"""
from typing import Any, Protocol, runtime_checkable from typing import Any, Protocol, runtime_checkable
+24 -28
View File
@@ -1,26 +1,16 @@
""" """
TLSSession one aioweb session, any tls backend TLSSession - one aioweb session, any tls backend
TLSSession subclasses aioweb.ExtendedSession and delegates only the four backend subclasses aioweb.ExtendedSession, delegating only the four backend seams to an
seams to an injected backend object (see protocol.py). everything else — header injected backend object (see protocol.py); everything else is inherited unchanged.
overwrites, domain rewriting, ephemeral headers, proxies, retries, previews — is see README for usage.
inherited from aioweb unchanged, because that logic operates on plain dicts and
never touches the backend.
from aioweb_tls import TLSSession, CurlCffi, Noble from aioweb_tls import TLSSession, CurlCffi
async with TLSSession(backend=CurlCffi(impersonate="chrome")) as s: async with TLSSession(backend=CurlCffi(impersonate="chrome")) as s:
resp = await s.request_with_retries("GET", "https://tls.peet.ws/api/all") resp = await s.request_with_retries("GET", "https://tls.peet.ws/api/all")
if resp: if resp:
print(resp.json()["tls"]["ja3"]) print(resp.json()["tls"]["ja3"])
s = TLSSession(backend=Noble(client="chrome_133"))
await s.setup() # fetch noble's Go lib once
...
a custom backend (e.g. a local Go TLS server) injects the same way — implement the
TLSBackend protocol and pass it as backend=. one TLSSession is the only session
class; the backend swaps the wire, not the session.
""" """
import logging import logging
@@ -46,12 +36,12 @@ class TLSSession(ExtendedSession):
super().__init__(*args, **kwargs) super().__init__(*args, **kwargs)
async def setup(self) -> None: async def setup(self) -> None:
"""run the backend's one-time setup if it has one (idempotent) """run the backend's one-time setup if it has one (idempotent, e.g. noble's Go lib fetch)
delegates to backend.setup() when defined (e.g. noble fetching its Go lib); optional to call upfront to pre-warm at startup. TLSSession itself never calls
a no-op for backends without it. also invoked lazily before the first request this automatically - a backend needing lazy setup must self-invoke it from its
by backends that guard their own setup, so calling this is optional but lets own raw_request, as Noble does; a custom backend that skips this will never run
callers pre-warm at startup. setup() unless the caller calls TLSSession.setup() explicitly.
""" """
setup = getattr(self.backend, "setup", None) setup = getattr(self.backend, "setup", None)
if setup is not None: if setup is not None:
@@ -76,15 +66,19 @@ class TLSSession(ExtendedSession):
return cookies_for_url(self.session, url) return cookies_for_url(self.session, url)
def _is_closed(self) -> bool: def _is_closed(self) -> bool:
"""closed if explicitly closed here or the backend reports it""" """closed if explicitly closed here, never built, or the backend reports it
an unbuilt session counts as closed: nothing was opened, nothing to leak.
"""
if self._closed: if self._closed:
return True return True
if self._session is None:
return True
return self.backend.is_closed(self.session) return self.backend.is_closed(self.session)
# ------------------------------------------------------------------------- # -------------------------------------------------------------------------
# mutable cookie api — the base's set_cookie/get_cookies/clear_cookies reach # mutable cookie api - base reaches into aiohttp-only session.cookie_jar, so
# into self.session.cookie_jar (aiohttp-only), so TLS backends override them # TLS backends override these to route through the backend instead
# to route through the backend instead of crashing with AttributeError
def set_cookie(self, name, value, domain=None, path="/"): def set_cookie(self, name, value, domain=None, path="/"):
"""set a cookie via the backend's own cookie store""" """set a cookie via the backend's own cookie store"""
@@ -114,13 +108,15 @@ class TLSSession(ExtendedSession):
clear_cookies(self.session) clear_cookies(self.session)
# ------------------------------------------------------------------------- # -------------------------------------------------------------------------
# lifecycle backends close differently, so route through the backend # lifecycle - backends close differently, so route through the backend
async def close(self) -> None: async def close(self) -> None:
"""close via the backend's close (falls back to session.close)""" """close via the backend's close (falls back to session.close); no-op if never built"""
self._closed = True self._closed = True
if self._session is None:
return
close = getattr(self.backend, "close", None) close = getattr(self.backend, "close", None)
if close is not None: if close is not None:
await close(self.session) await close(self._session)
else: else:
await self.session.close() await self._session.close()