24 Commits
Author SHA1 Message Date
dsql 7ee6cc9b88 release: 1.0.0
first stable release. pre-1.0.0 verification complete: all surviving MED regressions and
gaps resolved and independently re-fired, tree audited clean across the suite.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-09 18:53:15 -04:00
dsql b5cc6b9374 fix: _jar_to_dict prefers get_dict() to survive cross-domain cookie conflicts
the preview cookie map still used jar.items(), which raises curl_cffi CookieConflict when
the same cookie name lives on two domains - the blanket except then returned {}, so preview()
silently showed zero cookies in exactly the state ec1a20a adopted get_dict() to survive on
get_cookies. prefer get_dict() when the jar exposes it (hasattr-guarded), fall back to items()
for a plain mapping jar. preview-only; the items() path is unchanged for jars without get_dict.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 21:16:15 -04:00
dsql cc5e4a9414 docs: correct the CurlCffi redirect-history claim
curl_cffi never populates Response.history (it follows redirects in the native curl layer
and surfaces only the final URL/status), so the v0.1.7 claim that resp.history shows 'the
real hops on either backend' was false for CurlCffi - resp.history/redirect_chain are []
there even after a redirect. README + changelog now document this curl_cffi limitation
instead. Noble threads whatever noble_tls records. No code change (the adapter is correct;
curl_cffi just has no data to map).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:57:59 -04:00
dsql ec1a20a3f6 fix: CurlCffi preserves duplicate Set-Cookie and get_cookies survives cross-domain names
CIMultiDict(response.headers) consumed curl_cffi Headers.items(), which comma-joins
duplicate header keys, collapsing multiple Set-Cookie lines into one corrupted value -
now uses multi_items() so duplicates stay separate. get_cookies used dict(cookies.items()),
which raises curl_cffi CookieConflict when the same name exists on two domains - now uses
get_dict() (flattens, no raise). Noble.get_cookies prefers get_dict() where the jar exposes
it (unverified live-gap: the noble extra isn't installed here).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:28:40 -04:00
dsql 8ed97a185f docs: bump stale aioweb dependency pin; note setup() is not auto-invoked
aioweb's git+ssh pin was stale at v0.1.5 against aioweb's actual latest tag,
v0.1.10 - bumped the pin, no change to aioweb_tls's own version. Also updates the
README's backend-protocol table to state that TLSSession never auto-invokes a
backend's setup(), matching the session.py docstring fix.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 00:10:27 -04:00
dsql df48d1cea3 fix: TLS backends return case-insensitive response headers
CurlCffi.raw_request and Noble's _flatten_headers built Response.headers as a plain
case-sensitive dict, while aioweb's aiohttp-backed path returns a CIMultiDict -
resp.headers.get('content-type') silently returned None on TLS backends when the
server sent 'Content-Type', contradicting the "backends behave identically" claim.
Both paths now build a multidict.CIMultiDict instead.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 00:10:20 -04:00
dsql b23e1d399e fix: TLSSession guards unbuilt session in _is_closed/close; document setup() is not auto-invoked
_is_closed() and close() called self.session unconditionally, routing through the
lazy session property and building a real backend client even when the session was
never used - including during __del__ on GC of a constructed-but-unused TLSSession,
silently building (and leaking) a backend client nothing ever closes. Mirrors
aioweb.ExtendedSession's own _session is None guards. Also corrects setup()'s
docstring, which claimed TLSSession invokes a backend's setup() lazily before the
first request - it never does; only Noble self-invokes it from its own raw_request.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 00:10:11 -04:00
dsql d40be6928a refactor: derive __version__ from package metadata (single source)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 17:01:01 -04:00
dsql 76c3024ccc docs: compress residual internal helper docstrings
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:46:55 -04:00
dsql 92ddd5dc39 fix: thread redirect history through both TLS backends
CurlCffi.raw_request and Noble.raw_request built their Response without
history=, so resp.history/redirect_chain were always empty after a real
redirect despite inheriting aioweb's feature set unchanged. A shared
_history_entries() now maps each client's native history shape (curl_cffi
list[dict], noble_tls list[Response]) into aioweb's (status, url) tuples.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:19:46 -04:00
dsql 226f273695 docs: compress prose/module docstrings, em-dash->hyphen (de-bloat wave 1)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 00:16:00 -04:00
dsql ce240b0757 fix: label TLS-backend timeouts ServerTimeoutError, flatten noble multi headers, ValueError on unknown profile (v0.1.5)
Both backends' asyncio.TimeoutError except-branch was dead code (neither curl_cffi
nor noble_tls raises it), so a real wire timeout fell through to the generic OSError
branch and got mislabeled a plain aiohttp.ClientError instead of aioweb's own
ServerTimeoutError contract; now detected via curl_cffi's Timeout type or Go-side
timeout text and re-wrapped correctly. Noble's multi-valued response headers (e.g.
two Set-Cookie lines) arrived as Python lists instead of strings, breaking any
downstream .split()/.lower() call; now comma-joined per RFC 7230. An unknown Noble
client profile string raised a raw AttributeError from the enum lookup; now a
ValueError listing the valid profile names. Also compresses essay-length docstrings
and narrating comments across the module with no behavior change.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 23:28:42 -04:00
dsql eb7745ae9b fix: seam mutable cookie api, byte-safe noble bodies, drop header baking, coerce noble session timeout (v0.1.4)
TLSSession.set_cookie/get_cookies/clear_cookies crashed with AttributeError on
both backends (they reached into self.session.cookie_jar, which curl_cffi and
noble_tls sessions don't have); both backends now route the mutable cookie api
through their own requests-style session.cookies store.

Noble.raw_request now requests is_byte_response=True and decodes the resulting
base64 data-URI body, since noble_tls's default text response silently corrupts
any binary payload (image/zip/pdf) via lossy UTF-8 decoding on the Go side.

CurlCffi/Noble.create_session no longer bake session-default headers into the
underlying client; baking caused clear_headers()/get_headers() to lie about
what's actually still on the wire (a credential-leak divergence from the
aiohttp base, which never bakes). Headers flow through aioweb's per-request
merge only, matching the base's documented contract.

Noble.create_session now applies the same max(1, ceil()) timeout coercion
raw_request already had (extracted into a shared _noble_timeout_seconds
helper) — without it, a sub-second/float session-default timeout made every
request fail Go-side JSON unmarshal.

README corrected: dropped the 'every aioweb feature behaves identically'
overclaim re: cookies, documented the binary-body handling and the
no-header-baking rationale, bumped install pins to v0.1.4.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 17:09:11 -04:00
dsql ce0762a37c chore: ignore .claude/ dir (CLAUDE.md now lives under .claude/)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:55:13 -04:00
dsql ef9eb3010c fix: bump aioweb pin to v0.1.5 (seam-checked); narrow backend except
bump the aioweb dependency from the stale v0.1.0 to current v0.1.5 — seam-verified against
v0.1.5's actual API (request_with_retries -> Response on success, falsy FailureResponse on
failure, all four override seams present). backend raw_request catches narrowed from bare
Exception to OSError (covers curl_cffi RequestException / noble TLSClientException) and
re-raises ClientError/TimeoutError first, so a real bug isn't laundered into 'client error'.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:35:24 -04:00
dsql da8b0bf6f8 docs: pin install line to release, note unpinned-latest option
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 18:13:36 -04:00
dsql 612861b76c docs: show unpinned install line; note tag-pinning for reproducibility
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 18:07:21 -04:00
dsql ccea880df0 fix: wrap backend-native exceptions; correct setup/desc docs (v0.1.3)
- CurlCffi/Noble raw_request translate backend-native network errors (curl_cffi
  RequestException, noble_tls TLSClientException) into aiohttp.ClientError so the bare
  request() path gives the same typed-failure contract as the aiohttp backend (L6)
- Noble.setup uses download_if_necessary (the current noble_tls API), with
  update_if_necessary only as a fallback; docstring/CLAUDE.md no longer claim the dead
  'refreshes an existing one' path (L7)
- pyproject description says composition (one injectable TLSSession), not the old
  'ExtendedSession subclasses' (L8).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 17:57:54 -04:00
dsql 5eb689ba73 docs: narrow the per-call impersonate= claim (v0.1.2)
per-call impersonate= is honored only via the low-level request()/_raw_request path
(which forwards **kwargs to the backend), NOT request_with_retries — its inherited
aioweb signature is fixed with no **kwargs and raises TypeError on an extra kwarg.
docs-only across README + backend docstring + CLAUDE.md; for the retrying path, set
the profile on the CurlCffi/Noble instance. no code change.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 17:11:57 -04:00
dsql 87debe8465 fix: is_closed reads curl_cffi's private _closed (no public closed attr)
CurlCffi.is_closed read getattr(session, 'closed', False), but curl_cffi tracks closed state only in the private _closed and exposes no public 'closed' property, so it always returned False. it now reads _closed, falling back to a public 'closed' if a future version adds one. TLSSession's own flag remains the primary signal; this is the best-effort backend check for out-of-band closes.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 01:10:25 -04:00
dsql b2876d005e fix: Noble honors session-default headers and timeout like CurlCffi
noble_tls.Session takes neither headers nor timeout in its constructor, and Noble.create_session forwarded only client+kwargs, so TLSSession(backend=Noble(...), headers=...) silently dropped the headers while CurlCffi passed them through. create_session now applies headers via session.headers.update and sets timeout_seconds after construction. verified against the contract with a stubbed noble_tls; the real Go-lib + a live request remain an untested gap (noble_tls not installable in this env).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-28 18:45:25 -04:00
dsql ae4c653ecc fix: lock Noble.setup so concurrent first requests fetch the Go-lib once
Noble.setup guarded the one-time Go shared-library fetch with a bare 'if self._updated' flag — a TOCTOU race where concurrent first requests both passed the check before either set the flag, running the download multiple times. now guarded by a per-instance asyncio.Lock with a check-lock-recheck. verified under load: 2/10/100/500 concurrent setups run the fetch exactly once each (a no-lock control runs it N times).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-28 17:46:20 -04:00
dsql 7ea8ecf888 fix: don't truncate sub-second timeout to 0 in noble backend (v0.1.1)
int(timeout) truncated a fractional timeout (e.g. 0.5s) to 0, which noble treats as
no/instant timeout. round up with math.ceil and floor at 1 so a sub-second timeout
stays a real (>=1s) timeout.

verified: 0.5/0.1/0.001 -> 1 (was 0); whole seconds unchanged.
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-28 15:49:30 -04:00
dsql befa4cd196 add package: pyproject + src
TLSSession over aioweb's backend seam by composition: one session class
delegates the four seams to an injected backend. ships CurlCffi (curl_cffi
impersonate) and Noble (noble_tls Client) backends plus the TLSBackend
protocol for custom clients. tls clients are optional extras
([curl]/[noble]/[all]) with guarded imports; all aioweb features (domain/
header/ephemeral/proxy/retry/preview) inherited unchanged. src/ multi-module
layout, hatchling.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-24 21:36:43 -04:00
7 changed files with 643 additions and 16 deletions
+1 -1
View File
@@ -1,5 +1,5 @@
# claude
CLAUDE.md
.claude/
# python
__pycache__/
+82 -15
View File
@@ -22,17 +22,17 @@ you want; importing the package never fails because an extra is missing.
`requirements.txt` (pick the extra you need):
```
aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.0
aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.0
aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.0
aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v1.0.0
aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v1.0.0
aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v1.0.0
```
Direct:
```bash
pip install "aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.0"
pip install "aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.0"
pip install "aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.0"
pip install "aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v1.0.0"
pip install "aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v1.0.0"
pip install "aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v1.0.0"
```
- `[curl]` → curl_cffi backend · `[noble]` → noble_tls backend · `[all]` → both.
@@ -44,6 +44,8 @@ pip install "aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-publi
Constructing a backend whose client isn't installed raises that `RuntimeError` at
construction, never at import.
Drop the `@v1.0.0` suffix from the line above to install the latest unpinned.
## curl_cffi backend
```python
@@ -55,9 +57,16 @@ async with TLSSession(backend=CurlCffi(impersonate="chrome"), proxies={"https":
print(resp.json()["tls"]["ja3"])
```
- `CurlCffi(impersonate="chrome")` sets the forged profile; override per call by
passing `impersonate=` to any request method.
- `CurlCffi(impersonate="chrome")` sets the forged profile; override it per call by
passing `impersonate=` to the low-level `request()` (which forwards `**kwargs` to the
backend). `request_with_retries` has a fixed signature and does **not** accept extra
backend kwargs — passing `impersonate=` there raises `TypeError`; set the profile on
the `CurlCffi` instance for the retrying path.
- curl_cffi forges JA3/JA4 + HTTP/2 fingerprints via the bundled curl-impersonate binary.
- A wire-level timeout raises `aiohttp.ServerTimeoutError` (matching aioweb's own
contract), not a generic `aiohttp.ClientError` — both backends detect their
native timeout (curl_cffi's `Timeout` type, noble_tls's Go-side timeout text) and
re-wrap it before the fallback client-error path (v0.1.5).
## noble backend
@@ -71,10 +80,20 @@ async with TLSSession(backend=Noble(client="chrome_133")) as s:
print(resp.json()["tls"]["ja3"])
```
- `Noble(client="chrome_133")` — accepts a `noble_tls.Client` enum or a string name.
- `Noble(client="chrome_133")` — accepts a `noble_tls.Client` enum or a string name;
an unknown string raises `ValueError` listing the valid profile names (v0.1.5).
- noble_tls downloads a Go shared library on first use. `await s.setup()` fetches it
once at startup; if you skip it, the first request fetches it lazily (guarded to run
once).
once at startup; if you skip it, the first request fetches it lazily. The fetch is
guarded by a lock, so even concurrent first requests download it exactly once.
- Binary bodies (images, zips, PDFs, protobuf) round-trip as true bytes: noble_tls
returns response bodies as a UTF-8 JSON string by default, which mangles non-UTF-8
bytes (`U+FFFD` replacement, wrong length) even on a 200 response — the Noble
backend always requests `is_byte_response=True` and decodes the resulting
base64 data-URI back into raw bytes, so `resp.content` is never lossy.
- Multi-valued response headers (e.g. two `Set-Cookie` lines) arrive from noble_tls's
Go side as a Python list — the Noble backend flattens them to a single
comma-joined string per RFC 7230, so `resp.headers[...]` is always a plain string
(v0.1.5).
## Writing your own backend (the `TLSBackend` protocol)
@@ -94,7 +113,10 @@ for the authoritative contract):
| `raw_request` | **required** | `async (session, method, url, **kwargs) -> aioweb.Response` | send one request; adapt the client's response into an `aioweb.Response` |
| `is_closed` | **required** | `(session) -> bool` | whether the session is closed |
| `cookies_for_url` | optional | `(session, url) -> dict` | cookies for `preview()`; defaults to `{}` |
| `setup` | optional | `async () -> None` | one-time prep (e.g. fetch a native lib); idempotent |
| `set_cookie` | optional | `(session, name, value, domain=None, path="/") -> None` | backs `TLSSession.set_cookie()`; raises `NotImplementedError` if absent |
| `get_cookies` | optional | `(session) -> dict` | backs `TLSSession.get_cookies()`; raises `NotImplementedError` if absent |
| `clear_cookies` | optional | `(session) -> None` | backs `TLSSession.clear_cookies()`; raises `NotImplementedError` if absent |
| `setup` | optional | `async () -> None` | one-time prep (e.g. fetch a native lib); idempotent. `TLSSession` never calls this automatically - a backend needing lazy setup must self-invoke it from its own `raw_request`, as `Noble` does, or the caller must run `await session.setup()` explicitly |
| `close` | optional | `async (session) -> None` | close the session; defaults to `await session.close()` |
`raw_request` receives aioweb-shaped kwargs: the proxy is already resolved into
@@ -151,8 +173,9 @@ async with TLSSession(backend=GoTLSBackend("http://localhost:8080")) as s:
## Inherited features work unchanged
aioweb's overwrite/domain/ephemeral/proxy/retry/preview logic operates on plain dicts
and never touches the HTTP backend — only the seams do. Every aioweb feature behaves
identically on any backend:
and never touches the HTTP backend — only the seams do. Header overwrites, domain
rewriting, ephemeral headers, proxies, retries, and previews behave identically on
any backend:
```python
async with TLSSession(backend=CurlCffi(impersonate="chrome")) as s:
@@ -162,12 +185,56 @@ async with TLSSession(backend=CurlCffi(impersonate="chrome")) as s:
print(s.preview("GET", "https://internal.local/x").as_curl()) # reflects all of the above
```
Session-default headers are never baked into the underlying client (neither
`CurlCffi` nor `Noble` passes `headers=` to their client's constructor) — they flow
through aioweb's own per-request `_default_headers` merge instead. That keeps
`update_headers()` / `clear_headers()` accurate for both backends: what
`get_headers()` and `preview()` report is what actually goes out on the wire, with
no stale, already-baked value resurfacing after a clear.
The mutable cookie API — `set_cookie()` / `get_cookies()` / `clear_cookies()` — is
also backend-aware: `CurlCffi` and `Noble` each route it through their own client's
cookie store (both expose a `requests`-style `session.cookies` with `set()` /
`items()` / `clear()`), so these calls work the same way they do on the base
`aioweb.ExtendedSession`, not just `_cookies_for_url()` (used by `preview()`).
`resp.history` / `resp.redirect_chain` (`[(status, url), ...]`) and `resp.is_redirect`
are threaded through from whatever redirect history the backend exposes. **Caveat — the
CurlCffi backend has no per-hop history:** `curl_cffi` follows redirects internally in the
native curl layer and surfaces only the final URL/status, leaving `Response.history` an
empty list (it never populates it). So on `CurlCffi`, `resp.history`/`resp.redirect_chain`
are `[]` and `resp.is_redirect` reflects only the final response, even after a redirect —
this is a `curl_cffi` limitation, not a bug here, and it differs from the base aiohttp
`ExtendedSession` (which does record the hops). The `Noble` backend threads whatever
`noble_tls` exposes as its per-response history. If you need the redirect chain, use the
base backend or read the final URL.
## Honesty note
TLS fingerprinting changes one layer — the TLS/HTTP fingerprint. It does **not** by
itself defeat modern bot protection: behavioral analysis, captchas, and JS challenges
are separate signals. Use this as one component, not a complete anti-bot solution.
## Changelog
### v0.1.8
- Compressed 4 residual internal-helper docstrings (`_is_timeout_error`,
`_noble_timeout_seconds`, `_noble_content`, `_jar_to_dict`) to one-liners.
Cosmetic, zero behavior change.
### v0.1.7
- **Backends thread whatever redirect history the client exposes.** `CurlCffi.raw_request`
and `Noble.raw_request` built their `Response` without `history=`, so
`resp.history`/`resp.redirect_chain`/`resp.is_redirect`-after-follow were always empty
and aioweb's own `debug=True` "redirect chain:" log line was dead on both TLS backends.
Each backend now maps its client's history into aioweb's `(status, url)` tuple shape.
**Caveat:** `curl_cffi` never populates `Response.history` (it follows redirects in the
native curl layer and surfaces only the final URL/status), so on the `CurlCffi` backend
`resp.history` is `[]` even after a redirect — a `curl_cffi` limitation, not addressable
here. `Noble` passes through whatever `noble_tls` records.
## Versioning
Tagged `vX.Y.Z`. Pin the tag in `requirements.txt`.
Releases are tagged `vX.Y.Z`. The install line above pins a release; drop the `@vX.Y.Z` suffix to install the latest unpinned. Pin deliberately for reproducible installs.
+23
View File
@@ -0,0 +1,23 @@
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "aioweb_tls"
version = "1.0.0"
description = "TLS-fingerprinting backends (curl_cffi / noble_tls) for aioweb via one injectable TLSSession, config-free, installable."
requires-python = ">=3.10"
dependencies = [
"aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.10",
]
[project.optional-dependencies]
curl = ["curl_cffi>=0.15"]
noble = ["noble-tls>=0.1.5"]
all = ["curl_cffi>=0.15", "noble-tls>=0.1.5"]
[tool.hatch.build.targets.wheel]
packages = ["src/aioweb_tls"]
[tool.hatch.metadata]
allow-direct-references = true
+21
View File
@@ -0,0 +1,21 @@
"""
tls-fingerprinting backends for aioweb - see README for usage and the extras contract
from aioweb_tls import TLSSession, CurlCffi, Noble
async with TLSSession(backend=CurlCffi(impersonate="chrome")) as s: # [curl] extra
resp = await s.request_with_retries("GET", url)
"""
from importlib.metadata import version, PackageNotFoundError
from .session import TLSSession
from .backends import CurlCffi, Noble
from .protocol import TLSBackend
try:
__version__ = version("aioweb_tls")
except PackageNotFoundError:
__version__ = "0.0.0+unknown"
__all__ = ["TLSSession", "CurlCffi", "Noble", "TLSBackend"]
+372
View File
@@ -0,0 +1,372 @@
"""
tls backends for TLSSession - stateless config+behavior objects implementing the
TLSBackend protocol (see protocol.py); see README for the extras contract
"""
import asyncio
import base64
import logging
import math
import aiohttp
from multidict import CIMultiDict
from aioweb import Response
log = logging.getLogger(__name__)
def _as_client_error(error: Exception, backend: str) -> aiohttp.ClientError:
"""wrap a backend-native network exception (neither is an aiohttp.ClientError) as one"""
return aiohttp.ClientError(f"{backend} request failed: {error}")
_TIMEOUT_TEXT = ("timeout", "timed out", "deadline exceeded")
def _is_timeout_error(error: Exception) -> bool:
"""whether a backend-native OSError is a timeout, by type name (curl_cffi) or Go error text (noble_tls)"""
timeout_type = getattr(error, "__class__", None)
if timeout_type is not None and any(base.__name__ == "Timeout" for base in timeout_type.__mro__):
return True
return any(needle in str(error).lower() for needle in _TIMEOUT_TEXT)
def _as_timeout_error(error: Exception, backend: str) -> aiohttp.ServerTimeoutError:
"""wrap a backend-native timeout as aiohttp.ServerTimeoutError, matching aioweb's own contract"""
return aiohttp.ServerTimeoutError(f"{backend} request timed out: {error}")
try:
from curl_cffi import AsyncSession as _CurlAsyncSession
_CURL_ERROR = None
except ImportError as error:
_CurlAsyncSession = None
_CURL_ERROR = error
try:
import noble_tls
from noble_tls import Client as _NobleClient
_NOBLE_ERROR = None
except ImportError as error:
noble_tls = None
_NobleClient = None
_NOBLE_ERROR = error
def _coerce_timeout(value):
"""unwrap aioweb's aiohttp.ClientTimeout (or a plain number) to a bare number for the tls clients"""
total = getattr(value, "total", value)
return total if isinstance(total, (int, float)) else None
def _noble_timeout_seconds(value):
"""coerce a raw or wrapped timeout into whole seconds (min 1), rounding up so noble's Go int
field never truncates to 0/fails unmarshal; None if uncoercible; shared by both Noble timeout paths"""
timeout = _coerce_timeout(value)
if timeout is None:
return None
return max(1, math.ceil(timeout))
def _noble_content(response) -> bytes:
"""extract true response bytes from a noble_tls response fetched with is_byte_response, decoding
the base64 data-URI in response.text (falls back to a plain utf-8 encode if not a data-URI)"""
text = getattr(response, "text", "") or ""
if text.startswith("data:") and ";base64," in text:
_, _, payload = text.partition(";base64,")
return base64.b64decode(payload)
return text.encode()
def _flatten_headers(headers) -> CIMultiDict:
"""flatten a Go-style map[str][]str header dict (multi-valued headers as a list)
into plain str values, joined with ", " per RFC 7230, into a case-insensitive
mapping matching aioweb's aiohttp-backed Response.headers"""
return CIMultiDict(
(key, ", ".join(value) if isinstance(value, list) else value)
for key, value in headers.items()
)
def _history_entries(history) -> list:
"""map a backend-native redirect history into aioweb's `Response(history=...)` shape
aioweb's own `_raw_request` threads `[(status, url), ...]`; curl_cffi's
`Response.history` is `list[dict]` (dict-shaped hop records) and noble_tls's is
`list[Response]` (object-shaped, same accessors as the top-level response) - this
reads a hop's status/url either way. an unparseable hop is skipped rather than
raising, so a redirect-history quirk never breaks the response it's attached to.
"""
entries = []
for hop in history or []:
if isinstance(hop, dict):
status = hop.get("status_code", hop.get("status"))
url = hop.get("url")
else:
status = getattr(hop, "status_code", getattr(hop, "status", None))
url = getattr(hop, "url", None)
if status is None or url is None:
continue
entries.append((status, str(url)))
return entries
def _jar_to_dict(session):
"""best-effort map of a requests-style cookie jar on session to a plain dict, feeding preview()
only; a jar that fails to iterate degrades to {} rather than raising"""
jar = getattr(session, "cookies", None)
if not jar:
return {}
try:
# prefer get_dict() where the jar exposes it: items() raises curl_cffi CookieConflict
# when the same name lives on two domains, get_dict() flattens instead (mirrors get_cookies)
if hasattr(jar, "get_dict"):
return jar.get_dict()
return {k: v for k, v in jar.items()}
except Exception:
return {}
class CurlCffi:
"""curl_cffi backend: forges JA3/JA4 + HTTP/2 via the curl-impersonate binary
config:
impersonate: browser profile to forge (default "chrome"); override per call
via request()/_raw_request (forwards **kwargs) - NOT request_with_retries,
whose fixed signature raises TypeError on it. for the retrying path, set
the profile on the CurlCffi instance instead.
requires the [curl] extra (pip install "aioweb_tls[curl]").
"""
def __init__(self, impersonate: str = "chrome"):
if _CurlAsyncSession is None:
raise RuntimeError(
"curl_cffi is not installed; install the extra with "
"pip install \"aioweb_tls[curl]\""
) from _CURL_ERROR
self.impersonate = impersonate
def create_session(self, headers, timeout, **kwargs):
"""build the curl_cffi AsyncSession
deliberately does NOT pass `headers`: curl_cffi bakes a constructor
`headers=` into the client and re-merges it under per-request headers,
which would desync update_headers()/clear_headers() from what's actually on
the wire - aioweb's session-default headers already apply per request via
the base's _default_headers merge (see ExtendedSession._create_session).
"""
return _CurlAsyncSession(timeout=timeout, **kwargs)
async def raw_request(self, session, method, url, **kwargs) -> Response:
"""send via curl_cffi and adapt the result into an aioweb.Response"""
impersonate = kwargs.pop("impersonate", self.impersonate)
timeout = _coerce_timeout(kwargs.pop("timeout", None))
if timeout is not None:
kwargs["timeout"] = timeout
proxy = kwargs.pop("proxy", None)
if proxy:
kwargs["proxy"] = proxy
try:
response = await session.request(method, url, impersonate=impersonate, **kwargs)
except aiohttp.ClientError:
raise
except asyncio.TimeoutError:
raise
except OSError as error:
if _is_timeout_error(error):
raise _as_timeout_error(error, "curl_cffi") from error
raise _as_client_error(error, "curl_cffi") from error
content = response.content if response.content is not None else b""
return Response(
status_code=response.status_code,
headers=CIMultiDict(response.headers.multi_items()),
content=content,
url=str(response.url),
reason=getattr(response, "reason", None),
history=_history_entries(getattr(response, "history", None)),
cookies=getattr(response, "cookies", None),
)
def is_closed(self, session) -> bool:
"""whether the curl_cffi session is closed
curl_cffi tracks this in the private `_closed` (no public `closed` today);
falls back to a public `closed` if a future version adds one. best-effort -
TLSSession's own `_closed` flag is the primary signal.
"""
closed = getattr(session, "_closed", None)
if closed is None:
closed = getattr(session, "closed", False)
return bool(closed)
def cookies_for_url(self, session, url) -> dict:
"""cookies curl_cffi would send for url (best-effort)"""
return _jar_to_dict(session)
def set_cookie(self, session, name, value, domain=None, path="/") -> None:
"""set a cookie in curl_cffi's own cookie store"""
session.cookies.set(name, value, domain=domain or "", path=path)
def get_cookies(self, session) -> dict:
"""all cookies stored in curl_cffi's cookie store
uses get_dict() rather than dict(cookies.items()): items() raises curl_cffi
CookieConflict when the same name exists on two domains, get_dict() flattens
(last value wins) without raising.
"""
return session.cookies.get_dict()
def clear_cookies(self, session) -> None:
"""clear curl_cffi's cookie store"""
session.cookies.clear()
async def close(self, session) -> None:
"""close the curl_cffi session"""
await session.close()
class Noble:
"""noble_tls backend: forges a browser TLS fingerprint via the tls-client Go lib
config:
client: noble_tls Client profile (enum or string, default "chrome_133").
downloads a Go shared library on first use; setup() fetches it once (via
TLSSession.setup() or lazily before the first request).
requires the [noble] extra (pip install "aioweb_tls[noble]").
"""
def __init__(self, client="chrome_133"):
if noble_tls is None:
raise RuntimeError(
"noble_tls is not installed; install the extra with "
"pip install \"aioweb_tls[noble]\""
) from _NOBLE_ERROR
self.client = self._resolve_client(client)
self._updated = False
self._setup_lock = asyncio.Lock()
@staticmethod
def _resolve_client(client):
"""turn a string or Client enum into a noble_tls Client value, raising ValueError
naming the valid profiles for an unknown string"""
if not isinstance(client, str):
return client
name = client.upper()
resolved = getattr(_NobleClient, name, None)
if resolved is None:
valid = ", ".join(member.name for member in _NobleClient)
raise ValueError(f"unknown noble_tls client profile {client!r}; valid profiles: {valid}")
return resolved
async def setup(self) -> None:
"""fetch the noble_tls Go shared library once; idempotent and concurrency-safe
uses download_if_necessary (falls back to update_if_necessary on older
noble_tls). guarded by an asyncio.Lock with a check-lock-recheck so concurrent
first requests don't both run the fetch.
"""
if self._updated:
return
async with self._setup_lock:
if self._updated:
return
download = getattr(noble_tls, "download_if_necessary", None)
if download is not None:
await download()
elif hasattr(noble_tls, "update_if_necessary"):
await noble_tls.update_if_necessary()
self._updated = True
def create_session(self, headers, timeout, **kwargs):
"""build the noble_tls Session, honoring the session-default timeout
deliberately does NOT bake `headers` into session.headers - same header-baking
rationale as CurlCffi.create_session (see its docstring).
the coerced timeout IS applied here (matching raw_request's per-call path):
noble's Go field timeoutSeconds is an int, so an uncoerced sub-second/float
session-default would fail Go-side JSON unmarshal on any request that doesn't
override it per call; max(1, ceil()) mirrors raw_request's own guard.
"""
session = noble_tls.Session(client=self.client, **kwargs)
timeout_seconds = _noble_timeout_seconds(timeout)
if timeout_seconds is not None:
session.timeout_seconds = timeout_seconds
return session
async def raw_request(self, session, method, url, **kwargs) -> Response:
"""send via noble_tls and adapt the result into an aioweb.Response"""
await self.setup()
timeout_seconds = _noble_timeout_seconds(kwargs.pop("timeout", None))
if timeout_seconds is not None:
kwargs["timeout_seconds"] = timeout_seconds
proxy = kwargs.pop("proxy", None)
if proxy:
kwargs["proxy"] = proxy
kwargs.setdefault("is_byte_response", True)
try:
response = await session.execute_request(method=method.upper(), url=url, **kwargs)
except aiohttp.ClientError:
raise
except asyncio.TimeoutError:
raise
except OSError as error:
if _is_timeout_error(error):
raise _as_timeout_error(error, "noble_tls") from error
raise _as_client_error(error, "noble_tls") from error
content = _noble_content(response)
return Response(
status_code=response.status_code,
headers=_flatten_headers(getattr(response, "headers", {}) or {}),
content=content,
url=str(getattr(response, "url", url)),
reason=getattr(response, "reason", None),
history=_history_entries(getattr(response, "history", None)),
cookies=getattr(response, "cookies", None),
)
def is_closed(self, session) -> bool:
"""noble_tls has no closed flag; closed state is tracked by TLSSession"""
return bool(getattr(session, "closed", False))
def cookies_for_url(self, session, url) -> dict:
"""cookies noble_tls would send for url (best-effort)"""
return _jar_to_dict(session)
def set_cookie(self, session, name, value, domain=None, path="/") -> None:
"""set a cookie in noble_tls's own cookie jar"""
session.cookies.set(name, value, domain=domain or "", path=path)
def get_cookies(self, session) -> dict:
"""all cookies stored in noble_tls's cookie jar
prefers get_dict() when the jar exposes it (flattens cross-domain duplicate
names without raising, like curl_cffi); falls back to items() otherwise
"""
jar = session.cookies
if hasattr(jar, "get_dict"):
return jar.get_dict()
return dict(jar.items())
def clear_cookies(self, session) -> None:
"""clear noble_tls's cookie jar"""
session.cookies.clear()
async def close(self, session) -> None:
"""close the noble_tls session if it exposes a close"""
close = getattr(session, "close", None)
if close is None:
return
result = close()
if hasattr(result, "__await__"):
await result
+22
View File
@@ -0,0 +1,22 @@
"""the tls backend protocol - see TLSBackend below and README"""
from typing import Any, Protocol, runtime_checkable
from aioweb import Response
@runtime_checkable
class TLSBackend(Protocol):
"""the contract a TLSSession backend implements (see module docstring)"""
def create_session(self, headers: dict, timeout: Any, **kwargs) -> Any:
"""build and return the live client session object"""
...
async def raw_request(self, session: Any, method: str, url: str, **kwargs) -> Response:
"""send one request and adapt the result into an aioweb.Response"""
...
def is_closed(self, session: Any) -> bool:
"""whether the session is closed"""
...
+122
View File
@@ -0,0 +1,122 @@
"""
TLSSession - one aioweb session, any tls backend
subclasses aioweb.ExtendedSession, delegating only the four backend seams to an
injected backend object (see protocol.py); everything else is inherited unchanged.
see README for usage.
from aioweb_tls import TLSSession, CurlCffi
async with TLSSession(backend=CurlCffi(impersonate="chrome")) as s:
resp = await s.request_with_retries("GET", "https://tls.peet.ws/api/all")
if resp:
print(resp.json()["tls"]["ja3"])
"""
import logging
from aioweb import ExtendedSession
log = logging.getLogger(__name__)
class TLSSession(ExtendedSession):
"""aioweb session whose HTTP backend is an injected, swappable tls client"""
def __init__(self, backend, *args, **kwargs):
"""build a session over a tls backend
args:
backend: a TLSBackend (CurlCffi, Noble, or a custom one); owns its own
config vocabulary and the four seam behaviors
*args, **kwargs: forwarded to aioweb.ExtendedSession
"""
self.backend = backend
self._closed = False
super().__init__(*args, **kwargs)
async def setup(self) -> None:
"""run the backend's one-time setup if it has one (idempotent, e.g. noble's Go lib fetch)
optional to call upfront to pre-warm at startup. TLSSession itself never calls
this automatically - a backend needing lazy setup must self-invoke it from its
own raw_request, as Noble does; a custom backend that skips this will never run
setup() unless the caller calls TLSSession.setup() explicitly.
"""
setup = getattr(self.backend, "setup", None)
if setup is not None:
await setup()
# -------------------------------------------------------------------------
# the four backend seams, delegated to self.backend
def _create_session(self, headers, timeout, **kwargs):
"""delegate session creation to the backend"""
return self.backend.create_session(headers, timeout, **kwargs)
async def _raw_request(self, method, url, **kwargs):
"""delegate the wire request to the backend, returning its aioweb.Response"""
return await self.backend.raw_request(self.session, method, url, **kwargs)
def _cookies_for_url(self, url) -> dict:
"""delegate preview cookie lookup to the backend (default {})"""
cookies_for_url = getattr(self.backend, "cookies_for_url", None)
if cookies_for_url is None:
return {}
return cookies_for_url(self.session, url)
def _is_closed(self) -> bool:
"""closed if explicitly closed here, never built, or the backend reports it
an unbuilt session counts as closed: nothing was opened, nothing to leak.
"""
if self._closed:
return True
if self._session is None:
return True
return self.backend.is_closed(self.session)
# -------------------------------------------------------------------------
# mutable cookie api - base reaches into aiohttp-only session.cookie_jar, so
# TLS backends override these to route through the backend instead
def set_cookie(self, name, value, domain=None, path="/"):
"""set a cookie via the backend's own cookie store"""
set_cookie = getattr(self.backend, "set_cookie", None)
if set_cookie is None:
raise NotImplementedError(
f"{type(self.backend).__name__} does not support the mutable cookie api"
)
set_cookie(self.session, name, value, domain=domain, path=path)
def get_cookies(self) -> dict:
"""all cookies stored in the backend's cookie store"""
get_cookies = getattr(self.backend, "get_cookies", None)
if get_cookies is None:
raise NotImplementedError(
f"{type(self.backend).__name__} does not support the mutable cookie api"
)
return get_cookies(self.session)
def clear_cookies(self) -> None:
"""clear the backend's cookie store"""
clear_cookies = getattr(self.backend, "clear_cookies", None)
if clear_cookies is None:
raise NotImplementedError(
f"{type(self.backend).__name__} does not support the mutable cookie api"
)
clear_cookies(self.session)
# -------------------------------------------------------------------------
# lifecycle - backends close differently, so route through the backend
async def close(self) -> None:
"""close via the backend's close (falls back to session.close); no-op if never built"""
self._closed = True
if self._session is None:
return
close = getattr(self.backend, "close", None)
if close is not None:
await close(self._session)
else:
await self._session.close()