13 Commits
Author SHA1 Message Date
dsql ce240b0757 fix: label TLS-backend timeouts ServerTimeoutError, flatten noble multi headers, ValueError on unknown profile (v0.1.5)
Both backends' asyncio.TimeoutError except-branch was dead code (neither curl_cffi
nor noble_tls raises it), so a real wire timeout fell through to the generic OSError
branch and got mislabeled a plain aiohttp.ClientError instead of aioweb's own
ServerTimeoutError contract; now detected via curl_cffi's Timeout type or Go-side
timeout text and re-wrapped correctly. Noble's multi-valued response headers (e.g.
two Set-Cookie lines) arrived as Python lists instead of strings, breaking any
downstream .split()/.lower() call; now comma-joined per RFC 7230. An unknown Noble
client profile string raised a raw AttributeError from the enum lookup; now a
ValueError listing the valid profile names. Also compresses essay-length docstrings
and narrating comments across the module with no behavior change.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 23:28:42 -04:00
dsql eb7745ae9b fix: seam mutable cookie api, byte-safe noble bodies, drop header baking, coerce noble session timeout (v0.1.4)
TLSSession.set_cookie/get_cookies/clear_cookies crashed with AttributeError on
both backends (they reached into self.session.cookie_jar, which curl_cffi and
noble_tls sessions don't have); both backends now route the mutable cookie api
through their own requests-style session.cookies store.

Noble.raw_request now requests is_byte_response=True and decodes the resulting
base64 data-URI body, since noble_tls's default text response silently corrupts
any binary payload (image/zip/pdf) via lossy UTF-8 decoding on the Go side.

CurlCffi/Noble.create_session no longer bake session-default headers into the
underlying client; baking caused clear_headers()/get_headers() to lie about
what's actually still on the wire (a credential-leak divergence from the
aiohttp base, which never bakes). Headers flow through aioweb's per-request
merge only, matching the base's documented contract.

Noble.create_session now applies the same max(1, ceil()) timeout coercion
raw_request already had (extracted into a shared _noble_timeout_seconds
helper) — without it, a sub-second/float session-default timeout made every
request fail Go-side JSON unmarshal.

README corrected: dropped the 'every aioweb feature behaves identically'
overclaim re: cookies, documented the binary-body handling and the
no-header-baking rationale, bumped install pins to v0.1.4.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 17:09:11 -04:00
dsql ce0762a37c chore: ignore .claude/ dir (CLAUDE.md now lives under .claude/)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:55:13 -04:00
dsql ef9eb3010c fix: bump aioweb pin to v0.1.5 (seam-checked); narrow backend except
bump the aioweb dependency from the stale v0.1.0 to current v0.1.5 — seam-verified against
v0.1.5's actual API (request_with_retries -> Response on success, falsy FailureResponse on
failure, all four override seams present). backend raw_request catches narrowed from bare
Exception to OSError (covers curl_cffi RequestException / noble TLSClientException) and
re-raises ClientError/TimeoutError first, so a real bug isn't laundered into 'client error'.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:35:24 -04:00
dsql da8b0bf6f8 docs: pin install line to release, note unpinned-latest option
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 18:13:36 -04:00
dsql 612861b76c docs: show unpinned install line; note tag-pinning for reproducibility
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 18:07:21 -04:00
dsql ccea880df0 fix: wrap backend-native exceptions; correct setup/desc docs (v0.1.3)
- CurlCffi/Noble raw_request translate backend-native network errors (curl_cffi
  RequestException, noble_tls TLSClientException) into aiohttp.ClientError so the bare
  request() path gives the same typed-failure contract as the aiohttp backend (L6)
- Noble.setup uses download_if_necessary (the current noble_tls API), with
  update_if_necessary only as a fallback; docstring/CLAUDE.md no longer claim the dead
  'refreshes an existing one' path (L7)
- pyproject description says composition (one injectable TLSSession), not the old
  'ExtendedSession subclasses' (L8).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 17:57:54 -04:00
dsql 5eb689ba73 docs: narrow the per-call impersonate= claim (v0.1.2)
per-call impersonate= is honored only via the low-level request()/_raw_request path
(which forwards **kwargs to the backend), NOT request_with_retries — its inherited
aioweb signature is fixed with no **kwargs and raises TypeError on an extra kwarg.
docs-only across README + backend docstring + CLAUDE.md; for the retrying path, set
the profile on the CurlCffi/Noble instance. no code change.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 17:11:57 -04:00
dsql 87debe8465 fix: is_closed reads curl_cffi's private _closed (no public closed attr)
CurlCffi.is_closed read getattr(session, 'closed', False), but curl_cffi tracks closed state only in the private _closed and exposes no public 'closed' property, so it always returned False. it now reads _closed, falling back to a public 'closed' if a future version adds one. TLSSession's own flag remains the primary signal; this is the best-effort backend check for out-of-band closes.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 01:10:25 -04:00
dsql b2876d005e fix: Noble honors session-default headers and timeout like CurlCffi
noble_tls.Session takes neither headers nor timeout in its constructor, and Noble.create_session forwarded only client+kwargs, so TLSSession(backend=Noble(...), headers=...) silently dropped the headers while CurlCffi passed them through. create_session now applies headers via session.headers.update and sets timeout_seconds after construction. verified against the contract with a stubbed noble_tls; the real Go-lib + a live request remain an untested gap (noble_tls not installable in this env).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-28 18:45:25 -04:00
dsql ae4c653ecc fix: lock Noble.setup so concurrent first requests fetch the Go-lib once
Noble.setup guarded the one-time Go shared-library fetch with a bare 'if self._updated' flag — a TOCTOU race where concurrent first requests both passed the check before either set the flag, running the download multiple times. now guarded by a per-instance asyncio.Lock with a check-lock-recheck. verified under load: 2/10/100/500 concurrent setups run the fetch exactly once each (a no-lock control runs it N times).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-28 17:46:20 -04:00
dsql 7ea8ecf888 fix: don't truncate sub-second timeout to 0 in noble backend (v0.1.1)
int(timeout) truncated a fractional timeout (e.g. 0.5s) to 0, which noble treats as
no/instant timeout. round up with math.ceil and floor at 1 so a sub-second timeout
stays a real (>=1s) timeout.

verified: 0.5/0.1/0.001 -> 1 (was 0); whole seconds unchanged.
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-28 15:49:30 -04:00
dsql befa4cd196 add package: pyproject + src
TLSSession over aioweb's backend seam by composition: one session class
delegates the four seams to an injected backend. ships CurlCffi (curl_cffi
impersonate) and Noble (noble_tls Client) backends plus the TLSBackend
protocol for custom clients. tls clients are optional extras
([curl]/[noble]/[all]) with guarded imports; all aioweb features (domain/
header/ephemeral/proxy/retry/preview) inherited unchanged. src/ multi-module
layout, hatchling.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-24 21:36:43 -04:00
7 changed files with 688 additions and 15 deletions
+1 -1
View File
@@ -1,5 +1,5 @@
# claude
CLAUDE.md
.claude/
# python
__pycache__/
+50 -14
View File
@@ -22,17 +22,17 @@ you want; importing the package never fails because an extra is missing.
`requirements.txt` (pick the extra you need):
```
aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.0
aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.0
aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.0
aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.5
aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.5
aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.5
```
Direct:
```bash
pip install "aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.0"
pip install "aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.0"
pip install "aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.0"
pip install "aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.5"
pip install "aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.5"
pip install "aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.5"
```
- `[curl]` → curl_cffi backend · `[noble]` → noble_tls backend · `[all]` → both.
@@ -44,6 +44,8 @@ pip install "aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-publi
Constructing a backend whose client isn't installed raises that `RuntimeError` at
construction, never at import.
Drop the `@v0.1.5` suffix from the line above to install the latest unpinned.
## curl_cffi backend
```python
@@ -55,9 +57,16 @@ async with TLSSession(backend=CurlCffi(impersonate="chrome"), proxies={"https":
print(resp.json()["tls"]["ja3"])
```
- `CurlCffi(impersonate="chrome")` sets the forged profile; override per call by
passing `impersonate=` to any request method.
- `CurlCffi(impersonate="chrome")` sets the forged profile; override it per call by
passing `impersonate=` to the low-level `request()` (which forwards `**kwargs` to the
backend). `request_with_retries` has a fixed signature and does **not** accept extra
backend kwargs — passing `impersonate=` there raises `TypeError`; set the profile on
the `CurlCffi` instance for the retrying path.
- curl_cffi forges JA3/JA4 + HTTP/2 fingerprints via the bundled curl-impersonate binary.
- A wire-level timeout raises `aiohttp.ServerTimeoutError` (matching aioweb's own
contract), not a generic `aiohttp.ClientError` — both backends detect their
native timeout (curl_cffi's `Timeout` type, noble_tls's Go-side timeout text) and
re-wrap it before the fallback client-error path (v0.1.5).
## noble backend
@@ -71,10 +80,20 @@ async with TLSSession(backend=Noble(client="chrome_133")) as s:
print(resp.json()["tls"]["ja3"])
```
- `Noble(client="chrome_133")` — accepts a `noble_tls.Client` enum or a string name.
- `Noble(client="chrome_133")` — accepts a `noble_tls.Client` enum or a string name;
an unknown string raises `ValueError` listing the valid profile names (v0.1.5).
- noble_tls downloads a Go shared library on first use. `await s.setup()` fetches it
once at startup; if you skip it, the first request fetches it lazily (guarded to run
once).
once at startup; if you skip it, the first request fetches it lazily. The fetch is
guarded by a lock, so even concurrent first requests download it exactly once.
- Binary bodies (images, zips, PDFs, protobuf) round-trip as true bytes: noble_tls
returns response bodies as a UTF-8 JSON string by default, which mangles non-UTF-8
bytes (`U+FFFD` replacement, wrong length) even on a 200 response — the Noble
backend always requests `is_byte_response=True` and decodes the resulting
base64 data-URI back into raw bytes, so `resp.content` is never lossy.
- Multi-valued response headers (e.g. two `Set-Cookie` lines) arrive from noble_tls's
Go side as a Python list — the Noble backend flattens them to a single
comma-joined string per RFC 7230, so `resp.headers[...]` is always a plain string
(v0.1.5).
## Writing your own backend (the `TLSBackend` protocol)
@@ -94,6 +113,9 @@ for the authoritative contract):
| `raw_request` | **required** | `async (session, method, url, **kwargs) -> aioweb.Response` | send one request; adapt the client's response into an `aioweb.Response` |
| `is_closed` | **required** | `(session) -> bool` | whether the session is closed |
| `cookies_for_url` | optional | `(session, url) -> dict` | cookies for `preview()`; defaults to `{}` |
| `set_cookie` | optional | `(session, name, value, domain=None, path="/") -> None` | backs `TLSSession.set_cookie()`; raises `NotImplementedError` if absent |
| `get_cookies` | optional | `(session) -> dict` | backs `TLSSession.get_cookies()`; raises `NotImplementedError` if absent |
| `clear_cookies` | optional | `(session) -> None` | backs `TLSSession.clear_cookies()`; raises `NotImplementedError` if absent |
| `setup` | optional | `async () -> None` | one-time prep (e.g. fetch a native lib); idempotent |
| `close` | optional | `async (session) -> None` | close the session; defaults to `await session.close()` |
@@ -151,8 +173,9 @@ async with TLSSession(backend=GoTLSBackend("http://localhost:8080")) as s:
## Inherited features work unchanged
aioweb's overwrite/domain/ephemeral/proxy/retry/preview logic operates on plain dicts
and never touches the HTTP backend — only the seams do. Every aioweb feature behaves
identically on any backend:
and never touches the HTTP backend — only the seams do. Header overwrites, domain
rewriting, ephemeral headers, proxies, retries, and previews behave identically on
any backend:
```python
async with TLSSession(backend=CurlCffi(impersonate="chrome")) as s:
@@ -162,6 +185,19 @@ async with TLSSession(backend=CurlCffi(impersonate="chrome")) as s:
print(s.preview("GET", "https://internal.local/x").as_curl()) # reflects all of the above
```
Session-default headers are never baked into the underlying client (neither
`CurlCffi` nor `Noble` passes `headers=` to their client's constructor) — they flow
through aioweb's own per-request `_default_headers` merge instead. That keeps
`update_headers()` / `clear_headers()` accurate for both backends: what
`get_headers()` and `preview()` report is what actually goes out on the wire, with
no stale, already-baked value resurfacing after a clear.
The mutable cookie API — `set_cookie()` / `get_cookies()` / `clear_cookies()` — is
also backend-aware: `CurlCffi` and `Noble` each route it through their own client's
cookie store (both expose a `requests`-style `session.cookies` with `set()` /
`items()` / `clear()`), so these calls work the same way they do on the base
`aioweb.ExtendedSession`, not just `_cookies_for_url()` (used by `preview()`).
## Honesty note
TLS fingerprinting changes one layer — the TLS/HTTP fingerprint. It does **not** by
@@ -170,4 +206,4 @@ are separate signals. Use this as one component, not a complete anti-bot solutio
## Versioning
Tagged `vX.Y.Z`. Pin the tag in `requirements.txt`.
Releases are tagged `vX.Y.Z`. The install line above pins a release; drop the `@vX.Y.Z` suffix to install the latest unpinned. Pin deliberately for reproducible installs.
+23
View File
@@ -0,0 +1,23 @@
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "aioweb_tls"
version = "0.1.5"
description = "TLS-fingerprinting backends (curl_cffi / noble_tls) for aioweb via one injectable TLSSession, config-free, installable."
requires-python = ">=3.10"
dependencies = [
"aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.5",
]
[project.optional-dependencies]
curl = ["curl_cffi>=0.15"]
noble = ["noble-tls>=0.1.5"]
all = ["curl_cffi>=0.15", "noble-tls>=0.1.5"]
[tool.hatch.build.targets.wheel]
packages = ["src/aioweb_tls"]
[tool.hatch.metadata]
allow-direct-references = true
+28
View File
@@ -0,0 +1,28 @@
"""
tls-fingerprinting backends for aioweb
one session class, TLSSession, takes an injected backend that swaps the HTTP client
(and thus the TLS/HTTP fingerprint) while inheriting every aioweb feature — header
overwrites, domain rewriting, ephemeral headers, proxies, retries, previews —
unchanged.
from aioweb_tls import TLSSession, CurlCffi, Noble
async with TLSSession(backend=CurlCffi(impersonate="chrome")) as s: # [curl] extra
resp = await s.request_with_retries("GET", url)
async with TLSSession(backend=Noble(client="chrome_133")) as s: # [noble] extra
await s.setup() # fetch Go lib once
resp = await s.request_with_retries("GET", url)
the tls clients are optional extras, not base deps. importing this package never
fails because an extra is missing; the matching RuntimeError is raised only when you
construct a backend whose client isn't installed. custom backends implement the
TLSBackend protocol and inject the same way.
"""
from .session import TLSSession
from .backends import CurlCffi, Noble
from .protocol import TLSBackend
__all__ = ["TLSSession", "CurlCffi", "Noble", "TLSBackend"]
+394
View File
@@ -0,0 +1,394 @@
"""
tls backends for TLSSession
each backend is a stateless config+behavior object implementing the TLSBackend
protocol (see protocol.py). it owns its own config vocabulary — CurlCffi takes
impersonate=, Noble takes client= — so there is no shared kwarg-soup. TLSSession
owns the live session object and passes it into every method here.
the underlying tls clients are optional extras: constructing a backend whose client
is not installed raises a clear RuntimeError naming the extra to install. importing
this module never fails because an extra is missing.
"""
import asyncio
import base64
import logging
import math
import aiohttp
from aioweb import Response
log = logging.getLogger(__name__)
def _as_client_error(error: Exception, backend: str) -> aiohttp.ClientError:
"""wrap a backend-native network exception as an aiohttp.ClientError
curl_cffi raises RequestException(OSError), noble_tls raises
TLSClientException(IOError) — neither is an aiohttp.ClientError, which is all
aioweb's request() re-wraps; translating here gives both the same typed contract.
"""
return aiohttp.ClientError(f"{backend} request failed: {error}")
_TIMEOUT_TEXT = ("timeout", "timed out", "deadline exceeded")
def _is_timeout_error(error: Exception) -> bool:
"""whether a backend-native network error is a timeout
neither backend raises asyncio.TimeoutError for a wire timeout: curl_cffi's own
Timeout and noble_tls's Go-side TLSClientException are both plain OSError
subclasses with no dedicated timeout type — check curl_cffi's Timeout class by
name first (precise), then fall back to matching the error text (covers
noble_tls's Go messages, e.g. "context deadline exceeded").
"""
timeout_type = getattr(error, "__class__", None)
if timeout_type is not None and any(base.__name__ == "Timeout" for base in timeout_type.__mro__):
return True
return any(needle in str(error).lower() for needle in _TIMEOUT_TEXT)
def _as_timeout_error(error: Exception, backend: str) -> aiohttp.ServerTimeoutError:
"""wrap a backend-native timeout as aiohttp.ServerTimeoutError
matches aioweb's own contract (a total timeout raises ServerTimeoutError, both a
ClientError and a TimeoutError) instead of surfacing as a generic client error.
"""
return aiohttp.ServerTimeoutError(f"{backend} request timed out: {error}")
try:
from curl_cffi import AsyncSession as _CurlAsyncSession
_CURL_ERROR = None
except ImportError as error:
_CurlAsyncSession = None
_CURL_ERROR = error
try:
import noble_tls
from noble_tls import Client as _NobleClient
_NOBLE_ERROR = None
except ImportError as error:
noble_tls = None
_NobleClient = None
_NOBLE_ERROR = error
def _coerce_timeout(value):
"""unwrap aioweb's aiohttp ClientTimeout (or a plain number) to a number
aioweb.request() wraps a numeric timeout in aiohttp.ClientTimeout before the seam
sees it; the tls clients want a bare number.
"""
total = getattr(value, "total", value)
return total if isinstance(total, (int, float)) else None
def _noble_timeout_seconds(value):
"""coerce a raw or wrapped timeout into whole seconds for noble's Go int field
single source of truth for both Noble timeout paths (session-default and
per-call): the Go field timeoutSeconds is an int, so a sub-second float truncates
to 0 (= no timeout) and a non-integer fails Go-side JSON unmarshal outright.
rounds up so a sub-second timeout still waits at least 1s. None if uncoercible.
"""
timeout = _coerce_timeout(value)
if timeout is None:
return None
return max(1, math.ceil(timeout))
def _noble_content(response) -> bytes:
"""extract true response bytes from a noble_tls response fetched with is_byte_response
with is_byte_response=True, noble_tls's Go side returns the body as a
`data:<mime>;base64,<payload>` URI string in response.text (the whole Go response
is UTF-8-decoded before JSON parsing, so raw bytes travel as base64) —
response.content just re-encodes that string, so decode the data-URI here
instead. falls back to a plain utf-8 encode if the body isn't a data-URI.
"""
text = getattr(response, "text", "") or ""
if text.startswith("data:") and ";base64," in text:
_, _, payload = text.partition(";base64,")
return base64.b64decode(payload)
return text.encode()
def _flatten_headers(headers) -> dict:
"""flatten a Go-style map[str][]str header dict into plain str values
noble_tls keeps a multi-valued header (e.g. two Set-Cookie lines) as a Python
list, breaking any downstream .split()/.lower() call; join with ", " per RFC 7230
field-value combination, leaving single values untouched.
"""
return {
key: ", ".join(value) if isinstance(value, list) else value
for key, value in headers.items()
}
def _jar_to_dict(session):
"""best-effort map of a requests-style cookie jar on session to a plain dict
feeds preview() only; the two backends expose differently-shaped jars and a
cookie read must never crash a request, so a jar that fails to iterate degrades
to {} rather than raising.
"""
jar = getattr(session, "cookies", None)
if not jar:
return {}
try:
return {k: v for k, v in jar.items()}
except Exception:
return {}
class CurlCffi:
"""curl_cffi backend: forges JA3/JA4 + HTTP/2 via the curl-impersonate binary
config:
impersonate: browser profile to forge (default "chrome"); override per call
by passing impersonate= to the low-level request()/_raw_request path
(forwards **kwargs to the backend) — NOT request_with_retries, whose
fixed signature has no **kwargs and raises TypeError. for the retrying
path, set the profile on the CurlCffi instance instead.
requires the [curl] extra (pip install "aioweb_tls[curl]").
"""
def __init__(self, impersonate: str = "chrome"):
if _CurlAsyncSession is None:
raise RuntimeError(
"curl_cffi is not installed; install the extra with "
"pip install \"aioweb_tls[curl]\""
) from _CURL_ERROR
self.impersonate = impersonate
def create_session(self, headers, timeout, **kwargs):
"""build the curl_cffi AsyncSession
deliberately does NOT pass `headers`: curl_cffi bakes a constructor
`headers=` into the client and re-merges it under per-request headers,
which would desync update_headers()/clear_headers() from what's actually on
the wire — aioweb's session-default headers already apply per request via
the base's _default_headers merge (see ExtendedSession._create_session).
"""
return _CurlAsyncSession(timeout=timeout, **kwargs)
async def raw_request(self, session, method, url, **kwargs) -> Response:
"""send via curl_cffi and adapt the result into an aioweb.Response"""
impersonate = kwargs.pop("impersonate", self.impersonate)
timeout = _coerce_timeout(kwargs.pop("timeout", None))
if timeout is not None:
kwargs["timeout"] = timeout
proxy = kwargs.pop("proxy", None)
if proxy:
kwargs["proxy"] = proxy
try:
response = await session.request(method, url, impersonate=impersonate, **kwargs)
except aiohttp.ClientError:
raise
except asyncio.TimeoutError:
raise
except OSError as error:
# RequestException (incl. curl_cffi's own Timeout) subclasses OSError, never
# asyncio.TimeoutError — check for a timeout first so it surfaces as
# ServerTimeoutError; narrowed from bare Exception so a real bug surfaces
if _is_timeout_error(error):
raise _as_timeout_error(error, "curl_cffi") from error
raise _as_client_error(error, "curl_cffi") from error
content = response.content if response.content is not None else b""
return Response(
status_code=response.status_code,
headers=dict(response.headers),
content=content,
url=str(response.url),
reason=getattr(response, "reason", None),
cookies=getattr(response, "cookies", None),
)
def is_closed(self, session) -> bool:
"""whether the curl_cffi session is closed
curl_cffi tracks closed state in the private `_closed` (no public `closed`);
fall back to a public `closed` if a future version adds one. TLSSession's own
`_closed` flag is the primary signal — this is a best-effort out-of-band check.
"""
closed = getattr(session, "_closed", None)
if closed is None:
closed = getattr(session, "closed", False)
return bool(closed)
def cookies_for_url(self, session, url) -> dict:
"""cookies curl_cffi would send for url (best-effort)"""
return _jar_to_dict(session)
def set_cookie(self, session, name, value, domain=None, path="/") -> None:
"""set a cookie in curl_cffi's own cookie store"""
session.cookies.set(name, value, domain=domain or "", path=path)
def get_cookies(self, session) -> dict:
"""all cookies stored in curl_cffi's cookie store"""
return dict(session.cookies.items())
def clear_cookies(self, session) -> None:
"""clear curl_cffi's cookie store"""
session.cookies.clear()
async def close(self, session) -> None:
"""close the curl_cffi session"""
await session.close()
class Noble:
"""noble_tls backend: forges a browser TLS fingerprint via the tls-client Go lib
config:
client: noble_tls Client profile (enum or string, default "chrome_133").
noble_tls downloads a Go shared library on first use; setup() fetches it once
(run via TLSSession.setup() or lazily before the first request).
requires the [noble] extra (pip install "aioweb_tls[noble]").
"""
def __init__(self, client="chrome_133"):
if noble_tls is None:
raise RuntimeError(
"noble_tls is not installed; install the extra with "
"pip install \"aioweb_tls[noble]\""
) from _NOBLE_ERROR
self.client = self._resolve_client(client)
self._updated = False
self._setup_lock = asyncio.Lock()
@staticmethod
def _resolve_client(client):
"""turn a string or Client enum into a noble_tls Client value
raises ValueError naming the valid profiles for an unknown string, instead of
letting getattr's raw AttributeError leak the enum's internal lookup mechanics.
"""
if not isinstance(client, str):
return client
name = client.upper()
resolved = getattr(_NobleClient, name, None)
if resolved is None:
valid = ", ".join(member.name for member in _NobleClient)
raise ValueError(f"unknown noble_tls client profile {client!r}; valid profiles: {valid}")
return resolved
async def setup(self) -> None:
"""fetch the noble_tls Go shared library once; idempotent and concurrency-safe
uses noble_tls.download_if_necessary (fetches on first use, no-ops if
present); falls back to update_if_necessary on older noble_tls.
guarded by an asyncio.Lock with a check-lock-recheck so concurrent first
requests don't both run the fetch — only the first caller through the lock
does the work, the rest see _updated already set.
"""
if self._updated:
return
async with self._setup_lock:
if self._updated:
return
download = getattr(noble_tls, "download_if_necessary", None)
if download is not None:
await download()
elif hasattr(noble_tls, "update_if_necessary"):
await noble_tls.update_if_necessary()
self._updated = True
def create_session(self, headers, timeout, **kwargs):
"""build the noble_tls Session, honoring the session-default timeout
noble_tls.Session takes neither headers nor timeout in its constructor.
deliberately does NOT bake `headers` into session.headers, same rationale as
CurlCffi.create_session — aioweb's per-request _default_headers merge already
applies them, so baking here would desync update_headers()/clear_headers().
the coerced timeout IS applied here (matching raw_request's per-call path):
noble's Go field timeoutSeconds is an int, so an uncoerced sub-second/float
session-default would fail Go-side JSON unmarshal on any request that doesn't
override it per call; max(1, ceil()) mirrors raw_request's own guard.
"""
session = noble_tls.Session(client=self.client, **kwargs)
timeout_seconds = _noble_timeout_seconds(timeout)
if timeout_seconds is not None:
session.timeout_seconds = timeout_seconds
return session
async def raw_request(self, session, method, url, **kwargs) -> Response:
"""send via noble_tls and adapt the result into an aioweb.Response"""
await self.setup()
timeout_seconds = _noble_timeout_seconds(kwargs.pop("timeout", None))
if timeout_seconds is not None:
kwargs["timeout_seconds"] = timeout_seconds
proxy = kwargs.pop("proxy", None)
if proxy:
kwargs["proxy"] = proxy
# byte-safe transport: without this, noble_tls's Go side returns the body as a
# plain UTF-8 JSON string (U+FFFD-mangling any binary payload on a 200 response);
# see _noble_content() for the base64 data-URI decode this pairs with.
kwargs.setdefault("is_byte_response", True)
try:
response = await session.execute_request(method=method.upper(), url=url, **kwargs)
except aiohttp.ClientError:
raise
except asyncio.TimeoutError:
raise
except OSError as error:
# TLSClientException subclasses IOError (== OSError); the Go side has no
# distinct timeout type either, just a text body ("context deadline
# exceeded") — check for one before the generic client-error wrap
if _is_timeout_error(error):
raise _as_timeout_error(error, "noble_tls") from error
raise _as_client_error(error, "noble_tls") from error
content = _noble_content(response)
return Response(
status_code=response.status_code,
headers=_flatten_headers(getattr(response, "headers", {}) or {}),
content=content,
url=str(getattr(response, "url", url)),
reason=getattr(response, "reason", None),
cookies=getattr(response, "cookies", None),
)
def is_closed(self, session) -> bool:
"""noble_tls has no closed flag; closed state is tracked by TLSSession"""
return bool(getattr(session, "closed", False))
def cookies_for_url(self, session, url) -> dict:
"""cookies noble_tls would send for url (best-effort)"""
return _jar_to_dict(session)
def set_cookie(self, session, name, value, domain=None, path="/") -> None:
"""set a cookie in noble_tls's own cookie jar"""
session.cookies.set(name, value, domain=domain or "", path=path)
def get_cookies(self, session) -> dict:
"""all cookies stored in noble_tls's cookie jar"""
return dict(session.cookies.items())
def clear_cookies(self, session) -> None:
"""clear noble_tls's cookie jar"""
session.cookies.clear()
async def close(self, session) -> None:
"""close the noble_tls session if it exposes a close"""
close = getattr(session, "close", None)
if close is None:
return
result = close()
if hasattr(result, "__await__"):
await result
+66
View File
@@ -0,0 +1,66 @@
"""
the tls backend protocol
a backend is a stateless config+behavior object that teaches TLSSession how to talk
to one HTTP client. TLSSession owns the live session object (built by create_session)
and passes it into every backend call, so backends hold no per-request state.
implement this protocol to add a custom backend (e.g. a local Go TLS server); inject
it via TLSSession(backend=MyBackend(...)) and it inherits aioweb's domain / header /
ephemeral / proxy / retry / preview logic unchanged — those operate on plain dicts
and never touch the backend.
required:
create_session(headers: dict, timeout, **kwargs) -> session
build and return the live client session object, stored as self.session and
passed to every method below.
async raw_request(session, method, url, **kwargs) -> aioweb.Response
send one request; adapt the client's response into an aioweb.Response built
from primitives (status_code, headers, content bytes, url, reason). kwargs
arrive aioweb-shaped: proxy resolved into kwargs["proxy"], headers merged
into kwargs["headers"], numeric timeout wrapped in aiohttp.ClientTimeout
(unwrap .total).
is_closed(session) -> bool
whether `session` is closed.
optional:
cookies_for_url(session, url) -> dict
cookies the client would send for url, for preview(). default {}.
set_cookie(session, name, value, domain=None, path="/") -> None
get_cookies(session) -> dict
clear_cookies(session) -> None
back TLSSession's mutable cookie api. the aiohttp-only base reaches into
session.cookie_jar, which TLS backends lack, so an implementation without
these raises NotImplementedError instead of a private-attribute AttributeError.
async setup() -> None
one-time async preparation (e.g. fetch a native lib); idempotent. called via
TLSSession.setup() and lazily before the first request.
async close(session) -> None
close `session`. default awaits session.close() if present.
"""
from typing import Any, Protocol, runtime_checkable
from aioweb import Response
@runtime_checkable
class TLSBackend(Protocol):
"""the contract a TLSSession backend implements (see module docstring)"""
def create_session(self, headers: dict, timeout: Any, **kwargs) -> Any:
"""build and return the live client session object"""
...
async def raw_request(self, session: Any, method: str, url: str, **kwargs) -> Response:
"""send one request and adapt the result into an aioweb.Response"""
...
def is_closed(self, session: Any) -> bool:
"""whether the session is closed"""
...
+126
View File
@@ -0,0 +1,126 @@
"""
TLSSession — one aioweb session, any tls backend
TLSSession subclasses aioweb.ExtendedSession and delegates only the four backend
seams to an injected backend object (see protocol.py). everything else — header
overwrites, domain rewriting, ephemeral headers, proxies, retries, previews — is
inherited unchanged, since that logic operates on plain dicts and never touches
the backend.
from aioweb_tls import TLSSession, CurlCffi, Noble
async with TLSSession(backend=CurlCffi(impersonate="chrome")) as s:
resp = await s.request_with_retries("GET", "https://tls.peet.ws/api/all")
if resp:
print(resp.json()["tls"]["ja3"])
s = TLSSession(backend=Noble(client="chrome_133"))
await s.setup() # fetch noble's Go lib once
...
a custom backend (e.g. a local Go TLS server) injects the same way — implement the
TLSBackend protocol and pass it as backend=. one TLSSession is the only session
class; the backend swaps the wire, not the session.
"""
import logging
from aioweb import ExtendedSession
log = logging.getLogger(__name__)
class TLSSession(ExtendedSession):
"""aioweb session whose HTTP backend is an injected, swappable tls client"""
def __init__(self, backend, *args, **kwargs):
"""build a session over a tls backend
args:
backend: a TLSBackend (CurlCffi, Noble, or a custom one); owns its own
config vocabulary and the four seam behaviors
*args, **kwargs: forwarded to aioweb.ExtendedSession
"""
self.backend = backend
self._closed = False
super().__init__(*args, **kwargs)
async def setup(self) -> None:
"""run the backend's one-time setup if it has one (idempotent)
delegates to backend.setup() when defined (e.g. noble fetching its Go lib);
a no-op for backends without it. also invoked lazily before the first request
by backends that guard their own setup, so calling this is optional but lets
callers pre-warm at startup.
"""
setup = getattr(self.backend, "setup", None)
if setup is not None:
await setup()
# -------------------------------------------------------------------------
# the four backend seams, delegated to self.backend
def _create_session(self, headers, timeout, **kwargs):
"""delegate session creation to the backend"""
return self.backend.create_session(headers, timeout, **kwargs)
async def _raw_request(self, method, url, **kwargs):
"""delegate the wire request to the backend, returning its aioweb.Response"""
return await self.backend.raw_request(self.session, method, url, **kwargs)
def _cookies_for_url(self, url) -> dict:
"""delegate preview cookie lookup to the backend (default {})"""
cookies_for_url = getattr(self.backend, "cookies_for_url", None)
if cookies_for_url is None:
return {}
return cookies_for_url(self.session, url)
def _is_closed(self) -> bool:
"""closed if explicitly closed here or the backend reports it"""
if self._closed:
return True
return self.backend.is_closed(self.session)
# -------------------------------------------------------------------------
# mutable cookie api — the base's set_cookie/get_cookies/clear_cookies reach
# into self.session.cookie_jar (aiohttp-only), so TLS backends override them
# to route through the backend instead of crashing with AttributeError
def set_cookie(self, name, value, domain=None, path="/"):
"""set a cookie via the backend's own cookie store"""
set_cookie = getattr(self.backend, "set_cookie", None)
if set_cookie is None:
raise NotImplementedError(
f"{type(self.backend).__name__} does not support the mutable cookie api"
)
set_cookie(self.session, name, value, domain=domain, path=path)
def get_cookies(self) -> dict:
"""all cookies stored in the backend's cookie store"""
get_cookies = getattr(self.backend, "get_cookies", None)
if get_cookies is None:
raise NotImplementedError(
f"{type(self.backend).__name__} does not support the mutable cookie api"
)
return get_cookies(self.session)
def clear_cookies(self) -> None:
"""clear the backend's cookie store"""
clear_cookies = getattr(self.backend, "clear_cookies", None)
if clear_cookies is None:
raise NotImplementedError(
f"{type(self.backend).__name__} does not support the mutable cookie api"
)
clear_cookies(self.session)
# -------------------------------------------------------------------------
# lifecycle — backends close differently, so route through the backend
async def close(self) -> None:
"""close via the backend's close (falls back to session.close)"""
self._closed = True
close = getattr(self.backend, "close", None)
if close is not None:
await close(self.session)
else:
await self.session.close()