aioweb_tls/README.md

aioweb_tls

TLS-fingerprinting backends for aioweb. One session class — TLSSession — takes an injected backend that swaps the HTTP client (and thus the TLS/HTTP fingerprint), while inheriting every aioweb feature — header overwrites, domain rewriting, ephemeral headers, proxies, retries, previews — unchanged. The backend swaps the wire, not the session.

from aioweb_tls import TLSSession, CurlCffi, Noble   # the two bundled backends

TLSSession(backend=CurlCffi(impersonate="chrome"))
TLSSession(backend=Noble(client="chrome_133"))
TLSSession(backend=MyBackend(...))   # or one you write — see "Writing your own backend"

Install

Depends on aioweb. The TLS clients are optional extras — install the backend(s) you want; importing the package never fails because an extra is missing.

requirements.txt (pick the extra you need):

aioweb_tls[curl]  @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v1.0.0
aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v1.0.0
aioweb_tls[all]   @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v1.0.0

Direct:

pip install "aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v1.0.0"
pip install "aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v1.0.0"
pip install "aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v1.0.0"

• [curl] → curl_cffi backend · [noble] → noble_tls backend · [all] → both.
• No extra (pip install aioweb_tls) is also valid: you get TLSSession and the TLSBackend protocol — the framework and the bring-your-own-backend path — but no bundled client. CurlCffi() / Noble() then raise a clear RuntimeError naming the extra to install. Use a bare install when you only need a custom backend of your own.

Constructing a backend whose client isn't installed raises that RuntimeError at construction, never at import.

Drop the @v1.0.0 suffix from the line above to install the latest unpinned.

curl_cffi backend

from aioweb_tls import TLSSession, CurlCffi

async with TLSSession(backend=CurlCffi(impersonate="chrome"), proxies={"https": "http://..."}) as s:
    resp = await s.request_with_retries("GET", "https://tls.peet.ws/api/all")
    if resp:                          # FailureResponse is falsy
        print(resp.json()["tls"]["ja3"])

• CurlCffi(impersonate="chrome") sets the forged profile; override it per call by passing impersonate= to the low-level request() (which forwards **kwargs to the backend). request_with_retries has a fixed signature and does not accept extra backend kwargs — passing impersonate= there raises TypeError; set the profile on the CurlCffi instance for the retrying path.
• curl_cffi forges JA3/JA4 + HTTP/2 fingerprints via the bundled curl-impersonate binary.
• A wire-level timeout raises aiohttp.ServerTimeoutError (matching aioweb's own contract), not a generic aiohttp.ClientError — both backends detect their native timeout (curl_cffi's Timeout type, noble_tls's Go-side timeout text) and re-wrap it before the fallback client-error path (v0.1.5).

noble backend

from aioweb_tls import TLSSession, Noble

async with TLSSession(backend=Noble(client="chrome_133")) as s:
    await s.setup()                   # fetch noble's Go shared lib once (network)
    resp = await s.request_with_retries("GET", "https://tls.peet.ws/api/all")
    if resp:
        print(resp.json()["tls"]["ja3"])

• Noble(client="chrome_133") — accepts a noble_tls.Client enum or a string name; an unknown string raises ValueError listing the valid profile names (v0.1.5).
• noble_tls downloads a Go shared library on first use. await s.setup() fetches it once at startup; if you skip it, the first request fetches it lazily. The fetch is guarded by a lock, so even concurrent first requests download it exactly once.
• Binary bodies (images, zips, PDFs, protobuf) round-trip as true bytes: noble_tls returns response bodies as a UTF-8 JSON string by default, which mangles non-UTF-8 bytes (U+FFFD replacement, wrong length) even on a 200 response — the Noble backend always requests is_byte_response=True and decodes the resulting base64 data-URI back into raw bytes, so resp.content is never lossy.
• Multi-valued response headers (e.g. two Set-Cookie lines) arrive from noble_tls's Go side as a Python list — the Noble backend flattens them to a single comma-joined string per RFC 7230, so resp.headers[...] is always a plain string (v0.1.5).

Writing your own backend (the TLSBackend protocol)

aioweb_tls ships exactly two backends — CurlCffi and Noble — plus the TLSBackend protocol for writing your own. There is no Go, remote, or other bundled backend; anything beyond curl/noble is something you implement against the protocol. TLSSession itself is backend-agnostic, so a backend you write injects exactly like the built-ins (TLSSession(backend=YourBackend(...))) and inherits every aioweb feature for free.

To add a backend, write a small object satisfying TLSBackend (see protocol.py for the authoritative contract):

member	required?	signature	purpose
create_session	required	(headers, timeout, **kwargs) -> session	build the client object TLSSession stores as self.session
raw_request	required	async (session, method, url, **kwargs) -> aioweb.Response	send one request; adapt the client's response into an aioweb.Response
is_closed	required	(session) -> bool	whether the session is closed
cookies_for_url	optional	(session, url) -> dict	cookies for preview(); defaults to {}
set_cookie	optional	(session, name, value, domain=None, path="/") -> None	backs TLSSession.set_cookie(); raises NotImplementedError if absent
get_cookies	optional	(session) -> dict	backs TLSSession.get_cookies(); raises NotImplementedError if absent
clear_cookies	optional	(session) -> None	backs TLSSession.clear_cookies(); raises NotImplementedError if absent
setup	optional	async () -> None	one-time prep (e.g. fetch a native lib); idempotent. TLSSession never calls this automatically - a backend needing lazy setup must self-invoke it from its own raw_request, as Noble does, or the caller must run await session.setup() explicitly
close	optional	async (session) -> None	close the session; defaults to await session.close()

raw_request receives aioweb-shaped kwargs: the proxy is already resolved into kwargs["proxy"], headers are merged into kwargs["headers"], and a numeric timeout is wrapped in an aiohttp.ClientTimeout (unwrap .total).

Example — a backend you would write (not included)

The class below is illustrative template code you implement yourself, shown to make the protocol concrete. It is not part of aioweb_tls — there is no from aioweb_tls import GoTLSBackend. It sketches one case (talking to a local Go TLS service over HTTP) so the shape of a conformant backend is clear:

# example: a backend YOU write to drive a local Go TLS service.
# NOT shipped by aioweb_tls — this is a TLSBackend-conformance template.
from aioweb import Response
from aioweb_tls import TLSSession

class GoTLSBackend:
    """example user-written backend: drives a local Go tls-server over HTTP"""

    def __init__(self, endpoint: str):
        self.endpoint = endpoint

    def create_session(self, headers, timeout, **kwargs):
        # build/return whatever client object raw_request will use
        import aiohttp
        return aiohttp.ClientSession(headers=headers)

    async def raw_request(self, session, method, url, **kwargs) -> Response:
        # kwargs arrive aioweb-shaped: proxy in kwargs["proxy"], headers merged,
        # numeric timeout wrapped in an aiohttp.ClientTimeout (unwrap .total).
        payload = {"method": method, "url": url, "headers": kwargs.get("headers", {})}
        async with session.post(self.endpoint, json=payload) as r:
            body = await r.json()
            return Response(
                status_code=body["status"], headers=body["headers"],
                content=body["body"].encode(), url=url, reason=body.get("reason"),
            )

    def is_closed(self, session) -> bool:
        return session.closed

    async def close(self, session) -> None:
        await session.close()

# inject your backend — same shape as CurlCffi / Noble, all aioweb features inherited
async with TLSSession(backend=GoTLSBackend("http://localhost:8080")) as s:
    s.overwrite_domain("internal.local", "127.0.0.1")
    resp = await s.request_with_retries("GET", "https://internal.local/x")

Inherited features work unchanged

aioweb's overwrite/domain/ephemeral/proxy/retry/preview logic operates on plain dicts and never touches the HTTP backend — only the seams do. Header overwrites, domain rewriting, ephemeral headers, proxies, retries, and previews behave identically on any backend:

async with TLSSession(backend=CurlCffi(impersonate="chrome")) as s:
    s.overwrite_domain("internal.local", "127.0.0.1")
    s.set_ephemeral("X-Time", lambda: str(time.time()))
    s.overwrite_inject(True)
    print(s.preview("GET", "https://internal.local/x").as_curl())   # reflects all of the above

Session-default headers are never baked into the underlying client (neither CurlCffi nor Noble passes headers= to their client's constructor) — they flow through aioweb's own per-request _default_headers merge instead. That keeps update_headers() / clear_headers() accurate for both backends: what get_headers() and preview() report is what actually goes out on the wire, with no stale, already-baked value resurfacing after a clear.

The mutable cookie API — set_cookie() / get_cookies() / clear_cookies() — is also backend-aware: CurlCffi and Noble each route it through their own client's cookie store (both expose a requests-style session.cookies with set() / items() / clear()), so these calls work the same way they do on the base aioweb.ExtendedSession, not just _cookies_for_url() (used by preview()).

resp.history / resp.redirect_chain ([(status, url), ...]) and resp.is_redirect are threaded through from whatever redirect history the backend exposes. Caveat — the CurlCffi backend has no per-hop history: curl_cffi follows redirects internally in the native curl layer and surfaces only the final URL/status, leaving Response.history an empty list (it never populates it). So on CurlCffi, resp.history/resp.redirect_chain are [] and resp.is_redirect reflects only the final response, even after a redirect — this is a curl_cffi limitation, not a bug here, and it differs from the base aiohttp ExtendedSession (which does record the hops). The Noble backend threads whatever noble_tls exposes as its per-response history. If you need the redirect chain, use the base backend or read the final URL.

Honesty note

TLS fingerprinting changes one layer — the TLS/HTTP fingerprint. It does not by itself defeat modern bot protection: behavioral analysis, captchas, and JS challenges are separate signals. Use this as one component, not a complete anti-bot solution.

Changelog

v0.1.8

• Compressed 4 residual internal-helper docstrings (_is_timeout_error, _noble_timeout_seconds, _noble_content, _jar_to_dict) to one-liners. Cosmetic, zero behavior change.

v0.1.7

• Backends thread whatever redirect history the client exposes. CurlCffi.raw_request and Noble.raw_request built their Response without history=, so resp.history/resp.redirect_chain/resp.is_redirect-after-follow were always empty and aioweb's own debug=True "redirect chain:" log line was dead on both TLS backends. Each backend now maps its client's history into aioweb's (status, url) tuple shape. Caveat: curl_cffi never populates Response.history (it follows redirects in the native curl layer and surfaces only the final URL/status), so on the CurlCffi backend resp.history is [] even after a redirect — a curl_cffi limitation, not addressable here. Noble passes through whatever noble_tls records.

Versioning

Releases are tagged vX.Y.Z. The install line above pins a release; drop the @vX.Y.Z suffix to install the latest unpinned. Pin deliberately for reproducible installs.