fix: thread redirect history through both TLS backends
CurlCffi.raw_request and Noble.raw_request built their Response without history=, so resp.history/redirect_chain were always empty after a real redirect despite inheriting aioweb's feature set unchanged. A shared _history_entries() now maps each client's native history shape (curl_cffi list[dict], noble_tls list[Response]) into aioweb's (status, url) tuples. Signed-off-by: disqualifier <dev@disqualifier.me>
This commit is contained in:
@@ -22,17 +22,17 @@ you want; importing the package never fails because an extra is missing.
|
|||||||
`requirements.txt` (pick the extra you need):
|
`requirements.txt` (pick the extra you need):
|
||||||
|
|
||||||
```
|
```
|
||||||
aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.6
|
aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.7
|
||||||
aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.6
|
aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.7
|
||||||
aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.6
|
aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.7
|
||||||
```
|
```
|
||||||
|
|
||||||
Direct:
|
Direct:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pip install "aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.6"
|
pip install "aioweb_tls[curl] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.7"
|
||||||
pip install "aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.6"
|
pip install "aioweb_tls[noble] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.7"
|
||||||
pip install "aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.6"
|
pip install "aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb_tls.git@v0.1.7"
|
||||||
```
|
```
|
||||||
|
|
||||||
- `[curl]` → curl_cffi backend · `[noble]` → noble_tls backend · `[all]` → both.
|
- `[curl]` → curl_cffi backend · `[noble]` → noble_tls backend · `[all]` → both.
|
||||||
@@ -44,7 +44,7 @@ pip install "aioweb_tls[all] @ git+ssh://git@git.rethinkstudios.io/rethink-publi
|
|||||||
Constructing a backend whose client isn't installed raises that `RuntimeError` at
|
Constructing a backend whose client isn't installed raises that `RuntimeError` at
|
||||||
construction, never at import.
|
construction, never at import.
|
||||||
|
|
||||||
Drop the `@v0.1.6` suffix from the line above to install the latest unpinned.
|
Drop the `@v0.1.7` suffix from the line above to install the latest unpinned.
|
||||||
|
|
||||||
## curl_cffi backend
|
## curl_cffi backend
|
||||||
|
|
||||||
@@ -198,12 +198,32 @@ cookie store (both expose a `requests`-style `session.cookies` with `set()` /
|
|||||||
`items()` / `clear()`), so these calls work the same way they do on the base
|
`items()` / `clear()`), so these calls work the same way they do on the base
|
||||||
`aioweb.ExtendedSession`, not just `_cookies_for_url()` (used by `preview()`).
|
`aioweb.ExtendedSession`, not just `_cookies_for_url()` (used by `preview()`).
|
||||||
|
|
||||||
|
`resp.history` / `resp.redirect_chain` (`[(status, url), ...]`) and `resp.is_redirect`
|
||||||
|
are also threaded through on both backends: `CurlCffi.raw_request` maps curl_cffi's
|
||||||
|
`Response.history` (`list[dict]`) and `Noble.raw_request` maps noble_tls's
|
||||||
|
`list[Response]` into aioweb's `(status, url)` tuple shape, so `debug=True`'s "redirect
|
||||||
|
chain:" log line and any caller reading `resp.history` after a redirecting request see
|
||||||
|
the real hops on either backend, matching the base `aioweb.ExtendedSession` behavior.
|
||||||
|
|
||||||
## Honesty note
|
## Honesty note
|
||||||
|
|
||||||
TLS fingerprinting changes one layer — the TLS/HTTP fingerprint. It does **not** by
|
TLS fingerprinting changes one layer — the TLS/HTTP fingerprint. It does **not** by
|
||||||
itself defeat modern bot protection: behavioral analysis, captchas, and JS challenges
|
itself defeat modern bot protection: behavioral analysis, captchas, and JS challenges
|
||||||
are separate signals. Use this as one component, not a complete anti-bot solution.
|
are separate signals. Use this as one component, not a complete anti-bot solution.
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
|
||||||
|
### v0.1.7
|
||||||
|
|
||||||
|
- **Both backends now thread redirect history.** `CurlCffi.raw_request` and
|
||||||
|
`Noble.raw_request` built their `Response` without `history=`, so
|
||||||
|
`resp.history`/`resp.redirect_chain`/`resp.is_redirect`-after-follow were
|
||||||
|
always empty and aioweb's own `debug=True` "redirect chain:" log line was
|
||||||
|
permanently dead on both TLS backends, despite the "inherits every aioweb
|
||||||
|
feature unchanged" claim. Each backend's native history (curl_cffi
|
||||||
|
`list[dict]`, noble_tls `list[Response]`) is now mapped into aioweb's
|
||||||
|
`(status, url)` tuple shape and passed through.
|
||||||
|
|
||||||
## Versioning
|
## Versioning
|
||||||
|
|
||||||
Releases are tagged `vX.Y.Z`. The install line above pins a release; drop the `@vX.Y.Z` suffix to install the latest unpinned. Pin deliberately for reproducible installs.
|
Releases are tagged `vX.Y.Z`. The install line above pins a release; drop the `@vX.Y.Z` suffix to install the latest unpinned. Pin deliberately for reproducible installs.
|
||||||
|
|||||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "aioweb_tls"
|
name = "aioweb_tls"
|
||||||
version = "0.1.6"
|
version = "0.1.7"
|
||||||
description = "TLS-fingerprinting backends (curl_cffi / noble_tls) for aioweb via one injectable TLSSession, config-free, installable."
|
description = "TLS-fingerprinting backends (curl_cffi / noble_tls) for aioweb via one injectable TLSSession, config-free, installable."
|
||||||
requires-python = ">=3.10"
|
requires-python = ">=3.10"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
|
|||||||
@@ -98,6 +98,29 @@ def _flatten_headers(headers) -> dict:
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _history_entries(history) -> list:
|
||||||
|
"""map a backend-native redirect history into aioweb's `Response(history=...)` shape
|
||||||
|
|
||||||
|
aioweb's own `_raw_request` threads `[(status, url), ...]`; curl_cffi's
|
||||||
|
`Response.history` is `list[dict]` (dict-shaped hop records) and noble_tls's is
|
||||||
|
`list[Response]` (object-shaped, same accessors as the top-level response) - this
|
||||||
|
reads a hop's status/url either way. an unparseable hop is skipped rather than
|
||||||
|
raising, so a redirect-history quirk never breaks the response it's attached to.
|
||||||
|
"""
|
||||||
|
entries = []
|
||||||
|
for hop in history or []:
|
||||||
|
if isinstance(hop, dict):
|
||||||
|
status = hop.get("status_code", hop.get("status"))
|
||||||
|
url = hop.get("url")
|
||||||
|
else:
|
||||||
|
status = getattr(hop, "status_code", getattr(hop, "status", None))
|
||||||
|
url = getattr(hop, "url", None)
|
||||||
|
if status is None or url is None:
|
||||||
|
continue
|
||||||
|
entries.append((status, str(url)))
|
||||||
|
return entries
|
||||||
|
|
||||||
|
|
||||||
def _jar_to_dict(session):
|
def _jar_to_dict(session):
|
||||||
"""best-effort map of a requests-style cookie jar on session to a plain dict
|
"""best-effort map of a requests-style cookie jar on session to a plain dict
|
||||||
|
|
||||||
@@ -172,6 +195,7 @@ class CurlCffi:
|
|||||||
content=content,
|
content=content,
|
||||||
url=str(response.url),
|
url=str(response.url),
|
||||||
reason=getattr(response, "reason", None),
|
reason=getattr(response, "reason", None),
|
||||||
|
history=_history_entries(getattr(response, "history", None)),
|
||||||
cookies=getattr(response, "cookies", None),
|
cookies=getattr(response, "cookies", None),
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -310,6 +334,7 @@ class Noble:
|
|||||||
content=content,
|
content=content,
|
||||||
url=str(getattr(response, "url", url)),
|
url=str(getattr(response, "url", url)),
|
||||||
reason=getattr(response, "reason", None),
|
reason=getattr(response, "reason", None),
|
||||||
|
history=_history_entries(getattr(response, "history", None)),
|
||||||
cookies=getattr(response, "cookies", None),
|
cookies=getattr(response, "cookies", None),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user