Compare commits
17
Commits
v0.1.9
...
b009c0cf50
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b009c0cf50 | ||
|
|
e6eddf5bc3 | ||
|
|
952fe8a00f | ||
|
|
6ec132f97c | ||
|
|
6633d6cba3 | ||
|
|
1deb05ce4a | ||
|
|
45d8830833 | ||
|
|
eed607b08c | ||
|
|
ed773e2c1b | ||
|
|
85735023f5 | ||
|
|
51c421c9a2 | ||
|
|
59eadabe16 | ||
|
|
78dfba0962 | ||
|
|
a7302a6356 | ||
|
|
397d74efe3 | ||
|
|
7f73f21c93 | ||
|
|
27f0e49341 |
@@ -11,18 +11,19 @@ and swap the HTTP client while inheriting everything else.
|
||||
`requirements.txt`:
|
||||
|
||||
```
|
||||
aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.9
|
||||
aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v1.1.0
|
||||
```
|
||||
|
||||
Direct:
|
||||
|
||||
```bash
|
||||
pip install "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v0.1.9"
|
||||
pip install "aioweb @ git+ssh://git@git.rethinkstudios.io/rethink-public/aioweb.git@v1.1.0"
|
||||
```
|
||||
|
||||
Requires `aiohttp` and `yarl` (pulled transitively).
|
||||
Requires `aiohttp` and `yarl` (pulled transitively), plus the sibling `commons` library
|
||||
(a private `git+ssh` package — the install needs access to the `rethink-public` org).
|
||||
|
||||
Drop the `@v0.1.9` suffix from the line above to install the latest unpinned.
|
||||
Drop the `@v1.1.0` suffix from the line above to install the latest unpinned.
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -97,10 +98,15 @@ s.overwrite_domain("internal.local", "127.0.0.1") # host-substring rewrite
|
||||
print(s.preview("POST", url, json={"a": 1}).as_curl()) # equivalent cURL command
|
||||
```
|
||||
|
||||
`as_curl()` renders `params` (merged into the url's query string) and `timeout` (as
|
||||
`--max-time`) as well as headers/body/proxy, so the emitted command is faithful to what
|
||||
`request()` actually sends. Pass `debug=True` to `request_with_retries` to log the cURL
|
||||
preview and request flow.
|
||||
`preview()` is synchronous and never touches the backend session — it works before any
|
||||
event loop is running (e.g. building a preview at import/setup time), not just inside
|
||||
`asyncio.run()`. Its default cookie lookup only reads cookies from an already-built
|
||||
session; if the session hasn't been built yet there are no cookies to read, so it
|
||||
returns none by default (pass `cookies=` explicitly to preview cookies for a session
|
||||
that hasn't sent a request yet). `as_curl()` renders `params` (merged into the url's
|
||||
query string) and `timeout` (as `--max-time`) as well as headers/body/proxy, so the
|
||||
emitted command is faithful to what `request()` actually sends. Pass `debug=True` to
|
||||
`request_with_retries` to log the cURL preview and request flow.
|
||||
|
||||
## Custom backends
|
||||
|
||||
@@ -144,6 +150,65 @@ Two changes can't be shimmed without re-introducing the bugs they fix:
|
||||
|
||||
## Changelog
|
||||
|
||||
### v1.1.0
|
||||
|
||||
- **Hang-guard on `request()` — a wedged backend can no longer park the loop forever.**
|
||||
`request()` now wraps the backend call (`_raw_request`) in an outer `asyncio.wait_for`
|
||||
deadline sized at the effective timeout **+ 5s slack** (`_HANG_GUARD_SLACK`). The
|
||||
backend's own timeout still fires first in every healthy failure (slack, not
|
||||
replacement); the envelope only trips when the backend's timer is dead — e.g. a
|
||||
`curl_cffi` transfer that loses both its wakeup and its own enforcement in the same
|
||||
failure — and its cancellation is what unwedges the orphaned transfer. The guard's
|
||||
deadline is the per-call numeric `timeout` when given, else the session timeout
|
||||
(`Session(timeout=...)`, default 10s); it composes cleanly with `request_with_retries`
|
||||
(a wedge becomes one logged `ServerTimeoutError`, the next attempt proceeds, and an
|
||||
exhausted retry returns a `FailureResponse` rather than hanging). **Opt-out:** a
|
||||
session deliberately constructed unbounded (`Session(timeout=None)`) stays unguarded,
|
||||
for genuinely long-lived calls (long-poll / streaming). Additive; no new knob.
|
||||
|
||||
### v0.1.13
|
||||
|
||||
- **Session-default timeout no longer poisons pooled keep-alive connections.**
|
||||
The default `ClientTimeout` set `sock_read=timeout/2` alongside `total`. Under
|
||||
`aiohttp>=3.14`, that read timer re-arms on every request dispatched over a
|
||||
pooled protocol, including idle connections between requests; when it fires
|
||||
it permanently poisons the pooled connection (`SocketTimeoutError` on the next
|
||||
use, instantly, without contacting the server) — a real error from the server
|
||||
(e.g. a 503) could come back as a client-side `FailureResponse(status=0,
|
||||
reason='timeout')` instead. `sock_read` is now dropped from the session
|
||||
default; `total` (and `connect`/`sock_connect`) still bound every request, and
|
||||
the per-call `timeout=N` path (`ClientTimeout(total=N)`, no `sock_read`) was
|
||||
already unaffected.
|
||||
|
||||
### v0.1.12
|
||||
|
||||
- **Docstring-only.** Restored one-line docstrings on `FailureResponse`'s
|
||||
`redirect_chain`, `text()`, `json()`, and `raise_for_status()` — a prior
|
||||
de-bloat pass stripped them below the public tier while the `Response`
|
||||
twins kept theirs. No behavior change.
|
||||
|
||||
### v0.1.11
|
||||
|
||||
- **`preview()` no longer requires a running event loop.** Its default cookie
|
||||
lookup (`cookies=` not passed) previously called `_cookies_for_url()`, which
|
||||
reached the backend session and built it if absent — under aiohttp>=3.14 that
|
||||
needs a running loop, so `preview()` raised `RuntimeError('no running event
|
||||
loop')` when called before the loop starts, defeating its own pre-loop
|
||||
build/inspect use case (v0.1.7). Now the default cookie lookup is skipped
|
||||
entirely when the session hasn't been built yet (nothing could have been set
|
||||
on a session that doesn't exist); `cookies={}` and in-loop calls are unaffected.
|
||||
- **`set_cookie(domain=None)` now honors `path`.** The `domain=None` branch
|
||||
returned right after `update_cookies()`, before the line that sets the
|
||||
morsel's `path` — so a shared cookie (`set_cookie(name, value, path="/api")`,
|
||||
no `domain=`) always stored the `SimpleCookie` default `path="/"` instead. The
|
||||
morsel's `path` is now set before the `domain=None` early return. The
|
||||
domain-bound branch was already correct and is unchanged.
|
||||
|
||||
### v0.1.10
|
||||
|
||||
- Docs-only pass: compressed module/method docstrings and comments that restated
|
||||
README/CLAUDE prose, replaced em-dashes with hyphens. No behavior change.
|
||||
|
||||
### v0.1.9
|
||||
|
||||
- **`_get_proxy()`/`request()` proxy resolution now checks `is None`, not
|
||||
|
||||
+2
-2
@@ -4,13 +4,13 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "aioweb"
|
||||
version = "0.1.9"
|
||||
version = "1.2.0"
|
||||
description = "Async HTTP session wrapper over aiohttp — proxies, header overwrites, retries, previews. Config-free, installable."
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
"aiohttp>=3.9",
|
||||
"yarl>=1.9",
|
||||
"commons @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v0.2.1",
|
||||
"commons @ git+https://git.rethinkstudios.io/rethink-public/commons.git@v1.0.0",
|
||||
]
|
||||
|
||||
[tool.hatch.metadata]
|
||||
|
||||
@@ -1,7 +1,14 @@
|
||||
from importlib.metadata import version, PackageNotFoundError
|
||||
|
||||
from .session import ExtendedSession, DEFAULT_ATTEMPTS, RETRY_STATUSES
|
||||
from .responses import Response, FailureResponse, AiowebError, aiowebResponse
|
||||
from .preview import RequestPreview
|
||||
|
||||
try:
|
||||
__version__ = version("aioweb")
|
||||
except PackageNotFoundError:
|
||||
__version__ = "0.0.0+unknown"
|
||||
|
||||
__all__ = [
|
||||
"ExtendedSession",
|
||||
"Response",
|
||||
@@ -17,17 +24,11 @@ __all__ = [
|
||||
|
||||
|
||||
async def request_retries(session, method, url, **kwargs):
|
||||
"""back-compat wrapper for session.request_with_retries(...)
|
||||
|
||||
prefer calling session.request_with_retries(...) directly.
|
||||
"""
|
||||
"""back-compat wrapper for session.request_with_retries(...); prefer calling that directly"""
|
||||
return await session.request_with_retries(method, url, **kwargs)
|
||||
|
||||
|
||||
async def test_proxies(session, url="https://api.ipify.org?format=json"):
|
||||
"""fetch the public IP via the session (verifies proxy config)
|
||||
|
||||
url defaults to ipify; pass another IP-echo endpoint to point elsewhere.
|
||||
"""
|
||||
"""fetch the public IP via the session (verifies proxy config); url defaults to ipify"""
|
||||
response = await session.request("GET", url)
|
||||
return response.json()
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
"""
|
||||
request preview for aioweb — format or export a request without sending it
|
||||
request preview for aioweb - format or export a request without sending it
|
||||
"""
|
||||
|
||||
import json as _json
|
||||
import shlex
|
||||
from urllib.parse import urlencode
|
||||
|
||||
from yarl import URL
|
||||
|
||||
@@ -28,18 +29,17 @@ class RequestPreview:
|
||||
return "\n".join(f"{key}: {value}" for key, value in self.details.items())
|
||||
|
||||
def as_curl(self):
|
||||
"""equivalent cURL command for the request
|
||||
|
||||
every interpolated value is shlex.quote'd (non-injectable even with quotes/
|
||||
spaces/metacharacters). `params` is merged into the url's query string and
|
||||
`timeout` rendered as `--max-time`, so the command matches what request()
|
||||
actually sends.
|
||||
"""
|
||||
"""equivalent cURL command for the request (values shlex.quote'd; matches what request() sends)"""
|
||||
parts = [f"curl -X {shlex.quote(self.details['method'])}"]
|
||||
for header, value in (self.details["headers"] or {}).items():
|
||||
parts.append(f"-H {shlex.quote(f'{header}: {value}')}")
|
||||
if self.details["data"] is not None:
|
||||
parts.append(f"--data {shlex.quote(str(self.details['data']))}")
|
||||
data = self.details["data"]
|
||||
# aiohttp form-encodes a dict data= body (application/x-www-form-urlencoded);
|
||||
# render that wire form, not the python repr, so the curl replays identically.
|
||||
# doseq=True mirrors aiohttp's FormData: a list value becomes repeated k=v pairs
|
||||
rendered = urlencode(data, doseq=True) if isinstance(data, dict) else str(data)
|
||||
parts.append(f"--data {shlex.quote(rendered)}")
|
||||
elif self.details["json"] is not None:
|
||||
# is-not-None: an empty-but-valid body ({} / []) must still render
|
||||
parts.append(f"--data {shlex.quote(_json.dumps(self.details['json']))}")
|
||||
|
||||
+9
-15
@@ -1,8 +1,7 @@
|
||||
"""
|
||||
backend-agnostic response objects for aioweb — Response is built from primitives
|
||||
(status, headers, content, url, history), not a raw aiohttp object, so any backend can
|
||||
produce one; FailureResponse mirrors the same surface (every status/predicate a
|
||||
property on both) so callers can branch uniformly.
|
||||
backend-agnostic response objects for aioweb - Response and FailureResponse share
|
||||
the same surface (every status/predicate a property on both) so callers can branch
|
||||
uniformly regardless of which one they got.
|
||||
"""
|
||||
|
||||
import json as _json
|
||||
@@ -32,32 +31,26 @@ class Response:
|
||||
|
||||
@property
|
||||
def status_code(self) -> int:
|
||||
"""HTTP status code"""
|
||||
return self._status_code
|
||||
|
||||
@property
|
||||
def headers(self):
|
||||
"""response headers"""
|
||||
return self._headers
|
||||
|
||||
@property
|
||||
def url(self) -> str:
|
||||
"""final URL after redirects"""
|
||||
return self._url
|
||||
|
||||
@property
|
||||
def reason(self):
|
||||
"""reason phrase for the status code"""
|
||||
return self._reason
|
||||
|
||||
@property
|
||||
def cookies(self):
|
||||
"""cookies set in the response"""
|
||||
return self._cookies
|
||||
|
||||
@property
|
||||
def history(self):
|
||||
"""redirect history (list of (status, url) tuples)"""
|
||||
return self._history
|
||||
|
||||
@property
|
||||
@@ -67,17 +60,14 @@ class Response:
|
||||
|
||||
@property
|
||||
def is_redirect(self) -> bool:
|
||||
"""whether the status is a redirect"""
|
||||
return self._status_code in (301, 302, 303, 307, 308)
|
||||
|
||||
@property
|
||||
def is_success(self) -> bool:
|
||||
"""whether the status indicates success (2xx)"""
|
||||
return 200 <= self._status_code < 300
|
||||
|
||||
@property
|
||||
def content(self) -> bytes:
|
||||
"""raw response bytes"""
|
||||
return self._content
|
||||
|
||||
def text(self, encoding: Optional[str] = None) -> str:
|
||||
@@ -94,7 +84,7 @@ class Response:
|
||||
return _json.loads(self.text())
|
||||
except (_json.JSONDecodeError, UnicodeDecodeError):
|
||||
# text() decodes the body and can raise UnicodeDecodeError on a non-UTF-8
|
||||
# payload — that's a "not valid JSON" outcome, not an error to propagate
|
||||
# payload - that's a "not valid JSON" outcome, not an error to propagate
|
||||
return None
|
||||
|
||||
def raise_for_status(self):
|
||||
@@ -143,6 +133,7 @@ class FailureResponse:
|
||||
|
||||
@property
|
||||
def redirect_chain(self):
|
||||
"""list of (status, url) tuples for all redirects"""
|
||||
return []
|
||||
|
||||
@property
|
||||
@@ -158,12 +149,15 @@ class FailureResponse:
|
||||
return None
|
||||
|
||||
def text(self, encoding=None):
|
||||
"""always None"""
|
||||
return None
|
||||
|
||||
def json(self):
|
||||
"""always None"""
|
||||
return None
|
||||
|
||||
def raise_for_status(self):
|
||||
"""raise AiowebError(reason)"""
|
||||
raise AiowebError(self._reason)
|
||||
|
||||
def __bool__(self) -> bool:
|
||||
@@ -173,5 +167,5 @@ class FailureResponse:
|
||||
return f"<FailureResponse [{self._status_code}] {self._reason}>"
|
||||
|
||||
|
||||
# back-compat alias — the response class was renamed Response
|
||||
# back-compat alias - the response class was renamed Response
|
||||
aiowebResponse = Response
|
||||
|
||||
+180
-63
@@ -1,21 +1,15 @@
|
||||
"""
|
||||
async HTTP session wrapper over aiohttp — proxies, header overwrites, ephemeral
|
||||
headers, domain rewriting, previews, retry/backoff; byte-sending is isolated in
|
||||
_raw_request() so a subclass can swap backends and inherit everything else.
|
||||
async HTTP session wrapper over aiohttp - proxies, header overwrites, ephemeral
|
||||
headers, domain rewriting, previews, retry/backoff. See README for usage and contract.
|
||||
|
||||
async with ExtendedSession(proxies={"https": "http://..."}) as s:
|
||||
resp = await s.request_with_retries("GET", url)
|
||||
if resp: # FailureResponse is falsy
|
||||
data = resp.json()
|
||||
|
||||
config-free (proxies/headers/timeouts passed at construction or per call). the
|
||||
backend session is built lazily on first use, not in __init__, so construction is
|
||||
safe before an event loop is running (e.g. bot.http = ExtendedSession(...)).
|
||||
sessions must be closed explicitly (async with, or await s.close()) — no __del__
|
||||
auto-close (unsafe for async resources).
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import ipaddress
|
||||
import logging
|
||||
import warnings
|
||||
from http.cookies import SimpleCookie
|
||||
@@ -35,12 +29,29 @@ def _route_body(data):
|
||||
return data, None
|
||||
|
||||
|
||||
class _RetryStatus(Exception):
|
||||
"""internal signal: a retryable HTTP status; carries the real Response
|
||||
def _is_ip_host(host) -> bool:
|
||||
"""whether host is a literal IPv4/IPv6 address rather than a hostname"""
|
||||
if not host:
|
||||
return False
|
||||
try:
|
||||
ipaddress.ip_address(host)
|
||||
except ValueError:
|
||||
return False
|
||||
return True
|
||||
|
||||
raised inside an attempt so commons.aretry drives the backoff + cap; the caller
|
||||
catches the final one to return the REAL last response, not a synthetic failure.
|
||||
"""
|
||||
|
||||
def _is_ipv6_literal(host) -> bool:
|
||||
"""whether host is a bare IPv6 literal (must be a whole-host swap, never a substring splice)"""
|
||||
if not host:
|
||||
return False
|
||||
try:
|
||||
return isinstance(ipaddress.ip_address(host), ipaddress.IPv6Address)
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
class _RetryStatus(Exception):
|
||||
"""internal signal: a retryable HTTP status; carries the real Response through aretry"""
|
||||
|
||||
def __init__(self, response):
|
||||
super().__init__(f"retryable status {response.status_code}")
|
||||
@@ -51,8 +62,8 @@ log = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_ATTEMPTS = 3
|
||||
DEFAULT_BACKOFF_BASE = 2.0
|
||||
# statuses worth retrying: rate limit + transient server errors
|
||||
RETRY_STATUSES = frozenset({429, 500, 502, 503, 504})
|
||||
_HANG_GUARD_SLACK = 5
|
||||
|
||||
|
||||
class ExtendedSession:
|
||||
@@ -82,9 +93,7 @@ class ExtendedSession:
|
||||
self._default_headers = dict(headers or {})
|
||||
self._session_timeout = timeout
|
||||
self._session_kwargs = kwargs
|
||||
# aiohttp>=3.14 requires a running loop to build ClientSession; build lazily
|
||||
# (via `session` / _ensure_session()) so construction before the loop starts
|
||||
# (e.g. bot.http = ExtendedSession(...) in Bot.__init__) doesn't crash
|
||||
# aiohttp>=3.14 needs a running loop to build ClientSession; built lazily instead
|
||||
self._session = None
|
||||
|
||||
@property
|
||||
@@ -94,26 +103,29 @@ class ExtendedSession:
|
||||
return self._session
|
||||
|
||||
def _ensure_session(self):
|
||||
"""build the backend session on first use — idempotent"""
|
||||
"""build the backend session on first use - idempotent"""
|
||||
if self._session is None:
|
||||
self._session = self._create_session(
|
||||
self._default_headers, self._session_timeout, **self._session_kwargs,
|
||||
)
|
||||
|
||||
def _create_session(self, headers, timeout, **kwargs):
|
||||
"""create the backend HTTP session — override to swap HTTP clients
|
||||
"""create the backend HTTP session - override to swap HTTP clients
|
||||
|
||||
overwrite/domain/proxy/retry/preview logic never touches the session object
|
||||
directly (only _raw_request, cookies, close do), so those work unchanged on
|
||||
any backend. `headers` isn't baked in here — aiohttp would copy it into an
|
||||
immutable map update_headers/clear_headers can't touch; `_default_headers`
|
||||
is the mutable layer request()/preview() merge per call instead.
|
||||
`headers` isn't baked in here - aiohttp would copy it into an immutable map
|
||||
update_headers/clear_headers can't touch; `_default_headers` is the mutable
|
||||
layer request()/preview() merge per call instead.
|
||||
|
||||
no `sock_read` here - aiohttp re-arms that timer on every request dispatched
|
||||
over a pooled protocol, including idle keep-alive connections, and firing it
|
||||
calls `set_exception(SocketTimeoutError)` on the protocol, permanently
|
||||
poisoning that pooled connection; the next request on it fails instantly
|
||||
without contacting the server. `total` still bounds every request overall.
|
||||
"""
|
||||
return aiohttp.ClientSession(
|
||||
timeout=aiohttp.ClientTimeout(
|
||||
total=timeout,
|
||||
connect=timeout / 2,
|
||||
sock_read=timeout / 2,
|
||||
sock_connect=timeout / 2,
|
||||
),
|
||||
**kwargs,
|
||||
@@ -185,16 +197,47 @@ class ExtendedSession:
|
||||
# domain overwrites
|
||||
|
||||
def overwrite_domain(self, target, replacement):
|
||||
"""register a host-substring rewrite (target -> replacement)"""
|
||||
"""register a host rewrite (target -> replacement)
|
||||
|
||||
raises ValueError if the replacement is not a valid host for yarl's with_host()
|
||||
(e.g. a 'host:port' string or a pre-bracketed '[::1]') - caught here rather than
|
||||
deep in yarl at request time. a bare IPv6 literal ('::1', '2001:db8::1') is valid
|
||||
and accepted: yarl brackets it itself.
|
||||
|
||||
a hostname/IPv4 replacement rewrites any host CONTAINING target (substring splice,
|
||||
e.g. 'example.com' -> 'internal.example.com'). an IPv6 replacement can only ever be
|
||||
a WHOLE-HOST swap - splicing '::1' into the middle of a name yields an invalid host
|
||||
('internal::1') - so at request time an IPv6 replacement applies ONLY when target
|
||||
equals the whole request host; a request whose host merely CONTAINS the target (a
|
||||
substring) is left unrewritten instead of composing an invalid host and exploding
|
||||
deep in yarl. see _apply_domain_overwrites.
|
||||
"""
|
||||
try:
|
||||
URL("http://placeholder").with_host(replacement)
|
||||
except ValueError as exc:
|
||||
raise ValueError(
|
||||
f"overwrite_domain replacement {replacement!r} is not a valid host: {exc}; "
|
||||
"pass a bare hostname or IP (a bare IPv6 literal is fine), not a host:port or "
|
||||
"bracketed form - rewrite a port via a full URL override, not domain_overwrites"
|
||||
) from exc
|
||||
self.domain_overwrites[target] = replacement
|
||||
|
||||
def _apply_domain_overwrites(self, url: str) -> str:
|
||||
"""apply any host-substring rewrites to a url"""
|
||||
"""apply any host rewrites to a url
|
||||
|
||||
a hostname/IPv4 replacement splices on a substring match; an IPv6 replacement only
|
||||
applies on a whole-host (exact) match, since it can't be spliced mid-host - this pairs
|
||||
with overwrite_domain's registration guard so an IPv6 rewrite never composes an
|
||||
invalid host.
|
||||
"""
|
||||
parsed = URL(url)
|
||||
if not parsed.host:
|
||||
return url
|
||||
for target, replacement in self.domain_overwrites.items():
|
||||
if target in parsed.host:
|
||||
if _is_ipv6_literal(replacement):
|
||||
if target == parsed.host:
|
||||
return str(parsed.with_host(replacement))
|
||||
elif target in parsed.host:
|
||||
return str(parsed.with_host(parsed.host.replace(target, replacement)))
|
||||
return url
|
||||
|
||||
@@ -229,45 +272,84 @@ class ExtendedSession:
|
||||
"""all cookies in the session jar, regardless of domain binding
|
||||
|
||||
iterates the jar directly rather than filter_cookies() (which needs a url and,
|
||||
given none, returns only domain-less shared cookies — {} for any normal one).
|
||||
given none, returns only domain-less shared cookies - {} for any normal one).
|
||||
returns {} off-loop when no session was built yet (nothing could be set), mirroring
|
||||
preview() - reaching self.session would build it and need a running loop.
|
||||
"""
|
||||
if self._session is None:
|
||||
return {}
|
||||
return {c.key: c.value for c in self.session.cookie_jar}
|
||||
|
||||
def set_cookie(self, name, value, domain=None, path="/"):
|
||||
"""set a cookie in the session jar
|
||||
|
||||
domain=None stores a truly shared cookie sent to every host. domain='example.com'
|
||||
(scheme optional, defaulted to http://) scopes it to that host — a bare hostname
|
||||
(scheme optional, defaulted to http://) scopes it to that host - a bare hostname
|
||||
is normalized into a URL so the jar binds by host instead of silently storing
|
||||
another domain-less shared cookie. `path` is honored via the morsel itself.
|
||||
|
||||
raises:
|
||||
ValueError: if domain is a bare IP host (e.g. '127.0.0.1') and the jar is
|
||||
the default aiohttp.CookieJar(unsafe=False), which drops cookies bound
|
||||
to IP hosts with no store, no send, and no log - construct the session
|
||||
with cookie_jar=aiohttp.CookieJar(unsafe=True) to allow IP-host cookies
|
||||
"""
|
||||
if self._session is None:
|
||||
try:
|
||||
self._ensure_session()
|
||||
except RuntimeError as error:
|
||||
raise RuntimeError(
|
||||
"set_cookie needs the backend cookie jar, which builds on first use and "
|
||||
"requires a running event loop; call it inside an async context (unlike "
|
||||
"get_cookies/clear_cookies, which no-op off-loop)"
|
||||
) from error
|
||||
cookie = SimpleCookie()
|
||||
cookie[name] = value
|
||||
cookie[name]["path"] = path
|
||||
if domain is None:
|
||||
self.session.cookie_jar.update_cookies(cookie)
|
||||
return
|
||||
if "://" not in domain:
|
||||
domain = "http://" + domain
|
||||
cookie[name]["path"] = path
|
||||
self.session.cookie_jar.update_cookies(cookie, response_url=URL(domain))
|
||||
url = URL(domain)
|
||||
jar = self.session.cookie_jar
|
||||
if _is_ip_host(url.raw_host) and not getattr(jar, "unsafe", False):
|
||||
raise ValueError(
|
||||
f"set_cookie domain {domain!r} resolves to a bare IP host, but "
|
||||
"aiohttp's cookie jar rejects IP-address domains under unsafe=False "
|
||||
"(the default) - the cookie would be silently dropped; construct "
|
||||
"ExtendedSession with cookie_jar=aiohttp.CookieJar(unsafe=True) to "
|
||||
"allow IP-host cookies"
|
||||
)
|
||||
jar.update_cookies(cookie, response_url=url)
|
||||
|
||||
def clear_cookies(self):
|
||||
"""clear the session cookie jar"""
|
||||
"""clear the session cookie jar
|
||||
|
||||
a no-op off-loop when no session was built yet (an unbuilt jar is already empty),
|
||||
so callers can clear before the loop starts without a spurious session build.
|
||||
"""
|
||||
if self._session is None:
|
||||
return
|
||||
self.session.cookie_jar.clear()
|
||||
|
||||
def _cookies_for_url(self, url):
|
||||
"""dict of cookies the backend would send for url — override per backend (used by preview())"""
|
||||
"""dict of cookies the backend would send for url - override per backend (used by preview())"""
|
||||
return {k: v.value for k, v in self.session.cookie_jar.filter_cookies(URL(url)).items()}
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# preview
|
||||
|
||||
def preview(self, method, url, **kwargs):
|
||||
"""build a RequestPreview for a request without sending it
|
||||
"""build a RequestPreview for a request without sending it - synchronous, no loop required
|
||||
|
||||
url is domain-rewritten before cookies are resolved, matching the real
|
||||
request — cookie binding is host-based, so resolving against the pre-rewrite
|
||||
host could miss or misattribute cookies.
|
||||
request - cookie binding is host-based, so resolving against the pre-rewrite
|
||||
host could miss or misattribute cookies. the default cookie lookup never
|
||||
touches the backend session: building it needs a running loop (aiohttp>=3.14),
|
||||
which would defeat preview()'s own pre-loop inspection use case, and an unbuilt
|
||||
session has no cookies to report anyway - pass cookies= explicitly to preview
|
||||
cookies that would come from a not-yet-built session.
|
||||
"""
|
||||
url = self._apply_domain_overwrites(url)
|
||||
proxy = self._get_proxy(url, kwargs.pop("proxies", None))
|
||||
@@ -278,7 +360,7 @@ class ExtendedSession:
|
||||
timeout_total = timeout if isinstance(timeout, (int, float)) else None
|
||||
|
||||
cookies = kwargs.pop("cookies", None)
|
||||
if cookies is None:
|
||||
if cookies is None and self._session is not None:
|
||||
cookies = self._cookies_for_url(url)
|
||||
if cookies:
|
||||
cookie_header = "; ".join(f"{k}={v}" for k, v in cookies.items())
|
||||
@@ -328,12 +410,15 @@ class ExtendedSession:
|
||||
timeout; any other aiohttp.ClientError re-raised as-is, not flattened, so
|
||||
callers can branch by type or read subtype attributes like .status/.headers).
|
||||
a native proxy= is never silently clobbered by resolved proxies= (that would
|
||||
unmask the caller's real IP) — pass only one of them.
|
||||
unmask the caller's real IP) - pass only one of them.
|
||||
"""
|
||||
resolved_proxy = self._get_proxy(url, kwargs.pop("proxies", None))
|
||||
if "proxy" in kwargs and kwargs["proxy"] is not None and resolved_proxy is not None:
|
||||
if kwargs.get("proxy") is not None and resolved_proxy is not None:
|
||||
raise ValueError("pass only one of proxy= or proxies= (session/per-call), not both")
|
||||
kwargs.setdefault("proxy", resolved_proxy)
|
||||
# a bare proxy=None must not shadow a resolved proxies= (that would send direct and
|
||||
# unmask the caller's IP); setdefault can't fix it because the None key already exists
|
||||
if kwargs.get("proxy") is None:
|
||||
kwargs["proxy"] = resolved_proxy
|
||||
debug = kwargs.pop("debug", False)
|
||||
|
||||
merged = {**self._default_headers, **(kwargs.get("headers") or {})}
|
||||
@@ -341,11 +426,22 @@ class ExtendedSession:
|
||||
kwargs["headers"] = {str(k): str(v) for k, v in kwargs["headers"].items()}
|
||||
|
||||
timeout = kwargs.get("timeout")
|
||||
# the hang-guard deadline, captured as a bare number BEFORE the ClientTimeout
|
||||
# conversion below consumes it: a per-call numeric timeout, else the session
|
||||
# default. a timeout=None (omitted, or the explicit None request_with_retries
|
||||
# passes) means "use the session bound" - so it guards on self._session_timeout,
|
||||
# NOT unguarded. the ONLY unguarded case is a session deliberately constructed
|
||||
# unbounded (Session(timeout=None) -> self._session_timeout is None): the
|
||||
# documented opt-out for a genuinely unbounded call (long-poll/streaming).
|
||||
if isinstance(timeout, (int, float)):
|
||||
guard_secs = timeout
|
||||
else:
|
||||
guard_secs = self._session_timeout
|
||||
if isinstance(timeout, (int, float)):
|
||||
kwargs["timeout"] = aiohttp.ClientTimeout(total=timeout)
|
||||
elif timeout is None and "timeout" in kwargs:
|
||||
# explicit timeout=None would reach aiohttp as ClientTimeout(total=None) —
|
||||
# infinite, disabling the session default — so drop it instead (matters for
|
||||
# explicit timeout=None would reach aiohttp as ClientTimeout(total=None) -
|
||||
# infinite, disabling the session default - so drop it instead (matters for
|
||||
# request_with_retries, whose timeout kwarg defaults to None)
|
||||
del kwargs["timeout"]
|
||||
|
||||
@@ -354,19 +450,32 @@ class ExtendedSession:
|
||||
log.info("sending request to: %s", url)
|
||||
|
||||
try:
|
||||
# hang-guard: an outer asyncio deadline (session/curl timers can die in the
|
||||
# same failure that parks the transfer - a backend that loses both wakeup and
|
||||
# its own timeout enforcement would otherwise leave this await parked forever).
|
||||
# the backend's own timeout still fires first in every healthy failure (slack,
|
||||
# not replacement); wait_for only trips when the backend's enforcement is dead,
|
||||
# and its cancellation is what unwedges the orphaned transfer.
|
||||
if guard_secs is not None:
|
||||
result = await asyncio.wait_for(
|
||||
self._raw_request(method, url, **kwargs), guard_secs + _HANG_GUARD_SLACK
|
||||
)
|
||||
else:
|
||||
result = await self._raw_request(method, url, **kwargs)
|
||||
if debug and result.redirect_chain:
|
||||
log.info("redirect chain: %s", result.redirect_chain)
|
||||
return result
|
||||
except asyncio.TimeoutError as error:
|
||||
# a bare asyncio.TimeoutError isn't an aiohttp.ClientError subclass — wrap it
|
||||
# as ServerTimeoutError (both a ClientError AND a TimeoutError) so callers get
|
||||
# a typed failure and request_with_retries can still label it a timeout
|
||||
# not an aiohttp.ClientError subclass - wrap as ServerTimeoutError so callers
|
||||
# get a typed failure and request_with_retries can label it a timeout. now also
|
||||
# catches the hang-guard firing (asyncio.wait_for raises asyncio.TimeoutError).
|
||||
raise aiohttp.ServerTimeoutError(f"timeout for {url}: {error}") from error
|
||||
except aiohttp.ClientError as error:
|
||||
# re-raise the original subtype (not flattened) — request_with_retries still
|
||||
# catches the base aiohttp.ClientError below and is unaffected
|
||||
log.error("client error for %s: %s", url, error)
|
||||
except aiohttp.ClientError:
|
||||
# re-raise the original subtype (not flattened) - request_with_retries still
|
||||
# catches the base aiohttp.ClientError below and is unaffected. no log here:
|
||||
# this path RAISES, so the exception carries the failure (raise XOR log); on the
|
||||
# retrying path commons.aretry emits the per-attempt WARNING and the terminal
|
||||
# branch logs on exhaustion - logging here would double-report the same failure.
|
||||
raise
|
||||
|
||||
async def request_with_retries(
|
||||
@@ -400,7 +509,11 @@ class ExtendedSession:
|
||||
headers=headers, proxies=proxies, timeout=timeout, debug=debug,
|
||||
)
|
||||
if response.status_code in retry_statuses:
|
||||
log.warning("retryable status %s for %s", response.status_code, url)
|
||||
# debug, not warning: this RAISES _RetryStatus to trigger a retry, so it must
|
||||
# not also warn/error the same failure (raise XOR log). commons.aretry emits
|
||||
# the per-attempt WARNING with the attempt count when it catches and retries;
|
||||
# the terminal branch below logs on exhaustion.
|
||||
log.debug("retryable status %s for %s", response.status_code, url)
|
||||
raise _RetryStatus(response)
|
||||
return response
|
||||
|
||||
@@ -409,35 +522,39 @@ class ExtendedSession:
|
||||
attempt, attempts=attempts, backoff=1.0, factor=backoff_base,
|
||||
jitter=False, on=(Exception,),
|
||||
)
|
||||
# terminal path: retries are exhausted and the failure is SWALLOWED into a falsy
|
||||
# FailureResponse the caller branches on (the documented contract). rule 2 mandates a
|
||||
# log on a swallow; level is WARNING because the lib recovered cleanly into a
|
||||
# branchable value - the failing op is the caller's to escalate once it sees the falsy
|
||||
# result, so the lib does not claim ERROR on the caller's behalf. all four branches are
|
||||
# the same event class (exhausted retries) and stay at one uniform level.
|
||||
except _RetryStatus as exhausted:
|
||||
log.error("all %d attempts failed for %s (last status %s)",
|
||||
log.warning("all %d attempts failed for %s (last status %s)",
|
||||
attempts, url, exhausted.response.status_code)
|
||||
return exhausted.response
|
||||
except asyncio.TimeoutError:
|
||||
# request() wraps timeouts as ServerTimeoutError (ClientError + TimeoutError);
|
||||
# catch it first so it's labeled a timeout, not a generic client error below
|
||||
log.error("all %d attempts timed out for %s", attempts, url)
|
||||
# catch before ClientError so a timeout is labeled as such, not generic
|
||||
log.warning("all %d attempts timed out for %s", attempts, url)
|
||||
return FailureResponse(reason="timeout", url=url)
|
||||
except aiohttp.ClientError as error:
|
||||
log.error("all %d attempts failed for %s (client error: %s)", attempts, url, error)
|
||||
log.warning("all %d attempts failed for %s (client error: %s)", attempts, url, error)
|
||||
return FailureResponse(reason=f"client error: {error}", url=url)
|
||||
except Exception as error:
|
||||
log.error("all %d attempts failed for %s (unexpected: %s)", attempts, url, error)
|
||||
log.warning("all %d attempts failed for %s (unexpected: %s)", attempts, url, error)
|
||||
return FailureResponse(reason=f"unexpected error: {error}", url=url)
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# lifecycle
|
||||
|
||||
async def close(self):
|
||||
"""close the backend session — override if the backend's close differs; no-op if never built"""
|
||||
"""close the backend session - override if the backend's close differs; no-op if never built"""
|
||||
if self._session is not None:
|
||||
await self._session.close()
|
||||
|
||||
def _is_closed(self) -> bool:
|
||||
"""whether the backend session is closed — override for non-aiohttp backends
|
||||
"""whether the backend session is closed - override for non-aiohttp backends
|
||||
|
||||
an unbuilt session counts as closed: nothing was opened, nothing to leak,
|
||||
__del__ should not warn.
|
||||
an unbuilt session counts as closed: nothing was opened, nothing to leak.
|
||||
"""
|
||||
if self._session is None:
|
||||
return True
|
||||
@@ -450,7 +567,7 @@ class ExtendedSession:
|
||||
await self.close()
|
||||
|
||||
def __del__(self):
|
||||
# no async cleanup here — spinning event loops in a finalizer is unsafe;
|
||||
# no async cleanup here - spinning event loops in a finalizer is unsafe;
|
||||
# just warn so the leak is visible, callers must close explicitly
|
||||
try:
|
||||
closed = self._is_closed()
|
||||
|
||||
Reference in New Issue
Block a user