Compare commits
5
Commits
v0.1.3
...
6d4183948e
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6d4183948e | ||
|
|
90f67cf9fa | ||
|
|
585a432ae0 | ||
|
|
3d86fc249c | ||
|
|
ef20bc51f0 |
+1
-1
@@ -1,5 +1,5 @@
|
||||
# claude
|
||||
CLAUDE.md
|
||||
.claude/
|
||||
|
||||
# python
|
||||
__pycache__/
|
||||
|
||||
@@ -13,16 +13,16 @@ send to the core — inheriting rotation, proxy, retry, and result for free.
|
||||
## Install
|
||||
|
||||
```
|
||||
aiowebhooks @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiowebhooks.git@v0.1.3
|
||||
aiowebhooks @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiowebhooks.git@v0.1.6
|
||||
# discord embeds / identity helpers need the extra:
|
||||
aiowebhooks[discord] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiowebhooks.git@v0.1.3
|
||||
aiowebhooks[discord] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiowebhooks.git@v0.1.6
|
||||
```
|
||||
|
||||
The base pulls `aiohttp` and `commons` (for the retry/backoff engine). Only
|
||||
`aiowebhooks[discord]` adds `discord.py` (>=2.3, mainline — not discord.py-self), and
|
||||
only for `DiscordWebhook`.
|
||||
|
||||
Drop the `@v0.1.3` suffix from the line above to install the latest unpinned.
|
||||
Drop the `@v0.1.6` suffix from the line above to install the latest unpinned.
|
||||
|
||||
## Core sender
|
||||
|
||||
@@ -72,9 +72,17 @@ result.proxy # canonical proxy string used (host:port:user:pass / host:port)
|
||||
|
||||
Status retries run through `commons.aretry` (exponential backoff + cap):
|
||||
|
||||
- **429** — honors the `retry_after` from the body first (Discord sends seconds), then
|
||||
the `Retry-After` header, sleeping that exact value; capped by `max_retries`.
|
||||
- **429** — always retried, capped by `max_retries`. When a wait is parseable (body
|
||||
`retry_after` first — Discord sends seconds — then the `Retry-After` header) it sleeps
|
||||
that value before retrying; a 429 with no parseable wait (edge/Cloudflare/generic
|
||||
webhook) still retries under aretry's backoff rather than failing one-shot. A
|
||||
non-finite value (`inf`/`nan`) is rejected outright and a finite value is clamped to
|
||||
`MAX_RETRY_AFTER` (300s) — a server-controlled wait can never stall a `send()` past
|
||||
that ceiling, however large or malformed the value it sends.
|
||||
- **5xx** — retried with exponential backoff, capped by `max_retries`.
|
||||
- **connection/timeout errors** — retried with exponential backoff, capped by
|
||||
`max_retries`, same as a 5xx (with no proxy provider; see below for the
|
||||
proxy-rotation path).
|
||||
- **4xx** (other than 429) — fails immediately (no retry), returned as `ok=False`.
|
||||
|
||||
Exceeding a cap returns a failed result rather than looping — and the result carries the
|
||||
@@ -102,7 +110,8 @@ On a timeout/connection error the current proxy is burned and the next is tried,
|
||||
`max_proxy_retries`. Hitting the cap, or **any exception from the provider's
|
||||
`get()`/`burn()`** (the provider is duck-typed and never imported, so its exception
|
||||
types can't be caught by class), returns a failed result — never an infinite loop, never
|
||||
an escape. With no provider, a timeout just fails after normal retry.
|
||||
an escape. With no provider, a timeout/connection error is retried directly under
|
||||
`max_retries` (the same cap and backoff a 5xx gets) instead of burn+rotate.
|
||||
|
||||
## Discord (`aiowebhooks[discord]`)
|
||||
|
||||
@@ -139,6 +148,46 @@ Without the extra installed, importing `aiowebhooks` still works; constructing o
|
||||
|
||||
## Changelog
|
||||
|
||||
### v0.1.6
|
||||
|
||||
- **429 `retry_after` no longer sleeps on an exhausted final attempt:** the wait is
|
||||
now carried to the START of the next attempt instead of slept immediately after
|
||||
seeing the 429. Previously the last (exhausted) attempt slept the full
|
||||
`retry_after` before giving up — a pointless wait since no retry followed. The
|
||||
wait is still honored (additive with aretry's backoff) before every attempt that
|
||||
actually runs.
|
||||
- Docs: `errors.py` no longer implies the missing-`[discord]`-extra raise is one of
|
||||
the exported error types — it's a plain `RuntimeError`.
|
||||
- Docstrings/comments compressed; no behavior change.
|
||||
|
||||
### v0.1.5
|
||||
|
||||
- **429 `retry_after` bounded:** a non-finite server-controlled wait (`inf`/`nan`, from
|
||||
the body `retry_after` or the `Retry-After` header) is now rejected outright, and a
|
||||
finite wait is clamped to `MAX_RETRY_AFTER` (300s). Previously a bare `float()` parse
|
||||
slept the value verbatim, unbounded and outside `max_retries` accounting — an
|
||||
adversarial or ms-vs-s-misconfigured server could stall a `send()` for hours.
|
||||
- **Connection/timeout errors now retry without a proxy provider:** previously
|
||||
`aiohttp.ClientError`/`asyncio.TimeoutError` with no `proxies=` set failed one-shot,
|
||||
contradicting both the README and the single most retry-worthy failure class. Now it
|
||||
retries under `commons.aretry`'s backoff, capped by `max_retries`, same as a 5xx —
|
||||
and still returns `ok=False` (never raises) once retries are exhausted. Proxy-rotation
|
||||
behavior (burn + rotate, capped by `max_proxy_retries`) is unchanged when a provider
|
||||
is set.
|
||||
|
||||
### v0.1.4
|
||||
|
||||
- **Never-raises net widened:** an unexpected exception that escapes a send attempt (a
|
||||
closed injected session → `RuntimeError`, a malformed proxy URL → `ValueError`) now
|
||||
converts to a falsy `WebhookResult(ok=False, ...)` instead of propagating out of
|
||||
`send()`, restoring the documented contract for those edge triggers.
|
||||
|
||||
### v0.1.3
|
||||
|
||||
- **429 always retries:** every `429` is now retryable under aretry's backoff + cap, not
|
||||
only those with a parseable `retry_after`. A 429 with no body `retry_after` and no
|
||||
`Retry-After` header (edge/Cloudflare/generic webhook) previously failed one-shot.
|
||||
|
||||
### v0.1.2
|
||||
|
||||
- Removed a dead `clock` constructor param (it was stored but never used). Pinned
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "aiowebhooks"
|
||||
version = "0.1.3"
|
||||
version = "0.1.6"
|
||||
description = "async webhook sender (aiohttp) with round-robin urls, retry, and proxy rotation; optional discord.py embeds"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
|
||||
@@ -21,4 +21,4 @@ from .sender import Webhook
|
||||
|
||||
__all__ = ["Webhook", "WebhookResult", "WebhookError", "NoUrlsError"]
|
||||
|
||||
__version__ = "0.1.3"
|
||||
__version__ = "0.1.6"
|
||||
|
||||
@@ -3,8 +3,9 @@
|
||||
these are surfaced for callers that want to branch on a specific failure cause.
|
||||
note the core `Webhook.send` does NOT raise on a send failure — it returns a
|
||||
`WebhookResult` with `ok=False` and the cause captured in `error`. these types
|
||||
exist for the few raise paths (bad construction, missing extra) and as a base for
|
||||
any future raising surface.
|
||||
cover bad construction (e.g. `NoUrlsError`) and serve as a base for any future
|
||||
raising surface; the missing-`[discord]`-extra path raises a plain `RuntimeError`,
|
||||
not one of these.
|
||||
"""
|
||||
|
||||
|
||||
|
||||
+99
-41
@@ -1,13 +1,20 @@
|
||||
"""core async webhook sender (aiohttp only, no discord knowledge).
|
||||
|
||||
`Webhook` posts a JSON dict to a url (or round-robins a pool), handling 429/5xx
|
||||
retries and optional proxy rotation, and always returns a `WebhookResult` — it
|
||||
never raises on a send failure. the discord layer builds payloads and delegates
|
||||
the actual POST here so it inherits rotation / proxy / retry / result.
|
||||
retries, connection/timeout retries, and optional proxy rotation, and always
|
||||
returns a `WebhookResult` — it never raises on a send failure. the discord layer
|
||||
builds payloads and delegates the actual POST here so it inherits rotation / proxy
|
||||
/ retry / result.
|
||||
|
||||
a 429's server-controlled `retry_after` wait is bounded: non-finite values
|
||||
(inf/nan) are rejected and finite values are clamped to `MAX_RETRY_AFTER`, so an
|
||||
adversarial or misconfigured server can never stall a send() past a bounded
|
||||
ceiling.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import math
|
||||
from typing import Dict, List, Optional, Union
|
||||
from urllib.parse import unquote, urlsplit
|
||||
|
||||
@@ -19,6 +26,14 @@ from .result import WebhookResult
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
MAX_RETRY_AFTER = 300.0
|
||||
"""ceiling (seconds) honored from a server-controlled 429 retry_after/Retry-After
|
||||
|
||||
guards against an arbitrarily large or non-finite wait (inf/nan, or a ms-vs-s unit
|
||||
mismatch) stalling a send() past its ClientTimeout. non-finite values are rejected
|
||||
outright; finite values are clamped to this ceiling.
|
||||
"""
|
||||
|
||||
|
||||
class _Retryable(Exception):
|
||||
"""internal signal: a retryable HTTP status (429/5xx); carries the response
|
||||
@@ -90,18 +105,26 @@ class Webhook:
|
||||
|
||||
@staticmethod
|
||||
def _retry_after(status: int, headers, body) -> Optional[float]:
|
||||
"""seconds to wait on a 429, from body retry_after then Retry-After header"""
|
||||
"""seconds to wait on a 429, from body retry_after then Retry-After header
|
||||
|
||||
non-finite values (inf/nan) are rejected as unparseable; finite values are
|
||||
clamped to MAX_RETRY_AFTER.
|
||||
"""
|
||||
if status != 429:
|
||||
return None
|
||||
if isinstance(body, dict) and body.get("retry_after") is not None:
|
||||
try:
|
||||
return float(body["retry_after"])
|
||||
value = float(body["retry_after"])
|
||||
if math.isfinite(value):
|
||||
return max(0.0, min(value, MAX_RETRY_AFTER))
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
header = headers.get("Retry-After") if headers else None
|
||||
if header is not None:
|
||||
try:
|
||||
return float(header)
|
||||
value = float(header)
|
||||
if math.isfinite(value):
|
||||
return max(0.0, min(value, MAX_RETRY_AFTER))
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
return None
|
||||
@@ -110,9 +133,10 @@ class Webhook:
|
||||
"""post a json payload to the next pool url; always returns a result
|
||||
|
||||
handles 429 (wait + retry, capped by max_retries), 5xx (retry, capped), and
|
||||
timeout/connection errors with optional proxy rotation (burn + next, capped
|
||||
by max_proxy_retries). 4xx other than 429 fail immediately. never raises on a
|
||||
send failure.
|
||||
timeout/connection errors: with a proxy provider, burn + rotate to the next
|
||||
proxy (capped by max_proxy_retries) before failing; without one, retried
|
||||
directly under max_retries like a 5xx. 4xx other than 429 fail immediately.
|
||||
never raises on a send failure.
|
||||
"""
|
||||
url = self._next_url()
|
||||
session = self._session
|
||||
@@ -130,41 +154,66 @@ class Webhook:
|
||||
) -> WebhookResult:
|
||||
"""status-retry (via commons.aretry) wrapping proxy rotation; never raises
|
||||
|
||||
commons.aretry owns the 429/5xx backoff schedule + retry cap (max_retries),
|
||||
retrying on the internal _Retryable signal. on exhaustion it re-raises the
|
||||
last _Retryable, whose carried result is the REAL last response (not a
|
||||
synthetic status-0). proxy rotation on connection errors lives inside the
|
||||
attempt and is capped separately.
|
||||
commons.aretry owns the 429/5xx/no-provider-connection-error backoff schedule
|
||||
+ retry cap (max_retries), retrying on the internal _Retryable signal; on
|
||||
exhaustion it re-raises the last _Retryable, whose carried result is the real
|
||||
last response (not a synthetic status-0). proxy rotation on connection errors
|
||||
lives inside the attempt, capped separately (max_proxy_retries); once that cap
|
||||
is hit the failure is terminal, not retried again via aretry.
|
||||
"""
|
||||
counter = [0]
|
||||
pending_wait: List[Optional[float]] = [None]
|
||||
try:
|
||||
return await aretry(
|
||||
lambda: self._attempt(session, url, payload, counter),
|
||||
lambda: self._attempt(session, url, payload, counter, pending_wait),
|
||||
attempts=self.max_retries + 1,
|
||||
on=(_Retryable,),
|
||||
)
|
||||
except _Retryable as exhausted:
|
||||
return exhausted.result
|
||||
except Exception as error:
|
||||
# never-raises safety net: anything not aiohttp.ClientError/TimeoutError
|
||||
# (closed injected session, malformed proxy url, ...) comes back as a
|
||||
# failed result instead of escaping send()
|
||||
log.warning("webhook send failed unexpectedly on %s: %s", url, error, exc_info=True)
|
||||
return WebhookResult(
|
||||
ok=False, status=None, url=url, attempts=counter[0] or 1,
|
||||
error=f"{type(error).__name__}: {error}",
|
||||
)
|
||||
|
||||
async def _attempt(
|
||||
self, session: aiohttp.ClientSession, url: str, payload: Dict, counter: List[int]
|
||||
self, session: aiohttp.ClientSession, url: str, payload: Dict, counter: List[int],
|
||||
pending_wait: List[Optional[float]],
|
||||
) -> WebhookResult:
|
||||
"""one logical send: proxy rotation + a single POST; may raise _Retryable
|
||||
|
||||
raises _Retryable (carrying the real response) on a 429/5xx so the caller's
|
||||
aretry applies backoff; honors an explicit 429 retry_after by sleeping it
|
||||
before signalling. returns a final WebhookResult on success or a terminal
|
||||
(non-retryable) failure — never lets a provider/connection error escape.
|
||||
aretry applies backoff. an explicit 429 retry_after is carried over and slept
|
||||
at the START of the next attempt, never after the one that raises, so an
|
||||
exhausted final attempt never sleeps a wait it won't use. a connection/timeout
|
||||
error also raises _Retryable when no proxy provider is set (retries under
|
||||
aretry like a 5xx); with a provider it instead drives burn+rotate up to
|
||||
max_proxy_retries and only becomes terminal once that cap is hit. always
|
||||
returns a final WebhookResult or raises _Retryable — never lets a provider
|
||||
error escape.
|
||||
|
||||
`counter` is a per-call mutable cell ([0]) owned by the calling `_send_loop`,
|
||||
so the attempt count is local to one `send()` and concurrent sends on the
|
||||
same instance don't corrupt each other's tally.
|
||||
`counter`/`pending_wait` are per-call mutable cells ([0] / [None]) owned by
|
||||
`_send_loop` and threaded through because aretry calls `_attempt` fresh on
|
||||
every top-level retry, so a plain local wouldn't survive across calls.
|
||||
`counter` tallies attempts; `pending_wait` carries a 429 retry_after to the
|
||||
next attempt's sleep. keeps both local to one `send()`, safe for concurrent
|
||||
sends on the same instance.
|
||||
"""
|
||||
timeout = aiohttp.ClientTimeout(total=self.timeout)
|
||||
last_proxy: Optional[str] = None
|
||||
proxy_tries = 0
|
||||
|
||||
while True:
|
||||
if pending_wait[0] is not None:
|
||||
wait = pending_wait[0]
|
||||
pending_wait[0] = None
|
||||
log.warning("webhook 429 on %s; honoring retry_after %.3fs", url, wait)
|
||||
await asyncio.sleep(wait)
|
||||
counter[0] += 1
|
||||
attempts = counter[0]
|
||||
proxy_url = None
|
||||
@@ -172,8 +221,7 @@ class Webhook:
|
||||
try:
|
||||
proxy_dict = self._proxies.get()
|
||||
except Exception:
|
||||
# duck-typed provider; any error from get() means no proxy is
|
||||
# available — fail cleanly rather than escaping send().
|
||||
# duck-typed provider: any get() error means no proxy available
|
||||
log.warning("webhook: proxy get() failed; no proxy available",
|
||||
exc_info=True)
|
||||
return WebhookResult(
|
||||
@@ -202,13 +250,16 @@ class Webhook:
|
||||
)
|
||||
|
||||
if status == 429:
|
||||
# every 429 is retryable; honor an explicit retry_after by
|
||||
# sleeping it, but a 429 with no parseable wait (edge/Cloudflare/
|
||||
# generic webhook) still retries under aretry's backoff + cap
|
||||
# every 429 is retryable; an explicit retry_after is carried
|
||||
# to the NEXT attempt's sleep (never slept here), so an
|
||||
# exhausted final attempt never sleeps a wait it won't use.
|
||||
# no parseable wait still retries under aretry's backoff+cap;
|
||||
# an honored wait is additive with that backoff (over-waits,
|
||||
# never under-waits). _retry_after rejects non-finite values
|
||||
# and clamps finite ones to MAX_RETRY_AFTER.
|
||||
wait = self._retry_after(status, resp.headers, body)
|
||||
if wait is not None:
|
||||
log.warning("webhook 429 on %s; honoring retry_after %.3fs", url, wait)
|
||||
await asyncio.sleep(wait)
|
||||
pending_wait[0] = wait
|
||||
else:
|
||||
log.warning("webhook 429 on %s; no retry_after, backing off", url)
|
||||
raise _Retryable(result)
|
||||
@@ -217,27 +268,34 @@ class Webhook:
|
||||
return result
|
||||
|
||||
except (aiohttp.ClientError, asyncio.TimeoutError) as error:
|
||||
if self._proxies is not None and proxy_tries < self.max_proxy_retries:
|
||||
if self._burn(last_proxy):
|
||||
proxy_tries += 1
|
||||
continue
|
||||
if self._proxies is not None:
|
||||
if proxy_tries < self.max_proxy_retries:
|
||||
if self._burn(last_proxy):
|
||||
proxy_tries += 1
|
||||
continue
|
||||
return WebhookResult(
|
||||
ok=False, status=None, url=url, attempts=attempts,
|
||||
error="proxies exhausted", proxy=last_proxy,
|
||||
)
|
||||
return WebhookResult(
|
||||
ok=False, status=None, url=url, attempts=attempts,
|
||||
error="proxies exhausted", proxy=last_proxy,
|
||||
error=f"{type(error).__name__}: {error}", proxy=last_proxy,
|
||||
)
|
||||
return WebhookResult(
|
||||
# no proxy provider: retry a connection/timeout error like a 5xx,
|
||||
# under aretry's backoff + max_retries instead of failing one-shot;
|
||||
# _send_loop's `except _Retryable` returns the carried result once
|
||||
# exhausted, so never-raises still holds.
|
||||
raise _Retryable(WebhookResult(
|
||||
ok=False, status=None, url=url, attempts=attempts,
|
||||
error=f"{type(error).__name__}: {error}", proxy=last_proxy,
|
||||
)
|
||||
))
|
||||
|
||||
def _burn(self, proxy: Optional[str]) -> bool:
|
||||
"""burn the current proxy; return False if it can't be rotated
|
||||
|
||||
the provider is duck-typed and never imported, so we cannot catch its
|
||||
exception types by class. ANY exception from burn (a ProxiesExhaustedError
|
||||
on a dead pool, a ValueError when the proxy isn't in the pool, etc.) means
|
||||
we can't rotate — return False so the caller ends the loop with a failed
|
||||
result rather than letting it escape send() (which must never raise).
|
||||
the provider is duck-typed and never imported, so any exception from burn
|
||||
(dead pool, unknown proxy, ...) is caught broadly and means "can't rotate" —
|
||||
return False rather than let it escape send().
|
||||
"""
|
||||
try:
|
||||
self._proxies.burn(proxy)
|
||||
|
||||
Reference in New Issue
Block a user