fix: bound 429 retry_after and retry connection errors without a proxy (v0.1.5)
A server-controlled 429 retry_after/Retry-After was slept verbatim with no finiteness check or ceiling, so an inf or huge value (adversarial or a ms-vs-s unit mismatch) could stall send() for hours outside max_retries accounting. Non-finite values are now rejected and finite values clamped to MAX_RETRY_AFTER (300s). Connection/timeout errors with no proxy provider set failed one-shot, contradicting the README's documented "normal retry" behavior and skipping the single most retry-worthy failure class. They now raise the internal _Retryable signal so commons.aretry retries them under max_retries, same as a 5xx, while still returning a failed WebhookResult (never raising) once retries are exhausted. Signed-off-by: disqualifier <dev@disqualifier.me>
This commit is contained in:
@@ -13,16 +13,16 @@ send to the core — inheriting rotation, proxy, retry, and result for free.
|
|||||||
## Install
|
## Install
|
||||||
|
|
||||||
```
|
```
|
||||||
aiowebhooks @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiowebhooks.git@v0.1.4
|
aiowebhooks @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiowebhooks.git@v0.1.5
|
||||||
# discord embeds / identity helpers need the extra:
|
# discord embeds / identity helpers need the extra:
|
||||||
aiowebhooks[discord] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiowebhooks.git@v0.1.4
|
aiowebhooks[discord] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiowebhooks.git@v0.1.5
|
||||||
```
|
```
|
||||||
|
|
||||||
The base pulls `aiohttp` and `commons` (for the retry/backoff engine). Only
|
The base pulls `aiohttp` and `commons` (for the retry/backoff engine). Only
|
||||||
`aiowebhooks[discord]` adds `discord.py` (>=2.3, mainline — not discord.py-self), and
|
`aiowebhooks[discord]` adds `discord.py` (>=2.3, mainline — not discord.py-self), and
|
||||||
only for `DiscordWebhook`.
|
only for `DiscordWebhook`.
|
||||||
|
|
||||||
Drop the `@v0.1.4` suffix from the line above to install the latest unpinned.
|
Drop the `@v0.1.5` suffix from the line above to install the latest unpinned.
|
||||||
|
|
||||||
## Core sender
|
## Core sender
|
||||||
|
|
||||||
@@ -75,8 +75,14 @@ Status retries run through `commons.aretry` (exponential backoff + cap):
|
|||||||
- **429** — always retried, capped by `max_retries`. When a wait is parseable (body
|
- **429** — always retried, capped by `max_retries`. When a wait is parseable (body
|
||||||
`retry_after` first — Discord sends seconds — then the `Retry-After` header) it sleeps
|
`retry_after` first — Discord sends seconds — then the `Retry-After` header) it sleeps
|
||||||
that value before retrying; a 429 with no parseable wait (edge/Cloudflare/generic
|
that value before retrying; a 429 with no parseable wait (edge/Cloudflare/generic
|
||||||
webhook) still retries under aretry's backoff rather than failing one-shot.
|
webhook) still retries under aretry's backoff rather than failing one-shot. A
|
||||||
|
non-finite value (`inf`/`nan`) is rejected outright and a finite value is clamped to
|
||||||
|
`MAX_RETRY_AFTER` (300s) — a server-controlled wait can never stall a `send()` past
|
||||||
|
that ceiling, however large or malformed the value it sends.
|
||||||
- **5xx** — retried with exponential backoff, capped by `max_retries`.
|
- **5xx** — retried with exponential backoff, capped by `max_retries`.
|
||||||
|
- **connection/timeout errors** — retried with exponential backoff, capped by
|
||||||
|
`max_retries`, same as a 5xx (with no proxy provider; see below for the
|
||||||
|
proxy-rotation path).
|
||||||
- **4xx** (other than 429) — fails immediately (no retry), returned as `ok=False`.
|
- **4xx** (other than 429) — fails immediately (no retry), returned as `ok=False`.
|
||||||
|
|
||||||
Exceeding a cap returns a failed result rather than looping — and the result carries the
|
Exceeding a cap returns a failed result rather than looping — and the result carries the
|
||||||
@@ -104,7 +110,8 @@ On a timeout/connection error the current proxy is burned and the next is tried,
|
|||||||
`max_proxy_retries`. Hitting the cap, or **any exception from the provider's
|
`max_proxy_retries`. Hitting the cap, or **any exception from the provider's
|
||||||
`get()`/`burn()`** (the provider is duck-typed and never imported, so its exception
|
`get()`/`burn()`** (the provider is duck-typed and never imported, so its exception
|
||||||
types can't be caught by class), returns a failed result — never an infinite loop, never
|
types can't be caught by class), returns a failed result — never an infinite loop, never
|
||||||
an escape. With no provider, a timeout just fails after normal retry.
|
an escape. With no provider, a timeout/connection error is retried directly under
|
||||||
|
`max_retries` (the same cap and backoff a 5xx gets) instead of burn+rotate.
|
||||||
|
|
||||||
## Discord (`aiowebhooks[discord]`)
|
## Discord (`aiowebhooks[discord]`)
|
||||||
|
|
||||||
@@ -141,6 +148,21 @@ Without the extra installed, importing `aiowebhooks` still works; constructing o
|
|||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
### v0.1.5
|
||||||
|
|
||||||
|
- **429 `retry_after` bounded:** a non-finite server-controlled wait (`inf`/`nan`, from
|
||||||
|
the body `retry_after` or the `Retry-After` header) is now rejected outright, and a
|
||||||
|
finite wait is clamped to `MAX_RETRY_AFTER` (300s). Previously a bare `float()` parse
|
||||||
|
slept the value verbatim, unbounded and outside `max_retries` accounting — an
|
||||||
|
adversarial or ms-vs-s-misconfigured server could stall a `send()` for hours.
|
||||||
|
- **Connection/timeout errors now retry without a proxy provider:** previously
|
||||||
|
`aiohttp.ClientError`/`asyncio.TimeoutError` with no `proxies=` set failed one-shot,
|
||||||
|
contradicting both the README and the single most retry-worthy failure class. Now it
|
||||||
|
retries under `commons.aretry`'s backoff, capped by `max_retries`, same as a 5xx —
|
||||||
|
and still returns `ok=False` (never raises) once retries are exhausted. Proxy-rotation
|
||||||
|
behavior (burn + rotate, capped by `max_proxy_retries`) is unchanged when a provider
|
||||||
|
is set.
|
||||||
|
|
||||||
### v0.1.4
|
### v0.1.4
|
||||||
|
|
||||||
- **Never-raises net widened:** an unexpected exception that escapes a send attempt (a
|
- **Never-raises net widened:** an unexpected exception that escapes a send attempt (a
|
||||||
|
|||||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "aiowebhooks"
|
name = "aiowebhooks"
|
||||||
version = "0.1.4"
|
version = "0.1.5"
|
||||||
description = "async webhook sender (aiohttp) with round-robin urls, retry, and proxy rotation; optional discord.py embeds"
|
description = "async webhook sender (aiohttp) with round-robin urls, retry, and proxy rotation; optional discord.py embeds"
|
||||||
requires-python = ">=3.10"
|
requires-python = ">=3.10"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
|
|||||||
+73
-24
@@ -1,13 +1,20 @@
|
|||||||
"""core async webhook sender (aiohttp only, no discord knowledge).
|
"""core async webhook sender (aiohttp only, no discord knowledge).
|
||||||
|
|
||||||
`Webhook` posts a JSON dict to a url (or round-robins a pool), handling 429/5xx
|
`Webhook` posts a JSON dict to a url (or round-robins a pool), handling 429/5xx
|
||||||
retries and optional proxy rotation, and always returns a `WebhookResult` — it
|
retries, connection/timeout retries, and optional proxy rotation, and always
|
||||||
never raises on a send failure. the discord layer builds payloads and delegates
|
returns a `WebhookResult` — it never raises on a send failure. the discord layer
|
||||||
the actual POST here so it inherits rotation / proxy / retry / result.
|
builds payloads and delegates the actual POST here so it inherits rotation / proxy
|
||||||
|
/ retry / result.
|
||||||
|
|
||||||
|
a 429's server-controlled `retry_after` wait is bounded: non-finite values
|
||||||
|
(inf/nan) are rejected and finite values are clamped to `MAX_RETRY_AFTER`, so an
|
||||||
|
adversarial or misconfigured server can never stall a send() past a bounded
|
||||||
|
ceiling.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import logging
|
import logging
|
||||||
|
import math
|
||||||
from typing import Dict, List, Optional, Union
|
from typing import Dict, List, Optional, Union
|
||||||
from urllib.parse import unquote, urlsplit
|
from urllib.parse import unquote, urlsplit
|
||||||
|
|
||||||
@@ -19,6 +26,17 @@ from .result import WebhookResult
|
|||||||
|
|
||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
MAX_RETRY_AFTER = 300.0
|
||||||
|
"""ceiling (seconds) honored from a server-controlled 429 retry_after/Retry-After
|
||||||
|
|
||||||
|
a server can send an arbitrarily large or non-finite wait (float('inf'), a
|
||||||
|
Cloudflare-scale value, or a ms-vs-s unit mismatch turning 5s into 5000s); sleeping
|
||||||
|
that verbatim would stall a send() far past its ClientTimeout and outside
|
||||||
|
max_retries accounting, breaking the 'always returns a WebhookResult' contract.
|
||||||
|
non-finite values are rejected outright (treated as unparseable); finite values are
|
||||||
|
clamped to this ceiling.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
class _Retryable(Exception):
|
class _Retryable(Exception):
|
||||||
"""internal signal: a retryable HTTP status (429/5xx); carries the response
|
"""internal signal: a retryable HTTP status (429/5xx); carries the response
|
||||||
@@ -90,18 +108,27 @@ class Webhook:
|
|||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _retry_after(status: int, headers, body) -> Optional[float]:
|
def _retry_after(status: int, headers, body) -> Optional[float]:
|
||||||
"""seconds to wait on a 429, from body retry_after then Retry-After header"""
|
"""seconds to wait on a 429, from body retry_after then Retry-After header
|
||||||
|
|
||||||
|
non-finite values (inf/nan) are rejected as unparseable; finite values are
|
||||||
|
clamped to MAX_RETRY_AFTER so a server-controlled wait can never stall a
|
||||||
|
send() past a bounded ceiling.
|
||||||
|
"""
|
||||||
if status != 429:
|
if status != 429:
|
||||||
return None
|
return None
|
||||||
if isinstance(body, dict) and body.get("retry_after") is not None:
|
if isinstance(body, dict) and body.get("retry_after") is not None:
|
||||||
try:
|
try:
|
||||||
return float(body["retry_after"])
|
value = float(body["retry_after"])
|
||||||
|
if math.isfinite(value):
|
||||||
|
return max(0.0, min(value, MAX_RETRY_AFTER))
|
||||||
except (TypeError, ValueError):
|
except (TypeError, ValueError):
|
||||||
pass
|
pass
|
||||||
header = headers.get("Retry-After") if headers else None
|
header = headers.get("Retry-After") if headers else None
|
||||||
if header is not None:
|
if header is not None:
|
||||||
try:
|
try:
|
||||||
return float(header)
|
value = float(header)
|
||||||
|
if math.isfinite(value):
|
||||||
|
return max(0.0, min(value, MAX_RETRY_AFTER))
|
||||||
except (TypeError, ValueError):
|
except (TypeError, ValueError):
|
||||||
pass
|
pass
|
||||||
return None
|
return None
|
||||||
@@ -110,9 +137,10 @@ class Webhook:
|
|||||||
"""post a json payload to the next pool url; always returns a result
|
"""post a json payload to the next pool url; always returns a result
|
||||||
|
|
||||||
handles 429 (wait + retry, capped by max_retries), 5xx (retry, capped), and
|
handles 429 (wait + retry, capped by max_retries), 5xx (retry, capped), and
|
||||||
timeout/connection errors with optional proxy rotation (burn + next, capped
|
timeout/connection errors: with a proxy provider, burn + rotate to the next
|
||||||
by max_proxy_retries). 4xx other than 429 fail immediately. never raises on a
|
proxy (capped by max_proxy_retries) before failing; without one, retried
|
||||||
send failure.
|
directly under max_retries like a 5xx. 4xx other than 429 fail immediately.
|
||||||
|
never raises on a send failure.
|
||||||
"""
|
"""
|
||||||
url = self._next_url()
|
url = self._next_url()
|
||||||
session = self._session
|
session = self._session
|
||||||
@@ -130,11 +158,13 @@ class Webhook:
|
|||||||
) -> WebhookResult:
|
) -> WebhookResult:
|
||||||
"""status-retry (via commons.aretry) wrapping proxy rotation; never raises
|
"""status-retry (via commons.aretry) wrapping proxy rotation; never raises
|
||||||
|
|
||||||
commons.aretry owns the 429/5xx backoff schedule + retry cap (max_retries),
|
commons.aretry owns the 429/5xx/no-provider-connection-error backoff schedule
|
||||||
retrying on the internal _Retryable signal. on exhaustion it re-raises the
|
+ retry cap (max_retries), retrying on the internal _Retryable signal. on
|
||||||
last _Retryable, whose carried result is the REAL last response (not a
|
exhaustion it re-raises the last _Retryable, whose carried result is the REAL
|
||||||
synthetic status-0). proxy rotation on connection errors lives inside the
|
last response (not a synthetic status-0). proxy rotation on connection errors
|
||||||
attempt and is capped separately.
|
lives inside the attempt and is capped separately (max_proxy_retries); once a
|
||||||
|
provider's rotation cap is hit that failure is terminal, not retried again
|
||||||
|
via aretry.
|
||||||
"""
|
"""
|
||||||
counter = [0]
|
counter = [0]
|
||||||
try:
|
try:
|
||||||
@@ -163,8 +193,12 @@ class Webhook:
|
|||||||
|
|
||||||
raises _Retryable (carrying the real response) on a 429/5xx so the caller's
|
raises _Retryable (carrying the real response) on a 429/5xx so the caller's
|
||||||
aretry applies backoff; honors an explicit 429 retry_after by sleeping it
|
aretry applies backoff; honors an explicit 429 retry_after by sleeping it
|
||||||
before signalling. returns a final WebhookResult on success or a terminal
|
before signalling. a connection/timeout error also raises _Retryable when no
|
||||||
(non-retryable) failure — never lets a provider/connection error escape.
|
proxy provider is set, so it retries under aretry's backoff + max_retries
|
||||||
|
instead of failing one-shot; with a provider, the same error first drives
|
||||||
|
burn+rotate up to max_proxy_retries, and only becomes terminal (not retried
|
||||||
|
via aretry) once that cap is hit. returns a final WebhookResult on success or
|
||||||
|
a terminal (non-retryable) failure — never lets a provider error escape.
|
||||||
|
|
||||||
`counter` is a per-call mutable cell ([0]) owned by the calling `_send_loop`,
|
`counter` is a per-call mutable cell ([0]) owned by the calling `_send_loop`,
|
||||||
so the attempt count is local to one `send()` and concurrent sends on the
|
so the attempt count is local to one `send()` and concurrent sends on the
|
||||||
@@ -217,7 +251,10 @@ class Webhook:
|
|||||||
# generic webhook) still retries under aretry's backoff + cap.
|
# generic webhook) still retries under aretry's backoff + cap.
|
||||||
# note: aretry ALSO sleeps its backoff between retries, so an
|
# note: aretry ALSO sleeps its backoff between retries, so an
|
||||||
# honored retry_after is additive (retry_after + backoff) — this
|
# honored retry_after is additive (retry_after + backoff) — this
|
||||||
# only ever over-waits, never under-waits the server's hint
|
# only ever over-waits, never under-waits the server's hint.
|
||||||
|
# non-finite values (inf/nan) are rejected and finite values are
|
||||||
|
# clamped to MAX_RETRY_AFTER by _retry_after, so a server-controlled
|
||||||
|
# wait can never stall this attempt past a bounded ceiling
|
||||||
wait = self._retry_after(status, resp.headers, body)
|
wait = self._retry_after(status, resp.headers, body)
|
||||||
if wait is not None:
|
if wait is not None:
|
||||||
log.warning("webhook 429 on %s; honoring retry_after %.3fs", url, wait)
|
log.warning("webhook 429 on %s; honoring retry_after %.3fs", url, wait)
|
||||||
@@ -230,18 +267,30 @@ class Webhook:
|
|||||||
return result
|
return result
|
||||||
|
|
||||||
except (aiohttp.ClientError, asyncio.TimeoutError) as error:
|
except (aiohttp.ClientError, asyncio.TimeoutError) as error:
|
||||||
if self._proxies is not None and proxy_tries < self.max_proxy_retries:
|
if self._proxies is not None:
|
||||||
if self._burn(last_proxy):
|
if proxy_tries < self.max_proxy_retries:
|
||||||
proxy_tries += 1
|
if self._burn(last_proxy):
|
||||||
continue
|
proxy_tries += 1
|
||||||
|
continue
|
||||||
|
return WebhookResult(
|
||||||
|
ok=False, status=None, url=url, attempts=attempts,
|
||||||
|
error="proxies exhausted", proxy=last_proxy,
|
||||||
|
)
|
||||||
return WebhookResult(
|
return WebhookResult(
|
||||||
ok=False, status=None, url=url, attempts=attempts,
|
ok=False, status=None, url=url, attempts=attempts,
|
||||||
error="proxies exhausted", proxy=last_proxy,
|
error=f"{type(error).__name__}: {error}", proxy=last_proxy,
|
||||||
)
|
)
|
||||||
return WebhookResult(
|
# no proxy provider: a connection/timeout error is the single most
|
||||||
|
# retry-worthy failure class (the 5xx equivalent already retries under
|
||||||
|
# aretry) — raise _Retryable so aretry applies backoff + max_retries
|
||||||
|
# instead of failing one-shot. aretry re-raises this same _Retryable on
|
||||||
|
# exhaustion, and _send_loop's `except _Retryable` returns its carried
|
||||||
|
# result, so the never-raises / always-returns-a-WebhookResult contract
|
||||||
|
# still holds after retries run out.
|
||||||
|
raise _Retryable(WebhookResult(
|
||||||
ok=False, status=None, url=url, attempts=attempts,
|
ok=False, status=None, url=url, attempts=attempts,
|
||||||
error=f"{type(error).__name__}: {error}", proxy=last_proxy,
|
error=f"{type(error).__name__}: {error}", proxy=last_proxy,
|
||||||
)
|
))
|
||||||
|
|
||||||
def _burn(self, proxy: Optional[str]) -> bool:
|
def _burn(self, proxy: Optional[str]) -> bool:
|
||||||
"""burn the current proxy; return False if it can't be rotated
|
"""burn the current proxy; return False if it can't be rotated
|
||||||
|
|||||||
Reference in New Issue
Block a user