2 Commits
5 changed files with 78 additions and 251 deletions
+1 -1
View File
@@ -1,5 +1,5 @@
# claude
.claude/
CLAUDE.md
# python
__pycache__/
+18 -80
View File
@@ -13,16 +13,13 @@ send to the core — inheriting rotation, proxy, retry, and result for free.
## Install
```
aiowebhooks @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiowebhooks.git@v0.1.5
aiowebhooks @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiowebhooks.git@v0.1.0
# discord embeds / identity helpers need the extra:
aiowebhooks[discord] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiowebhooks.git@v0.1.5
aiowebhooks[discord] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiowebhooks.git@v0.1.0
```
The base pulls `aiohttp` and `commons` (for the retry/backoff engine). Only
`aiowebhooks[discord]` adds `discord.py` (>=2.3, mainline — not discord.py-self), and
only for `DiscordWebhook`.
Drop the `@v0.1.5` suffix from the line above to install the latest unpinned.
The base pulls `aiohttp`. Only `aiowebhooks[discord]` adds `discord.py` (>=2.3,
mainline — not discord.py-self), and only for `DiscordWebhook`.
## Core sender
@@ -50,9 +47,7 @@ Webhook(
```
Inject a shared `session` for throughput (one session per process); without one, each
send opens and closes its own. A single `Webhook` instance is safe to drive from many
concurrent `send()` calls — each call tracks its own attempt count, so concurrent sends
don't corrupt each other's `WebhookResult.attempts`.
send opens and closes its own.
## WebhookResult
@@ -70,23 +65,12 @@ result.proxy # canonical proxy string used (host:port:user:pass / host:port)
## Retry & rate limits
Status retries run through `commons.aretry` (exponential backoff + cap):
- **429** — always retried, capped by `max_retries`. When a wait is parseable (body
`retry_after` first — Discord sends seconds — then the `Retry-After` header) it sleeps
that value before retrying; a 429 with no parseable wait (edge/Cloudflare/generic
webhook) still retries under aretry's backoff rather than failing one-shot. A
non-finite value (`inf`/`nan`) is rejected outright and a finite value is clamped to
`MAX_RETRY_AFTER` (300s) — a server-controlled wait can never stall a `send()` past
that ceiling, however large or malformed the value it sends.
- **5xx** — retried with exponential backoff, capped by `max_retries`.
- **connection/timeout errors** — retried with exponential backoff, capped by
`max_retries`, same as a 5xx (with no proxy provider; see below for the
proxy-rotation path).
- **429** — waits the `retry_after` from the body first (Discord sends seconds), then
the `Retry-After` header, then retries; capped by `max_retries`.
- **5xx** — retried, capped by `max_retries`.
- **4xx** (other than 429) — fails immediately (no retry), returned as `ok=False`.
Exceeding a cap returns a failed result rather than looping — and the result carries the
**real** last status/body (not a synthetic placeholder).
Exceeding a cap returns a failed result rather than looping.
## Proxy rotation (optional, duck-typed)
@@ -107,11 +91,9 @@ result = await wh.send(payload) # sends through pm.get(); on a timeout/connect
```
On a timeout/connection error the current proxy is burned and the next is tried, up to
`max_proxy_retries`. Hitting the cap, or **any exception from the provider's
`get()`/`burn()`** (the provider is duck-typed and never imported, so its exception
types can't be caught by class), returns a failed result — never an infinite loop, never
an escape. With no provider, a timeout/connection error is retried directly under
`max_retries` (the same cap and backoff a 5xx gets) instead of burn+rotate.
`max_proxy_retries`. A provider `ProxiesExhaustedError` (or hitting the cap) returns a
failed result — never an infinite loop. With no provider, a timeout just fails after
normal retry.
## Discord (`aiowebhooks[discord]`)
@@ -142,56 +124,12 @@ Without the extra installed, importing `aiowebhooks` still works; constructing o
- Every send returns a `WebhookResult`; the core never raises on a send failure and
never prints. Callers check `result.ok`.
- JSON-only: **files/attachments, `tts`, and `allowed_mentions` are out** (deliberate
scope cut, addable later). The Discord surface is content + embeds + identity.
- Rotation is round-robin only; try-next-on-failure across URLs is a later feature.
## Changelog
### v0.1.5
- **429 `retry_after` bounded:** a non-finite server-controlled wait (`inf`/`nan`, from
the body `retry_after` or the `Retry-After` header) is now rejected outright, and a
finite wait is clamped to `MAX_RETRY_AFTER` (300s). Previously a bare `float()` parse
slept the value verbatim, unbounded and outside `max_retries` accounting — an
adversarial or ms-vs-s-misconfigured server could stall a `send()` for hours.
- **Connection/timeout errors now retry without a proxy provider:** previously
`aiohttp.ClientError`/`asyncio.TimeoutError` with no `proxies=` set failed one-shot,
contradicting both the README and the single most retry-worthy failure class. Now it
retries under `commons.aretry`'s backoff, capped by `max_retries`, same as a 5xx —
and still returns `ok=False` (never raises) once retries are exhausted. Proxy-rotation
behavior (burn + rotate, capped by `max_proxy_retries`) is unchanged when a provider
is set.
### v0.1.4
- **Never-raises net widened:** an unexpected exception that escapes a send attempt (a
closed injected session → `RuntimeError`, a malformed proxy URL → `ValueError`) now
converts to a falsy `WebhookResult(ok=False, ...)` instead of propagating out of
`send()`, restoring the documented contract for those edge triggers.
### v0.1.3
- **429 always retries:** every `429` is now retryable under aretry's backoff + cap, not
only those with a parseable `retry_after`. A 429 with no body `retry_after` and no
`Retry-After` header (edge/Cloudflare/generic webhook) previously failed one-shot.
### v0.1.2
- Removed a dead `clock` constructor param (it was stored but never used). Pinned
`commons` to v0.2.1.
### v0.1.1
- **Never-raises contract hardened:** an error from a duck-typed proxy provider's
`get()`/`burn()` (e.g. `aioproxies.burn()` raising `ValueError` for an unknown proxy)
used to escape `send()`. Now any provider exception is caught and converted to a
failed result.
- **Retry via `commons.aretry`:** 429/5xx retry moved onto the shared backoff engine —
5xx now backs off (was a tight no-backoff loop), and exhausted retries return the
**real** last status/body instead of a synthetic placeholder. Adds a `commons`
dependency.
- v0.1.0 is JSON-only: **files/attachments, `tts`, and `allowed_mentions` are out**
(deliberate scope cut, addable later). The Discord surface is content + embeds +
identity.
- v0.1.0 rotation is round-robin only; try-next-on-failure across URLs is a later
feature.
## Versioning
Releases are tagged `vX.Y.Z`. The install line above pins a release; drop the `@vX.Y.Z` suffix to install the latest unpinned. Pin deliberately for reproducible installs.
Tagged `vX.Y.Z`. Pin the tag.
+1 -5
View File
@@ -4,12 +4,11 @@ build-backend = "hatchling.build"
[project]
name = "aiowebhooks"
version = "0.1.5"
version = "0.1.0"
description = "async webhook sender (aiohttp) with round-robin urls, retry, and proxy rotation; optional discord.py embeds"
requires-python = ">=3.10"
dependencies = [
"aiohttp>=3.9",
"commons @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v0.2.1",
]
[project.optional-dependencies]
@@ -17,8 +16,5 @@ discord = [
"discord.py>=2.3",
]
[tool.hatch.metadata]
allow-direct-references = true
[tool.hatch.build.targets.wheel]
packages = ["src/aiowebhooks"]
+1 -1
View File
@@ -21,4 +21,4 @@ from .sender import Webhook
__all__ = ["Webhook", "WebhookResult", "WebhookError", "NoUrlsError"]
__version__ = "0.1.4"
__version__ = "0.1.0"
+47 -154
View File
@@ -1,54 +1,24 @@
"""core async webhook sender (aiohttp only, no discord knowledge).
`Webhook` posts a JSON dict to a url (or round-robins a pool), handling 429/5xx
retries, connection/timeout retries, and optional proxy rotation, and always
returns a `WebhookResult` — it never raises on a send failure. the discord layer
builds payloads and delegates the actual POST here so it inherits rotation / proxy
/ retry / result.
a 429's server-controlled `retry_after` wait is bounded: non-finite values
(inf/nan) are rejected and finite values are clamped to `MAX_RETRY_AFTER`, so an
adversarial or misconfigured server can never stall a send() past a bounded
ceiling.
retries and optional proxy rotation, and always returns a `WebhookResult` — it
never raises on a send failure. the discord layer builds payloads and delegates
the actual POST here so it inherits rotation / proxy / retry / result.
"""
import asyncio
import logging
import math
import time
from typing import Dict, List, Optional, Union
from urllib.parse import unquote, urlsplit
import aiohttp
from commons import aretry
from .errors import NoUrlsError
from .result import WebhookResult
log = logging.getLogger(__name__)
MAX_RETRY_AFTER = 300.0
"""ceiling (seconds) honored from a server-controlled 429 retry_after/Retry-After
a server can send an arbitrarily large or non-finite wait (float('inf'), a
Cloudflare-scale value, or a ms-vs-s unit mismatch turning 5s into 5000s); sleeping
that verbatim would stall a send() far past its ClientTimeout and outside
max_retries accounting, breaking the 'always returns a WebhookResult' contract.
non-finite values are rejected outright (treated as unparseable); finite values are
clamped to this ceiling.
"""
class _Retryable(Exception):
"""internal signal: a retryable HTTP status (429/5xx); carries the response
raised inside an attempt so commons.aretry drives the backoff + cap; the loop
catches the final one to return the REAL last response, not a synthetic result.
"""
def __init__(self, result: WebhookResult):
super().__init__(f"retryable status {result.status}")
self.result = result
def _proxy_string(proxies_dict: Optional[Dict[str, str]]) -> Optional[str]:
"""canonical host:port:user:pass (or host:port) from an aiohttp proxies dict
@@ -67,12 +37,12 @@ def _proxy_string(proxies_dict: Optional[Dict[str, str]]) -> Optional[str]:
if host is None:
return None
host = host.lower()
hostport = f"{host}:{parts.port}" if parts.port is not None else host
port = str(parts.port) if parts.port is not None else ""
if parts.username:
user = unquote(parts.username)
password = unquote(parts.password) if parts.password is not None else ""
return f"{hostport}:{user}:{password}"
return hostport
return f"{host}:{port}:{user}:{password}"
return f"{host}:{port}"
except ValueError:
return None
@@ -89,6 +59,7 @@ class Webhook:
timeout: float = 15,
max_retries: int = 3,
max_proxy_retries: int = 3,
clock=time.monotonic,
):
self._urls = [urls] if isinstance(urls, str) else list(urls)
if not self._urls:
@@ -98,6 +69,7 @@ class Webhook:
self.timeout = timeout
self.max_retries = max_retries
self.max_proxy_retries = max_proxy_retries
self._clock = clock
self._index = 0
def _next_url(self) -> str:
@@ -108,27 +80,18 @@ class Webhook:
@staticmethod
def _retry_after(status: int, headers, body) -> Optional[float]:
"""seconds to wait on a 429, from body retry_after then Retry-After header
non-finite values (inf/nan) are rejected as unparseable; finite values are
clamped to MAX_RETRY_AFTER so a server-controlled wait can never stall a
send() past a bounded ceiling.
"""
"""seconds to wait on a 429, from body retry_after then Retry-After header"""
if status != 429:
return None
if isinstance(body, dict) and body.get("retry_after") is not None:
try:
value = float(body["retry_after"])
if math.isfinite(value):
return max(0.0, min(value, MAX_RETRY_AFTER))
return float(body["retry_after"])
except (TypeError, ValueError):
pass
header = headers.get("Retry-After") if headers else None
if header is not None:
try:
value = float(header)
if math.isfinite(value):
return max(0.0, min(value, MAX_RETRY_AFTER))
return float(header)
except (TypeError, ValueError):
pass
return None
@@ -137,10 +100,9 @@ class Webhook:
"""post a json payload to the next pool url; always returns a result
handles 429 (wait + retry, capped by max_retries), 5xx (retry, capped), and
timeout/connection errors: with a proxy provider, burn + rotate to the next
proxy (capped by max_proxy_retries) before failing; without one, retried
directly under max_retries like a 5xx. 4xx other than 429 fail immediately.
never raises on a send failure.
timeout/connection errors with optional proxy rotation (burn + next, capped
by max_proxy_retries). 4xx other than 429 fail immediately. never raises on a
send failure.
"""
url = self._next_url()
session = self._session
@@ -156,77 +118,29 @@ class Webhook:
async def _send_loop(
self, session: aiohttp.ClientSession, url: str, payload: Dict
) -> WebhookResult:
"""status-retry (via commons.aretry) wrapping proxy rotation; never raises
commons.aretry owns the 429/5xx/no-provider-connection-error backoff schedule
+ retry cap (max_retries), retrying on the internal _Retryable signal. on
exhaustion it re-raises the last _Retryable, whose carried result is the REAL
last response (not a synthetic status-0). proxy rotation on connection errors
lives inside the attempt and is capped separately (max_proxy_retries); once a
provider's rotation cap is hit that failure is terminal, not retried again
via aretry.
"""
counter = [0]
try:
return await aretry(
lambda: self._attempt(session, url, payload, counter),
attempts=self.max_retries + 1,
on=(_Retryable,),
)
except _Retryable as exhausted:
return exhausted.result
except Exception as error:
# never-raises safety net: an unexpected error that escapes the attempt (a
# closed injected session -> RuntimeError, a malformed proxy url -> ValueError,
# anything not aiohttp.ClientError/TimeoutError) must come back as a failed
# result, not propagate out of send()
log.warning("webhook send failed unexpectedly on %s: %s", url, error, exc_info=True)
return WebhookResult(
ok=False, status=None, url=url, attempts=counter[0] or 1,
error=f"{type(error).__name__}: {error}",
)
async def _attempt(
self, session: aiohttp.ClientSession, url: str, payload: Dict, counter: List[int]
) -> WebhookResult:
"""one logical send: proxy rotation + a single POST; may raise _Retryable
raises _Retryable (carrying the real response) on a 429/5xx so the caller's
aretry applies backoff; honors an explicit 429 retry_after by sleeping it
before signalling. a connection/timeout error also raises _Retryable when no
proxy provider is set, so it retries under aretry's backoff + max_retries
instead of failing one-shot; with a provider, the same error first drives
burn+rotate up to max_proxy_retries, and only becomes terminal (not retried
via aretry) once that cap is hit. returns a final WebhookResult on success or
a terminal (non-retryable) failure — never lets a provider error escape.
`counter` is a per-call mutable cell ([0]) owned by the calling `_send_loop`,
so the attempt count is local to one `send()` and concurrent sends on the
same instance don't corrupt each other's tally.
"""
timeout = aiohttp.ClientTimeout(total=self.timeout)
last_proxy: Optional[str] = None
"""retry/rotation loop for a single send"""
attempts = 0
retries = 0
proxy_tries = 0
last_proxy: Optional[str] = None
timeout = aiohttp.ClientTimeout(total=self.timeout)
while True:
counter[0] += 1
attempts = counter[0]
proxy_url = None
if self._proxies is not None:
try:
proxy_dict = self._proxies.get()
except Exception:
# duck-typed provider; any error from get() means no proxy is
# available — fail cleanly rather than escaping send().
log.warning("webhook: proxy get() failed; no proxy available",
exc_info=True)
except Exception as error:
if type(error).__name__ == "ProxiesExhaustedError":
return WebhookResult(
ok=False, status=None, url=url, attempts=attempts,
error="proxies unavailable", proxy=last_proxy,
error="proxies exhausted", proxy=last_proxy,
)
raise
last_proxy = _proxy_string(proxy_dict)
proxy_url = (proxy_dict or {}).get("http") or (proxy_dict or {}).get("https")
attempts += 1
try:
async with session.post(
url, json=payload, proxy=proxy_url, timeout=timeout
@@ -240,35 +154,27 @@ class Webhook:
response=body, proxy=last_proxy,
)
result = WebhookResult(
wait = self._retry_after(status, resp.headers, body)
if wait is not None and retries < self.max_retries:
retries += 1
log.warning("webhook 429 on %s; waiting %.3fs (retry %d/%d)",
url, wait, retries, self.max_retries)
await asyncio.sleep(wait)
continue
if status >= 500 and retries < self.max_retries:
retries += 1
log.warning("webhook %d on %s; retry %d/%d",
status, url, retries, self.max_retries)
continue
return WebhookResult(
ok=False, status=status, url=url, attempts=attempts,
error=f"http {status}", response=body, proxy=last_proxy,
)
if status == 429:
# every 429 is retryable; honor an explicit retry_after by
# sleeping it, but a 429 with no parseable wait (edge/Cloudflare/
# generic webhook) still retries under aretry's backoff + cap.
# note: aretry ALSO sleeps its backoff between retries, so an
# honored retry_after is additive (retry_after + backoff) — this
# only ever over-waits, never under-waits the server's hint.
# non-finite values (inf/nan) are rejected and finite values are
# clamped to MAX_RETRY_AFTER by _retry_after, so a server-controlled
# wait can never stall this attempt past a bounded ceiling
wait = self._retry_after(status, resp.headers, body)
if wait is not None:
log.warning("webhook 429 on %s; honoring retry_after %.3fs", url, wait)
await asyncio.sleep(wait)
else:
log.warning("webhook 429 on %s; no retry_after, backing off", url)
raise _Retryable(result)
if status >= 500:
raise _Retryable(result)
return result
except (aiohttp.ClientError, asyncio.TimeoutError) as error:
if self._proxies is not None:
if proxy_tries < self.max_proxy_retries:
if self._proxies is not None and proxy_tries < self.max_proxy_retries:
if self._burn(last_proxy):
proxy_tries += 1
continue
@@ -280,33 +186,20 @@ class Webhook:
ok=False, status=None, url=url, attempts=attempts,
error=f"{type(error).__name__}: {error}", proxy=last_proxy,
)
# no proxy provider: a connection/timeout error is the single most
# retry-worthy failure class (the 5xx equivalent already retries under
# aretry) — raise _Retryable so aretry applies backoff + max_retries
# instead of failing one-shot. aretry re-raises this same _Retryable on
# exhaustion, and _send_loop's `except _Retryable` returns its carried
# result, so the never-raises / always-returns-a-WebhookResult contract
# still holds after retries run out.
raise _Retryable(WebhookResult(
ok=False, status=None, url=url, attempts=attempts,
error=f"{type(error).__name__}: {error}", proxy=last_proxy,
))
def _burn(self, proxy: Optional[str]) -> bool:
"""burn the current proxy; return False if it can't be rotated
"""burn the current proxy; return False if the provider is exhausted
the provider is duck-typed and never imported, so we cannot catch its
exception types by class. ANY exception from burn (a ProxiesExhaustedError
on a dead pool, a ValueError when the proxy isn't in the pool, etc.) means
we can't rotate — return False so the caller ends the loop with a failed
result rather than letting it escape send() (which must never raise).
catches a provider ProxiesExhaustedError duck-typed by class name (the
provider is never imported), so a dying pool ends the loop cleanly.
"""
try:
self._proxies.burn(proxy)
return True
except Exception:
log.warning("webhook: proxy burn failed; ending rotation", exc_info=True)
except Exception as error:
if type(error).__name__ == "ProxiesExhaustedError":
return False
raise
@staticmethod
async def _read_body(resp) -> Optional[Union[str, Dict]]: