Compare commits
No commits in common. "main" and "v0.1.0" have entirely different histories.
2
.gitignore
vendored
2
.gitignore
vendored
@ -1,5 +1,5 @@
|
|||||||
# claude
|
# claude
|
||||||
.claude/
|
CLAUDE.md
|
||||||
|
|
||||||
# python
|
# python
|
||||||
__pycache__/
|
__pycache__/
|
||||||
|
|||||||
76
README.md
76
README.md
@ -13,16 +13,13 @@ send to the core — inheriting rotation, proxy, retry, and result for free.
|
|||||||
## Install
|
## Install
|
||||||
|
|
||||||
```
|
```
|
||||||
aiowebhooks @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiowebhooks.git@v0.1.4
|
aiowebhooks @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiowebhooks.git@v0.1.0
|
||||||
# discord embeds / identity helpers need the extra:
|
# discord embeds / identity helpers need the extra:
|
||||||
aiowebhooks[discord] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiowebhooks.git@v0.1.4
|
aiowebhooks[discord] @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiowebhooks.git@v0.1.0
|
||||||
```
|
```
|
||||||
|
|
||||||
The base pulls `aiohttp` and `commons` (for the retry/backoff engine). Only
|
The base pulls `aiohttp`. Only `aiowebhooks[discord]` adds `discord.py` (>=2.3,
|
||||||
`aiowebhooks[discord]` adds `discord.py` (>=2.3, mainline — not discord.py-self), and
|
mainline — not discord.py-self), and only for `DiscordWebhook`.
|
||||||
only for `DiscordWebhook`.
|
|
||||||
|
|
||||||
Drop the `@v0.1.4` suffix from the line above to install the latest unpinned.
|
|
||||||
|
|
||||||
## Core sender
|
## Core sender
|
||||||
|
|
||||||
@ -50,9 +47,7 @@ Webhook(
|
|||||||
```
|
```
|
||||||
|
|
||||||
Inject a shared `session` for throughput (one session per process); without one, each
|
Inject a shared `session` for throughput (one session per process); without one, each
|
||||||
send opens and closes its own. A single `Webhook` instance is safe to drive from many
|
send opens and closes its own.
|
||||||
concurrent `send()` calls — each call tracks its own attempt count, so concurrent sends
|
|
||||||
don't corrupt each other's `WebhookResult.attempts`.
|
|
||||||
|
|
||||||
## WebhookResult
|
## WebhookResult
|
||||||
|
|
||||||
@ -70,17 +65,12 @@ result.proxy # canonical proxy string used (host:port:user:pass / host:port)
|
|||||||
|
|
||||||
## Retry & rate limits
|
## Retry & rate limits
|
||||||
|
|
||||||
Status retries run through `commons.aretry` (exponential backoff + cap):
|
- **429** — waits the `retry_after` from the body first (Discord sends seconds), then
|
||||||
|
the `Retry-After` header, then retries; capped by `max_retries`.
|
||||||
- **429** — always retried, capped by `max_retries`. When a wait is parseable (body
|
- **5xx** — retried, capped by `max_retries`.
|
||||||
`retry_after` first — Discord sends seconds — then the `Retry-After` header) it sleeps
|
|
||||||
that value before retrying; a 429 with no parseable wait (edge/Cloudflare/generic
|
|
||||||
webhook) still retries under aretry's backoff rather than failing one-shot.
|
|
||||||
- **5xx** — retried with exponential backoff, capped by `max_retries`.
|
|
||||||
- **4xx** (other than 429) — fails immediately (no retry), returned as `ok=False`.
|
- **4xx** (other than 429) — fails immediately (no retry), returned as `ok=False`.
|
||||||
|
|
||||||
Exceeding a cap returns a failed result rather than looping — and the result carries the
|
Exceeding a cap returns a failed result rather than looping.
|
||||||
**real** last status/body (not a synthetic placeholder).
|
|
||||||
|
|
||||||
## Proxy rotation (optional, duck-typed)
|
## Proxy rotation (optional, duck-typed)
|
||||||
|
|
||||||
@ -101,10 +91,9 @@ result = await wh.send(payload) # sends through pm.get(); on a timeout/connect
|
|||||||
```
|
```
|
||||||
|
|
||||||
On a timeout/connection error the current proxy is burned and the next is tried, up to
|
On a timeout/connection error the current proxy is burned and the next is tried, up to
|
||||||
`max_proxy_retries`. Hitting the cap, or **any exception from the provider's
|
`max_proxy_retries`. A provider `ProxiesExhaustedError` (or hitting the cap) returns a
|
||||||
`get()`/`burn()`** (the provider is duck-typed and never imported, so its exception
|
failed result — never an infinite loop. With no provider, a timeout just fails after
|
||||||
types can't be caught by class), returns a failed result — never an infinite loop, never
|
normal retry.
|
||||||
an escape. With no provider, a timeout just fails after normal retry.
|
|
||||||
|
|
||||||
## Discord (`aiowebhooks[discord]`)
|
## Discord (`aiowebhooks[discord]`)
|
||||||
|
|
||||||
@ -135,41 +124,12 @@ Without the extra installed, importing `aiowebhooks` still works; constructing o
|
|||||||
|
|
||||||
- Every send returns a `WebhookResult`; the core never raises on a send failure and
|
- Every send returns a `WebhookResult`; the core never raises on a send failure and
|
||||||
never prints. Callers check `result.ok`.
|
never prints. Callers check `result.ok`.
|
||||||
- JSON-only: **files/attachments, `tts`, and `allowed_mentions` are out** (deliberate
|
- v0.1.0 is JSON-only: **files/attachments, `tts`, and `allowed_mentions` are out**
|
||||||
scope cut, addable later). The Discord surface is content + embeds + identity.
|
(deliberate scope cut, addable later). The Discord surface is content + embeds +
|
||||||
- Rotation is round-robin only; try-next-on-failure across URLs is a later feature.
|
identity.
|
||||||
|
- v0.1.0 rotation is round-robin only; try-next-on-failure across URLs is a later
|
||||||
## Changelog
|
feature.
|
||||||
|
|
||||||
### v0.1.4
|
|
||||||
|
|
||||||
- **Never-raises net widened:** an unexpected exception that escapes a send attempt (a
|
|
||||||
closed injected session → `RuntimeError`, a malformed proxy URL → `ValueError`) now
|
|
||||||
converts to a falsy `WebhookResult(ok=False, ...)` instead of propagating out of
|
|
||||||
`send()`, restoring the documented contract for those edge triggers.
|
|
||||||
|
|
||||||
### v0.1.3
|
|
||||||
|
|
||||||
- **429 always retries:** every `429` is now retryable under aretry's backoff + cap, not
|
|
||||||
only those with a parseable `retry_after`. A 429 with no body `retry_after` and no
|
|
||||||
`Retry-After` header (edge/Cloudflare/generic webhook) previously failed one-shot.
|
|
||||||
|
|
||||||
### v0.1.2
|
|
||||||
|
|
||||||
- Removed a dead `clock` constructor param (it was stored but never used). Pinned
|
|
||||||
`commons` to v0.2.1.
|
|
||||||
|
|
||||||
### v0.1.1
|
|
||||||
|
|
||||||
- **Never-raises contract hardened:** an error from a duck-typed proxy provider's
|
|
||||||
`get()`/`burn()` (e.g. `aioproxies.burn()` raising `ValueError` for an unknown proxy)
|
|
||||||
used to escape `send()`. Now any provider exception is caught and converted to a
|
|
||||||
failed result.
|
|
||||||
- **Retry via `commons.aretry`:** 429/5xx retry moved onto the shared backoff engine —
|
|
||||||
5xx now backs off (was a tight no-backoff loop), and exhausted retries return the
|
|
||||||
**real** last status/body instead of a synthetic placeholder. Adds a `commons`
|
|
||||||
dependency.
|
|
||||||
|
|
||||||
## Versioning
|
## Versioning
|
||||||
|
|
||||||
Releases are tagged `vX.Y.Z`. The install line above pins a release; drop the `@vX.Y.Z` suffix to install the latest unpinned. Pin deliberately for reproducible installs.
|
Tagged `vX.Y.Z`. Pin the tag.
|
||||||
|
|||||||
@ -4,12 +4,11 @@ build-backend = "hatchling.build"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "aiowebhooks"
|
name = "aiowebhooks"
|
||||||
version = "0.1.4"
|
version = "0.1.0"
|
||||||
description = "async webhook sender (aiohttp) with round-robin urls, retry, and proxy rotation; optional discord.py embeds"
|
description = "async webhook sender (aiohttp) with round-robin urls, retry, and proxy rotation; optional discord.py embeds"
|
||||||
requires-python = ">=3.10"
|
requires-python = ">=3.10"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"aiohttp>=3.9",
|
"aiohttp>=3.9",
|
||||||
"commons @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v0.2.1",
|
|
||||||
]
|
]
|
||||||
|
|
||||||
[project.optional-dependencies]
|
[project.optional-dependencies]
|
||||||
@ -17,8 +16,5 @@ discord = [
|
|||||||
"discord.py>=2.3",
|
"discord.py>=2.3",
|
||||||
]
|
]
|
||||||
|
|
||||||
[tool.hatch.metadata]
|
|
||||||
allow-direct-references = true
|
|
||||||
|
|
||||||
[tool.hatch.build.targets.wheel]
|
[tool.hatch.build.targets.wheel]
|
||||||
packages = ["src/aiowebhooks"]
|
packages = ["src/aiowebhooks"]
|
||||||
|
|||||||
@ -21,4 +21,4 @@ from .sender import Webhook
|
|||||||
|
|
||||||
__all__ = ["Webhook", "WebhookResult", "WebhookError", "NoUrlsError"]
|
__all__ = ["Webhook", "WebhookResult", "WebhookError", "NoUrlsError"]
|
||||||
|
|
||||||
__version__ = "0.1.4"
|
__version__ = "0.1.0"
|
||||||
|
|||||||
@ -8,11 +8,11 @@ the actual POST here so it inherits rotation / proxy / retry / result.
|
|||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import logging
|
import logging
|
||||||
|
import time
|
||||||
from typing import Dict, List, Optional, Union
|
from typing import Dict, List, Optional, Union
|
||||||
from urllib.parse import unquote, urlsplit
|
from urllib.parse import unquote, urlsplit
|
||||||
|
|
||||||
import aiohttp
|
import aiohttp
|
||||||
from commons import aretry
|
|
||||||
|
|
||||||
from .errors import NoUrlsError
|
from .errors import NoUrlsError
|
||||||
from .result import WebhookResult
|
from .result import WebhookResult
|
||||||
@ -20,18 +20,6 @@ from .result import WebhookResult
|
|||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
class _Retryable(Exception):
|
|
||||||
"""internal signal: a retryable HTTP status (429/5xx); carries the response
|
|
||||||
|
|
||||||
raised inside an attempt so commons.aretry drives the backoff + cap; the loop
|
|
||||||
catches the final one to return the REAL last response, not a synthetic result.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, result: WebhookResult):
|
|
||||||
super().__init__(f"retryable status {result.status}")
|
|
||||||
self.result = result
|
|
||||||
|
|
||||||
|
|
||||||
def _proxy_string(proxies_dict: Optional[Dict[str, str]]) -> Optional[str]:
|
def _proxy_string(proxies_dict: Optional[Dict[str, str]]) -> Optional[str]:
|
||||||
"""canonical host:port:user:pass (or host:port) from an aiohttp proxies dict
|
"""canonical host:port:user:pass (or host:port) from an aiohttp proxies dict
|
||||||
|
|
||||||
@ -49,12 +37,12 @@ def _proxy_string(proxies_dict: Optional[Dict[str, str]]) -> Optional[str]:
|
|||||||
if host is None:
|
if host is None:
|
||||||
return None
|
return None
|
||||||
host = host.lower()
|
host = host.lower()
|
||||||
hostport = f"{host}:{parts.port}" if parts.port is not None else host
|
port = str(parts.port) if parts.port is not None else ""
|
||||||
if parts.username:
|
if parts.username:
|
||||||
user = unquote(parts.username)
|
user = unquote(parts.username)
|
||||||
password = unquote(parts.password) if parts.password is not None else ""
|
password = unquote(parts.password) if parts.password is not None else ""
|
||||||
return f"{hostport}:{user}:{password}"
|
return f"{host}:{port}:{user}:{password}"
|
||||||
return hostport
|
return f"{host}:{port}"
|
||||||
except ValueError:
|
except ValueError:
|
||||||
return None
|
return None
|
||||||
|
|
||||||
@ -71,6 +59,7 @@ class Webhook:
|
|||||||
timeout: float = 15,
|
timeout: float = 15,
|
||||||
max_retries: int = 3,
|
max_retries: int = 3,
|
||||||
max_proxy_retries: int = 3,
|
max_proxy_retries: int = 3,
|
||||||
|
clock=time.monotonic,
|
||||||
):
|
):
|
||||||
self._urls = [urls] if isinstance(urls, str) else list(urls)
|
self._urls = [urls] if isinstance(urls, str) else list(urls)
|
||||||
if not self._urls:
|
if not self._urls:
|
||||||
@ -80,6 +69,7 @@ class Webhook:
|
|||||||
self.timeout = timeout
|
self.timeout = timeout
|
||||||
self.max_retries = max_retries
|
self.max_retries = max_retries
|
||||||
self.max_proxy_retries = max_proxy_retries
|
self.max_proxy_retries = max_proxy_retries
|
||||||
|
self._clock = clock
|
||||||
self._index = 0
|
self._index = 0
|
||||||
|
|
||||||
def _next_url(self) -> str:
|
def _next_url(self) -> str:
|
||||||
@ -128,71 +118,29 @@ class Webhook:
|
|||||||
async def _send_loop(
|
async def _send_loop(
|
||||||
self, session: aiohttp.ClientSession, url: str, payload: Dict
|
self, session: aiohttp.ClientSession, url: str, payload: Dict
|
||||||
) -> WebhookResult:
|
) -> WebhookResult:
|
||||||
"""status-retry (via commons.aretry) wrapping proxy rotation; never raises
|
"""retry/rotation loop for a single send"""
|
||||||
|
attempts = 0
|
||||||
commons.aretry owns the 429/5xx backoff schedule + retry cap (max_retries),
|
retries = 0
|
||||||
retrying on the internal _Retryable signal. on exhaustion it re-raises the
|
|
||||||
last _Retryable, whose carried result is the REAL last response (not a
|
|
||||||
synthetic status-0). proxy rotation on connection errors lives inside the
|
|
||||||
attempt and is capped separately.
|
|
||||||
"""
|
|
||||||
counter = [0]
|
|
||||||
try:
|
|
||||||
return await aretry(
|
|
||||||
lambda: self._attempt(session, url, payload, counter),
|
|
||||||
attempts=self.max_retries + 1,
|
|
||||||
on=(_Retryable,),
|
|
||||||
)
|
|
||||||
except _Retryable as exhausted:
|
|
||||||
return exhausted.result
|
|
||||||
except Exception as error:
|
|
||||||
# never-raises safety net: an unexpected error that escapes the attempt (a
|
|
||||||
# closed injected session -> RuntimeError, a malformed proxy url -> ValueError,
|
|
||||||
# anything not aiohttp.ClientError/TimeoutError) must come back as a failed
|
|
||||||
# result, not propagate out of send()
|
|
||||||
log.warning("webhook send failed unexpectedly on %s: %s", url, error, exc_info=True)
|
|
||||||
return WebhookResult(
|
|
||||||
ok=False, status=None, url=url, attempts=counter[0] or 1,
|
|
||||||
error=f"{type(error).__name__}: {error}",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _attempt(
|
|
||||||
self, session: aiohttp.ClientSession, url: str, payload: Dict, counter: List[int]
|
|
||||||
) -> WebhookResult:
|
|
||||||
"""one logical send: proxy rotation + a single POST; may raise _Retryable
|
|
||||||
|
|
||||||
raises _Retryable (carrying the real response) on a 429/5xx so the caller's
|
|
||||||
aretry applies backoff; honors an explicit 429 retry_after by sleeping it
|
|
||||||
before signalling. returns a final WebhookResult on success or a terminal
|
|
||||||
(non-retryable) failure — never lets a provider/connection error escape.
|
|
||||||
|
|
||||||
`counter` is a per-call mutable cell ([0]) owned by the calling `_send_loop`,
|
|
||||||
so the attempt count is local to one `send()` and concurrent sends on the
|
|
||||||
same instance don't corrupt each other's tally.
|
|
||||||
"""
|
|
||||||
timeout = aiohttp.ClientTimeout(total=self.timeout)
|
|
||||||
last_proxy: Optional[str] = None
|
|
||||||
proxy_tries = 0
|
proxy_tries = 0
|
||||||
|
last_proxy: Optional[str] = None
|
||||||
|
timeout = aiohttp.ClientTimeout(total=self.timeout)
|
||||||
|
|
||||||
while True:
|
while True:
|
||||||
counter[0] += 1
|
|
||||||
attempts = counter[0]
|
|
||||||
proxy_url = None
|
proxy_url = None
|
||||||
if self._proxies is not None:
|
if self._proxies is not None:
|
||||||
try:
|
try:
|
||||||
proxy_dict = self._proxies.get()
|
proxy_dict = self._proxies.get()
|
||||||
except Exception:
|
except Exception as error:
|
||||||
# duck-typed provider; any error from get() means no proxy is
|
if type(error).__name__ == "ProxiesExhaustedError":
|
||||||
# available — fail cleanly rather than escaping send().
|
return WebhookResult(
|
||||||
log.warning("webhook: proxy get() failed; no proxy available",
|
ok=False, status=None, url=url, attempts=attempts,
|
||||||
exc_info=True)
|
error="proxies exhausted", proxy=last_proxy,
|
||||||
return WebhookResult(
|
)
|
||||||
ok=False, status=None, url=url, attempts=attempts,
|
raise
|
||||||
error="proxies unavailable", proxy=last_proxy,
|
|
||||||
)
|
|
||||||
last_proxy = _proxy_string(proxy_dict)
|
last_proxy = _proxy_string(proxy_dict)
|
||||||
proxy_url = (proxy_dict or {}).get("http") or (proxy_dict or {}).get("https")
|
proxy_url = (proxy_dict or {}).get("http") or (proxy_dict or {}).get("https")
|
||||||
|
|
||||||
|
attempts += 1
|
||||||
try:
|
try:
|
||||||
async with session.post(
|
async with session.post(
|
||||||
url, json=payload, proxy=proxy_url, timeout=timeout
|
url, json=payload, proxy=proxy_url, timeout=timeout
|
||||||
@ -206,29 +154,25 @@ class Webhook:
|
|||||||
response=body, proxy=last_proxy,
|
response=body, proxy=last_proxy,
|
||||||
)
|
)
|
||||||
|
|
||||||
result = WebhookResult(
|
wait = self._retry_after(status, resp.headers, body)
|
||||||
|
if wait is not None and retries < self.max_retries:
|
||||||
|
retries += 1
|
||||||
|
log.warning("webhook 429 on %s; waiting %.3fs (retry %d/%d)",
|
||||||
|
url, wait, retries, self.max_retries)
|
||||||
|
await asyncio.sleep(wait)
|
||||||
|
continue
|
||||||
|
|
||||||
|
if status >= 500 and retries < self.max_retries:
|
||||||
|
retries += 1
|
||||||
|
log.warning("webhook %d on %s; retry %d/%d",
|
||||||
|
status, url, retries, self.max_retries)
|
||||||
|
continue
|
||||||
|
|
||||||
|
return WebhookResult(
|
||||||
ok=False, status=status, url=url, attempts=attempts,
|
ok=False, status=status, url=url, attempts=attempts,
|
||||||
error=f"http {status}", response=body, proxy=last_proxy,
|
error=f"http {status}", response=body, proxy=last_proxy,
|
||||||
)
|
)
|
||||||
|
|
||||||
if status == 429:
|
|
||||||
# every 429 is retryable; honor an explicit retry_after by
|
|
||||||
# sleeping it, but a 429 with no parseable wait (edge/Cloudflare/
|
|
||||||
# generic webhook) still retries under aretry's backoff + cap.
|
|
||||||
# note: aretry ALSO sleeps its backoff between retries, so an
|
|
||||||
# honored retry_after is additive (retry_after + backoff) — this
|
|
||||||
# only ever over-waits, never under-waits the server's hint
|
|
||||||
wait = self._retry_after(status, resp.headers, body)
|
|
||||||
if wait is not None:
|
|
||||||
log.warning("webhook 429 on %s; honoring retry_after %.3fs", url, wait)
|
|
||||||
await asyncio.sleep(wait)
|
|
||||||
else:
|
|
||||||
log.warning("webhook 429 on %s; no retry_after, backing off", url)
|
|
||||||
raise _Retryable(result)
|
|
||||||
if status >= 500:
|
|
||||||
raise _Retryable(result)
|
|
||||||
return result
|
|
||||||
|
|
||||||
except (aiohttp.ClientError, asyncio.TimeoutError) as error:
|
except (aiohttp.ClientError, asyncio.TimeoutError) as error:
|
||||||
if self._proxies is not None and proxy_tries < self.max_proxy_retries:
|
if self._proxies is not None and proxy_tries < self.max_proxy_retries:
|
||||||
if self._burn(last_proxy):
|
if self._burn(last_proxy):
|
||||||
@ -244,20 +188,18 @@ class Webhook:
|
|||||||
)
|
)
|
||||||
|
|
||||||
def _burn(self, proxy: Optional[str]) -> bool:
|
def _burn(self, proxy: Optional[str]) -> bool:
|
||||||
"""burn the current proxy; return False if it can't be rotated
|
"""burn the current proxy; return False if the provider is exhausted
|
||||||
|
|
||||||
the provider is duck-typed and never imported, so we cannot catch its
|
catches a provider ProxiesExhaustedError duck-typed by class name (the
|
||||||
exception types by class. ANY exception from burn (a ProxiesExhaustedError
|
provider is never imported), so a dying pool ends the loop cleanly.
|
||||||
on a dead pool, a ValueError when the proxy isn't in the pool, etc.) means
|
|
||||||
we can't rotate — return False so the caller ends the loop with a failed
|
|
||||||
result rather than letting it escape send() (which must never raise).
|
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
self._proxies.burn(proxy)
|
self._proxies.burn(proxy)
|
||||||
return True
|
return True
|
||||||
except Exception:
|
except Exception as error:
|
||||||
log.warning("webhook: proxy burn failed; ending rotation", exc_info=True)
|
if type(error).__name__ == "ProxiesExhaustedError":
|
||||||
return False
|
return False
|
||||||
|
raise
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
async def _read_body(resp) -> Optional[Union[str, Dict]]:
|
async def _read_body(resp) -> Optional[Union[str, Dict]]:
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user