fix: reject duplicate-name uploads and rebuild fresh Files per send
cache() silently lost a FileRef when a list input had two sources resolving to the same name (two paths with the same basename, or two discord.File objects with the same .filename): both files were uploaded (real cost) but the merge step kept only the last, orphaning the first. _normalize now rejects a duplicate name with DPYCacheError before any upload happens. _to_file returned the same discord.File object for a discord.File source, only mutating .filename. Reusing one File under two keys swapped filename/URL between refs; reusing a BytesIO-backed File re-sent 0 bytes; reusing a path-backed File after discord.py's own close() raised ValueError. _to_file now builds a genuinely distinct discord.File per send, re-opening a path-backed source or seeking and copying a caller-supplied stream; an unrebuildable opaque stream now raises DPYCacheError instead of corrupting or crashing a later send. Also: lookup() accepts a digit-only message-id string (previously only bare int fetched directly, a numeric string routed into the jump-url parser and raised); dropped a redundant discord.Forbidden member from an except tuple (Forbidden already subclasses HTTPException); tightened _parse_jump_url's return type off a dead Optional and added `from err` to its raise. Signed-off-by: disqualifier <dev@disqualifier.me>
This commit is contained in:
@@ -9,7 +9,7 @@ bites you.
|
|||||||
## Install
|
## Install
|
||||||
|
|
||||||
```
|
```
|
||||||
dpy_cache @ git+ssh://git@git.rethinkstudios.io/rethink-public/dpy_cache.git@v0.1.0
|
dpy_cache @ git+ssh://git@git.rethinkstudios.io/rethink-public/dpy_cache.git@v0.1.1
|
||||||
```
|
```
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
@@ -46,6 +46,9 @@ refs = await cache.lookup(result.message_url) # jump url, message id, or disco
|
|||||||
refs["avatar.png"].url
|
refs["avatar.png"].url
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`lookup()` accepts a message id as an `int` or as a digit-only `str` (the snowflake shape
|
||||||
|
JSON/DBs store), e.g. `cache.lookup("123456789012345678")`.
|
||||||
|
|
||||||
## What you inject
|
## What you inject
|
||||||
|
|
||||||
A resolved Discord channel object (`discord.abc.Messageable` that supports `.send` and
|
A resolved Discord channel object (`discord.abc.Messageable` that supports `.send` and
|
||||||
@@ -58,7 +61,16 @@ first use.
|
|||||||
|
|
||||||
`cache_one` / `cache` accept, per file: `bytes`, a filesystem path (`str` or
|
`cache_one` / `cache` accept, per file: `bytes`, a filesystem path (`str` or
|
||||||
`os.PathLike`), or a `discord.File`. In a **list** input, raw `bytes` have no filename and
|
`os.PathLike`), or a `discord.File`. In a **list** input, raw `bytes` have no filename and
|
||||||
raise `ValueError` — pass a dict `{name: bytes}` or a `discord.File` to name bytes.
|
raise `ValueError` — pass a dict `{name: bytes}` or a `discord.File` to name bytes. A
|
||||||
|
**list** input whose sources resolve to the same name (two paths with the same basename,
|
||||||
|
two `discord.File`s with the same `.filename`, ...) raises `DPYCacheError` before any
|
||||||
|
upload — a dict input can't collide since dict keys are already unique.
|
||||||
|
|
||||||
|
A `discord.File` source is safe to reuse across multiple names/batches: `cache()` always
|
||||||
|
builds a genuinely fresh `discord.File` per send (re-opening a path-backed source, or
|
||||||
|
seeking + copying a caller-supplied stream) rather than reusing the object you passed in.
|
||||||
|
An opaque stream that can't be safely re-read (already closed, not seekable) raises
|
||||||
|
`DPYCacheError` instead of silently sending stale or empty data.
|
||||||
|
|
||||||
## Why `resolve()` exists
|
## Why `resolve()` exists
|
||||||
|
|
||||||
@@ -80,12 +92,26 @@ API. In short:
|
|||||||
- `lookup(message) -> dict[str, FileRef]`
|
- `lookup(message) -> dict[str, FileRef]`
|
||||||
|
|
||||||
**Fail-loud.** Nothing is swallowed. Lib-specific invariants raise `DPYCacheError`
|
**Fail-loud.** Nothing is swallowed. Lib-specific invariants raise `DPYCacheError`
|
||||||
(attachment-count mismatch after a send, `cache_url` non-200, a `lookup` jump-URL pointing
|
(attachment-count mismatch after a send, a duplicate filename in a list input to `cache()`,
|
||||||
at a different channel than the injected one). Unnamed `bytes` in a list raise `ValueError`.
|
`cache_url` non-200, a `lookup` jump-URL pointing at a different channel than the injected
|
||||||
Raw Discord errors (`Forbidden` / `HTTPException` / `NotFound`) propagate **unwrapped** so
|
one, a `discord.File` source that can't be safely rebuilt for a repeat send). Unnamed
|
||||||
you can still branch on Discord's own types.
|
`bytes` in a list raise `ValueError`. Raw Discord errors (`Forbidden` / `HTTPException` /
|
||||||
|
`NotFound`) propagate **unwrapped** so you can still branch on Discord's own types.
|
||||||
|
|
||||||
## Versioning
|
## Versioning
|
||||||
|
|
||||||
Tagged `vX.Y.Z`; pin a tag in your install line. Targets `discord.py>=2.0` (not
|
Tagged `vX.Y.Z`; pin a tag in your install line. Targets `discord.py>=2.0` (not
|
||||||
`discord.py-self`).
|
`discord.py-self`).
|
||||||
|
|
||||||
|
### v0.1.1
|
||||||
|
|
||||||
|
- `cache()` now rejects a duplicate filename in a **list** input with `DPYCacheError`
|
||||||
|
before any upload, instead of silently uploading both files and losing the first
|
||||||
|
`FileRef` at merge time.
|
||||||
|
- `_to_file` now builds a genuinely fresh `discord.File` per send for `discord.File`
|
||||||
|
sources (previously returned the same object, which could swap filename/URL when one
|
||||||
|
`File` was reused under two names, or re-send 0 bytes / raise `ValueError` on a spent
|
||||||
|
stream).
|
||||||
|
- `lookup()` now accepts a digit-only message-id string in addition to `int`.
|
||||||
|
- Dropped the redundant `discord.Forbidden` member from an `except` tuple (it already
|
||||||
|
subclasses `discord.HTTPException`); no behavior change.
|
||||||
|
|||||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "dpy_cache"
|
name = "dpy_cache"
|
||||||
version = "0.1.0"
|
version = "0.1.1"
|
||||||
description = "Persist images/files to a Discord storage channel and get durable, re-resolvable CDN references — config-free, injectable, installable."
|
description = "Persist images/files to a Discord storage channel and get durable, re-resolvable CDN references — config-free, injectable, installable."
|
||||||
requires-python = ">=3.10"
|
requires-python = ">=3.10"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
|
|||||||
+72
-22
@@ -40,10 +40,11 @@ error on first use.
|
|||||||
error contract (fail-loud)
|
error contract (fail-loud)
|
||||||
--------------------------
|
--------------------------
|
||||||
nothing is swallowed. lib-specific invariants raise :class:`DPYCacheError` (attachment-count
|
nothing is swallowed. lib-specific invariants raise :class:`DPYCacheError` (attachment-count
|
||||||
mismatch after a send, ``cache_url`` non-200, a ``lookup`` jump-url pointing at a different
|
mismatch after a send, a duplicate filename in a list input to ``cache()``, ``cache_url``
|
||||||
channel than the injected one). unnamed ``bytes`` in a list input raises ``ValueError``. raw
|
non-200, a ``lookup`` jump-url pointing at a different channel than the injected one, a
|
||||||
discord errors (``Forbidden`` / ``HTTPException`` / ``NotFound``) propagate UNWRAPPED so
|
``discord.File`` source that can't be safely rebuilt for a repeat send). unnamed ``bytes`` in
|
||||||
callers can still branch on discord's types.
|
a list input raises ``ValueError``. raw discord errors (``Forbidden`` / ``HTTPException`` /
|
||||||
|
``NotFound``) propagate UNWRAPPED so callers can still branch on discord's types.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
@@ -64,8 +65,9 @@ FileContent = Union[bytes, str, "os.PathLike[str]", discord.File]
|
|||||||
|
|
||||||
|
|
||||||
class DPYCacheError(Exception):
|
class DPYCacheError(Exception):
|
||||||
"""raised for dpy_cache's own invariants (count mismatch, remote fetch failure,
|
"""raised for dpy_cache's own invariants (count mismatch, duplicate filename, remote
|
||||||
channel mismatch); raw discord errors propagate unwrapped instead"""
|
fetch failure, channel mismatch, an unrebuildable discord.File source); raw discord
|
||||||
|
errors propagate unwrapped instead"""
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
@@ -119,8 +121,10 @@ class DPYCache:
|
|||||||
content: "str | None" = None,
|
content: "str | None" = None,
|
||||||
) -> CacheResult:
|
) -> CacheResult:
|
||||||
"""upload files to the storage channel (batching over 10/message) and return refs
|
"""upload files to the storage channel (batching over 10/message) and return refs
|
||||||
keyed by original filename; raises on unnamed bytes, count mismatch, or a discord
|
keyed by original filename; raises ValueError on unnamed bytes in a list input,
|
||||||
send error"""
|
DPYCacheError on a duplicate filename in a list input (checked before any upload, so
|
||||||
|
a collision costs zero requests) or an attachment-count mismatch after a send, and
|
||||||
|
propagates discord's own Forbidden/HTTPException unwrapped on a send failure"""
|
||||||
items = self._normalize(files)
|
items = self._normalize(files)
|
||||||
merged: dict[str, FileRef] = {}
|
merged: dict[str, FileRef] = {}
|
||||||
jump_urls: list[str] = []
|
jump_urls: list[str] = []
|
||||||
@@ -130,7 +134,7 @@ class DPYCache:
|
|||||||
discord_files = [self._to_file(name, source) for name, source in group]
|
discord_files = [self._to_file(name, source) for name, source in group]
|
||||||
try:
|
try:
|
||||||
message = await self._channel.send(content=content, files=discord_files)
|
message = await self._channel.send(content=content, files=discord_files)
|
||||||
except (discord.Forbidden, discord.HTTPException):
|
except discord.HTTPException:
|
||||||
log.error("dpy_cache: send of %d file(s) to the storage channel failed", len(group))
|
log.error("dpy_cache: send of %d file(s) to the storage channel failed", len(group))
|
||||||
raise
|
raise
|
||||||
|
|
||||||
@@ -194,7 +198,8 @@ class DPYCache:
|
|||||||
)
|
)
|
||||||
|
|
||||||
async def lookup(self, message: "str | int | discord.Message") -> dict[str, FileRef]:
|
async def lookup(self, message: "str | int | discord.Message") -> dict[str, FileRef]:
|
||||||
"""reverse lookup: a jump-url, message id, or Message -> {filename: FileRef}; raises
|
"""reverse lookup: a jump-url, message id (``int`` or a digit-only ``str``, matching
|
||||||
|
the snowflake shape stored in JSON/DBs), or Message -> {filename: FileRef}; raises
|
||||||
(NotFound) if the message is gone, never an empty map for a missing message"""
|
(NotFound) if the message is gone, never an empty map for a missing message"""
|
||||||
resolved = await self._resolve_message(message)
|
resolved = await self._resolve_message(message)
|
||||||
return {
|
return {
|
||||||
@@ -204,16 +209,18 @@ class DPYCache:
|
|||||||
|
|
||||||
def _normalize(self, files: "dict[str, FileContent] | list[FileContent]") -> "list[tuple[str, FileContent]]":
|
def _normalize(self, files: "dict[str, FileContent] | list[FileContent]") -> "list[tuple[str, FileContent]]":
|
||||||
"""normalize dict/list input to an ordered list of (name, source); raises ValueError
|
"""normalize dict/list input to an ordered list of (name, source); raises ValueError
|
||||||
on unnamed bytes in a list"""
|
on unnamed bytes in a list, or DPYCacheError on a duplicate name in a list (dict keys
|
||||||
|
can't collide by construction)"""
|
||||||
if isinstance(files, dict):
|
if isinstance(files, dict):
|
||||||
return list(files.items())
|
return list(files.items())
|
||||||
|
|
||||||
items: "list[tuple[str, FileContent]]" = []
|
items: "list[tuple[str, FileContent]]" = []
|
||||||
|
seen: "set[str]" = set()
|
||||||
for source in files:
|
for source in files:
|
||||||
if isinstance(source, discord.File):
|
if isinstance(source, discord.File):
|
||||||
items.append((source.filename, source))
|
name = source.filename
|
||||||
elif isinstance(source, (str, os.PathLike)):
|
elif isinstance(source, (str, os.PathLike)):
|
||||||
items.append((_basename(source), source))
|
name = _basename(source)
|
||||||
elif isinstance(source, (bytes, bytearray)):
|
elif isinstance(source, (bytes, bytearray)):
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
"raw bytes in a list input have no filename; pass a dict {name: bytes} "
|
"raw bytes in a list input have no filename; pass a dict {name: bytes} "
|
||||||
@@ -221,29 +228,72 @@ class DPYCache:
|
|||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
raise ValueError(f"unsupported file content type in list: {type(source).__name__}")
|
raise ValueError(f"unsupported file content type in list: {type(source).__name__}")
|
||||||
|
if name in seen:
|
||||||
|
raise DPYCacheError(
|
||||||
|
f"duplicate filename {name!r} in list input; every list source must resolve "
|
||||||
|
f"to a unique name, or the later upload silently orphans the earlier one"
|
||||||
|
)
|
||||||
|
seen.add(name)
|
||||||
|
items.append((name, source))
|
||||||
return items
|
return items
|
||||||
|
|
||||||
def _to_file(self, name: str, source: FileContent) -> discord.File:
|
def _to_file(self, name: str, source: FileContent) -> discord.File:
|
||||||
"""build a fresh discord.File for ``source`` named ``name``; a fresh File is built
|
"""build a genuinely fresh discord.File for ``source`` named ``name``, distinct from
|
||||||
per send so the source survives multi-batch and retries"""
|
any File previously handed to a send; a fresh File is built per send so the source
|
||||||
|
survives multi-batch, retries, and reuse of the same source under multiple names"""
|
||||||
if isinstance(source, discord.File):
|
if isinstance(source, discord.File):
|
||||||
source.filename = name
|
return self._fresh_file_from_source(source, name)
|
||||||
return source
|
|
||||||
if isinstance(source, (bytes, bytearray)):
|
if isinstance(source, (bytes, bytearray)):
|
||||||
return discord.File(io.BytesIO(bytes(source)), filename=name)
|
return discord.File(io.BytesIO(bytes(source)), filename=name)
|
||||||
if isinstance(source, (str, os.PathLike)):
|
if isinstance(source, (str, os.PathLike)):
|
||||||
return discord.File(source, filename=name)
|
return discord.File(source, filename=name)
|
||||||
raise ValueError(f"unsupported file content type: {type(source).__name__}")
|
raise ValueError(f"unsupported file content type: {type(source).__name__}")
|
||||||
|
|
||||||
|
def _fresh_file_from_source(self, source: discord.File, name: str) -> discord.File:
|
||||||
|
"""rebuild a distinct discord.File from an existing one without mutating or
|
||||||
|
re-sending ``source`` itself; re-opens the path for an owner-opened File, or
|
||||||
|
seeks to the original position and copies the bytes for a caller-supplied stream;
|
||||||
|
raises DPYCacheError if the underlying stream can't be safely re-read"""
|
||||||
|
if source._owner and hasattr(source.fp, "name"):
|
||||||
|
return discord.File(
|
||||||
|
source.fp.name,
|
||||||
|
filename=name,
|
||||||
|
spoiler=source.spoiler,
|
||||||
|
description=source.description,
|
||||||
|
)
|
||||||
|
if source.fp.closed:
|
||||||
|
raise DPYCacheError(
|
||||||
|
f"discord.File source for {name!r} is already closed and can't be safely "
|
||||||
|
f"reused; pass the raw bytes or path instead of a spent discord.File"
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
source.fp.seek(source._original_pos)
|
||||||
|
data = source.fp.read()
|
||||||
|
source.fp.seek(source._original_pos)
|
||||||
|
except (OSError, ValueError) as err:
|
||||||
|
raise DPYCacheError(
|
||||||
|
f"discord.File source for {name!r} could not be re-read (opaque or "
|
||||||
|
f"already-consumed stream); pass the raw bytes instead of a discord.File"
|
||||||
|
) from err
|
||||||
|
return discord.File(
|
||||||
|
io.BytesIO(data),
|
||||||
|
filename=name,
|
||||||
|
spoiler=source.spoiler,
|
||||||
|
description=source.description,
|
||||||
|
)
|
||||||
|
|
||||||
async def _resolve_message(self, message: "str | int | discord.Message") -> discord.Message:
|
async def _resolve_message(self, message: "str | int | discord.Message") -> discord.Message:
|
||||||
"""turn a jump-url / id / Message into a fetched Message on the injected channel"""
|
"""turn a jump-url / id / digit-only id string / Message into a fetched Message on
|
||||||
|
the injected channel"""
|
||||||
if isinstance(message, discord.Message):
|
if isinstance(message, discord.Message):
|
||||||
return message
|
return message
|
||||||
if isinstance(message, int):
|
if isinstance(message, int):
|
||||||
return await self._channel.fetch_message(message)
|
return await self._channel.fetch_message(message)
|
||||||
if isinstance(message, str):
|
if isinstance(message, str):
|
||||||
|
if message.isdigit():
|
||||||
|
return await self._channel.fetch_message(int(message))
|
||||||
channel_id, message_id = self._parse_jump_url(message)
|
channel_id, message_id = self._parse_jump_url(message)
|
||||||
if channel_id is not None and channel_id != self._channel.id:
|
if channel_id != self._channel.id:
|
||||||
raise DPYCacheError(
|
raise DPYCacheError(
|
||||||
f"lookup jump-url targets channel {channel_id} but the injected channel is "
|
f"lookup jump-url targets channel {channel_id} but the injected channel is "
|
||||||
f"{self._channel.id}; this lib holds no client and can only fetch its own channel"
|
f"{self._channel.id}; this lib holds no client and can only fetch its own channel"
|
||||||
@@ -252,13 +302,13 @@ class DPYCache:
|
|||||||
raise ValueError(f"unsupported lookup input type: {type(message).__name__}")
|
raise ValueError(f"unsupported lookup input type: {type(message).__name__}")
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _parse_jump_url(url: str) -> "tuple[int | None, int]":
|
def _parse_jump_url(url: str) -> "tuple[int, int]":
|
||||||
"""parse (channel_id, message_id) out of a discord jump url; raises DPYCacheError on
|
"""parse (channel_id, message_id) out of a discord jump url; raises DPYCacheError on
|
||||||
a malformed url"""
|
a malformed url"""
|
||||||
parts = url.rstrip("/").split("/")
|
parts = url.rstrip("/").split("/")
|
||||||
try:
|
try:
|
||||||
message_id = int(parts[-1])
|
message_id = int(parts[-1])
|
||||||
channel_id = int(parts[-2])
|
channel_id = int(parts[-2])
|
||||||
except (IndexError, ValueError):
|
except (IndexError, ValueError) as err:
|
||||||
raise DPYCacheError(f"lookup: not a valid discord jump url: {url}")
|
raise DPYCacheError(f"lookup: not a valid discord jump url: {url}") from err
|
||||||
return channel_id, message_id
|
return channel_id, message_id
|
||||||
|
|||||||
Reference in New Issue
Block a user