7 Commits
Author SHA1 Message Date
dsql 4ba768a3bf release: 1.0.0
first stable release. pre-1.0.0 verification complete: all surviving MED regressions and
gaps resolved and independently re-fired, tree audited clean across the suite.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-09 18:53:15 -04:00
dsql 68be8f7101 fix: _pid_alive treats an out-of-C-int-range pid as not alive (no OverflowError)
os.kill(pid, 0) raises OverflowError for a pid too large for a C int, which _pid_alive did
not catch; a foreign <base>.<huge-number>.tmp file in the store dir then made every stale-tmp
sweep (so every _save) crash. an over-range pid can't name a live process, so it's treated as
not-alive like ProcessLookupError.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 17:17:39 -04:00
dsql 98f7190f18 fix: _save no longer resolves the store's realpath twice per call
_save() already resolves target = realpath(self.file) for the write path;
_sweep_stale_tmp then independently recomputed the same realpath to derive its
match basename. _sweep_stale_tmp now takes that resolved basename as a
parameter instead of re-resolving it, cutting one syscall per save. the sweep
itself still runs on every save (unchanged) - gating it to first-save-per-
instance would change when a crash-orphaned tmp from before this process
started actually gets swept, so that part is left alone.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 00:18:16 -04:00
dsql ddf76d8241 fix: clear() and stale-tmp sweep resolve the real symlink target and match only <base>.<pid>.tmp
clear() removed the raw path, so on a symlinked store it unlinked only the
symlink while the real data file kept every value - clear() returned True
but the data resurfaced if the symlink was recreated. It now realpaths the
target first, mirroring _save()'s symlink-safe writes.

_sweep_stale_tmp globbed {base}.*.tmp, which crosses dots and can match an
unrelated neighbor file (deleting it outside the lib's own artifacts), and
interpolated the store's basename unescaped, so glob metacharacters in the
filename (e.g. state[prod].json) either missed the store's own orphans or
cross-matched a different store's. The sweep now lists the directory and
matches a re.escape'd regex anchored to the exact <base>.<pid>.tmp shape
_save() creates.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 19:05:46 -04:00
dsql 5e3eb4e704 refactor: derive __version__ from package metadata (single source)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:58:41 -04:00
dsql 8381b2ecbd docs: compress prose/module docstrings, em-dash->hyphen (de-bloat wave 1)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 00:10:19 -04:00
dsql 5c2fdd80f9 fix: str-only keys, symlink-safe atomic writes, strict JSON, stale-tmp sweep
_load() masked permission/IO errors as an empty store via os.path.exists();
now raises FileNotFoundError only, propagating real failures. Non-str keys
silently never round-tripped through get() since JSON object keys are always
strings; set()/get() now raise ValueError on a non-str key. _save() clobbered
a symlinked store file with os.replace(); now realpaths the target first.
json.dumps() allows NaN/Infinity by default, producing invalid JSON for
strict readers; now allow_nan=False. A whitespace-only file raised while a
zero-byte file returned {}; both now return {}. Orphaned .<pid>.tmp files
from a hard crash of a dead process are swept on save. JSON (de)serialization
now runs via asyncio.to_thread instead of on the event loop.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 23:23:17 -04:00
4 changed files with 112 additions and 67 deletions
+22 -12
View File
@@ -12,18 +12,18 @@ you `delete` or `clear` them.
`requirements.txt`: `requirements.txt`:
``` ```
aiokv @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiokv.git@v0.1.1 aiokv @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiokv.git@v1.0.0
``` ```
Direct: Direct:
```bash ```bash
pip install "aiokv @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiokv.git@v0.1.1" pip install "aiokv @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiokv.git@v1.0.0"
``` ```
Requires `aiofiles` (pulled transitively). Requires `aiofiles` (pulled transitively).
Drop the `@v0.1.1` suffix from the line above to install the latest unpinned. Drop the `@v1.0.0` suffix from the line above to install the latest unpinned.
## Usage ## Usage
@@ -62,14 +62,21 @@ Prefer `AioKV` in new code.
## Durability ## Durability
Writes are **atomic**: data is written to a temp file in the same directory and Writes are **atomic**: data is written to a temp file in the same directory and
`os.replace()`d over the target (atomic on POSIX). A **process** crash mid-write leaves `os.replace()`d over the target's realpath — symlink-safe, so a symlinked store file is
the previous good file intact, and a reader never observes a partial file. (This is written through rather than clobbered. A **process** crash mid-write leaves the previous
process-crash safety, not power-loss durability — there's no `fsync`, so an OS/power good file intact, and a reader never observes a partial file. (This is process-crash
failure could still lose the last write; fine for reconstructible single-process state.) safety, not power-loss durability — there's no `fsync`, so an OS/power failure could
A single still lose the last write; fine for reconstructible single-process state.) `clear()` is
`asyncio.Lock` guards every read and write, so concurrent operations on one instance symlink-safe the same way: it removes the realpath'd target, not the symlink itself, so
are consistent and no update is lost. All blocking filesystem calls run via a symlinked store is genuinely erased rather than just having its symlink unlinked.
`asyncio.to_thread`, so nothing stalls the event loop. Orphaned `.<pid>.tmp` files left by a hard crash of a *different, dead* process are swept
on the next save; the sweep matches only the exact `<realpath basename>.<pid>.tmp` shape
it creates, so a differently-shaped neighbor file is never touched and a store filename
containing glob-like characters (e.g. `state[prod].json`) doesn't miss its own orphans or
cross-match another store's. A single `asyncio.Lock` guards every read and write, so
concurrent operations on one instance are consistent and no update is lost. All blocking
filesystem calls and JSON (de)serialization run via `asyncio.to_thread`, so nothing stalls
the event loop.
## Scope — read this ## Scope — read this
@@ -82,10 +89,13 @@ are consistent and no update is lost. All blocking filesystem calls run via
## Error contract ## Error contract
- Keys must be `str``get` / `set` raise `ValueError` on a non-str key (JSON object
keys are always strings, so a non-str key would silently never round-trip).
- `get` / `set` / `get_all` raise on unexpected I/O. `_load` raises `JSONDecodeError` - `get` / `set` / `get_all` raise on unexpected I/O. `_load` raises `JSONDecodeError`
on a truncated/corrupt file, and `ValueError` when the file holds valid JSON that on a truncated/corrupt file, and `ValueError` when the file holds valid JSON that
isn't an object (a bare list/number/string/null) — so corruption or a wrong-shaped isn't an object (a bare list/number/string/null) — so corruption or a wrong-shaped
file is visible rather than silently masked. file is visible rather than silently masked. Non-finite floats (`NaN`/`Infinity`)
raise `ValueError` on `set` rather than persisting invalid JSON.
- `delete` / `clear` log the exception and return `False` on error, `True` otherwise. - `delete` / `clear` log the exception and return `False` on error, `True` otherwise.
## Versioning ## Versioning
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project] [project]
name = "aiokv" name = "aiokv"
version = "0.1.1" version = "1.0.0"
description = "Async file-backed key-value store for single-process local state — atomic writes, no TTL, config-free, installable." description = "Async file-backed key-value store for single-process local state — atomic writes, no TTL, config-free, installable."
requires-python = ">=3.10" requires-python = ">=3.10"
dependencies = [ dependencies = [
+7
View File
@@ -1,3 +1,10 @@
from importlib.metadata import version, PackageNotFoundError
from .aiokv import AioKV, aiocache from .aiokv import AioKV, aiocache
try:
__version__ = version("aiokv")
except PackageNotFoundError:
__version__ = "0.0.0+unknown"
__all__ = ["AioKV", "aiocache"] __all__ = ["AioKV", "aiocache"]
+82 -54
View File
@@ -1,34 +1,23 @@
""" """
async file-backed key-value store for single-process local state async file-backed key-value store for single-process local state
a persist-forever KV store (last-used-command, rate-limit timestamps, seen-ids, a persist-forever KV store backed by a JSON file, not a cache (no TTL/expiry/eviction).
simple bot state) backed by a JSON file. NOT a cache: no TTL, no expiry, no see README for usage and the full API.
eviction — values live until you delete or clear them.
from aiokv import AioKV from aiokv import AioKV
kv = AioKV("state.json") kv = AioKV("state.json")
await kv.set("last_seen", 12345) await kv.set("last_seen", 12345)
await kv.set("ran_cleanup") # value omitted -> stores int(time.time()) when = await kv.get("last_seen")
when = await kv.get("ran_cleanup")
await kv.delete("last_seen")
durability: writes are atomic — data is written to a temp file in the same atomic writes are process-crash safe (temp file + os.replace) but NOT power-loss safe
directory and os.replace()d over the target, so a crash mid-write never corrupts (no fsync). the lock is per-instance only - two AioKV instances or processes on the
the store and readers never see a partial file. a single asyncio.Lock guards every same file will clobber each other; this is single-process local state, not shared
read and write, so concurrent operations on one instance are consistent. cross-process storage.
scope: SINGLE-PROCESS, single-instance local state only. the lock is per-instance —
two AioKV instances (or two processes) pointing at the same file are NOT safe and
will clobber each other. for shared cross-process/cross-bot state, use a database
(e.g. mongo), not this.
config-free: the file path is passed at construction; nothing is read from a global
config. errors in delete/clear are logged and swallowed (returning False); get/set
raise on unexpected i/o so a real failure is visible.
""" """
import os import os
import re
import json import json
import time import time
import asyncio import asyncio
@@ -53,10 +42,8 @@ class AioKV:
self.lock = asyncio.Lock() self.lock = asyncio.Lock()
async def set(self, key: str, value: Any = None) -> None: async def set(self, key: str, value: Any = None) -> None:
"""set a value; if value is omitted or None, stores int(time.time()) """set a value; if value is omitted or None, stores int(time.time())"""
self._check_key(key)
the timestamp default exists for "mark that i saw/did X at time T" usage.
"""
async with self.lock: async with self.lock:
cache = await self._load() cache = await self._load()
cache[key] = value if value is not None else int(time.time()) cache[key] = value if value is not None else int(time.time())
@@ -64,6 +51,7 @@ class AioKV:
async def get(self, key: str, default: Any = None) -> Any: async def get(self, key: str, default: Any = None) -> Any:
"""return the value for key, or default if absent""" """return the value for key, or default if absent"""
self._check_key(key)
async with self.lock: async with self.lock:
cache = await self._load() cache = await self._load()
return cache.get(key, default) return cache.get(key, default)
@@ -71,6 +59,7 @@ class AioKV:
async def delete(self, key: str) -> bool: async def delete(self, key: str) -> bool:
"""delete a key; returns True if removed or absent, False on error""" """delete a key; returns True if removed or absent, False on error"""
try: try:
self._check_key(key)
async with self.lock: async with self.lock:
cache = await self._load() cache = await self._load()
if key in cache: if key in cache:
@@ -82,15 +71,15 @@ class AioKV:
return False return False
async def clear(self) -> bool: async def clear(self) -> bool:
"""remove the backing file entirely; returns True on success, False on error """remove the backing file entirely; True on success or if already absent, False on error
a file already absent (or removed concurrently between the check and the remove) symlink-safe: removes the realpath'd target rather than the symlink itself, mirroring
is success — the goal state, no file, is reached. _save()'s symlink-safe writes - a symlinked store's real data file is what gets erased"""
"""
try: try:
async with self.lock: async with self.lock:
target = await asyncio.to_thread(os.path.realpath, self.file)
try: try:
await asyncio.to_thread(os.remove, self.file) await asyncio.to_thread(os.remove, target)
except FileNotFoundError: except FileNotFoundError:
pass pass
return True return True
@@ -103,43 +92,43 @@ class AioKV:
async with self.lock: async with self.lock:
return await self._load() return await self._load()
async def _load(self) -> Dict[str, Any]: @staticmethod
"""load the store from disk, returning {} if the file is absent or empty def _check_key(key: str) -> None:
"""raise ValueError if key is not str"""
if not isinstance(key, str):
raise ValueError(f"aiokv keys must be str, got {type(key).__name__}")
a truncated/corrupt file raises JSONDecodeError, and a file holding valid async def _load(self) -> Dict[str, Any]:
JSON that is not an object (e.g. a bare list, number, or null) raises """load the store, returning {} if absent or blank; raises JSONDecodeError on
ValueError — both surfaced to the caller rather than silently masking a corrupt content and ValueError on valid-but-non-object JSON rather than masking it"""
real corruption or returning a non-dict that breaks every other method. try:
""" async with aiofiles.open(self.file, mode="r", encoding="utf-8") as f:
if not await asyncio.to_thread(os.path.exists, self.file): data = await f.read()
except FileNotFoundError:
return {} return {}
async with aiofiles.open(self.file, mode="r", encoding="utf-8") as f: if not data.strip():
data = await f.read()
if not data:
return {} return {}
loaded = json.loads(data) loaded = await asyncio.to_thread(json.loads, data)
if not isinstance(loaded, dict): if not isinstance(loaded, dict):
raise ValueError(f"store file {self.file} does not hold a JSON object") raise ValueError(f"store file {self.file} does not hold a JSON object")
return loaded return loaded
async def _save(self, cache: Dict[str, Any]) -> None: async def _save(self, cache: Dict[str, Any]) -> None:
"""write the store atomically: temp file in the same dir, then os.replace """write atomically: temp file in the same dir, then os.replace over the realpath'd
target (symlink-safe). process-crash safe (a crash mid-write leaves the prior good
os.replace is atomic on POSIX, so a reader never sees a partial file and a file intact), NOT power-loss safe - no fsync, so an OS/power failure can still lose
process crash mid-write leaves the previous good file intact. note this is the last write (acceptable: reconstructible single-process state, not a db)"""
process-crash safety, NOT power-loss durability — there is no fsync of the temp target = await asyncio.to_thread(os.path.realpath, self.file)
file or the directory, so an OS/power failure could still lose the most recent directory = os.path.dirname(target) or "."
write (acceptable here: this is reconstructible single-process state, not a db).
"""
directory = os.path.dirname(self.file) or "."
await asyncio.to_thread(os.makedirs, directory, exist_ok=True) await asyncio.to_thread(os.makedirs, directory, exist_ok=True)
await self._sweep_stale_tmp(directory, os.path.basename(target))
payload = json.dumps(cache) payload = await asyncio.to_thread(json.dumps, cache, allow_nan=False)
tmp = f"{self.file}.{os.getpid()}.tmp" tmp = f"{target}.{os.getpid()}.tmp"
try: try:
async with aiofiles.open(tmp, mode="w", encoding="utf-8") as f: async with aiofiles.open(tmp, mode="w", encoding="utf-8") as f:
await f.write(payload) await f.write(payload)
await asyncio.to_thread(os.replace, tmp, self.file) await asyncio.to_thread(os.replace, tmp, target)
except Exception: except Exception:
if await asyncio.to_thread(os.path.exists, tmp): if await asyncio.to_thread(os.path.exists, tmp):
try: try:
@@ -148,7 +137,46 @@ class AioKV:
log.exception("aiokv: failed to clean up temp file %s", tmp) log.exception("aiokv: failed to clean up temp file %s", tmp)
raise raise
async def _sweep_stale_tmp(self, directory: str, base: str) -> None:
"""remove orphaned .<pid>.tmp files left by a hard crash of a different, dead process
# back-compat: this lib was originally named aiocache; legacy call sites using matches only the exact shape _save() creates (<realpath basename>.<pid>.tmp), via a
# `aiocache(...)` keep working via this alias. prefer AioKV in new code. re.escape'd regex over a directory listing rather than a raw glob - a glob pattern
both crosses unrelated dots (matching non-aiokv neighbors like foo.backup.999.tmp) and
mistreats glob metacharacters in the store's own filename (e.g. state[prod].json).
`base` is the realpath'd basename _save() already resolved - passed in rather than
re-resolved here so a save does one realpath call, not two."""
candidate = re.compile(rf"^{re.escape(base)}\.(\d+)\.tmp$")
names = await asyncio.to_thread(os.listdir, directory)
for name in names:
match = candidate.match(name)
if not match:
continue
pid = int(match.group(1))
if pid == os.getpid():
continue
if await asyncio.to_thread(self._pid_alive, pid):
continue
path = os.path.join(directory, name)
try:
await asyncio.to_thread(os.remove, path)
except FileNotFoundError:
pass
except Exception:
log.exception("aiokv: failed to sweep stale temp file %s", path)
@staticmethod
def _pid_alive(pid: int) -> bool:
"""check whether pid refers to a live process, without permission to signal counting as alive"""
try:
os.kill(pid, 0)
except (ProcessLookupError, OverflowError):
# OverflowError: a pid too large for a C int can't name a live process
return False
except PermissionError:
return True
return True
# back-compat: originally named aiocache; prefer AioKV in new code.
aiocache = AioKV aiocache = AioKV