12 Commits
Author SHA1 Message Date
dsql 4ba768a3bf release: 1.0.0
first stable release. pre-1.0.0 verification complete: all surviving MED regressions and
gaps resolved and independently re-fired, tree audited clean across the suite.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-09 18:53:15 -04:00
dsql 68be8f7101 fix: _pid_alive treats an out-of-C-int-range pid as not alive (no OverflowError)
os.kill(pid, 0) raises OverflowError for a pid too large for a C int, which _pid_alive did
not catch; a foreign <base>.<huge-number>.tmp file in the store dir then made every stale-tmp
sweep (so every _save) crash. an over-range pid can't name a live process, so it's treated as
not-alive like ProcessLookupError.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 17:17:39 -04:00
dsql 98f7190f18 fix: _save no longer resolves the store's realpath twice per call
_save() already resolves target = realpath(self.file) for the write path;
_sweep_stale_tmp then independently recomputed the same realpath to derive its
match basename. _sweep_stale_tmp now takes that resolved basename as a
parameter instead of re-resolving it, cutting one syscall per save. the sweep
itself still runs on every save (unchanged) - gating it to first-save-per-
instance would change when a crash-orphaned tmp from before this process
started actually gets swept, so that part is left alone.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 00:18:16 -04:00
dsql ddf76d8241 fix: clear() and stale-tmp sweep resolve the real symlink target and match only <base>.<pid>.tmp
clear() removed the raw path, so on a symlinked store it unlinked only the
symlink while the real data file kept every value - clear() returned True
but the data resurfaced if the symlink was recreated. It now realpaths the
target first, mirroring _save()'s symlink-safe writes.

_sweep_stale_tmp globbed {base}.*.tmp, which crosses dots and can match an
unrelated neighbor file (deleting it outside the lib's own artifacts), and
interpolated the store's basename unescaped, so glob metacharacters in the
filename (e.g. state[prod].json) either missed the store's own orphans or
cross-matched a different store's. The sweep now lists the directory and
matches a re.escape'd regex anchored to the exact <base>.<pid>.tmp shape
_save() creates.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 19:05:46 -04:00
dsql 5e3eb4e704 refactor: derive __version__ from package metadata (single source)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:58:41 -04:00
dsql 8381b2ecbd docs: compress prose/module docstrings, em-dash->hyphen (de-bloat wave 1)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 00:10:19 -04:00
dsql 5c2fdd80f9 fix: str-only keys, symlink-safe atomic writes, strict JSON, stale-tmp sweep
_load() masked permission/IO errors as an empty store via os.path.exists();
now raises FileNotFoundError only, propagating real failures. Non-str keys
silently never round-tripped through get() since JSON object keys are always
strings; set()/get() now raise ValueError on a non-str key. _save() clobbered
a symlinked store file with os.replace(); now realpaths the target first.
json.dumps() allows NaN/Infinity by default, producing invalid JSON for
strict readers; now allow_nan=False. A whitespace-only file raised while a
zero-byte file returned {}; both now return {}. Orphaned .<pid>.tmp files
from a hard crash of a dead process are swept on save. JSON (de)serialization
now runs via asyncio.to_thread instead of on the event loop.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 23:23:17 -04:00
dsql e68e0b9ccf chore: ignore .claude/ dir (CLAUDE.md now lives under .claude/)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:54:14 -04:00
dsql 1e364fcfdb fix: clear() treats a concurrent delete as success; explicit utf-8; durability prose
clear() handles FileNotFoundError as success (the goal state — no file — is reached)
instead of returning False. read/write open with explicit encoding='utf-8'. atomic-write
prose scoped to process-crash safety (NOT power-loss durability — no fsync), in module,
README, and CLAUDE.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:35:23 -04:00
dsql 8747b61705 docs: pin install line to release, note unpinned-latest option
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 18:13:30 -04:00
dsql 22e91d2b2d docs: show unpinned install line; note tag-pinning for reproducibility
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 18:07:15 -04:00
dsql 5ee0292dcb docs: document _load's ValueError-on-non-object-JSON in the error contract (v0.1.1)
README + CLAUDE.md error contract now note that _load raises ValueError on valid-but-
non-object JSON (bare list/number/string/null) in addition to JSONDecodeError on a
corrupt file, matching the module docstring (L1).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 17:57:37 -04:00
5 changed files with 123 additions and 62 deletions
+1 -1
View File
@@ -1,5 +1,5 @@
# claude
CLAUDE.md
.claude/
# python
__pycache__/
+27 -10
View File
@@ -12,17 +12,19 @@ you `delete` or `clear` them.
`requirements.txt`:
```
aiokv @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiokv.git@v0.1.0
aiokv @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiokv.git@v1.0.0
```
Direct:
```bash
pip install "aiokv @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiokv.git@v0.1.0"
pip install "aiokv @ git+ssh://git@git.rethinkstudios.io/rethink-public/aiokv.git@v1.0.0"
```
Requires `aiofiles` (pulled transitively).
Drop the `@v1.0.0` suffix from the line above to install the latest unpinned.
## Usage
```python
@@ -60,11 +62,21 @@ Prefer `AioKV` in new code.
## Durability
Writes are **atomic**: data is written to a temp file in the same directory and
`os.replace()`d over the target (atomic on POSIX). A crash mid-write leaves the
previous good file intact, and a reader never observes a partial file. A single
`asyncio.Lock` guards every read and write, so concurrent operations on one instance
are consistent and no update is lost. All blocking filesystem calls run via
`asyncio.to_thread`, so nothing stalls the event loop.
`os.replace()`d over the target's realpath — symlink-safe, so a symlinked store file is
written through rather than clobbered. A **process** crash mid-write leaves the previous
good file intact, and a reader never observes a partial file. (This is process-crash
safety, not power-loss durability — there's no `fsync`, so an OS/power failure could
still lose the last write; fine for reconstructible single-process state.) `clear()` is
symlink-safe the same way: it removes the realpath'd target, not the symlink itself, so
a symlinked store is genuinely erased rather than just having its symlink unlinked.
Orphaned `.<pid>.tmp` files left by a hard crash of a *different, dead* process are swept
on the next save; the sweep matches only the exact `<realpath basename>.<pid>.tmp` shape
it creates, so a differently-shaped neighbor file is never touched and a store filename
containing glob-like characters (e.g. `state[prod].json`) doesn't miss its own orphans or
cross-match another store's. A single `asyncio.Lock` guards every read and write, so
concurrent operations on one instance are consistent and no update is lost. All blocking
filesystem calls and JSON (de)serialization run via `asyncio.to_thread`, so nothing stalls
the event loop.
## Scope — read this
@@ -77,10 +89,15 @@ are consistent and no update is lost. All blocking filesystem calls run via
## Error contract
- `get` / `set` / `get_all` raise on unexpected I/O (and `_load` raises on a
truncated/corrupt file) so a real failure is visible rather than silently masked.
- Keys must be `str``get` / `set` raise `ValueError` on a non-str key (JSON object
keys are always strings, so a non-str key would silently never round-trip).
- `get` / `set` / `get_all` raise on unexpected I/O. `_load` raises `JSONDecodeError`
on a truncated/corrupt file, and `ValueError` when the file holds valid JSON that
isn't an object (a bare list/number/string/null) — so corruption or a wrong-shaped
file is visible rather than silently masked. Non-finite floats (`NaN`/`Infinity`)
raise `ValueError` on `set` rather than persisting invalid JSON.
- `delete` / `clear` log the exception and return `False` on error, `True` otherwise.
## Versioning
Tagged `vX.Y.Z`. Pin the tag in `requirements.txt`.
Releases are tagged `vX.Y.Z`. The install line above pins a release; drop the `@vX.Y.Z` suffix to install the latest unpinned. Pin deliberately for reproducible installs.
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "aiokv"
version = "0.1.0"
version = "1.0.0"
description = "Async file-backed key-value store for single-process local state — atomic writes, no TTL, config-free, installable."
requires-python = ">=3.10"
dependencies = [
+7
View File
@@ -1,3 +1,10 @@
from importlib.metadata import version, PackageNotFoundError
from .aiokv import AioKV, aiocache
try:
__version__ = version("aiokv")
except PackageNotFoundError:
__version__ = "0.0.0+unknown"
__all__ = ["AioKV", "aiocache"]
+87 -50
View File
@@ -1,34 +1,23 @@
"""
async file-backed key-value store for single-process local state
a persist-forever KV store (last-used-command, rate-limit timestamps, seen-ids,
simple bot state) backed by a JSON file. NOT a cache: no TTL, no expiry, no
eviction — values live until you delete or clear them.
a persist-forever KV store backed by a JSON file, not a cache (no TTL/expiry/eviction).
see README for usage and the full API.
from aiokv import AioKV
kv = AioKV("state.json")
await kv.set("last_seen", 12345)
await kv.set("ran_cleanup") # value omitted -> stores int(time.time())
when = await kv.get("ran_cleanup")
await kv.delete("last_seen")
when = await kv.get("last_seen")
durability: writes are atomic — data is written to a temp file in the same
directory and os.replace()d over the target, so a crash mid-write never corrupts
the store and readers never see a partial file. a single asyncio.Lock guards every
read and write, so concurrent operations on one instance are consistent.
scope: SINGLE-PROCESS, single-instance local state only. the lock is per-instance —
two AioKV instances (or two processes) pointing at the same file are NOT safe and
will clobber each other. for shared cross-process/cross-bot state, use a database
(e.g. mongo), not this.
config-free: the file path is passed at construction; nothing is read from a global
config. errors in delete/clear are logged and swallowed (returning False); get/set
raise on unexpected i/o so a real failure is visible.
atomic writes are process-crash safe (temp file + os.replace) but NOT power-loss safe
(no fsync). the lock is per-instance only - two AioKV instances or processes on the
same file will clobber each other; this is single-process local state, not shared
cross-process storage.
"""
import os
import re
import json
import time
import asyncio
@@ -53,10 +42,8 @@ class AioKV:
self.lock = asyncio.Lock()
async def set(self, key: str, value: Any = None) -> None:
"""set a value; if value is omitted or None, stores int(time.time())
the timestamp default exists for "mark that i saw/did X at time T" usage.
"""
"""set a value; if value is omitted or None, stores int(time.time())"""
self._check_key(key)
async with self.lock:
cache = await self._load()
cache[key] = value if value is not None else int(time.time())
@@ -64,6 +51,7 @@ class AioKV:
async def get(self, key: str, default: Any = None) -> Any:
"""return the value for key, or default if absent"""
self._check_key(key)
async with self.lock:
cache = await self._load()
return cache.get(key, default)
@@ -71,6 +59,7 @@ class AioKV:
async def delete(self, key: str) -> bool:
"""delete a key; returns True if removed or absent, False on error"""
try:
self._check_key(key)
async with self.lock:
cache = await self._load()
if key in cache:
@@ -82,11 +71,17 @@ class AioKV:
return False
async def clear(self) -> bool:
"""remove the backing file entirely; returns True on success, False on error"""
"""remove the backing file entirely; True on success or if already absent, False on error
symlink-safe: removes the realpath'd target rather than the symlink itself, mirroring
_save()'s symlink-safe writes - a symlinked store's real data file is what gets erased"""
try:
async with self.lock:
if await asyncio.to_thread(os.path.exists, self.file):
await asyncio.to_thread(os.remove, self.file)
target = await asyncio.to_thread(os.path.realpath, self.file)
try:
await asyncio.to_thread(os.remove, target)
except FileNotFoundError:
pass
return True
except Exception:
log.exception("aiokv.clear() failed")
@@ -97,40 +92,43 @@ class AioKV:
async with self.lock:
return await self._load()
async def _load(self) -> Dict[str, Any]:
"""load the store from disk, returning {} if the file is absent or empty
@staticmethod
def _check_key(key: str) -> None:
"""raise ValueError if key is not str"""
if not isinstance(key, str):
raise ValueError(f"aiokv keys must be str, got {type(key).__name__}")
a truncated/corrupt file raises JSONDecodeError, and a file holding valid
JSON that is not an object (e.g. a bare list, number, or null) raises
ValueError — both surfaced to the caller rather than silently masking a
real corruption or returning a non-dict that breaks every other method.
"""
if not await asyncio.to_thread(os.path.exists, self.file):
async def _load(self) -> Dict[str, Any]:
"""load the store, returning {} if absent or blank; raises JSONDecodeError on
corrupt content and ValueError on valid-but-non-object JSON rather than masking it"""
try:
async with aiofiles.open(self.file, mode="r", encoding="utf-8") as f:
data = await f.read()
except FileNotFoundError:
return {}
async with aiofiles.open(self.file, mode="r") as f:
data = await f.read()
if not data:
if not data.strip():
return {}
loaded = json.loads(data)
loaded = await asyncio.to_thread(json.loads, data)
if not isinstance(loaded, dict):
raise ValueError(f"store file {self.file} does not hold a JSON object")
return loaded
async def _save(self, cache: Dict[str, Any]) -> None:
"""write the store atomically: temp file in the same dir, then os.replace
os.replace is atomic on POSIX, so a reader never sees a partial file and a
crash mid-write leaves the previous good file intact.
"""
directory = os.path.dirname(self.file) or "."
"""write atomically: temp file in the same dir, then os.replace over the realpath'd
target (symlink-safe). process-crash safe (a crash mid-write leaves the prior good
file intact), NOT power-loss safe - no fsync, so an OS/power failure can still lose
the last write (acceptable: reconstructible single-process state, not a db)"""
target = await asyncio.to_thread(os.path.realpath, self.file)
directory = os.path.dirname(target) or "."
await asyncio.to_thread(os.makedirs, directory, exist_ok=True)
await self._sweep_stale_tmp(directory, os.path.basename(target))
payload = json.dumps(cache)
tmp = f"{self.file}.{os.getpid()}.tmp"
payload = await asyncio.to_thread(json.dumps, cache, allow_nan=False)
tmp = f"{target}.{os.getpid()}.tmp"
try:
async with aiofiles.open(tmp, mode="w") as f:
async with aiofiles.open(tmp, mode="w", encoding="utf-8") as f:
await f.write(payload)
await asyncio.to_thread(os.replace, tmp, self.file)
await asyncio.to_thread(os.replace, tmp, target)
except Exception:
if await asyncio.to_thread(os.path.exists, tmp):
try:
@@ -139,7 +137,46 @@ class AioKV:
log.exception("aiokv: failed to clean up temp file %s", tmp)
raise
async def _sweep_stale_tmp(self, directory: str, base: str) -> None:
"""remove orphaned .<pid>.tmp files left by a hard crash of a different, dead process
# back-compat: this lib was originally named aiocache; legacy call sites using
# `aiocache(...)` keep working via this alias. prefer AioKV in new code.
matches only the exact shape _save() creates (<realpath basename>.<pid>.tmp), via a
re.escape'd regex over a directory listing rather than a raw glob - a glob pattern
both crosses unrelated dots (matching non-aiokv neighbors like foo.backup.999.tmp) and
mistreats glob metacharacters in the store's own filename (e.g. state[prod].json).
`base` is the realpath'd basename _save() already resolved - passed in rather than
re-resolved here so a save does one realpath call, not two."""
candidate = re.compile(rf"^{re.escape(base)}\.(\d+)\.tmp$")
names = await asyncio.to_thread(os.listdir, directory)
for name in names:
match = candidate.match(name)
if not match:
continue
pid = int(match.group(1))
if pid == os.getpid():
continue
if await asyncio.to_thread(self._pid_alive, pid):
continue
path = os.path.join(directory, name)
try:
await asyncio.to_thread(os.remove, path)
except FileNotFoundError:
pass
except Exception:
log.exception("aiokv: failed to sweep stale temp file %s", path)
@staticmethod
def _pid_alive(pid: int) -> bool:
"""check whether pid refers to a live process, without permission to signal counting as alive"""
try:
os.kill(pid, 0)
except (ProcessLookupError, OverflowError):
# OverflowError: a pid too large for a C int can't name a live process
return False
except PermissionError:
return True
return True
# back-compat: originally named aiocache; prefer AioKV in new code.
aiocache = AioKV