2 Commits
Author SHA1 Message Date
dsql 8bf1866ca2 fix: gzip crash-safety + size rotation always bounds live file (logsetup-7/8/9)
_gzip_file now writes to a .tmp sibling and atomically os.replaces it onto
the final .gz path, so a crash/OOM mid-write can never leave a truncated
.gz where retention trusts it; retier's plain/gz dedupe additionally
verifies a .gz decompresses cleanly before removing its plain twin, so a
corrupt .gz is never preferred over the last intact copy.

rotate="size" now always forces the handler's backupCount to fire
regardless of backup_count, so backup_count=0 means "keep zero rolled
files" (each roll deletes itself immediately) instead of silently
disabling rotation and growing the live file unbounded.

Also mirrors retier's mtime+roll-counter sort key into prune() so a
same-second burst with tied mtimes prunes the oldest files instead of
arbitrary listdir order (logsetup-9, adjacent one-line fix).

Bumps to v0.5.1.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 17:24:26 -04:00
dsql 1207c53742 feat: historic logs named off the project namespace (cwd basename / history_name)
the live file keeps its defined name (latest.log); rolled/historic files are now named off
the PROJECT namespace so you can tell which service a log came from at a glance. default =
os.path.basename(os.getcwd()) (run from bestbuy/ -> historic bestbuy.<stamp>.log[.gz]);
override with the new history_name= param.

- decouple the rolled stem (history) from the live stem (name) in setup.py; thread
  history_stem into BOTH the namers AND prune/retier (same stem, or retention breaks).
- unify size + daily on make_history_namer -> <stem>.<stamp>.log[.gz] (was stdlib .N /
  .log.<date>); rotate_on_start takes history_stem.

behavior-visible: rolled names differ from pre-0.5.0 (were the live stem). pass
history_name=name for the old naming. execute-verified across on_start/daily/size (tiered +
non-tiered), default-cwd + explicit + history==name + normalization edges; v0.4.x fixes
(rotation/tiering/retention) all still hold. bump v0.4.3 -> v0.5.0

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-01 01:58:10 -04:00
5 changed files with 238 additions and 75 deletions
+52 -10
View File
@@ -13,12 +13,12 @@ and emit; their records flow into the handlers `log_setup` wired.
## Install ## Install
``` ```
log_setup @ git+ssh://git@git.rethinkstudios.io/rethink-public/log_setup.git@v0.4.3 log_setup @ git+ssh://git@git.rethinkstudios.io/rethink-public/log_setup.git@v0.5.1
``` ```
No dependencies — stdlib only. No dependencies — stdlib only.
Drop the `@v0.4.3` suffix from the line above to install the latest unpinned. Drop the `@v0.5.1` suffix from the line above to install the latest unpinned.
## Quick start ## Quick start
@@ -43,22 +43,48 @@ emits; the records land in the configured root.
- **Format:** `2026-06-27 19:55:05 | module.name | INFO | message`. `%(name)s` is the - **Format:** `2026-06-27 19:55:05 | module.name | INFO | message`. `%(name)s` is the
`getLogger` name each module used, so you see which lib/module logged. `getLogger` name each module used, so you see which lib/module logged.
- **Rotation** (`rotate=`): - **Rotation** (`rotate=`):
- `"daily"` (default) — rolls at midnight, dated name into `log_dir`, keeps - `"daily"` (default) — rolls at midnight into `log_dir`, keeps `backup_count` days.
`backup_count` days. - `"size"` — rolls at `max_bytes` into `log_dir`, keeps `backup_count`. `backup_count=0`
- `"size"` — rolls at `max_bytes`; numbered backups (`run.log.1`, `.2`, …) in `log_dir` means **keep no rolled history**: the live file still rolls at `max_bytes` (size is
for the default flat retention, or timestamped names when tiered retention is on (below). always bounded), each rolled file is deleted immediately after landing — it does not
- `"on_start"` — on startup, moves an existing `run.log` into `log_dir` disable rotation (see **Retention** below).
(`run.<timestamp>.log[.gz]`) and starts fresh; prunes to `backup_count`. - `"on_start"` — on startup, moves an existing live file into `log_dir` and starts fresh;
prunes to `backup_count`.
- `None` — single file, no rotation. - `None` — single file, no rotation.
- **compress=True** (default) gzips each rolled file (`run.log.2026-06-27.gz`). - **Historic files are named off the project** — see below. Every rolled file is
`<project>.<timestamp>.log[.gz]`; the live file keeps its own name.
- **compress=True** (default) gzips each rolled file.
- **Retention** = `backup_count` (default 14) for every mode — unless tiered retention is - **Retention** = `backup_count` (default 14) for every mode — unless tiered retention is
enabled (below). enabled (below). For `rotate="size"`, `backup_count=0` is "keep none" (not "disable
rotation") — see the `size` bullet above and the note at the bottom of this section.
- **console=True** (off by default) also logs to stdout in the same format — opt in when - **console=True** (off by default) also logs to stdout in the same format — opt in when
you want live terminal output alongside the file. you want live terminal output alongside the file.
The `name` you pass is normalized so it produces exactly one `.log`: `name="latest"` and The `name` you pass is normalized so it produces exactly one `.log`: `name="latest"` and
`name="latest.log"` both yield the live file `latest.log` (never `latest.log.log`). `name="latest.log"` both yield the live file `latest.log` (never `latest.log.log`).
## Historic files are named off the project (`history_name`)
The **live** file keeps its defined `name` (`latest.log`). The **historic** (rolled/gz)
files are named off the **project namespace** — by default the current directory's basename
— so you can tell at a glance which service a log came from:
```python
# app run from bestbuy/run.py , with name="latest":
setup_logging(name="latest", rotate="daily")
# logs/
# latest.log <- live (the tail -f target)
# bestbuy.2026-07-01_02-00-00.log <- historic, named off the project dir
# bestbuy.2026-06-30_02-00-00.log.gz
```
- **Default** = `os.path.basename(os.getcwd())` (the project directory). Zero config.
- Override with **`history_name="foo"`** → historic files become `foo.<timestamp>.log[.gz]`.
- This changed in **v0.5.0**: historic files used to reuse the live `name`. To keep the old
behavior, pass `history_name=name`.
- Retention (tier counts / `backup_count`) is unchanged — it's just keyed to the project
stem now.
## Tiered retention (`keep_uncompressed` / `keep_compressed`) ## Tiered retention (`keep_uncompressed` / `keep_compressed`)
The default is a flat `backup_count`: every rolled file is gzipped on roll and the oldest The default is a flat `backup_count`: every rolled file is gzipped on roll and the oldest
@@ -134,6 +160,7 @@ setup_logging(
level="INFO", # root level everything inherits (str name or logging constant) level="INFO", # root level everything inherits (str name or logging constant)
module_levels=None, # {logger_name: level} per-logger overrides (exact name match) module_levels=None, # {logger_name: level} per-logger overrides (exact name match)
rotate="daily", # "daily" | "size" | "on_start" | None rotate="daily", # "daily" | "size" | "on_start" | None
history_name=None, # stem for rolled/historic files; None -> cwd basename (project)
backup_count=14, # rotated files to keep (flat retention; ignored if tiered) backup_count=14, # rotated files to keep (flat retention; ignored if tiered)
keep_uncompressed=None, # tiered: newest N rolled logs kept PLAIN (opt-in) keep_uncompressed=None, # tiered: newest N rolled logs kept PLAIN (opt-in)
keep_compressed=None, # tiered: next M rolled logs kept GZIPPED (opt-in) keep_compressed=None, # tiered: next M rolled logs kept GZIPPED (opt-in)
@@ -205,6 +232,21 @@ setup_logging(name="run", queue=True)
duplicate lines) and leaves handlers your app added itself alone. duplicate lines) and leaves handlers your app added itself alone.
- **Never crashes the app over logging:** if `log_dir` isn't writable, it falls back to - **Never crashes the app over logging:** if `log_dir` isn't writable, it falls back to
console-only with a warning instead of raising. console-only with a warning instead of raising.
- **`rotate="size"` always bounds the live file (v0.5.1+).** Previously, `backup_count=0`
with `rotate="size"` silently disabled rotation entirely (the live file grew forever,
ignoring `max_bytes`). As of v0.5.1, the live file always rolls at `max_bytes`
regardless of `backup_count`; `backup_count=0` means "keep zero rolled files" (each roll
is deleted right after it lands) rather than "never roll." `backup_count>=1` behaves as
documented (keeps that many rolled files). This does not change `"daily"`/`"on_start"`,
where `backup_count=0` still means "roll, but don't prune the rolled files" (unbounded
`log_dir` growth) — that is a separate, pre-existing knob, not this fix's scope.
- **Gzip writes are crash-safe (v0.5.1+).** `_gzip_file` now writes to a `.tmp` sibling and
atomically `os.replace`s it onto the final `.gz` path, so a crash/OOM/power-loss mid-write
can never leave a truncated `.gz` at the path retention logic trusts. Tiered retention's
plain/gz dedupe additionally verifies a `.gz` decompresses cleanly before deleting its
plain twin — a corrupt `.gz` (from before this fix, or an external cause) is never
preferred over an intact plain copy; the plain is kept and the `.gz` gets rewritten
cleanly on the next retier pass instead of being deleted.
## Scope — what this is NOT ## Scope — what this is NOT
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project] [project]
name = "log_setup" name = "log_setup"
version = "0.4.3" version = "0.5.1"
description = "stdlib app-entry-point logging setup: live run.log, rotation, gzip, retention, consistent format" description = "stdlib app-entry-point logging setup: live run.log, rotation, gzip, retention, consistent format"
requires-python = ">=3.10" requires-python = ">=3.10"
dependencies = [] dependencies = []
+1 -1
View File
@@ -19,4 +19,4 @@ from .setup import setup_logging
__all__ = ["setup_logging"] __all__ = ["setup_logging"]
__version__ = "0.4.3" __version__ = "0.5.0"
+127 -49
View File
@@ -4,6 +4,13 @@ the stdlib rotating handlers roll a file next to the live file; these helpers
override the namer/rotator so rolled files land in `log_dir` and are gzipped when override the namer/rotator so rolled files land in `log_dir` and are gzipped when
asked, keep the live file at its stable path, and handle the on-start and prune asked, keep the live file at its stable path, and handle the on-start and prune
paths the handlers don't manage themselves. paths the handlers don't manage themselves.
gzip writes are crash-safe: `_gzip_file` writes to a `.tmp` sibling and atomically
`os.replace`s it onto the final `.gz` path, so a crash/OOM/power-loss mid-write never
leaves a truncated `.gz` where retention logic would trust it. `retier`'s plain/gz
dedupe additionally verifies a `.gz` decompresses cleanly (`_gz_intact`) before removing
its plain twin, so a corrupt `.gz` (pre-existing or externally caused) is never preferred
over an intact plain copy.
""" """
import gzip import gzip
@@ -53,13 +60,28 @@ def _free_dest(dest: str) -> str:
def _gzip_file(source: str, dest: str) -> None: def _gzip_file(source: str, dest: str) -> None:
"""gzip source into dest then remove source (the rolled-file compression idiom) """gzip source into dest then remove source (the rolled-file compression idiom)
writes to `dest + ".tmp"` first and `os.replace`s it onto `dest` once the gzip
write is complete, so a crash/OOM/power-loss mid-write never leaves a truncated
`.gz` at `dest` — the partial write stays quarantined in the `.tmp` name and the
source is untouched (safe to retry). os.replace is atomic on the same filesystem,
which the `.tmp` sibling always is.
the source mtime is carried onto dest so a file keeps its position when it crosses the source mtime is carried onto dest so a file keeps its position when it crosses
the plain->gz tier boundary — retier ranks by mtime, and a fresh write would the plain->gz tier boundary — retier ranks by mtime, and a fresh write would
otherwise make a just-compressed file look like the newest one and reshuffle tiers. otherwise make a just-compressed file look like the newest one and reshuffle tiers.
""" """
mtime = _safe_mtime(source) mtime = _safe_mtime(source)
with open(source, "rb") as src, gzip.open(dest, "wb") as dst: tmp_dest = dest + ".tmp"
shutil.copyfileobj(src, dst) try:
with open(source, "rb") as src, gzip.open(tmp_dest, "wb") as dst:
shutil.copyfileobj(src, dst)
except BaseException:
try:
os.remove(tmp_dest)
except OSError:
pass
raise
os.replace(tmp_dest, dest)
os.remove(source) os.remove(source)
try: try:
os.utime(dest, (mtime, mtime)) os.utime(dest, (mtime, mtime))
@@ -67,6 +89,24 @@ def _gzip_file(source: str, dest: str) -> None:
pass pass
def _gz_intact(path: str) -> bool:
"""return True if the gzip file at path decompresses cleanly end to end
belt-and-suspenders check before a dedupe site removes a plain twin in favor of
its .gz — a truncated/corrupt .gz (partial write, bad copy, disk error) must never
be trusted over an intact plain copy. reads the whole stream since gzip.open only
validates the end-of-stream marker on a full read; any failure (BadGzipFile, EOFError,
OSError, zlib error) is treated as "not intact" so the caller keeps the plain source.
"""
try:
with gzip.open(path, "rb") as handle:
while handle.read(1 << 20):
pass
return True
except Exception:
return False
def make_namer(log_dir: str, compress: bool) -> Callable[[str], str]: def make_namer(log_dir: str, compress: bool) -> Callable[[str], str]:
"""namer: redirect a rolled filename into log_dir, adding .gz when compressing """namer: redirect a rolled filename into log_dir, adding .gz when compressing
@@ -80,27 +120,32 @@ def make_namer(log_dir: str, compress: bool) -> Callable[[str], str]:
return namer return namer
def make_size_namer( def make_history_namer(
stem: str, log_dir: str, clock=time.localtime, stem: str, log_dir: str, compress: bool = False, plain: bool = False,
clock=time.localtime,
) -> Callable[[str], str]: ) -> Callable[[str], str]:
"""namer for tiered SIZE mode: a unique timestamped dest per roll, plain (no .gz) """namer minting historic rolled files `<stem>.<Y-m-d_H-M-S>.log[.gz]` in log_dir
stdlib RotatingFileHandler names rolls `<live>.1`, `<live>.2`, ... and shifts them — used by size and daily (and their tiered variants). `stem` is the HISTORY stem (the
a scheme that breaks once files are redirected into log_dir (the shift can't find project namespace), independent of the live file's name — the returned rolled files
them, so every roll reuses slot 1). tiered retention wants unique per-roll names it are keyed off it, and prune/retier glob the same stem.
can rank + tier like the daily/on_start paths, so ignore the handler's `.N` suffix
entirely and mint `<stem>.<Y-m-d_H-M-S>.log`, disambiguating a same-second collision the stdlib handler's own rolled name (`.N` for size, `.log.<date>` for daily) is
(against both the .log and .log.gz forms) with a counter. always plain — retier ignored — we mint our own uniform timestamped name so all modes converge on one shape
decides compression. and retier can rank/tier them. `plain=True` (tiered mode) always lands `.log` and lets
retier decide compression; otherwise `.gz` is appended when `compress`. same-second
collisions are disambiguated with a counter, checking both .log and .log.gz forms.
""" """
def namer(default_name: str) -> str: def namer(default_name: str) -> str:
stamp = time.strftime("%Y-%m-%d_%H-%M-%S", clock()) stamp = time.strftime("%Y-%m-%d_%H-%M-%S", clock())
dest = os.path.join(log_dir, f"{stem}.{stamp}.log") base = os.path.join(log_dir, f"{stem}.{stamp}")
candidate = base
counter = 1 counter = 1
while os.path.exists(dest) or os.path.exists(dest + ".gz"): while os.path.exists(candidate + ".log") or os.path.exists(candidate + ".log.gz"):
dest = os.path.join(log_dir, f"{stem}.{stamp}.{counter}.log") candidate = f"{base}.{counter}"
counter += 1 counter += 1
return dest suffix = ".log.gz" if (compress and not plain) else ".log"
return candidate + suffix
return namer return namer
@@ -115,7 +160,10 @@ def make_rotator(
`backup_count` newest rolled files. the stdlib handler's own retention `backup_count` newest rolled files. the stdlib handler's own retention
(`getFilesToDelete`) only scans the live file's directory, so it never sees the (`getFilesToDelete`) only scans the live file's directory, so it never sees the
rolled files we redirect into `log_dir` — pruning here is what bounds retention for rolled files we redirect into `log_dir` — pruning here is what bounds retention for
the daily and size rolling modes. the daily and size rolling modes. `backup_count <= 0` means "keep no rolled history":
`prune()` itself no-ops at `backup_count <= 0` (it means "don't touch history"), so a
zero-retention roll is deleted by the rotator directly right after landing, rather than
relying on prune to do it.
tiered mode (when `keep_uncompressed`/`keep_compressed` are given): land the rolled tiered mode (when `keep_uncompressed`/`keep_compressed` are given): land the rolled
file PLAIN and re-tier `log_dir` — newest `keep_uncompressed` stay uncompressed, the file PLAIN and re-tier `log_dir` — newest `keep_uncompressed` stay uncompressed, the
@@ -142,7 +190,15 @@ def make_rotator(
_gzip_file(source, dest) _gzip_file(source, dest)
else: else:
_move(source, dest) _move(source, dest)
if log_dir is not None and prune_stem is not None: if backup_count <= 0:
# "keep no history": prune() no-ops at backup_count <= 0 (that's its "leave
# history alone" sentinel, not "delete everything"), so a zero-retention roll
# deletes its own just-landed file directly instead of relying on prune.
try:
os.remove(dest)
except OSError:
pass
elif log_dir is not None and prune_stem is not None:
prune(log_dir, prune_stem, backup_count) prune(log_dir, prune_stem, backup_count)
return rotator return rotator
@@ -150,9 +206,14 @@ def make_rotator(
def rotate_on_start( def rotate_on_start(
live_path: str, log_dir: str, compress: bool, clock=time.localtime, live_path: str, log_dir: str, compress: bool, clock=time.localtime,
keep_uncompressed: Optional[int] = None, keep_compressed: Optional[int] = None, keep_uncompressed: Optional[int] = None, keep_compressed: Optional[int] = None,
history_stem: Optional[str] = None,
) -> None: ) -> None:
"""move an existing live file into log_dir with a timestamp, gzipped if asked """move an existing live file into log_dir with a timestamp, gzipped if asked
the rolled file is named off `history_stem` (the project namespace) when given, so
historic files carry the project name independent of the live file's stem; falls back
to the live file's own stem when history_stem is None/empty.
no-op if the live file doesn't exist. used by rotate="on_start" before the fresh no-op if the live file doesn't exist. used by rotate="on_start" before the fresh
handler opens a new live file. the timestamp form is run.<%Y-%m-%d_%H-%M-%S>.log. handler opens a new live file. the timestamp form is run.<%Y-%m-%d_%H-%M-%S>.log.
@@ -164,7 +225,8 @@ def rotate_on_start(
if not os.path.exists(live_path): if not os.path.exists(live_path):
return return
tiered = keep_uncompressed is not None or keep_compressed is not None tiered = keep_uncompressed is not None or keep_compressed is not None
stem = os.path.splitext(os.path.basename(live_path))[0] live_stem = os.path.splitext(os.path.basename(live_path))[0]
stem = os.path.basename(history_stem) if history_stem else live_stem
stamp = time.strftime("%Y-%m-%d_%H-%M-%S", clock()) stamp = time.strftime("%Y-%m-%d_%H-%M-%S", clock())
suffix = ".log.gz" if (compress and not tiered) else ".log" suffix = ".log.gz" if (compress and not tiered) else ".log"
# the stamp is 1-second resolution; two starts in the same second would collide # the stamp is 1-second resolution; two starts in the same second would collide
@@ -220,16 +282,24 @@ def retier(log_dir: str, stem: str, keep_uncompressed: int, keep_compressed: int
entries = [os.path.join(log_dir, name) for name in names] entries = [os.path.join(log_dir, name) for name in names]
# dedupe plain/gz twins FIRST (a crash between _gzip_file's write and its os.remove can # dedupe plain/gz twins FIRST (a crash between _gzip_file's write and its os.remove can
# leave <x>.log beside <x>.log.gz). drop the redundant plain copy up front so the # leave <x>.log beside <x>.log.gz). drop the redundant plain copy up front so the
# phantom twin never occupies a retention slot and evicts a distinct older roll. # phantom twin never occupies a retention slot and evicts a distinct older roll — but
# ONLY once the .gz is verified to decompress cleanly. _gzip_file now writes atomically
# (temp+os.replace) so a fresh truncated twin can't occur, but a pre-existing corrupt
# .gz (older data, bad copy, disk error) must never win over an intact plain copy: that
# would delete the only good copy and keep garbage, strictly worse than doing nothing.
present = set(entries) present = set(entries)
kept = [] kept = []
for p in entries: for p in entries:
if not p.endswith(".gz") and (p + ".gz") in present: if not p.endswith(".gz") and (p + ".gz") in present:
try: if _gz_intact(p + ".gz"):
os.remove(p) try:
except OSError: os.remove(p)
kept.append(p) # couldn't remove — keep it in the accounting except OSError:
continue kept.append(p) # couldn't remove — keep it in the accounting
continue
# .gz twin is corrupt/truncated — keep the intact plain, don't touch either
# file here (the second dedupe pass below or a future retier will retry the
# compress once the plain is re-gzipped cleanly)
kept.append(p) kept.append(p)
files = [(p, _safe_mtime(p), _roll_counter(p)) for p in kept if os.path.isfile(p)] files = [(p, _safe_mtime(p), _roll_counter(p)) for p in kept if os.path.isfile(p)]
# newest-first: higher mtime first, and within a tied second the higher roll counter # newest-first: higher mtime first, and within a tied second the higher roll counter
@@ -249,12 +319,17 @@ def retier(log_dir: str, stem: str, keep_uncompressed: int, keep_compressed: int
# a crash/power-loss between _gzip_file's write and its os.remove can leave # a crash/power-loss between _gzip_file's write and its os.remove can leave
# a plain source beside a fresh .gz. don't keep both (they'd double-count # a plain source beside a fresh .gz. don't keep both (they'd double-count
# toward retention and evict a distinct older roll) — drop the redundant # toward retention and evict a distinct older roll) — drop the redundant
# plain twin, keeping the compressed copy. # plain twin, but ONLY once the .gz is verified intact (see _gz_intact):
try: # a corrupt/truncated .gz here must never win over the last good copy.
os.remove(path) if _gz_intact(dest):
except OSError: try:
pass os.remove(path)
continue except OSError:
pass
continue
# .gz is corrupt — fall through and re-gzip the plain over the bad dest
# (_gzip_file writes atomically, so the corrupt dest is only replaced once
# a fully valid archive exists)
try: try:
_gzip_file(path, dest) _gzip_file(path, dest)
except OSError: except OSError:
@@ -283,8 +358,9 @@ def _roll_counter(path: str) -> int:
def prune(log_dir: str, stem: str, backup_count: int) -> None: def prune(log_dir: str, stem: str, backup_count: int) -> None:
"""keep only the newest `backup_count` rolled files for a given stem in log_dir """keep only the newest `backup_count` rolled files for a given stem in log_dir
matches files beginning with `<stem>.` (e.g. run.*), sorted by mtime, deleting the matches files beginning with `<stem>.` (e.g. run.*), sorted by mtime (then by the
oldest beyond the count. used for on_start, which the handlers don't auto-prune. roll counter parsed from the name, mirroring retier — see _roll_counter), deleting
the oldest beyond the count. used for on_start, which the handlers don't auto-prune.
`stem` is reduced to its basename so a `name` containing a directory (e.g. "sub/run") `stem` is reduced to its basename so a `name` containing a directory (e.g. "sub/run")
still matches the basenamed rolled files in log_dir (else nothing matches and old still matches the basenamed rolled files in log_dir (else nothing matches and old
@@ -301,9 +377,11 @@ def prune(log_dir: str, stem: str, backup_count: int) -> None:
] ]
except OSError: except OSError:
return return
files = [(p, _safe_mtime(p)) for p in entries if os.path.isfile(p)] files = [(p, _safe_mtime(p), _roll_counter(p)) for p in entries if os.path.isfile(p)]
files.sort(key=lambda pair: pair[1], reverse=True) # newest-first: higher mtime first, and within a tied second the higher roll counter
for path, _ in files[backup_count:]: # (a later same-second roll) is newer — mirrors retier's ordering (line ~289)
files.sort(key=lambda t: (t[1], t[2]), reverse=True)
for path, _, _ in files[backup_count:]:
try: try:
os.remove(path) os.remove(path)
except OSError: except OSError:
@@ -322,24 +400,24 @@ def attach_rolling(
handler, log_dir: str, compress: bool, handler, log_dir: str, compress: bool,
prune_stem: Optional[str] = None, backup_count: int = 0, prune_stem: Optional[str] = None, backup_count: int = 0,
keep_uncompressed: Optional[int] = None, keep_compressed: Optional[int] = None, keep_uncompressed: Optional[int] = None, keep_compressed: Optional[int] = None,
size_tiered: bool = False, tiered: bool = False,
) -> Tuple[Callable, Callable]: ) -> Tuple[Callable, Callable]:
"""wire the custom namer + rotator onto a rotating handler; return them """wire the custom namer + rotator onto a rotating handler; return them
pass `prune_stem`/`backup_count` so the rotator prunes `log_dir` after each roll rolled files are named off `prune_stem` (the HISTORY stem — the project namespace),
(the handler's own retention can't see the redirected rolled files). pass independent of the live file name, via make_history_namer: `<stem>.<stamp>.log[.gz]`
`keep_uncompressed`/`keep_compressed` instead to use tiered retention (newest plain, uniform across size and daily. this replaces the stdlib handler's own rolled-name
next gzipped, rest deleted) — see make_rotator. scheme (`.N` for size, `.log.<date>` for daily), which can't inject a project stem and
(for size) can't be managed once files are redirected into log_dir.
`size_tiered` uses a timestamped per-roll namer (make_size_namer) instead of the pass `prune_stem`/`backup_count` so the rotator prunes `log_dir` after each roll (the
default one, for a tiered RotatingFileHandler (size mode): stdlib's `.1/.2` numbered handler's own retention can't see the redirected rolled files). pass
shift can't manage files redirected into log_dir, so each roll gets a unique dated `keep_uncompressed`/`keep_compressed` for tiered retention (newest plain, next gzipped,
name that retier ranks/tiers like the daily/on_start paths. rest deleted) — see make_rotator. `tiered=True` lands rolls plain (retier compresses).
""" """
if size_tiered: namer = make_history_namer(
namer = make_size_namer(os.path.basename(prune_stem or ""), log_dir) os.path.basename(prune_stem or ""), log_dir, compress, plain=tiered,
else: )
namer = make_namer(log_dir, compress)
rotator = make_rotator( rotator = make_rotator(
compress, log_dir, prune_stem, backup_count, keep_uncompressed, keep_compressed, compress, log_dir, prune_stem, backup_count, keep_uncompressed, keep_compressed,
) )
+57 -14
View File
@@ -6,6 +6,10 @@ of rolled files, retention, console output, and a consistent format. it is calle
the APPLICATION, not by reusable libraries (those stay emit-only). it is idempotent the APPLICATION, not by reusable libraries (those stay emit-only). it is idempotent
(no duplicate handlers on repeat calls), never crashes the app over logging, and can (no duplicate handlers on repeat calls), never crashes the app over logging, and can
route through a background queue so an async event loop doesn't block on file I/O. route through a background queue so an async event loop doesn't block on file I/O.
`rotate="size"` always bounds the live file: the roll fires at `max_bytes` regardless
of `backup_count`, including `backup_count=0` (which means "keep zero rolled files",
not "never roll" — each roll is deleted right after landing).
""" """
import atexit import atexit
@@ -120,45 +124,69 @@ def _normalize_name(name: str) -> str:
return name return name
def _history_stem() -> str:
"""the project namespace for historic files: the cwd basename
a service run from bestbuy/ gives historic files bestbuy.<stamp>.log[.gz]. falls back
to an empty string only for a degenerate cwd (e.g. "/"), which the caller resolves to
the live stem.
"""
try:
return os.path.basename(os.getcwd().rstrip(os.sep))
except OSError:
return ""
def _file_handler( def _file_handler(
name: str, live_path: str, log_dir: str, rotate: Optional[str], name: str, history_stem: str, live_path: str, log_dir: str, rotate: Optional[str],
backup_count: int, max_bytes: int, compress: bool, backup_count: int, max_bytes: int, compress: bool,
keep_uncompressed: Optional[int], keep_compressed: Optional[int], keep_uncompressed: Optional[int], keep_compressed: Optional[int],
) -> logging.Handler: ) -> logging.Handler:
"""build the configured file handler with custom rolling into log_dir""" """build the configured file handler with custom rolling into log_dir
`name` is the LIVE stem (drives live_path); `history_stem` is the PROJECT stem that
rolled/historic files are named off + the retention glob keys on. they are decoupled:
the live file keeps its defined name, historic files carry the project namespace.
"""
tiered = keep_uncompressed is not None or keep_compressed is not None tiered = keep_uncompressed is not None or keep_compressed is not None
if rotate == "size": if rotate == "size":
# stdlib doRollover is a no-op when backupCount == 0, and its numbered .1/.2 shift # stdlib doRollover is a no-op when backupCount == 0, and its numbered .1/.2 shift
# can't manage files redirected into log_dir. in tiered size mode force a nonzero # can't manage files redirected into log_dir. force a nonzero backupCount so the
# backupCount so the roll always fires (retier bounds retention, not backupCount) # roll always fires regardless of mode — tiered AND legacy — and let
# and use the timestamped size-namer (size_tiered) instead of the .N shift. # attach_rolling's history namer name + retier/prune bound retention (keyed to
size_backup = backup_count if not tiered else max(backup_count, 1) # history_stem). the ORIGINAL backup_count (which may be 0) still flows into
# attach_rolling below: make_rotator treats backup_count <= 0 there as "keep no
# rolled history" and deletes each roll right after it lands, rather than passing
# 0 to prune() (whose own backup_count <= 0 means "leave history alone", a no-op —
# that mismatch is exactly what silently disabled rotation before).
size_backup = max(backup_count, 1)
handler = logging.handlers.RotatingFileHandler( handler = logging.handlers.RotatingFileHandler(
live_path, maxBytes=max_bytes, backupCount=size_backup, encoding="utf-8", live_path, maxBytes=max_bytes, backupCount=size_backup, encoding="utf-8",
) )
attach_rolling( attach_rolling(
handler, log_dir, compress, prune_stem=name, backup_count=backup_count, handler, log_dir, compress, prune_stem=history_stem, backup_count=backup_count,
keep_uncompressed=keep_uncompressed, keep_compressed=keep_compressed, keep_uncompressed=keep_uncompressed, keep_compressed=keep_compressed,
size_tiered=tiered, tiered=tiered,
) )
elif rotate == "daily": elif rotate == "daily":
handler = logging.handlers.TimedRotatingFileHandler( handler = logging.handlers.TimedRotatingFileHandler(
live_path, when="midnight", backupCount=backup_count, encoding="utf-8", live_path, when="midnight", backupCount=backup_count, encoding="utf-8",
) )
attach_rolling( attach_rolling(
handler, log_dir, compress, prune_stem=name, backup_count=backup_count, handler, log_dir, compress, prune_stem=history_stem, backup_count=backup_count,
keep_uncompressed=keep_uncompressed, keep_compressed=keep_compressed, keep_uncompressed=keep_uncompressed, keep_compressed=keep_compressed,
tiered=tiered,
) )
else: else:
if rotate == "on_start": if rotate == "on_start":
if tiered: if tiered:
rotate_on_start( rotate_on_start(
live_path, log_dir, compress, live_path, log_dir, compress, history_stem=history_stem,
keep_uncompressed=keep_uncompressed, keep_compressed=keep_compressed, keep_uncompressed=keep_uncompressed, keep_compressed=keep_compressed,
) )
else: else:
rotate_on_start(live_path, log_dir, compress) rotate_on_start(live_path, log_dir, compress, history_stem=history_stem)
prune(log_dir, name, backup_count) prune(log_dir, history_stem, backup_count)
elif rotate is not None: elif rotate is not None:
# a typo'd rotate value (e.g. "hourly") would otherwise silently fall through # a typo'd rotate value (e.g. "hourly") would otherwise silently fall through
# to a non-rotating FileHandler and grow forever — warn, matching the # to a non-rotating FileHandler and grow forever — warn, matching the
@@ -177,6 +205,7 @@ def setup_logging(
level: Union[int, str] = "INFO", level: Union[int, str] = "INFO",
module_levels: Optional[Dict[str, Union[int, str]]] = None, module_levels: Optional[Dict[str, Union[int, str]]] = None,
rotate: Optional[str] = "daily", rotate: Optional[str] = "daily",
history_name: Optional[str] = None,
backup_count: int = 14, backup_count: int = 14,
keep_uncompressed: Optional[int] = None, keep_uncompressed: Optional[int] = None,
keep_compressed: Optional[int] = None, keep_compressed: Optional[int] = None,
@@ -193,6 +222,10 @@ def setup_logging(
`name` -> <name>.log live file at cwd; rolled/compressed copies go to `log_dir`. a `name` -> <name>.log live file at cwd; rolled/compressed copies go to `log_dir`. a
trailing ".log" in `name` is stripped so "latest" and "latest.log" both produce the trailing ".log" in `name` is stripped so "latest" and "latest.log" both produce the
live file latest.log (never latest.log.log). live file latest.log (never latest.log.log).
`history_name` names the rolled/historic files (`<history_name>.<timestamp>.log[.gz]`),
independent of the live file: it defaults to the PROJECT namespace = the cwd basename
(run from bestbuy/ -> historic files bestbuy.<stamp>...), and can be set explicitly. the
live file always keeps `name`; only the historic files carry the project name.
`keep_uncompressed`/`keep_compressed` (default None) enable TIERED retention: when `keep_uncompressed`/`keep_compressed` (default None) enable TIERED retention: when
either is given, rolled files are kept as the newest `keep_uncompressed` uncompressed either is given, rolled files are kept as the newest `keep_uncompressed` uncompressed
+ the next `keep_compressed` gzipped, and the rest are deleted (total retained = + the next `keep_compressed` gzipped, and the rest are deleted (total retained =
@@ -209,7 +242,11 @@ def setup_logging(
applies, so naming a parent ("aiohttp") quiets its whole subtree (aiohttp.client, applies, so naming a parent ("aiohttp") quiets its whole subtree (aiohttp.client,
aiohttp.access, ...). each entry accepts a str or int level; a bad value for one aiohttp.access, ...). each entry accepts a str or int level; a bad value for one
entry is skipped with a warning and never aborts the others or the setup. entry is skipped with a warning and never aborts the others or the setup.
`rotate` is "daily" (default), "size", "on_start", or None. `console=True` adds a `rotate` is "daily" (default), "size", "on_start", or None. for `rotate="size"`, the
live file always rolls at `max_bytes` regardless of `backup_count`: `backup_count=0`
means "keep zero rolled files" (each roll lands then is deleted immediately), NOT
"disable rotation" — the live file is always bounded. `backup_count>=1` keeps that
many rolled files as before. `console=True` adds a
stdout handler (off by default — the file is the output). `queue=True` routes records stdout handler (off by default — the file is the output). `queue=True` routes records
through a background QueueListener so file I/O never blocks the caller (the listener through a background QueueListener so file I/O never blocks the caller (the listener
is stopped at exit). `output` is "text" (default, human `time | module | level | is stopped at exit). `output` is "text" (default, human `time | module | level |
@@ -231,6 +268,12 @@ def setup_logging(
formatter = build_formatter(output, fmt, datefmt) formatter = build_formatter(output, fmt, datefmt)
stem = _normalize_name(name) stem = _normalize_name(name)
live_path = f"{stem}.log" live_path = f"{stem}.log"
# historic/rolled files are named off the project namespace, independent of the live
# file: history_name if given, else the cwd basename (e.g. run from bestbuy/ -> historic
# files bestbuy.<stamp>.log[.gz]). normalized + basenamed like `name`; falls back to the
# live stem for a degenerate cwd so naming/retention never break.
history_source = history_name if history_name is not None else _history_stem()
history_stem = os.path.basename(_normalize_name(history_source)) or stem
handlers = [] handlers = []
@@ -243,7 +286,7 @@ def setup_logging(
if file_ok: if file_ok:
try: try:
fh = _file_handler( fh = _file_handler(
stem, live_path, log_dir, rotate, backup_count, max_bytes, compress, stem, history_stem, live_path, log_dir, rotate, backup_count, max_bytes, compress,
keep_uncompressed, keep_compressed, keep_uncompressed, keep_compressed,
) )
fh.setFormatter(formatter) fh.setFormatter(formatter)