Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8bf1866ca2 | ||
|
|
1207c53742 | ||
|
|
b52c1d37fa |
@@ -13,12 +13,12 @@ and emit; their records flow into the handlers `log_setup` wired.
|
||||
## Install
|
||||
|
||||
```
|
||||
log_setup @ git+ssh://git@git.rethinkstudios.io/rethink-public/log_setup.git@v0.4.2
|
||||
log_setup @ git+ssh://git@git.rethinkstudios.io/rethink-public/log_setup.git@v0.5.1
|
||||
```
|
||||
|
||||
No dependencies — stdlib only.
|
||||
|
||||
Drop the `@v0.4.2` suffix from the line above to install the latest unpinned.
|
||||
Drop the `@v0.5.1` suffix from the line above to install the latest unpinned.
|
||||
|
||||
## Quick start
|
||||
|
||||
@@ -43,21 +43,48 @@ emits; the records land in the configured root.
|
||||
- **Format:** `2026-06-27 19:55:05 | module.name | INFO | message`. `%(name)s` is the
|
||||
`getLogger` name each module used, so you see which lib/module logged.
|
||||
- **Rotation** (`rotate=`):
|
||||
- `"daily"` (default) — rolls at midnight, dated name into `log_dir`, keeps
|
||||
`backup_count` days.
|
||||
- `"size"` — rolls at `max_bytes`, numbered backups in `log_dir`.
|
||||
- `"on_start"` — on startup, moves an existing `run.log` into `log_dir`
|
||||
(`run.<timestamp>.log[.gz]`) and starts fresh; prunes to `backup_count`.
|
||||
- `"daily"` (default) — rolls at midnight into `log_dir`, keeps `backup_count` days.
|
||||
- `"size"` — rolls at `max_bytes` into `log_dir`, keeps `backup_count`. `backup_count=0`
|
||||
means **keep no rolled history**: the live file still rolls at `max_bytes` (size is
|
||||
always bounded), each rolled file is deleted immediately after landing — it does not
|
||||
disable rotation (see **Retention** below).
|
||||
- `"on_start"` — on startup, moves an existing live file into `log_dir` and starts fresh;
|
||||
prunes to `backup_count`.
|
||||
- `None` — single file, no rotation.
|
||||
- **compress=True** (default) gzips each rolled file (`run.log.2026-06-27.gz`).
|
||||
- **Historic files are named off the project** — see below. Every rolled file is
|
||||
`<project>.<timestamp>.log[.gz]`; the live file keeps its own name.
|
||||
- **compress=True** (default) gzips each rolled file.
|
||||
- **Retention** = `backup_count` (default 14) for every mode — unless tiered retention is
|
||||
enabled (below).
|
||||
enabled (below). For `rotate="size"`, `backup_count=0` is "keep none" (not "disable
|
||||
rotation") — see the `size` bullet above and the note at the bottom of this section.
|
||||
- **console=True** (off by default) also logs to stdout in the same format — opt in when
|
||||
you want live terminal output alongside the file.
|
||||
|
||||
The `name` you pass is normalized so it produces exactly one `.log`: `name="latest"` and
|
||||
`name="latest.log"` both yield the live file `latest.log` (never `latest.log.log`).
|
||||
|
||||
## Historic files are named off the project (`history_name`)
|
||||
|
||||
The **live** file keeps its defined `name` (`latest.log`). The **historic** (rolled/gz)
|
||||
files are named off the **project namespace** — by default the current directory's basename
|
||||
— so you can tell at a glance which service a log came from:
|
||||
|
||||
```python
|
||||
# app run from bestbuy/run.py , with name="latest":
|
||||
setup_logging(name="latest", rotate="daily")
|
||||
# logs/
|
||||
# latest.log <- live (the tail -f target)
|
||||
# bestbuy.2026-07-01_02-00-00.log <- historic, named off the project dir
|
||||
# bestbuy.2026-06-30_02-00-00.log.gz
|
||||
```
|
||||
|
||||
- **Default** = `os.path.basename(os.getcwd())` (the project directory). Zero config.
|
||||
- Override with **`history_name="foo"`** → historic files become `foo.<timestamp>.log[.gz]`.
|
||||
- This changed in **v0.5.0**: historic files used to reuse the live `name`. To keep the old
|
||||
behavior, pass `history_name=name`.
|
||||
- Retention (tier counts / `backup_count`) is unchanged — it's just keyed to the project
|
||||
stem now.
|
||||
|
||||
## Tiered retention (`keep_uncompressed` / `keep_compressed`)
|
||||
|
||||
The default is a flat `backup_count`: every rolled file is gzipped on roll and the oldest
|
||||
@@ -133,6 +160,7 @@ setup_logging(
|
||||
level="INFO", # root level everything inherits (str name or logging constant)
|
||||
module_levels=None, # {logger_name: level} per-logger overrides (exact name match)
|
||||
rotate="daily", # "daily" | "size" | "on_start" | None
|
||||
history_name=None, # stem for rolled/historic files; None -> cwd basename (project)
|
||||
backup_count=14, # rotated files to keep (flat retention; ignored if tiered)
|
||||
keep_uncompressed=None, # tiered: newest N rolled logs kept PLAIN (opt-in)
|
||||
keep_compressed=None, # tiered: next M rolled logs kept GZIPPED (opt-in)
|
||||
@@ -204,6 +232,21 @@ setup_logging(name="run", queue=True)
|
||||
duplicate lines) and leaves handlers your app added itself alone.
|
||||
- **Never crashes the app over logging:** if `log_dir` isn't writable, it falls back to
|
||||
console-only with a warning instead of raising.
|
||||
- **`rotate="size"` always bounds the live file (v0.5.1+).** Previously, `backup_count=0`
|
||||
with `rotate="size"` silently disabled rotation entirely (the live file grew forever,
|
||||
ignoring `max_bytes`). As of v0.5.1, the live file always rolls at `max_bytes`
|
||||
regardless of `backup_count`; `backup_count=0` means "keep zero rolled files" (each roll
|
||||
is deleted right after it lands) rather than "never roll." `backup_count>=1` behaves as
|
||||
documented (keeps that many rolled files). This does not change `"daily"`/`"on_start"`,
|
||||
where `backup_count=0` still means "roll, but don't prune the rolled files" (unbounded
|
||||
`log_dir` growth) — that is a separate, pre-existing knob, not this fix's scope.
|
||||
- **Gzip writes are crash-safe (v0.5.1+).** `_gzip_file` now writes to a `.tmp` sibling and
|
||||
atomically `os.replace`s it onto the final `.gz` path, so a crash/OOM/power-loss mid-write
|
||||
can never leave a truncated `.gz` at the path retention logic trusts. Tiered retention's
|
||||
plain/gz dedupe additionally verifies a `.gz` decompresses cleanly before deleting its
|
||||
plain twin — a corrupt `.gz` (from before this fix, or an external cause) is never
|
||||
preferred over an intact plain copy; the plain is kept and the `.gz` gets rewritten
|
||||
cleanly on the next retier pass instead of being deleted.
|
||||
|
||||
## Scope — what this is NOT
|
||||
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "log_setup"
|
||||
version = "0.4.2"
|
||||
version = "0.5.1"
|
||||
description = "stdlib app-entry-point logging setup: live run.log, rotation, gzip, retention, consistent format"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = []
|
||||
|
||||
@@ -19,4 +19,4 @@ from .setup import setup_logging
|
||||
|
||||
__all__ = ["setup_logging"]
|
||||
|
||||
__version__ = "0.4.2"
|
||||
__version__ = "0.5.0"
|
||||
|
||||
+188
-43
@@ -4,6 +4,13 @@ the stdlib rotating handlers roll a file next to the live file; these helpers
|
||||
override the namer/rotator so rolled files land in `log_dir` and are gzipped when
|
||||
asked, keep the live file at its stable path, and handle the on-start and prune
|
||||
paths the handlers don't manage themselves.
|
||||
|
||||
gzip writes are crash-safe: `_gzip_file` writes to a `.tmp` sibling and atomically
|
||||
`os.replace`s it onto the final `.gz` path, so a crash/OOM/power-loss mid-write never
|
||||
leaves a truncated `.gz` where retention logic would trust it. `retier`'s plain/gz
|
||||
dedupe additionally verifies a `.gz` decompresses cleanly (`_gz_intact`) before removing
|
||||
its plain twin, so a corrupt `.gz` (pre-existing or externally caused) is never preferred
|
||||
over an intact plain copy.
|
||||
"""
|
||||
|
||||
import gzip
|
||||
@@ -32,16 +39,49 @@ def _move(source: str, dest: str) -> None:
|
||||
shutil.move(source, dest)
|
||||
|
||||
|
||||
def _free_dest(dest: str) -> str:
|
||||
"""return `dest`, or a `.N`-suffixed variant if it (or its .gz twin) already exists
|
||||
|
||||
used by the tiered rotator so a second roll landing on the same dated/stamped name
|
||||
(e.g. two daily rolls in one day) doesn't clobber the earlier file. the suffix goes
|
||||
before nothing here (dest is already the plain path) — checks both the plain and .gz
|
||||
forms of each candidate.
|
||||
"""
|
||||
if not os.path.exists(dest) and not os.path.exists(dest + ".gz"):
|
||||
return dest
|
||||
counter = 1
|
||||
while True:
|
||||
candidate = f"{dest}.{counter}"
|
||||
if not os.path.exists(candidate) and not os.path.exists(candidate + ".gz"):
|
||||
return candidate
|
||||
counter += 1
|
||||
|
||||
|
||||
def _gzip_file(source: str, dest: str) -> None:
|
||||
"""gzip source into dest then remove source (the rolled-file compression idiom)
|
||||
|
||||
writes to `dest + ".tmp"` first and `os.replace`s it onto `dest` once the gzip
|
||||
write is complete, so a crash/OOM/power-loss mid-write never leaves a truncated
|
||||
`.gz` at `dest` — the partial write stays quarantined in the `.tmp` name and the
|
||||
source is untouched (safe to retry). os.replace is atomic on the same filesystem,
|
||||
which the `.tmp` sibling always is.
|
||||
|
||||
the source mtime is carried onto dest so a file keeps its position when it crosses
|
||||
the plain->gz tier boundary — retier ranks by mtime, and a fresh write would
|
||||
otherwise make a just-compressed file look like the newest one and reshuffle tiers.
|
||||
"""
|
||||
mtime = _safe_mtime(source)
|
||||
with open(source, "rb") as src, gzip.open(dest, "wb") as dst:
|
||||
shutil.copyfileobj(src, dst)
|
||||
tmp_dest = dest + ".tmp"
|
||||
try:
|
||||
with open(source, "rb") as src, gzip.open(tmp_dest, "wb") as dst:
|
||||
shutil.copyfileobj(src, dst)
|
||||
except BaseException:
|
||||
try:
|
||||
os.remove(tmp_dest)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
os.replace(tmp_dest, dest)
|
||||
os.remove(source)
|
||||
try:
|
||||
os.utime(dest, (mtime, mtime))
|
||||
@@ -49,6 +89,24 @@ def _gzip_file(source: str, dest: str) -> None:
|
||||
pass
|
||||
|
||||
|
||||
def _gz_intact(path: str) -> bool:
|
||||
"""return True if the gzip file at path decompresses cleanly end to end
|
||||
|
||||
belt-and-suspenders check before a dedupe site removes a plain twin in favor of
|
||||
its .gz — a truncated/corrupt .gz (partial write, bad copy, disk error) must never
|
||||
be trusted over an intact plain copy. reads the whole stream since gzip.open only
|
||||
validates the end-of-stream marker on a full read; any failure (BadGzipFile, EOFError,
|
||||
OSError, zlib error) is treated as "not intact" so the caller keeps the plain source.
|
||||
"""
|
||||
try:
|
||||
with gzip.open(path, "rb") as handle:
|
||||
while handle.read(1 << 20):
|
||||
pass
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def make_namer(log_dir: str, compress: bool) -> Callable[[str], str]:
|
||||
"""namer: redirect a rolled filename into log_dir, adding .gz when compressing
|
||||
|
||||
@@ -62,27 +120,32 @@ def make_namer(log_dir: str, compress: bool) -> Callable[[str], str]:
|
||||
return namer
|
||||
|
||||
|
||||
def make_size_namer(
|
||||
stem: str, log_dir: str, clock=time.localtime,
|
||||
def make_history_namer(
|
||||
stem: str, log_dir: str, compress: bool = False, plain: bool = False,
|
||||
clock=time.localtime,
|
||||
) -> Callable[[str], str]:
|
||||
"""namer for tiered SIZE mode: a unique timestamped dest per roll, plain (no .gz)
|
||||
"""namer minting historic rolled files `<stem>.<Y-m-d_H-M-S>.log[.gz]` in log_dir
|
||||
|
||||
stdlib RotatingFileHandler names rolls `<live>.1`, `<live>.2`, ... and shifts them —
|
||||
a scheme that breaks once files are redirected into log_dir (the shift can't find
|
||||
them, so every roll reuses slot 1). tiered retention wants unique per-roll names it
|
||||
can rank + tier like the daily/on_start paths, so ignore the handler's `.N` suffix
|
||||
entirely and mint `<stem>.<Y-m-d_H-M-S>.log`, disambiguating a same-second collision
|
||||
(against both the .log and .log.gz forms) with a counter. always plain — retier
|
||||
decides compression.
|
||||
used by size and daily (and their tiered variants). `stem` is the HISTORY stem (the
|
||||
project namespace), independent of the live file's name — the returned rolled files
|
||||
are keyed off it, and prune/retier glob the same stem.
|
||||
|
||||
the stdlib handler's own rolled name (`.N` for size, `.log.<date>` for daily) is
|
||||
ignored — we mint our own uniform timestamped name so all modes converge on one shape
|
||||
and retier can rank/tier them. `plain=True` (tiered mode) always lands `.log` and lets
|
||||
retier decide compression; otherwise `.gz` is appended when `compress`. same-second
|
||||
collisions are disambiguated with a counter, checking both .log and .log.gz forms.
|
||||
"""
|
||||
def namer(default_name: str) -> str:
|
||||
stamp = time.strftime("%Y-%m-%d_%H-%M-%S", clock())
|
||||
dest = os.path.join(log_dir, f"{stem}.{stamp}.log")
|
||||
base = os.path.join(log_dir, f"{stem}.{stamp}")
|
||||
candidate = base
|
||||
counter = 1
|
||||
while os.path.exists(dest) or os.path.exists(dest + ".gz"):
|
||||
dest = os.path.join(log_dir, f"{stem}.{stamp}.{counter}.log")
|
||||
while os.path.exists(candidate + ".log") or os.path.exists(candidate + ".log.gz"):
|
||||
candidate = f"{base}.{counter}"
|
||||
counter += 1
|
||||
return dest
|
||||
suffix = ".log.gz" if (compress and not plain) else ".log"
|
||||
return candidate + suffix
|
||||
return namer
|
||||
|
||||
|
||||
@@ -97,7 +160,10 @@ def make_rotator(
|
||||
`backup_count` newest rolled files. the stdlib handler's own retention
|
||||
(`getFilesToDelete`) only scans the live file's directory, so it never sees the
|
||||
rolled files we redirect into `log_dir` — pruning here is what bounds retention for
|
||||
the daily and size rolling modes.
|
||||
the daily and size rolling modes. `backup_count <= 0` means "keep no rolled history":
|
||||
`prune()` itself no-ops at `backup_count <= 0` (it means "don't touch history"), so a
|
||||
zero-retention roll is deleted by the rotator directly right after landing, rather than
|
||||
relying on prune to do it.
|
||||
|
||||
tiered mode (when `keep_uncompressed`/`keep_compressed` are given): land the rolled
|
||||
file PLAIN and re-tier `log_dir` — newest `keep_uncompressed` stay uncompressed, the
|
||||
@@ -111,8 +177,11 @@ def make_rotator(
|
||||
return
|
||||
if tiered:
|
||||
# dest carries the namer's .gz suffix in compress mode; strip it so the
|
||||
# freshly-rolled file lands plain and retier decides its tier
|
||||
plain_dest = dest[:-3] if dest.endswith(".gz") else dest
|
||||
# freshly-rolled file lands plain and retier decides its tier. disambiguate a
|
||||
# dest that already exists (a second same-interval daily roll reuses the same
|
||||
# dated name) with a counter, checking both .log and .log.gz forms, so the
|
||||
# earlier roll isn't clobbered.
|
||||
plain_dest = _free_dest(dest[:-3] if dest.endswith(".gz") else dest)
|
||||
_move(source, plain_dest)
|
||||
if log_dir is not None and prune_stem is not None:
|
||||
retier(log_dir, prune_stem, keep_uncompressed or 0, keep_compressed or 0)
|
||||
@@ -121,7 +190,15 @@ def make_rotator(
|
||||
_gzip_file(source, dest)
|
||||
else:
|
||||
_move(source, dest)
|
||||
if log_dir is not None and prune_stem is not None:
|
||||
if backup_count <= 0:
|
||||
# "keep no history": prune() no-ops at backup_count <= 0 (that's its "leave
|
||||
# history alone" sentinel, not "delete everything"), so a zero-retention roll
|
||||
# deletes its own just-landed file directly instead of relying on prune.
|
||||
try:
|
||||
os.remove(dest)
|
||||
except OSError:
|
||||
pass
|
||||
elif log_dir is not None and prune_stem is not None:
|
||||
prune(log_dir, prune_stem, backup_count)
|
||||
return rotator
|
||||
|
||||
@@ -129,9 +206,14 @@ def make_rotator(
|
||||
def rotate_on_start(
|
||||
live_path: str, log_dir: str, compress: bool, clock=time.localtime,
|
||||
keep_uncompressed: Optional[int] = None, keep_compressed: Optional[int] = None,
|
||||
history_stem: Optional[str] = None,
|
||||
) -> None:
|
||||
"""move an existing live file into log_dir with a timestamp, gzipped if asked
|
||||
|
||||
the rolled file is named off `history_stem` (the project namespace) when given, so
|
||||
historic files carry the project name independent of the live file's stem; falls back
|
||||
to the live file's own stem when history_stem is None/empty.
|
||||
|
||||
no-op if the live file doesn't exist. used by rotate="on_start" before the fresh
|
||||
handler opens a new live file. the timestamp form is run.<%Y-%m-%d_%H-%M-%S>.log.
|
||||
|
||||
@@ -143,7 +225,8 @@ def rotate_on_start(
|
||||
if not os.path.exists(live_path):
|
||||
return
|
||||
tiered = keep_uncompressed is not None or keep_compressed is not None
|
||||
stem = os.path.splitext(os.path.basename(live_path))[0]
|
||||
live_stem = os.path.splitext(os.path.basename(live_path))[0]
|
||||
stem = os.path.basename(history_stem) if history_stem else live_stem
|
||||
stamp = time.strftime("%Y-%m-%d_%H-%M-%S", clock())
|
||||
suffix = ".log.gz" if (compress and not tiered) else ".log"
|
||||
# the stamp is 1-second resolution; two starts in the same second would collide
|
||||
@@ -183,6 +266,10 @@ def retier(log_dir: str, stem: str, keep_uncompressed: int, keep_compressed: int
|
||||
(the namer/rotate_on_start basename them), so a `name` containing a directory (e.g.
|
||||
"sub/run") must be matched by "run." here or nothing matches and retention silently
|
||||
never fires (unbounded pileup).
|
||||
|
||||
ordering is by mtime, then by the roll counter parsed from the name so a same-second
|
||||
burst (tied mtimes, counter-disambiguated stamps like run.<t>.log / run.<t>.1.log)
|
||||
still tiers newest-first correctly rather than falling back to arbitrary listdir order.
|
||||
"""
|
||||
stem = os.path.basename(stem)
|
||||
try:
|
||||
@@ -193,11 +280,34 @@ def retier(log_dir: str, stem: str, keep_uncompressed: int, keep_compressed: int
|
||||
except OSError:
|
||||
return
|
||||
entries = [os.path.join(log_dir, name) for name in names]
|
||||
files = [(p, _safe_mtime(p)) for p in entries if os.path.isfile(p)]
|
||||
files.sort(key=lambda pair: pair[1], reverse=True)
|
||||
# dedupe plain/gz twins FIRST (a crash between _gzip_file's write and its os.remove can
|
||||
# leave <x>.log beside <x>.log.gz). drop the redundant plain copy up front so the
|
||||
# phantom twin never occupies a retention slot and evicts a distinct older roll — but
|
||||
# ONLY once the .gz is verified to decompress cleanly. _gzip_file now writes atomically
|
||||
# (temp+os.replace) so a fresh truncated twin can't occur, but a pre-existing corrupt
|
||||
# .gz (older data, bad copy, disk error) must never win over an intact plain copy: that
|
||||
# would delete the only good copy and keep garbage, strictly worse than doing nothing.
|
||||
present = set(entries)
|
||||
kept = []
|
||||
for p in entries:
|
||||
if not p.endswith(".gz") and (p + ".gz") in present:
|
||||
if _gz_intact(p + ".gz"):
|
||||
try:
|
||||
os.remove(p)
|
||||
except OSError:
|
||||
kept.append(p) # couldn't remove — keep it in the accounting
|
||||
continue
|
||||
# .gz twin is corrupt/truncated — keep the intact plain, don't touch either
|
||||
# file here (the second dedupe pass below or a future retier will retry the
|
||||
# compress once the plain is re-gzipped cleanly)
|
||||
kept.append(p)
|
||||
files = [(p, _safe_mtime(p), _roll_counter(p)) for p in kept if os.path.isfile(p)]
|
||||
# newest-first: higher mtime first, and within a tied second the higher roll counter
|
||||
# (a later same-second roll) is newer
|
||||
files.sort(key=lambda t: (t[1], t[2]), reverse=True)
|
||||
|
||||
keep = keep_uncompressed + keep_compressed
|
||||
for index, (path, _) in enumerate(files):
|
||||
for index, (path, _, _) in enumerate(files):
|
||||
if index >= keep:
|
||||
try:
|
||||
os.remove(path)
|
||||
@@ -206,18 +316,51 @@ def retier(log_dir: str, stem: str, keep_uncompressed: int, keep_compressed: int
|
||||
elif index >= keep_uncompressed and not path.endswith(".gz"):
|
||||
dest = path + ".gz"
|
||||
if os.path.exists(dest):
|
||||
continue
|
||||
# a crash/power-loss between _gzip_file's write and its os.remove can leave
|
||||
# a plain source beside a fresh .gz. don't keep both (they'd double-count
|
||||
# toward retention and evict a distinct older roll) — drop the redundant
|
||||
# plain twin, but ONLY once the .gz is verified intact (see _gz_intact):
|
||||
# a corrupt/truncated .gz here must never win over the last good copy.
|
||||
if _gz_intact(dest):
|
||||
try:
|
||||
os.remove(path)
|
||||
except OSError:
|
||||
pass
|
||||
continue
|
||||
# .gz is corrupt — fall through and re-gzip the plain over the bad dest
|
||||
# (_gzip_file writes atomically, so the corrupt dest is only replaced once
|
||||
# a fully valid archive exists)
|
||||
try:
|
||||
_gzip_file(path, dest)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def _roll_counter(path: str) -> int:
|
||||
"""parse the same-second disambiguation counter out of a rolled filename
|
||||
|
||||
only the on_start / size-namer shape carries a counter: `<stem>.<stamp>[.<counter>].log`
|
||||
(optionally `.gz`), where a colliding same-second roll gets `.1`, `.2`, ... and a higher
|
||||
counter is the later (newer) roll. the first roll of a second has no counter (0).
|
||||
|
||||
daily's dated names (`<stem>.log.<Y-m-d>`) do NOT end in `.log` and are second+-granular
|
||||
(distinct mtimes), so they never need the counter tie-break — return 0 for them rather
|
||||
than misparsing the trailing date component as a counter.
|
||||
"""
|
||||
base = path[:-3] if path.endswith(".gz") else path
|
||||
if not base.endswith(".log"):
|
||||
return 0
|
||||
base = base[:-4]
|
||||
tail = base.rsplit(".", 1)[-1]
|
||||
return int(tail) if tail.isdigit() else 0
|
||||
|
||||
|
||||
def prune(log_dir: str, stem: str, backup_count: int) -> None:
|
||||
"""keep only the newest `backup_count` rolled files for a given stem in log_dir
|
||||
|
||||
matches files beginning with `<stem>.` (e.g. run.*), sorted by mtime, deleting the
|
||||
oldest beyond the count. used for on_start, which the handlers don't auto-prune.
|
||||
matches files beginning with `<stem>.` (e.g. run.*), sorted by mtime (then by the
|
||||
roll counter parsed from the name, mirroring retier — see _roll_counter), deleting
|
||||
the oldest beyond the count. used for on_start, which the handlers don't auto-prune.
|
||||
|
||||
`stem` is reduced to its basename so a `name` containing a directory (e.g. "sub/run")
|
||||
still matches the basenamed rolled files in log_dir (else nothing matches and old
|
||||
@@ -234,9 +377,11 @@ def prune(log_dir: str, stem: str, backup_count: int) -> None:
|
||||
]
|
||||
except OSError:
|
||||
return
|
||||
files = [(p, _safe_mtime(p)) for p in entries if os.path.isfile(p)]
|
||||
files.sort(key=lambda pair: pair[1], reverse=True)
|
||||
for path, _ in files[backup_count:]:
|
||||
files = [(p, _safe_mtime(p), _roll_counter(p)) for p in entries if os.path.isfile(p)]
|
||||
# newest-first: higher mtime first, and within a tied second the higher roll counter
|
||||
# (a later same-second roll) is newer — mirrors retier's ordering (line ~289)
|
||||
files.sort(key=lambda t: (t[1], t[2]), reverse=True)
|
||||
for path, _, _ in files[backup_count:]:
|
||||
try:
|
||||
os.remove(path)
|
||||
except OSError:
|
||||
@@ -255,24 +400,24 @@ def attach_rolling(
|
||||
handler, log_dir: str, compress: bool,
|
||||
prune_stem: Optional[str] = None, backup_count: int = 0,
|
||||
keep_uncompressed: Optional[int] = None, keep_compressed: Optional[int] = None,
|
||||
size_tiered: bool = False,
|
||||
tiered: bool = False,
|
||||
) -> Tuple[Callable, Callable]:
|
||||
"""wire the custom namer + rotator onto a rotating handler; return them
|
||||
|
||||
pass `prune_stem`/`backup_count` so the rotator prunes `log_dir` after each roll
|
||||
(the handler's own retention can't see the redirected rolled files). pass
|
||||
`keep_uncompressed`/`keep_compressed` instead to use tiered retention (newest plain,
|
||||
next gzipped, rest deleted) — see make_rotator.
|
||||
rolled files are named off `prune_stem` (the HISTORY stem — the project namespace),
|
||||
independent of the live file name, via make_history_namer: `<stem>.<stamp>.log[.gz]`
|
||||
uniform across size and daily. this replaces the stdlib handler's own rolled-name
|
||||
scheme (`.N` for size, `.log.<date>` for daily), which can't inject a project stem and
|
||||
(for size) can't be managed once files are redirected into log_dir.
|
||||
|
||||
`size_tiered` uses a timestamped per-roll namer (make_size_namer) instead of the
|
||||
default one, for a tiered RotatingFileHandler (size mode): stdlib's `.1/.2` numbered
|
||||
shift can't manage files redirected into log_dir, so each roll gets a unique dated
|
||||
name that retier ranks/tiers like the daily/on_start paths.
|
||||
pass `prune_stem`/`backup_count` so the rotator prunes `log_dir` after each roll (the
|
||||
handler's own retention can't see the redirected rolled files). pass
|
||||
`keep_uncompressed`/`keep_compressed` for tiered retention (newest plain, next gzipped,
|
||||
rest deleted) — see make_rotator. `tiered=True` lands rolls plain (retier compresses).
|
||||
"""
|
||||
if size_tiered:
|
||||
namer = make_size_namer(os.path.basename(prune_stem or ""), log_dir)
|
||||
else:
|
||||
namer = make_namer(log_dir, compress)
|
||||
namer = make_history_namer(
|
||||
os.path.basename(prune_stem or ""), log_dir, compress, plain=tiered,
|
||||
)
|
||||
rotator = make_rotator(
|
||||
compress, log_dir, prune_stem, backup_count, keep_uncompressed, keep_compressed,
|
||||
)
|
||||
|
||||
+65
-14
@@ -6,6 +6,10 @@ of rolled files, retention, console output, and a consistent format. it is calle
|
||||
the APPLICATION, not by reusable libraries (those stay emit-only). it is idempotent
|
||||
(no duplicate handlers on repeat calls), never crashes the app over logging, and can
|
||||
route through a background queue so an async event loop doesn't block on file I/O.
|
||||
|
||||
`rotate="size"` always bounds the live file: the roll fires at `max_bytes` regardless
|
||||
of `backup_count`, including `backup_count=0` (which means "keep zero rolled files",
|
||||
not "never roll" — each roll is deleted right after landing).
|
||||
"""
|
||||
|
||||
import atexit
|
||||
@@ -120,45 +124,77 @@ def _normalize_name(name: str) -> str:
|
||||
return name
|
||||
|
||||
|
||||
def _history_stem() -> str:
|
||||
"""the project namespace for historic files: the cwd basename
|
||||
|
||||
a service run from bestbuy/ gives historic files bestbuy.<stamp>.log[.gz]. falls back
|
||||
to an empty string only for a degenerate cwd (e.g. "/"), which the caller resolves to
|
||||
the live stem.
|
||||
"""
|
||||
try:
|
||||
return os.path.basename(os.getcwd().rstrip(os.sep))
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
|
||||
def _file_handler(
|
||||
name: str, live_path: str, log_dir: str, rotate: Optional[str],
|
||||
name: str, history_stem: str, live_path: str, log_dir: str, rotate: Optional[str],
|
||||
backup_count: int, max_bytes: int, compress: bool,
|
||||
keep_uncompressed: Optional[int], keep_compressed: Optional[int],
|
||||
) -> logging.Handler:
|
||||
"""build the configured file handler with custom rolling into log_dir"""
|
||||
"""build the configured file handler with custom rolling into log_dir
|
||||
|
||||
`name` is the LIVE stem (drives live_path); `history_stem` is the PROJECT stem that
|
||||
rolled/historic files are named off + the retention glob keys on. they are decoupled:
|
||||
the live file keeps its defined name, historic files carry the project namespace.
|
||||
"""
|
||||
tiered = keep_uncompressed is not None or keep_compressed is not None
|
||||
if rotate == "size":
|
||||
# stdlib doRollover is a no-op when backupCount == 0, and its numbered .1/.2 shift
|
||||
# can't manage files redirected into log_dir. in tiered size mode force a nonzero
|
||||
# backupCount so the roll always fires (retier bounds retention, not backupCount)
|
||||
# and use the timestamped size-namer (size_tiered) instead of the .N shift.
|
||||
size_backup = backup_count if not tiered else max(backup_count, 1)
|
||||
# can't manage files redirected into log_dir. force a nonzero backupCount so the
|
||||
# roll always fires regardless of mode — tiered AND legacy — and let
|
||||
# attach_rolling's history namer name + retier/prune bound retention (keyed to
|
||||
# history_stem). the ORIGINAL backup_count (which may be 0) still flows into
|
||||
# attach_rolling below: make_rotator treats backup_count <= 0 there as "keep no
|
||||
# rolled history" and deletes each roll right after it lands, rather than passing
|
||||
# 0 to prune() (whose own backup_count <= 0 means "leave history alone", a no-op —
|
||||
# that mismatch is exactly what silently disabled rotation before).
|
||||
size_backup = max(backup_count, 1)
|
||||
handler = logging.handlers.RotatingFileHandler(
|
||||
live_path, maxBytes=max_bytes, backupCount=size_backup, encoding="utf-8",
|
||||
)
|
||||
attach_rolling(
|
||||
handler, log_dir, compress, prune_stem=name, backup_count=backup_count,
|
||||
handler, log_dir, compress, prune_stem=history_stem, backup_count=backup_count,
|
||||
keep_uncompressed=keep_uncompressed, keep_compressed=keep_compressed,
|
||||
size_tiered=tiered,
|
||||
tiered=tiered,
|
||||
)
|
||||
elif rotate == "daily":
|
||||
handler = logging.handlers.TimedRotatingFileHandler(
|
||||
live_path, when="midnight", backupCount=backup_count, encoding="utf-8",
|
||||
)
|
||||
attach_rolling(
|
||||
handler, log_dir, compress, prune_stem=name, backup_count=backup_count,
|
||||
handler, log_dir, compress, prune_stem=history_stem, backup_count=backup_count,
|
||||
keep_uncompressed=keep_uncompressed, keep_compressed=keep_compressed,
|
||||
tiered=tiered,
|
||||
)
|
||||
else:
|
||||
if rotate == "on_start":
|
||||
if tiered:
|
||||
rotate_on_start(
|
||||
live_path, log_dir, compress,
|
||||
live_path, log_dir, compress, history_stem=history_stem,
|
||||
keep_uncompressed=keep_uncompressed, keep_compressed=keep_compressed,
|
||||
)
|
||||
else:
|
||||
rotate_on_start(live_path, log_dir, compress)
|
||||
prune(log_dir, name, backup_count)
|
||||
rotate_on_start(live_path, log_dir, compress, history_stem=history_stem)
|
||||
prune(log_dir, history_stem, backup_count)
|
||||
elif rotate is not None:
|
||||
# a typo'd rotate value (e.g. "hourly") would otherwise silently fall through
|
||||
# to a non-rotating FileHandler and grow forever — warn, matching the
|
||||
# unknown-`output` convention, rather than degrade silently
|
||||
log.warning(
|
||||
"log_setup: unknown rotate %r; expected 'daily'/'size'/'on_start'/None — "
|
||||
"no rotation applied (single growing file)", rotate,
|
||||
)
|
||||
handler = logging.FileHandler(live_path, encoding="utf-8")
|
||||
return handler
|
||||
|
||||
@@ -169,6 +205,7 @@ def setup_logging(
|
||||
level: Union[int, str] = "INFO",
|
||||
module_levels: Optional[Dict[str, Union[int, str]]] = None,
|
||||
rotate: Optional[str] = "daily",
|
||||
history_name: Optional[str] = None,
|
||||
backup_count: int = 14,
|
||||
keep_uncompressed: Optional[int] = None,
|
||||
keep_compressed: Optional[int] = None,
|
||||
@@ -185,6 +222,10 @@ def setup_logging(
|
||||
`name` -> <name>.log live file at cwd; rolled/compressed copies go to `log_dir`. a
|
||||
trailing ".log" in `name` is stripped so "latest" and "latest.log" both produce the
|
||||
live file latest.log (never latest.log.log).
|
||||
`history_name` names the rolled/historic files (`<history_name>.<timestamp>.log[.gz]`),
|
||||
independent of the live file: it defaults to the PROJECT namespace = the cwd basename
|
||||
(run from bestbuy/ -> historic files bestbuy.<stamp>...), and can be set explicitly. the
|
||||
live file always keeps `name`; only the historic files carry the project name.
|
||||
`keep_uncompressed`/`keep_compressed` (default None) enable TIERED retention: when
|
||||
either is given, rolled files are kept as the newest `keep_uncompressed` uncompressed
|
||||
+ the next `keep_compressed` gzipped, and the rest are deleted (total retained =
|
||||
@@ -201,7 +242,11 @@ def setup_logging(
|
||||
applies, so naming a parent ("aiohttp") quiets its whole subtree (aiohttp.client,
|
||||
aiohttp.access, ...). each entry accepts a str or int level; a bad value for one
|
||||
entry is skipped with a warning and never aborts the others or the setup.
|
||||
`rotate` is "daily" (default), "size", "on_start", or None. `console=True` adds a
|
||||
`rotate` is "daily" (default), "size", "on_start", or None. for `rotate="size"`, the
|
||||
live file always rolls at `max_bytes` regardless of `backup_count`: `backup_count=0`
|
||||
means "keep zero rolled files" (each roll lands then is deleted immediately), NOT
|
||||
"disable rotation" — the live file is always bounded. `backup_count>=1` keeps that
|
||||
many rolled files as before. `console=True` adds a
|
||||
stdout handler (off by default — the file is the output). `queue=True` routes records
|
||||
through a background QueueListener so file I/O never blocks the caller (the listener
|
||||
is stopped at exit). `output` is "text" (default, human `time | module | level |
|
||||
@@ -223,6 +268,12 @@ def setup_logging(
|
||||
formatter = build_formatter(output, fmt, datefmt)
|
||||
stem = _normalize_name(name)
|
||||
live_path = f"{stem}.log"
|
||||
# historic/rolled files are named off the project namespace, independent of the live
|
||||
# file: history_name if given, else the cwd basename (e.g. run from bestbuy/ -> historic
|
||||
# files bestbuy.<stamp>.log[.gz]). normalized + basenamed like `name`; falls back to the
|
||||
# live stem for a degenerate cwd so naming/retention never break.
|
||||
history_source = history_name if history_name is not None else _history_stem()
|
||||
history_stem = os.path.basename(_normalize_name(history_source)) or stem
|
||||
|
||||
handlers = []
|
||||
|
||||
@@ -235,7 +286,7 @@ def setup_logging(
|
||||
if file_ok:
|
||||
try:
|
||||
fh = _file_handler(
|
||||
stem, live_path, log_dir, rotate, backup_count, max_bytes, compress,
|
||||
stem, history_stem, live_path, log_dir, rotate, backup_count, max_bytes, compress,
|
||||
keep_uncompressed, keep_compressed,
|
||||
)
|
||||
fh.setFormatter(formatter)
|
||||
|
||||
Reference in New Issue
Block a user