feat: structured JSON output mode (output="json")
add a selectable output format to setup_logging: text (default, human,
local time) stays unchanged; output="json" emits one-JSON-object-per-line
(JSON Lines) for the Grafana/Loki path. json fields are time (UTC ISO-8601
with Z), level, module, message, plus any extra={...} keys surfaced as
top-level fields and a rendered exc_info traceback on error records. both
file and console use the chosen format; the live-file name is unchanged so
the Promtail glob and tail command don't break across text/json. an unknown
output falls back to text and warns, never crashes. stdlib json only, zero
new deps. minor bump to v0.2.0.
Signed-off-by: disqualifier <dev@disqualifier.me>
This commit is contained in:
parent
84e1744d6f
commit
54151b9835
43
README.md
43
README.md
@ -13,7 +13,7 @@ and emit; their records flow into the handlers `log_setup` wired.
|
|||||||
## Install
|
## Install
|
||||||
|
|
||||||
```
|
```
|
||||||
log_setup @ git+ssh://git@git.rethinkstudios.io/rethink-public/log_setup.git@v0.1.0
|
log_setup @ git+ssh://git@git.rethinkstudios.io/rethink-public/log_setup.git@v0.2.0
|
||||||
```
|
```
|
||||||
|
|
||||||
No dependencies — stdlib only.
|
No dependencies — stdlib only.
|
||||||
@ -52,6 +52,37 @@ emits; the records land in the configured root.
|
|||||||
- **console=True** (off by default) also logs to stdout in the same format — opt in when
|
- **console=True** (off by default) also logs to stdout in the same format — opt in when
|
||||||
you want live terminal output alongside the file.
|
you want live terminal output alongside the file.
|
||||||
|
|
||||||
|
## Output format (`output=`)
|
||||||
|
|
||||||
|
Two formats, two needs. Default is `"text"`; the live-file name is the same either way
|
||||||
|
(`run.log`, never auto-renamed), so a service can switch text↔json without breaking the
|
||||||
|
Promtail glob, bind-mount path, or your `tail` command.
|
||||||
|
|
||||||
|
- **`output="text"`** (default) — human-readable
|
||||||
|
`2026-06-27 19:55:05 | module.name | INFO | message`, **local time**. The
|
||||||
|
single-machine `tail -f` path. `fmt`/`datefmt` override it. Unchanged from v0.1.x.
|
||||||
|
- **`output="json"`** — structured **one JSON object per line** (JSON Lines) for the
|
||||||
|
Grafana/Loki pipeline (Promtail → Loki → Grafana); Loki parses JSON fields into labels
|
||||||
|
natively, no regex.
|
||||||
|
|
||||||
|
```python
|
||||||
|
setup_logging(name="run", output="json")
|
||||||
|
logging.getLogger("bot.core").info("ready", extra={"monitor": "heartbeat"})
|
||||||
|
# -> {"time": "2026-06-28T14:03:11Z", "level": "INFO", "module": "bot.core",
|
||||||
|
# "message": "ready", "monitor": "heartbeat"}
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Fields:** `time`, `level`, `module`, `message` always; any `extra={...}` keys land
|
||||||
|
as **top-level** fields (stamp `monitor`/`service`/request-id for Loki labels — the lib
|
||||||
|
stays domain-agnostic); error records carry the traceback in `exc_info` (never dropped).
|
||||||
|
- **Time is UTC ISO-8601 with a `Z`** (`2026-06-28T14:03:11Z`), not local. json is the
|
||||||
|
aggregation path — logs from many servers/containers sort unambiguously only in UTC;
|
||||||
|
Grafana converts to local for display. (Text mode stays local — that's a human on one
|
||||||
|
box.)
|
||||||
|
- Both file and console use the chosen format. `fmt`/`datefmt` apply to text only (json
|
||||||
|
builds fields, not a format string). An unknown `output` falls back to text + warns,
|
||||||
|
never crashes. **Zero new deps** — stdlib `json` only.
|
||||||
|
|
||||||
## Signature
|
## Signature
|
||||||
|
|
||||||
```python
|
```python
|
||||||
@ -65,8 +96,9 @@ setup_logging(
|
|||||||
compress=True, # gzip rolled files
|
compress=True, # gzip rolled files
|
||||||
console=False, # also log to stdout (off by default; opt in)
|
console=False, # also log to stdout (off by default; opt in)
|
||||||
queue=False, # route through a background QueueListener (async-friendly)
|
queue=False, # route through a background QueueListener (async-friendly)
|
||||||
fmt=None, # override the format string
|
output="text", # "text" (human, local time) | "json" (structured, UTC)
|
||||||
datefmt=None, # override the date format
|
fmt=None, # override the text format string (text mode only)
|
||||||
|
datefmt=None, # override the text date format (text mode only)
|
||||||
) -> logging.Logger # returns the configured root logger
|
) -> logging.Logger # returns the configured root logger
|
||||||
```
|
```
|
||||||
|
|
||||||
@ -97,8 +129,9 @@ handlers. Getting files to a backend is a separate concern (e.g. Promtail tails
|
|||||||
backend can change without touching any app, and the consistent format here is what
|
backend can change without touching any app, and the consistent format here is what
|
||||||
makes downstream parsing and alerting easy.
|
makes downstream parsing and alerting easy.
|
||||||
|
|
||||||
Also out of v0.1.0 (possible later additions): structured/JSON logging, color
|
Structured/JSON output is **in** as of v0.2.0 (`output="json"`) — text and json only.
|
||||||
formatting, per-logger filters, remote handlers.
|
Still deliberately out: logfmt or other formats, a format DSL, per-handler formats,
|
||||||
|
color formatting, per-logger filters, remote handlers.
|
||||||
|
|
||||||
## Versioning
|
## Versioning
|
||||||
|
|
||||||
|
|||||||
@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "log_setup"
|
name = "log_setup"
|
||||||
version = "0.1.1"
|
version = "0.2.0"
|
||||||
description = "stdlib app-entry-point logging setup: live run.log, rotation, gzip, retention, consistent format"
|
description = "stdlib app-entry-point logging setup: live run.log, rotation, gzip, retention, consistent format"
|
||||||
requires-python = ">=3.10"
|
requires-python = ">=3.10"
|
||||||
dependencies = []
|
dependencies = []
|
||||||
|
|||||||
@ -1,15 +1,67 @@
|
|||||||
"""default log format + datefmt for the app-wide setup.
|
"""log formats for the app-wide setup: human-readable text + structured JSON lines.
|
||||||
|
|
||||||
one format for v0.1.0, used on both console and file. `%(name)s` is the getLogger
|
two output formats, two proven needs. `text` (default) is the human `tail -f` format
|
||||||
name the emitting module used, so each library/module shows in the line.
|
(`time | module | level | message`, local time). `json` is the Grafana/Loki path —
|
||||||
|
one JSON object per line (JSON Lines), fields parsed into labels natively, UTC
|
||||||
|
timestamps so logs aggregated from many machines/containers sort unambiguously.
|
||||||
|
`%(name)s` is the getLogger name the emitting module used, so each module shows.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
import datetime
|
||||||
|
import json
|
||||||
import logging
|
import logging
|
||||||
|
|
||||||
DEFAULT_FORMAT = "%(asctime)s | %(name)s | %(levelname)s | %(message)s"
|
DEFAULT_FORMAT = "%(asctime)s | %(name)s | %(levelname)s | %(message)s"
|
||||||
DEFAULT_DATEFMT = "%Y-%m-%d %H:%M:%S"
|
DEFAULT_DATEFMT = "%Y-%m-%d %H:%M:%S"
|
||||||
|
|
||||||
|
_RESERVED = frozenset(vars(logging.makeLogRecord({})).keys()) | {"message", "asctime"}
|
||||||
|
|
||||||
def build_formatter(fmt=None, datefmt=None) -> logging.Formatter:
|
|
||||||
"""build a logging.Formatter from overrides, falling back to the defaults"""
|
class JsonLinesFormatter(logging.Formatter):
|
||||||
|
"""format each record as a single-line JSON object (JSON Lines / .jsonl)
|
||||||
|
|
||||||
|
emits at minimum time/level/module/message. time is UTC ISO-8601 with a `Z`
|
||||||
|
suffix (e.g. 2026-06-28T14:03:11Z) so logs aggregated across machines and
|
||||||
|
containers sort unambiguously — Grafana converts to local for display. any
|
||||||
|
field passed via logging `extra={...}` lands as a top-level JSON field, which
|
||||||
|
is how a caller stamps monitor/service/request-id for Loki labels without the
|
||||||
|
lib knowing those domain concepts. a traceback (exc_info) is rendered into an
|
||||||
|
`exc_info` string field rather than dropped.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def format(self, record: logging.LogRecord) -> str:
|
||||||
|
when = datetime.datetime.fromtimestamp(record.created, datetime.timezone.utc)
|
||||||
|
payload = {
|
||||||
|
"time": when.strftime("%Y-%m-%dT%H:%M:%SZ"),
|
||||||
|
"level": record.levelname,
|
||||||
|
"module": record.name,
|
||||||
|
"message": record.getMessage(),
|
||||||
|
}
|
||||||
|
for key, value in record.__dict__.items():
|
||||||
|
if key not in _RESERVED and not key.startswith("_"):
|
||||||
|
payload[key] = value
|
||||||
|
if record.exc_info:
|
||||||
|
payload["exc_info"] = self.formatException(record.exc_info)
|
||||||
|
elif record.exc_text:
|
||||||
|
payload["exc_info"] = record.exc_text
|
||||||
|
if record.stack_info:
|
||||||
|
payload["stack_info"] = self.formatStack(record.stack_info)
|
||||||
|
return json.dumps(payload, default=str)
|
||||||
|
|
||||||
|
|
||||||
|
def build_formatter(output: str = "text", fmt=None, datefmt=None) -> logging.Formatter:
|
||||||
|
"""build the formatter for the chosen output format
|
||||||
|
|
||||||
|
`output="text"` (default) returns the human-readable text formatter, honoring
|
||||||
|
the raw `fmt`/`datefmt` format-string overrides. `output="json"` returns the
|
||||||
|
structured `JsonLinesFormatter` (which ignores `fmt`/`datefmt` — it builds
|
||||||
|
fields, not a format string). an unrecognized `output` falls back to text and
|
||||||
|
warns, never raising — a bad format arg must not take the app down.
|
||||||
|
"""
|
||||||
|
if output == "json":
|
||||||
|
return JsonLinesFormatter()
|
||||||
|
if output != "text":
|
||||||
|
logging.getLogger(__name__).warning(
|
||||||
|
"log_setup: unknown output %r; falling back to 'text'", output
|
||||||
|
)
|
||||||
return logging.Formatter(fmt or DEFAULT_FORMAT, datefmt or DEFAULT_DATEFMT)
|
return logging.Formatter(fmt or DEFAULT_FORMAT, datefmt or DEFAULT_DATEFMT)
|
||||||
|
|||||||
@ -90,6 +90,7 @@ def setup_logging(
|
|||||||
compress: bool = True,
|
compress: bool = True,
|
||||||
console: bool = False,
|
console: bool = False,
|
||||||
queue: bool = False,
|
queue: bool = False,
|
||||||
|
output: str = "text",
|
||||||
fmt: Optional[str] = None,
|
fmt: Optional[str] = None,
|
||||||
datefmt: Optional[str] = None,
|
datefmt: Optional[str] = None,
|
||||||
) -> logging.Logger:
|
) -> logging.Logger:
|
||||||
@ -99,9 +100,14 @@ def setup_logging(
|
|||||||
`rotate` is "daily" (default), "size", "on_start", or None. `console=True` adds a
|
`rotate` is "daily" (default), "size", "on_start", or None. `console=True` adds a
|
||||||
stdout handler (off by default — the file is the output). `queue=True` routes records
|
stdout handler (off by default — the file is the output). `queue=True` routes records
|
||||||
through a background QueueListener so file I/O never blocks the caller (the listener
|
through a background QueueListener so file I/O never blocks the caller (the listener
|
||||||
is stopped at exit). idempotent: a repeat call clears only the handlers this function
|
is stopped at exit). `output` is "text" (default, human `time | module | level |
|
||||||
added. never raises over logging — an unwritable `log_dir` falls back to console-only
|
message`, local time) or "json" (structured one-JSON-object-per-line for the
|
||||||
with a warning even when `console` is off, so output is never silently lost.
|
Grafana/Loki path, UTC timestamps, `extra=` fields surfaced as top-level keys); both
|
||||||
|
file and console use the chosen format and the live-file name is the same regardless.
|
||||||
|
the raw `fmt`/`datefmt` overrides apply to text output only. idempotent: a repeat call
|
||||||
|
clears only the handlers this function added. never raises over logging — an
|
||||||
|
unwritable `log_dir` falls back to console-only with a warning even when `console` is
|
||||||
|
off, so output is never silently lost; an unknown `output` falls back to text.
|
||||||
"""
|
"""
|
||||||
global _listener
|
global _listener
|
||||||
|
|
||||||
@ -109,7 +115,7 @@ def setup_logging(
|
|||||||
root.setLevel(_level_value(level))
|
root.setLevel(_level_value(level))
|
||||||
_clear_owned(root)
|
_clear_owned(root)
|
||||||
|
|
||||||
formatter = build_formatter(fmt, datefmt)
|
formatter = build_formatter(output, fmt, datefmt)
|
||||||
live_path = f"{name}.log"
|
live_path = f"{name}.log"
|
||||||
|
|
||||||
handlers = []
|
handlers = []
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user