18 Commits
Author SHA1 Message Date
dsql bbfcc4818b release: 1.0.0
first stable release. pre-1.0.0 verification complete: all surviving MED regressions and
gaps resolved and independently re-fired, tree audited clean across the suite.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-09 18:53:15 -04:00
dsql a6bd95bda7 fix: mask_url returns a malformed url unchanged instead of raising
the 0030dae rewrite dropped the try/except ValueError guard around urlsplit, so a
malformed url (an unbalanced ipv6 literal like http://[::1) raised ValueError out of a
display/logging masker instead of being returned unchanged - crashing the log call it
feeds. restore the guard: urlsplit failure returns the input verbatim, matching the
module's fail-safe display-only intent. a valid url still strips query+fragment.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 21:01:22 -04:00
dsql a1702eede8 fix: stable_id rejects an empty call; hosts(limit=0) returns []
stable_id() with no parts skipped the per-part loop and returned a constant
sha256(b''), colliding on every empty call - now raises ValueError like the other
weak-id guards. hosts(cidr, limit=0) checked the cap after appending, so limit=0
yielded one host - move the check before the append so limit=0 returns [].

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:21:14 -04:00
dsql 0030daeb7b refactor: simplify masking to dumb mask_url/mask_proxy + add stable_id
The selective-key URL/proxy machinery (_mask_qs, SENSITIVE_QUERY_KEYS,
_split_host, _bullet_trailing_fields) tried to redact tokens inside arbitrary
URL structure and leaked on shapes it didn't anticipate (SPA-route fragments,
colon-passwords). Replace with purpose-honest functions that cannot leak or
over-mask:

- mask_url: strips query + fragment, keeps scheme/host/path. Does not parse or
  hunt for sensitive params - a secret in a URL is the caller's bug.
- mask_proxy: partial user-reportable proxy id - drops any userinfo credentials
  and masks an IPv4 host's middle octets (A.***.***.D, port kept); non-IPv4
  host passes through; no parseable host raises ValueError.
- stable_id: deterministic sha256-derived id from ordered str parts (\x00-joined,
  regenerable); empty/non-str part or non-positive length raises ValueError.

credit/cvv/phantom/provider unchanged. Export stable_id from the facade.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 18:25:06 -04:00
dsql e6e655335e fix: mask_proxy/mask_url never return a credential in the clear (structure-preserving)
three credential-leak shapes where a redactor returned the secret verbatim (all 718c8a7):
- mask_proxy host:port:cred (3 parts) fell through to a verbatim return; and a >4-part spec
  bulleted only the LAST field, leaking a colon-bearing password's earlier halves. now
  everything after host:port is credential material and collapses to a single :**** for any
  shape (3/4/n parts), so no credential field survives. plain host:port/bare host unchanged.
- mask_url's fragment used a lossless parse_qsl->urlencode round-trip gate that a =-padded
  base64 token failed, passing the whole fragment (token included) through verbatim. _mask_qs
  is now structure-preserving: it splits on & and the first =, replaces only a sensitive key's
  value with *** (covering a =-padded/embedded-= token whole), and leaves every other segment
  byte-identical - so a non-secret value is never corrupted and the round-trip gate (the root of
  the leak) is gone. fragment and query mask identically now.
invariant: never emit a credential in the clear; over-mask when the shape is ambiguous.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 17:54:33 -04:00
dsql 7fa5916eda fix: deep_set fails loud on a tuple instead of silently replacing it; add deep_gets/deep_sets wildcard verbs
deep_set stepping into or through a tuple used to fall through to the dict branch and
replace the tuple with {}, silently corrupting data on a deep_get/deep_set round-trip
(deep_get traverses tuples read-only). it now raises TypeError - tuples are immutable, so
an in-place update is impossible. also adds deep_gets/deep_sets: '*' path segments for
bulk get (always a list) and bulk set (plain value or fn(current)->new); a '*' passed to
the scalar deep_get/deep_set raises ValueError so the return type is never ambiguous.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 16:35:58 -04:00
dsql 718c8a79b0 fix: mask_proxy and mask_url no longer leak credentials in non-4-part specs or url fragments
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 19:08:39 -04:00
dsql 33ade498da refactor: derive __version__ from package metadata (single source)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:59:45 -04:00
dsql e86184986f fix: mask_url re-encodes non-sensitive query values on rebuild (v0.3.4)
the identity quote_via lambda disabled percent-encoding on re-emit, so a
non-sensitive value with a reserved character (e.g. x=%26%3D) came out
structurally corrupted (x=&=). re-encode with quote_plus (safe="*" so the
masked "***" stays literal) instead of the identity function - display
fidelity only, the secret is still always masked to "***" first.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:20:50 -04:00
dsql 18646f313c docs: compress prose/module docstrings, em-dash->hyphen (de-bloat wave 1)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 00:14:44 -04:00
dsql 4f835ff003 fix: provider() unicode-digit crash + phantom() short-value false-mask (v0.3.2)
provider() used str.isdigit(), which accepts unicode digit lookalikes (e.g. superscript
²) that int() can't parse, crashing on digit-only-looking input; _digits() now filters
on isdecimal()+isascii() so only real ASCII digits pass through. phantom() revealed the
entire value for len <= 10 while still looking masked (first6+last4 fully covers or
double-covers short strings); those now fully mask instead. Also compresses several
essay-length docstrings (retry, masking, paths) to a line plus the real nuance, with
zero behavior change, and bumps the README install pin.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 23:21:51 -04:00
dsql 19e6f4aa06 feat: mask_url/mask_proxy display maskers; fix: deep_set now indexes into lists on numeric segments (commons-2)
mask_url/mask_proxy redact credentials from connection strings for logging
(display only, not a security control). deep_set previously destroyed a list
when a numeric path segment landed on it (isinstance(nxt, dict) check
replaced the list wholesale); it now mirrors deep_get's list indexing so a
get/set round-trip on the same dotted path never corrupts data, raising
IndexError on an out-of-range segment instead of silently mis-storing.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 17:36:03 -04:00
dsql 1dc27ebc1a docs: soften pyproject desc to 'stdlib-based' (base is zero-dep; [addr] adds aiohttp)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 22:02:53 -04:00
dsql f8476fe8d4 chore: ignore .claude/ dir (CLAUDE.md now lives under .claude/)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:55:13 -04:00
dsql 12cf07919f fix: C1 coerce non-str state in _state_slug so geo parse can't TypeError
a malformed non-string 'state' reached unicodedata.normalize and raised TypeError out of
fetch_location, breaking the 'None on any parse failure' contract; str() it first.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:34:40 -04:00
dsql 4be69f3c95 docs: pin install line to release, note unpinned-latest option
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 18:13:38 -04:00
dsql 5d444eaf16 docs: show unpinned install line; note tag-pinning for reproducibility
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 18:07:22 -04:00
dsql 449f790571 fix: guard geo parse helpers against non-dict JSON; de-dup timeout build (v0.2.3)
- _parse_ipify/_parse_reverse return None on a truthy non-dict body instead of raising
  AttributeError, honoring the documented 'None on any parse failure' contract (L9)
- build the geo request ClientTimeout once instead of twice (nit)
- drop the stale 'live proxy region- contract' wording from _state_slug's docstring (nit).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 17:57:54 -04:00
11 changed files with 373 additions and 121 deletions
+1 -1
View File
@@ -1,5 +1,5 @@
# claude
CLAUDE.md
.claude/
# python
__pycache__/
+74 -4
View File
@@ -12,14 +12,16 @@ Small sync helpers shared across projects. Base is stdlib only — **no dependen
## Install
```
commons @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v0.2.2
commons @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v1.0.0
# async address/geo lookups (fetch_ip / ip_location / fetch_location) need the extra:
commons[addr] @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v0.2.2
commons[addr] @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v1.0.0
```
The base install pulls **nothing** (stdlib). Only `commons[addr]` adds `aiohttp`, and
only for the geo lookups — the pure `commons.addr.ip` utilities ship in base.
Drop the `@v1.0.0` suffix from the line above to install the latest unpinned.
## timing
Unix ints stay the storable value; datetimes are produced on demand in whatever
@@ -87,32 +89,100 @@ timing.FAST_MODE = True # in test setup
## paths
Two pairs of verbs: **single-path** (scalar in/out) and **wildcard** (bulk, always a list).
### single-path — `deep_get` / `deep_set`
```python
from commons import deep_get, deep_set
data = {"in": {"this": {"old": {"notation": 42}}}, "items": [{"id": "a"}, {"id": "b"}]}
deep_get(data, "in.this.old.notation") # 42
deep_get(data, "items.1.id") # "b" (numeric segment indexes a list)
deep_get(data, "items.1.id") # "b" (numeric segment indexes a list/tuple)
deep_get(data, "in.nope.here", "DEF") # "DEF" (missing -> default, no raise)
deep_set({}, "a.b.c", 9) # {"a": {"b": {"c": 9}}}
deep_set(data, "items.1.id", "B") # updates the list element in place
deep_get(data, "items.1.id") # "B" (get/set share the same grammar)
deep_set(data, "items.9.id", "X") # raises IndexError (out of range, no
# silent mis-store)
```
`deep_get` steps into both lists and tuples on a numeric segment. `deep_set` updates a
**list** element in place; setting into (or through) a **tuple** raises `TypeError`
tuples are immutable, so "update in place" is impossible; flatten to a list first. An
out-of-range numeric segment raises `IndexError` instead of silently corrupting the
structure.
### wildcard — `deep_gets` / `deep_sets`
A `*` segment iterates every element at that level (dict values or list/tuple items);
multiple `*` fan out cartesian. These are **separate verbs** with a fixed list/bulk
return type — a `*` passed to `deep_get`/`deep_set` raises `ValueError` (use the plural
verb).
```python
from commons import deep_gets, deep_sets
data = {"users": [{"username": "al"}, {"username": "bo"}, {"username": "cy"}]}
deep_gets(data, "users.*.username") # ["al", "bo", "cy"] (ALWAYS a list, in order)
deep_gets(data, "users.*.nope") # [] (no match -> empty list, no raise)
deep_sets(data, "users.*.username", "X") # set every match to "X"
deep_sets(data, "users.*.username", str.upper) # callable fn(current)->new per match
```
`deep_sets`'s second arg is either a plain value (write it to every match) or a callable
`fn(current) -> new` (compute each). It writes only matches that already exist (it does
not create missing keys). Setting through/into a tuple raises `TypeError`.
## masking
Display helpers only — they format a value for showing; they are **not a security
control** (the underlying value is unchanged and still needs proper handling).
```python
from commons import credit, cvv, phantom, provider
from commons import credit, cvv, phantom, provider, mask_url, mask_proxy, stable_id
credit("4111 1111 1111 1234") # "•••• •••• •••• 1234"
cvv("123") # "•••"
phantom("abcdef1234567890") # "abcdef...7890"
phantom("1234567890") # "••••••••••" (len <= 10 fully masks, never a false reveal)
provider("4111111111111111") # "VISA" (VISA/MC/AMEX/UPAY/DISC/JCB/DNRS/UNKW)
provider("4111²111111111234") # "VISA" (unicode digit lookalikes ignored, never raises)
# strip query + fragment off a url before logging it
mask_url("https://u:pw@api.x/v2?apiKey=SECRET&ip=8.8.8.8") # -> "https://u:pw@api.x/v2"
mask_url("https://api.x/v2") # -> "https://api.x/v2" (unchanged)
# partial, user-reportable proxy id: creds stripped, ipv4 middle octets masked
mask_proxy("172.58.32.1:8080") # -> "172.***.***.1:8080"
mask_proxy("user:supersecret@172.58.32.1:8080") # -> "172.***.***.1:8080" (creds gone)
mask_proxy("proxy.host.net:8080") # -> "proxy.host.net:8080" (hostname untouched)
# deterministic, regenerable id from ordered parts
stable_id("profit_lounge", "user_123") # -> "d1f4…" (same 16 hex chars every call)
```
`mask_url` drops the query and fragment, keeping scheme/host/path. It does **not** parse
or hunt for sensitive params — a secret in a URL is the caller's bug, not this function's
to detect — so it structurally cannot leak a query param and cannot over-mask a legit one.
A URL with no query/fragment is returned unchanged. (Note: `user:pass@` userinfo is part of
the authority and is **not** stripped; keep credentials out of the URL you log.)
`mask_proxy` yields a partial identifier a user can safely report for debugging. Any
`user:pass@` userinfo (or a `scheme://` prefix) is dropped entirely, then an IPv4 host
`A.B.C.D` is masked to `A.***.***.D` with the port kept in full; a non-IPv4 host (hostname
or IPv6) passes through unchanged with only its credentials stripped. A spec with no
parseable host raises `ValueError`.
`stable_id` derives a fixed-length hex id from ordered string parts via SHA-256 — same
inputs always produce the same id, no state, so it's regenerable anywhere. Parts join with
a `\x00` separator so `("ab","c")` and `("a","bc")` differ; an empty or non-str part, or a
non-positive `length`, raises `ValueError`.
## retry
Exponential-backoff retry, sync (`retry`) and async (`aretry`). Each works as a **call
+2 -2
View File
@@ -4,8 +4,8 @@ build-backend = "hatchling.build"
[project]
name = "commons"
version = "0.2.2"
description = "small stdlib-only sync helpers: time/timezone deltas, dotted-path dict access, display masking, ip/address tooling, and retry/backoff"
version = "1.0.0"
description = "small stdlib-based sync helpers: time/timezone deltas, dotted-path dict access, display masking, ip/address tooling, and retry/backoff"
requires-python = ">=3.10"
dependencies = []
+15 -17
View File
@@ -1,23 +1,13 @@
"""commons small sync helpers shared across projects.
timing: unix-timestamp deltas + timezone-aware datetime conversions, bare
functions and a configurable Clock.
paths: nested dict/list access by dotted path (deep_get / deep_set).
masking: display masking for cards / cvv / tokens (cosmetic, not a security control).
addr: ip/address tooling — pure stdlib ip utils (`commons.addr.ip`) in base,
async geo lookups (`commons.addr.geo`) behind the `commons[addr]` extra.
retry: exponential-backoff retry, sync (`retry`) and async (`aretry`), call or
decorator form; re-raises the last exception (fail loud), never swallows.
"""commons - small sync helpers shared across projects: timing, paths, masking, addr, retry.
base is stdlib only, no dependencies (the addr geo lookups add aiohttp via the
extra). to toggle the timing test mode for bare calls, set it on the module —
`from commons import timing; timing.FAST_MODE = True` — or use `Clock(fast=True)`
for instance-scoped control. addr is exposed as a submodule (`from commons import
addr`); its ip helpers live under `commons.addr.ip` to keep top-level uncluttered.
`commons[addr]` extra). see each submodule's docstring for its api.
"""
from importlib.metadata import PackageNotFoundError, version
from . import addr, masking, paths, timing
from .masking import credit, cvv, phantom, provider
from .paths import deep_get, deep_set
from .masking import credit, cvv, mask_proxy, mask_url, phantom, provider, stable_id
from .paths import deep_get, deep_gets, deep_set, deep_sets
from .retry import aretry, retry
from .timing import (
UTC,
@@ -55,12 +45,20 @@ __all__ = [
"Clock",
"deep_get",
"deep_set",
"deep_gets",
"deep_sets",
"credit",
"cvv",
"phantom",
"provider",
"mask_url",
"mask_proxy",
"stable_id",
"retry",
"aretry",
]
__version__ = "0.2.2"
try:
__version__ = version("commons")
except PackageNotFoundError:
__version__ = "0.0.0+unknown"
+6 -9
View File
@@ -1,13 +1,10 @@
"""addr ip/address tooling for commons.
"""addr - ip/address tooling for commons.
two concerns:
- `ip` — pure stdlib ipaddress utilities (validation, membership, cidr shape,
conversions). ships in the base install, no dependencies.
- `geo` — async network lookups (public ip, ip->geo, reverse geocode). needs
aiohttp, gated behind the `commons[addr]` extra; importing this package is fine
without it, but calling a geo function raises until it is installed.
the geo.ipify api_key is always injected by the caller — nothing is hardcoded.
`ip`: pure stdlib ipaddress utilities, ships in the base install, no dependencies.
`geo`: async network lookups (aiohttp), gated behind the `commons[addr]` extra;
importing this package is fine without it, but calling a geo function raises until
it is installed. the geo.ipify api_key is always injected by the caller, never
hardcoded.
"""
from . import geo, ip
from .geo import fetch_ip, fetch_location, ip_location
+16 -16
View File
@@ -1,15 +1,10 @@
"""async ip/geo network lookups (aiohttp, gated behind the [addr] extra).
importing this module without aiohttp is fine; only calling a lookup without it
raises a clear RuntimeError. each call may reuse a caller-supplied aiohttp
ClientSession (`session=`) or create and close one internally.
url-building and json->dict parsing are pulled into pure helpers (`_*_url`,
`_parse_*`) so they unit-test without network. every lookup returns None on any
request/parse failure (logged, never raised, never printed).
security: the only secret is geo.ipify's `api_key`, which is a REQUIRED keyword on
`ip_location` — the caller injects it. nothing is hardcoded here.
raises a clear RuntimeError. every lookup returns None on any request/parse
failure (logged, never raised, never printed). the only secret is geo.ipify's
`api_key`, a REQUIRED keyword on `ip_location` - the caller injects it, nothing
is hardcoded here.
"""
import logging
import unicodedata
@@ -52,18 +47,23 @@ def _reverse_url(lat: float, lon: float) -> str:
def _parse_ipify(data: dict) -> Optional[str]:
"""pull the ip string out of an ipify response"""
if not isinstance(data, dict):
return None
ip = data.get("ip")
return ip or None
def _state_slug(state: str) -> str:
"""lowercase ascii-folded state slug with underscores, matching the live proxy region- contract"""
folded = unicodedata.normalize("NFKD", state).encode("ascii", "ignore").decode("ascii")
def _state_slug(state) -> str:
"""lowercase ascii-folded state slug with underscores (e.g. 'New York' -> 'new_york');
coerces to str first so a non-string `state` can't TypeError out of the parse contract"""
folded = unicodedata.normalize("NFKD", str(state)).encode("ascii", "ignore").decode("ascii")
return folded.lower().replace(" ", "_")
def _parse_reverse(data: dict) -> Optional[dict]:
"""parse a nominatim reverse response into {country: iso2 lower, state: slug|None}"""
if not isinstance(data, dict):
return None
address = data.get("address")
if not isinstance(address, dict):
return None
@@ -84,10 +84,10 @@ async def _get_json(
if not _HAVE_AIOHTTP:
raise RuntimeError(_MISSING)
owns = session is None
if owns:
session = aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=timeout))
try:
request_timeout = aiohttp.ClientTimeout(total=timeout)
if owns:
session = aiohttp.ClientSession(timeout=request_timeout)
try:
async with session.get(url, headers=headers, timeout=request_timeout) as resp:
if resp.status != 200:
log.warning("address lookup %s -> %s", url, resp.status)
@@ -113,7 +113,7 @@ async def ip_location(
"""look up geo data for an ip via geo.ipify; api_key is required and injected
returns the raw geo.ipify json dict, or None on failure. raises ValueError if
no api_key is supplied there is no default and nothing hardcoded.
no api_key is supplied - there is no default and nothing hardcoded.
"""
if not api_key:
raise ValueError("ip_location requires an api_key (inject it; never hardcode)")
+6 -8
View File
@@ -1,12 +1,10 @@
"""pure ip/address utilities over stdlib ipaddress (no network, no deps).
every function accepts strings. validation/membership helpers
(`is_valid`/`version`/`in_network`/`in_any`) never raise on bad input they return
a falsy value. functions that require a valid address or cidr (`to_int`/`set_bits`/
the network-shape helpers) let `ValueError` propagate so misuse is visible.
cidr parsing uses `ip_network(cidr, strict=False)` throughout, so host bits set in
the network string are tolerated (e.g. "10.0.0.5/24").
validation/membership helpers (`is_valid`/`version`/`in_network`/`in_any`) never
raise on bad input - they return a falsy value. functions that require a valid
address or cidr (`to_int`/`set_bits`/the network-shape helpers) let `ValueError`
propagate so misuse is visible. cidr parsing uses `ip_network(cidr, strict=False)`
throughout, so host bits set in the network string are tolerated (e.g. "10.0.0.5/24").
"""
import ipaddress
from typing import List, Optional
@@ -117,7 +115,7 @@ def hosts(cidr: str, *, limit: Optional[int] = None) -> List[str]:
gen = ipaddress.ip_network(cidr, strict=False).hosts()
out: List[str] = []
for host in gen:
out.append(str(host))
if limit is not None and len(out) >= limit:
break
out.append(str(host))
return out
+95 -8
View File
@@ -1,14 +1,23 @@
"""display masking for sensitive-looking values.
"""display masking for sensitive-looking values (e.g. "•••• •••• •••• 1234").
these are DISPLAY helpers only — they format a value for showing in a UI or log
(e.g. "•••• •••• •••• 1234"). they are not a security control: the underlying
value is unchanged and still needs proper handling (encryption at rest, etc.).
DISPLAY only, not a security control: the underlying value is unchanged and still
needs proper handling (encryption at rest, etc.).
- ``credit``/``cvv``/``phantom``/``provider`` mask card-shaped values for display.
- ``mask_url`` strips the query and fragment off a url (keeping scheme/host/path),
so a url can be logged without leaking whatever a caller stuck in its params - it
does NOT hunt for tokens; a secret in a url is the caller's bug, not this to detect.
- ``mask_proxy`` gives a partial, user-reportable proxy identifier: credentials are
stripped and an ipv4 host's middle octets are masked (first/last octet + port kept).
- ``stable_id`` derives a deterministic, regenerable id from ordered string parts.
"""
import hashlib
from urllib.parse import urlsplit, urlunsplit
def _digits(value: str) -> str:
"""keep only the digit characters of a string"""
return "".join(c for c in value if c.isdigit())
"""keep only ascii-decimal characters of a string (excludes unicode digit lookalikes)"""
return "".join(c for c in value if c.isdecimal() and c.isascii())
def credit(card_number: str) -> str:
@@ -23,14 +32,20 @@ def cvv(value: str) -> str:
def phantom(value: str) -> str:
"""show a long token as first-six...last-four (e.g. a hash or id)"""
"""show a long token as first-six...last-four (e.g. a hash or id)
values of length <= 10 fully mask instead - first6+last4 would otherwise
reveal (or double-reveal) every character while still looking masked.
"""
if len(value) <= 10:
return "" * len(value)
return f"{value[:6]}...{value[-4:]}"
def provider(card_number: str) -> str:
"""short card brand from the number's prefix, or UNKW if undetermined
a BIN-prefix heuristic for display/labeling not authoritative validation.
a BIN-prefix heuristic for display/labeling - not authoritative validation.
tolerates spaces/dashes and short or non-numeric input (returns UNKW).
"""
n = _digits(card_number)
@@ -51,3 +66,75 @@ def provider(card_number: str) -> str:
if n.startswith(("30", "36", "38", "39")):
return "DNRS"
return "UNKW"
def mask_url(url: str) -> str:
"""strip query and fragment from a url, keeping scheme, host and path
does NOT parse or hunt for sensitive params - it simply drops everything after the
path, so it structurally cannot leak a query param and cannot over-mask a legit one.
a url with no query/fragment is returned unchanged; a url urlsplit cannot parse
(a malformed ipv6 literal) is also returned unchanged rather than crashing the
display/log call this feeds.
"""
try:
parts = urlsplit(url)
except ValueError:
return url
return urlunsplit((parts.scheme, parts.netloc, parts.path, "", ""))
def _is_ipv4(host: str) -> bool:
"""true if host is a dotted-quad ipv4 literal (four 0-255 octets)"""
octets = host.split(".")
if len(octets) != 4:
return False
for octet in octets:
if not (octet.isdigit() and octet.isascii()):
return False
if not 0 <= int(octet) <= 255:
return False
return True
def mask_proxy(spec: str) -> str:
"""mask the middle octets of a proxy ip, keeping first/last octet and full port; strips any credentials
drops a ``scheme://`` prefix and any ``user:pass@`` userinfo entirely (a credential is
removed, never partial-masked). an ipv4 host ``A.B.C.D`` becomes ``A.***.***.D`` with the
port kept as-is (``172.58.32.1:8080`` -> ``172.***.***.1:8080``); a non-ipv4 host (hostname,
ipv6) passes through unchanged with only its credentials stripped. raises ValueError on a
spec with no parseable host.
"""
rest = spec
if "://" in rest:
rest = rest.split("://", 1)[1]
if "@" in rest:
rest = rest.rsplit("@", 1)[1]
host, sep, port = rest.partition(":")
if not host:
raise ValueError(f"mask_proxy: no host in proxy spec {spec!r}")
if _is_ipv4(host):
octets = host.split(".")
host = f"{octets[0]}.***.***.{octets[3]}"
return f"{host}:{port}" if sep else host
def stable_id(*parts: str, length: int = 16) -> str:
"""deterministic id from ordered parts; same inputs always produce the same id
joins the parts with a ``\\x00`` separator (so ``("ab","c")`` != ``("a","bc")``) and
returns the leading ``length`` hex chars of their sha256. raises ValueError on no parts,
an empty part, a non-str part, or a non-positive length - fail loud rather than emit a
weak or constant id.
"""
if length <= 0:
raise ValueError(f"stable_id: length must be positive, got {length}")
if not parts:
raise ValueError("stable_id: needs at least one part")
for part in parts:
if not isinstance(part, str):
raise ValueError(f"stable_id: parts must be str, got {type(part).__name__}")
if part == "":
raise ValueError("stable_id: parts must be non-empty")
return hashlib.sha256("\x00".join(parts).encode()).hexdigest()[:length]
+144 -12
View File
@@ -1,12 +1,17 @@
"""nested access by dotted path.
"""nested dict/list access by dotted path
`deep_get(data, "in.this.old.notation")` walks dicts (and lists, when a segment is
a number) and returns a default instead of raising on a missing/!wrong path.
`deep_set` writes a nested value, creating intermediate dicts.
- deep_get/deep_set: single path, scalar in/out. numeric segment indexes a list;
setting into a tuple raises (immutable).
- deep_gets/deep_sets: `*` segments iterate every element at a level (bulk get -> list,
bulk set -> value or fn(current)->new); multiple `*` fan out cartesian.
a `*` in the scalar verbs raises ValueError - use the plural verbs so the return type
is never ambiguous.
"""
from typing import Any
from typing import Any, Callable, List, Union
_MISSING = object()
_WILDCARD = "*"
def deep_get(data: Any, path: str, default: Any = None, *, sep: str = ".") -> Any:
@@ -14,10 +19,14 @@ def deep_get(data: Any, path: str, default: Any = None, *, sep: str = ".") -> An
dict keys are matched by name; a numeric segment indexes a list/tuple
(e.g. "items.0.id"). any missing key, out-of-range index, or non-container
along the way yields `default` rather than raising.
along the way yields `default` rather than raising. a `*` segment raises
ValueError - use deep_gets for wildcard paths.
"""
segments = path.split(sep)
if _WILDCARD in segments:
raise ValueError(f"deep_get: wildcard '*' in path {path!r}; use deep_gets for wildcard paths")
cur = data
for seg in path.split(sep):
for seg in segments:
if isinstance(cur, dict):
cur = cur.get(seg, _MISSING)
if cur is _MISSING:
@@ -33,18 +42,141 @@ def deep_get(data: Any, path: str, default: Any = None, *, sep: str = ".") -> An
def deep_set(data: dict, path: str, value: Any, *, sep: str = ".") -> dict:
"""set a nested value by dotted path, creating intermediate dicts
"""set a nested value by dotted path, creating intermediate dicts; returns data for chaining
returns the same top-level dict for chaining. overwrites any non-dict value
sitting where an intermediate dict is needed.
a numeric segment updates a LIST element in place (out-of-range raises IndexError);
setting through or into a tuple raises TypeError (immutable). a `*` raises ValueError -
use deep_sets for wildcard paths.
"""
segments = path.split(sep)
if _WILDCARD in segments:
raise ValueError(f"deep_set: wildcard '*' in path {path!r}; use deep_sets for wildcard paths")
cur = data
for seg in segments[:-1]:
if isinstance(cur, tuple):
raise TypeError(f"deep_set: cannot set into an immutable tuple at {path!r}")
if isinstance(cur, list):
idx = int(seg)
if not -len(cur) <= idx < len(cur):
raise IndexError(f"deep_set: index {seg!r} out of range for list of length {len(cur)}")
nxt = cur[idx]
if not isinstance(nxt, (dict, list, tuple)):
nxt = {}
cur[idx] = nxt
cur = nxt
else:
nxt = cur.get(seg)
if not isinstance(nxt, dict):
if not isinstance(nxt, (dict, list, tuple)):
nxt = {}
cur[seg] = nxt
cur = nxt
cur[segments[-1]] = value
last = segments[-1]
if isinstance(cur, tuple):
raise TypeError(f"deep_set: cannot set into an immutable tuple at {path!r}")
if isinstance(cur, list):
idx = int(last)
if not -len(cur) <= idx < len(cur):
raise IndexError(f"deep_set: index {last!r} out of range for list of length {len(cur)}")
cur[idx] = value
else:
cur[last] = value
return data
def _iter_children(node: Any):
"""yield (key, child) pairs for a `*` step: dict items, or list/tuple index/item pairs"""
if isinstance(node, dict):
yield from node.items()
elif isinstance(node, (list, tuple)):
yield from enumerate(node)
def deep_gets(data: Any, path: str, *, sep: str = ".") -> List[Any]:
"""get every value matching a dotted path with `*` wildcards, as a list (may be empty)
a `*` iterates every element at that level; multiple `*` fan out cartesian. a branch
that dead-ends (missing key, out-of-range index, non-container) is skipped, not raised.
a path with no `*` returns a one- or zero-element list.
"""
segments = path.split(sep)
frontier = [data]
for seg in segments:
nxt: List[Any] = []
if seg == _WILDCARD:
for node in frontier:
for _, child in _iter_children(node):
nxt.append(child)
else:
for node in frontier:
if isinstance(node, dict):
got = node.get(seg, _MISSING)
if got is not _MISSING:
nxt.append(got)
elif isinstance(node, (list, tuple)):
try:
nxt.append(node[int(seg)])
except (ValueError, IndexError):
pass
frontier = nxt
return frontier
def deep_sets(data: Any, path: str, value: "Union[Any, Callable[[Any], Any]]", *, sep: str = ".") -> Any:
"""set every value matching a dotted path with `*` wildcards; returns data for chaining
`value` is a plain value written to every match, or a callable `fn(current) -> new`
(a bare callable is always invoked - to store one, wrap it `lambda _c, f=fn: f`). a `*`
iterates every element at that level; multiple `*` fan out cartesian. best-effort: any
dead-end (missing key, out-of-range index) is skipped, not raised. setting through or
into a tuple raises TypeError (immutable).
"""
compute: Callable[[Any], Any] = value if callable(value) else (lambda _cur, _v=value: _v)
segments = path.split(sep)
parents = segments[:-1]
last = segments[-1]
frontier = [data]
for seg in parents:
nxt: List[Any] = []
if seg == _WILDCARD:
for node in frontier:
if isinstance(node, tuple):
raise TypeError(f"deep_sets: cannot set through an immutable tuple at {path!r}")
for _, child in _iter_children(node):
nxt.append(child)
else:
for node in frontier:
if isinstance(node, tuple):
raise TypeError(f"deep_sets: cannot set through an immutable tuple at {path!r}")
if isinstance(node, dict):
got = node.get(seg, _MISSING)
if got is not _MISSING:
nxt.append(got)
elif isinstance(node, list):
try:
nxt.append(node[int(seg)])
except (ValueError, IndexError):
pass
frontier = nxt
for node in frontier:
if isinstance(node, tuple):
raise TypeError(f"deep_sets: cannot set into an immutable tuple at {path!r}")
if last == _WILDCARD:
if isinstance(node, dict):
for key in list(node.keys()):
node[key] = compute(node[key])
elif isinstance(node, list):
for idx in range(len(node)):
node[idx] = compute(node[idx])
elif isinstance(node, dict):
if node.get(last, _MISSING) is not _MISSING:
node[last] = compute(node[last])
elif isinstance(node, list):
try:
idx = int(last)
except ValueError:
continue
if -len(node) <= idx < len(node):
node[idx] = compute(node[idx])
return data
+5 -29
View File
@@ -1,25 +1,7 @@
"""retry with exponential backoff sync and async, one backoff engine.
de-duplicates retry logic that was written divergently across several libs (HTTP
429/5xx caps, proxy burn/rotate caps, IMAP reconnects). both a call form and a
decorator form share one implementation per flavor; the backoff schedule is a pure
generator so it tests without real sleeps.
from commons import retry, aretry
# call form
rows = retry(lambda: db_read(), attempts=5, on=(IOError,))
data = await aretry(lambda: fetch(url), attempts=3, on=(TimeoutError,))
# decorator form (same kwargs)
@aretry(attempts=4, backoff=0.5, on=(ConnectionError,))
async def pull():
...
"""retry with exponential backoff, sync (`retry`) and async (`aretry`), call or decorator form.
after `attempts` are exhausted the LAST exception is re-raised (fail loud, never
swallowed). `on` narrows which exceptions retry; `give_up(exc) -> bool` stops early
on a non-retryable error (e.g. a 400 vs a 429). each retry is logged (emit-only),
never printed.
swallowed). see README for usage examples.
"""
import asyncio
@@ -38,7 +20,7 @@ def _delays(attempts: int, backoff: float, factor: float, max_backoff: float):
"""yield the wait before each retry: min(backoff * factor**n, max_backoff)
yields `attempts - 1` delays (one before each retry after the first try). the
raw, un-jittered schedule jitter is applied at call time so the schedule stays
raw, un-jittered schedule - jitter is applied at call time so the schedule stays
pure and testable.
"""
for n in range(max(0, attempts - 1)):
@@ -72,11 +54,8 @@ def retry(
sleep: Callable[[float], None] = time.sleep,
rand: Callable[[], float] = random.random,
):
"""retry a sync callable with exponential backoff; call form or decorator
"""retry a sync callable with exponential backoff; call form (`retry(fn, ...)`) or decorator
`retry(fn, ...)` runs immediately; `@retry(...)` wraps a function. retries on the
`on` exceptions, stops early if `give_up(exc)` is true, re-raises the last
exception once `attempts` are exhausted. `sleep`/`rand` are injectable for tests.
`attempts` is floored at 1 so the callable always runs at least once.
"""
types = _as_types(on)
@@ -125,11 +104,8 @@ def aretry(
sleep: Callable[[float], "asyncio.Future"] = asyncio.sleep,
rand: Callable[[], float] = random.random,
):
"""retry an async callable with exponential backoff; call form or decorator
"""async twin of `retry`; call form (`await aretry(coro_fn, ...)`) or decorator, same semantics
async twin of `retry`. `await aretry(coro_fn, ...)` runs immediately;
`@aretry(...)` wraps a coroutine function. same semantics: retry on `on`, stop on
`give_up`, re-raise the last exception after `attempts`. `sleep`/`rand` injectable.
`attempts` is floored at 1 so the callable always runs at least once.
"""
types = _as_types(on)
+6 -12
View File
@@ -1,22 +1,16 @@
"""time helpers built on unix timestamps with timezone-aware datetime support.
one engine, two ergonomics:
- bare module functions (now/add/ahead/ago/is_expired/to_dt/...) operate on unix
ints for quick, stateless use. all delta math routes through `_delta_seconds`.
- `Clock` holds a timezone + fast flag and delegates to the same functions, so a
configured clock and the bare functions never diverge.
unix ints stay the storable value; datetimes are produced on demand in whatever
timezone you ask for (stdlib zoneinfo, no pytz).
one engine, two ergonomics: bare module functions for stateless unix-int math
(all delta math routes through `_delta_seconds`, the single source of truth), and
`Clock`, which holds a timezone + fast flag and delegates to the same functions.
"""
import time as _time
from datetime import datetime, timezone
from typing import Optional, Union
from zoneinfo import ZoneInfo
# default fast flag for bare module calls. when True, every unit (day/hour/minute)
# counts as one second so time-based flows run fast in tests. set
# `commons.timing.FAST_MODE` from test setup; off by default. not a config import.
# default fast flag for bare calls (test-only: collapses each unit to 1s); set
# `commons.timing.FAST_MODE` from test setup, off by default, not a config import.
FAST_MODE = False
UTC = timezone.utc
@@ -42,7 +36,7 @@ def now() -> int:
def _delta_seconds(days: int, hours: int, minutes: int, seconds: int, fast: bool) -> int:
"""seconds for a unit combination; collapses to 1s/unit when fast is True
single source of truth for unit math every delta helper routes through here.
single source of truth for unit math - every delta helper routes through here.
"""
if fast:
return days + hours + minutes + seconds