22 Commits
Author SHA1 Message Date
dsql 5a8ca8327d release: 1.0.0
first stable release. pre-1.0.0 verification complete: all surviving MED regressions and
gaps resolved and independently re-fired, tree audited clean across the suite.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 21:21:00 -04:00
dsql a6bd95bda7 fix: mask_url returns a malformed url unchanged instead of raising
the 0030dae rewrite dropped the try/except ValueError guard around urlsplit, so a
malformed url (an unbalanced ipv6 literal like http://[::1) raised ValueError out of a
display/logging masker instead of being returned unchanged - crashing the log call it
feeds. restore the guard: urlsplit failure returns the input verbatim, matching the
module's fail-safe display-only intent. a valid url still strips query+fragment.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 21:01:22 -04:00
dsql a1702eede8 fix: stable_id rejects an empty call; hosts(limit=0) returns []
stable_id() with no parts skipped the per-part loop and returned a constant
sha256(b''), colliding on every empty call - now raises ValueError like the other
weak-id guards. hosts(cidr, limit=0) checked the cap after appending, so limit=0
yielded one host - move the check before the append so limit=0 returns [].

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 19:21:14 -04:00
dsql 0030daeb7b refactor: simplify masking to dumb mask_url/mask_proxy + add stable_id
The selective-key URL/proxy machinery (_mask_qs, SENSITIVE_QUERY_KEYS,
_split_host, _bullet_trailing_fields) tried to redact tokens inside arbitrary
URL structure and leaked on shapes it didn't anticipate (SPA-route fragments,
colon-passwords). Replace with purpose-honest functions that cannot leak or
over-mask:

- mask_url: strips query + fragment, keeps scheme/host/path. Does not parse or
  hunt for sensitive params - a secret in a URL is the caller's bug.
- mask_proxy: partial user-reportable proxy id - drops any userinfo credentials
  and masks an IPv4 host's middle octets (A.***.***.D, port kept); non-IPv4
  host passes through; no parseable host raises ValueError.
- stable_id: deterministic sha256-derived id from ordered str parts (\x00-joined,
  regenerable); empty/non-str part or non-positive length raises ValueError.

credit/cvv/phantom/provider unchanged. Export stable_id from the facade.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 18:25:06 -04:00
dsql e6e655335e fix: mask_proxy/mask_url never return a credential in the clear (structure-preserving)
three credential-leak shapes where a redactor returned the secret verbatim (all 718c8a7):
- mask_proxy host:port:cred (3 parts) fell through to a verbatim return; and a >4-part spec
  bulleted only the LAST field, leaking a colon-bearing password's earlier halves. now
  everything after host:port is credential material and collapses to a single :**** for any
  shape (3/4/n parts), so no credential field survives. plain host:port/bare host unchanged.
- mask_url's fragment used a lossless parse_qsl->urlencode round-trip gate that a =-padded
  base64 token failed, passing the whole fragment (token included) through verbatim. _mask_qs
  is now structure-preserving: it splits on & and the first =, replaces only a sensitive key's
  value with *** (covering a =-padded/embedded-= token whole), and leaves every other segment
  byte-identical - so a non-secret value is never corrupted and the round-trip gate (the root of
  the leak) is gone. fragment and query mask identically now.
invariant: never emit a credential in the clear; over-mask when the shape is ambiguous.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 17:54:33 -04:00
dsql 7fa5916eda fix: deep_set fails loud on a tuple instead of silently replacing it; add deep_gets/deep_sets wildcard verbs
deep_set stepping into or through a tuple used to fall through to the dict branch and
replace the tuple with {}, silently corrupting data on a deep_get/deep_set round-trip
(deep_get traverses tuples read-only). it now raises TypeError - tuples are immutable, so
an in-place update is impossible. also adds deep_gets/deep_sets: '*' path segments for
bulk get (always a list) and bulk set (plain value or fn(current)->new); a '*' passed to
the scalar deep_get/deep_set raises ValueError so the return type is never ambiguous.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-06 16:35:58 -04:00
dsql 718c8a79b0 fix: mask_proxy and mask_url no longer leak credentials in non-4-part specs or url fragments
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 19:08:39 -04:00
dsql 33ade498da refactor: derive __version__ from package metadata (single source)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:59:45 -04:00
dsql e86184986f fix: mask_url re-encodes non-sensitive query values on rebuild (v0.3.4)
the identity quote_via lambda disabled percent-encoding on re-emit, so a
non-sensitive value with a reserved character (e.g. x=%26%3D) came out
structurally corrupted (x=&=). re-encode with quote_plus (safe="*" so the
masked "***" stays literal) instead of the identity function - display
fidelity only, the secret is still always masked to "***" first.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 16:20:50 -04:00
dsql 18646f313c docs: compress prose/module docstrings, em-dash->hyphen (de-bloat wave 1)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-03 00:14:44 -04:00
dsql 4f835ff003 fix: provider() unicode-digit crash + phantom() short-value false-mask (v0.3.2)
provider() used str.isdigit(), which accepts unicode digit lookalikes (e.g. superscript
²) that int() can't parse, crashing on digit-only-looking input; _digits() now filters
on isdecimal()+isascii() so only real ASCII digits pass through. phantom() revealed the
entire value for len <= 10 while still looking masked (first6+last4 fully covers or
double-covers short strings); those now fully mask instead. Also compresses several
essay-length docstrings (retry, masking, paths) to a line plus the real nuance, with
zero behavior change, and bumps the README install pin.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 23:21:51 -04:00
dsql 19e6f4aa06 feat: mask_url/mask_proxy display maskers; fix: deep_set now indexes into lists on numeric segments (commons-2)
mask_url/mask_proxy redact credentials from connection strings for logging
(display only, not a security control). deep_set previously destroyed a list
when a numeric path segment landed on it (isinstance(nxt, dict) check
replaced the list wholesale); it now mirrors deep_get's list indexing so a
get/set round-trip on the same dotted path never corrupts data, raising
IndexError on an out-of-range segment instead of silently mis-storing.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-07-02 17:36:03 -04:00
dsql 1dc27ebc1a docs: soften pyproject desc to 'stdlib-based' (base is zero-dep; [addr] adds aiohttp)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 22:02:53 -04:00
dsql f8476fe8d4 chore: ignore .claude/ dir (CLAUDE.md now lives under .claude/)
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:55:13 -04:00
dsql 12cf07919f fix: C1 coerce non-str state in _state_slug so geo parse can't TypeError
a malformed non-string 'state' reached unicodedata.normalize and raised TypeError out of
fetch_location, breaking the 'None on any parse failure' contract; str() it first.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 21:34:40 -04:00
dsql 4be69f3c95 docs: pin install line to release, note unpinned-latest option
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 18:13:38 -04:00
dsql 5d444eaf16 docs: show unpinned install line; note tag-pinning for reproducibility
Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 18:07:22 -04:00
dsql 449f790571 fix: guard geo parse helpers against non-dict JSON; de-dup timeout build (v0.2.3)
- _parse_ipify/_parse_reverse return None on a truthy non-dict body instead of raising
  AttributeError, honoring the documented 'None on any parse failure' contract (L9)
- build the geo request ClientTimeout once instead of twice (nit)
- drop the stale 'live proxy region- contract' wording from _state_slug's docstring (nit).

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 17:57:54 -04:00
dsql a5b91bed0d fix: state slug uses underscore + ascii-fold to match the proxy region- contract
addr.geo.fetch_location produced a hyphen slug with no unicode fold (new-york / québec), but the live proxy region- token the original utils.py fed expects underscore + ascii-fold (new_york / quebec). a multi-word state silently routed to an unrecognized region with no error. added _state_slug (NFKD ascii-fold, lowercase, spaces->underscore) and routed _parse_reverse through it. bump to v0.2.2.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 03:25:14 -04:00
dsql 83a156fd31 fix: forward per-request timeout to geo lookups on an injected session
_get_json applied the timeout only when it created the session; when the caller passed their own session=, the timeout was silently dropped and the session default (aiohttp's 300s) governed. the per-request timeout is now passed to session.get(timeout=...) on both the owned and injected paths.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-29 01:10:25 -04:00
dsql de6911fb05 fix: retry log uses total attempts as the denominator
the retry warning logged index/last_index (attempts-1), so a 3-attempt retry showed 'retry 1/2'. now logs index+1 of attempts. both retry and aretry.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-28 17:18:28 -04:00
dsql c6e3dd1b54 fix: floor retry/aretry attempts at 1 (v0.2.1)
retry(fn, attempts=0) (or negative) silently returned None without ever calling fn,
looking like success. floor attempts at max(1, attempts) so the callable always runs
at least once; a failing call now fails loud after one try instead of no-op None.

verified: attempts=0/-5 -> 1 call (sync + async); failing fn raises after 1 try;
31/31 retry regression intact.

Signed-off-by: disqualifier <dev@disqualifier.me>
2026-06-28 16:11:34 -04:00
11 changed files with 385 additions and 121 deletions
+1 -1
View File
@@ -1,5 +1,5 @@
# claude # claude
CLAUDE.md .claude/
# python # python
__pycache__/ __pycache__/
+74 -4
View File
@@ -12,14 +12,16 @@ Small sync helpers shared across projects. Base is stdlib only — **no dependen
## Install ## Install
``` ```
commons @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v0.2.0 commons @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v0.3.5
# async address/geo lookups (fetch_ip / ip_location / fetch_location) need the extra: # async address/geo lookups (fetch_ip / ip_location / fetch_location) need the extra:
commons[addr] @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v0.2.0 commons[addr] @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v0.3.5
``` ```
The base install pulls **nothing** (stdlib). Only `commons[addr]` adds `aiohttp`, and The base install pulls **nothing** (stdlib). Only `commons[addr]` adds `aiohttp`, and
only for the geo lookups — the pure `commons.addr.ip` utilities ship in base. only for the geo lookups — the pure `commons.addr.ip` utilities ship in base.
Drop the `@v0.3.5` suffix from the line above to install the latest unpinned.
## timing ## timing
Unix ints stay the storable value; datetimes are produced on demand in whatever Unix ints stay the storable value; datetimes are produced on demand in whatever
@@ -87,32 +89,100 @@ timing.FAST_MODE = True # in test setup
## paths ## paths
Two pairs of verbs: **single-path** (scalar in/out) and **wildcard** (bulk, always a list).
### single-path — `deep_get` / `deep_set`
```python ```python
from commons import deep_get, deep_set from commons import deep_get, deep_set
data = {"in": {"this": {"old": {"notation": 42}}}, "items": [{"id": "a"}, {"id": "b"}]} data = {"in": {"this": {"old": {"notation": 42}}}, "items": [{"id": "a"}, {"id": "b"}]}
deep_get(data, "in.this.old.notation") # 42 deep_get(data, "in.this.old.notation") # 42
deep_get(data, "items.1.id") # "b" (numeric segment indexes a list) deep_get(data, "items.1.id") # "b" (numeric segment indexes a list/tuple)
deep_get(data, "in.nope.here", "DEF") # "DEF" (missing -> default, no raise) deep_get(data, "in.nope.here", "DEF") # "DEF" (missing -> default, no raise)
deep_set({}, "a.b.c", 9) # {"a": {"b": {"c": 9}}} deep_set({}, "a.b.c", 9) # {"a": {"b": {"c": 9}}}
deep_set(data, "items.1.id", "B") # updates the list element in place
deep_get(data, "items.1.id") # "B" (get/set share the same grammar)
deep_set(data, "items.9.id", "X") # raises IndexError (out of range, no
# silent mis-store)
``` ```
`deep_get` steps into both lists and tuples on a numeric segment. `deep_set` updates a
**list** element in place; setting into (or through) a **tuple** raises `TypeError`
tuples are immutable, so "update in place" is impossible; flatten to a list first. An
out-of-range numeric segment raises `IndexError` instead of silently corrupting the
structure.
### wildcard — `deep_gets` / `deep_sets`
A `*` segment iterates every element at that level (dict values or list/tuple items);
multiple `*` fan out cartesian. These are **separate verbs** with a fixed list/bulk
return type — a `*` passed to `deep_get`/`deep_set` raises `ValueError` (use the plural
verb).
```python
from commons import deep_gets, deep_sets
data = {"users": [{"username": "al"}, {"username": "bo"}, {"username": "cy"}]}
deep_gets(data, "users.*.username") # ["al", "bo", "cy"] (ALWAYS a list, in order)
deep_gets(data, "users.*.nope") # [] (no match -> empty list, no raise)
deep_sets(data, "users.*.username", "X") # set every match to "X"
deep_sets(data, "users.*.username", str.upper) # callable fn(current)->new per match
```
`deep_sets`'s second arg is either a plain value (write it to every match) or a callable
`fn(current) -> new` (compute each). It writes only matches that already exist (it does
not create missing keys). Setting through/into a tuple raises `TypeError`.
## masking ## masking
Display helpers only — they format a value for showing; they are **not a security Display helpers only — they format a value for showing; they are **not a security
control** (the underlying value is unchanged and still needs proper handling). control** (the underlying value is unchanged and still needs proper handling).
```python ```python
from commons import credit, cvv, phantom, provider from commons import credit, cvv, phantom, provider, mask_url, mask_proxy, stable_id
credit("4111 1111 1111 1234") # "•••• •••• •••• 1234" credit("4111 1111 1111 1234") # "•••• •••• •••• 1234"
cvv("123") # "•••" cvv("123") # "•••"
phantom("abcdef1234567890") # "abcdef...7890" phantom("abcdef1234567890") # "abcdef...7890"
phantom("1234567890") # "••••••••••" (len <= 10 fully masks, never a false reveal)
provider("4111111111111111") # "VISA" (VISA/MC/AMEX/UPAY/DISC/JCB/DNRS/UNKW) provider("4111111111111111") # "VISA" (VISA/MC/AMEX/UPAY/DISC/JCB/DNRS/UNKW)
provider("4111²111111111234") # "VISA" (unicode digit lookalikes ignored, never raises)
# strip query + fragment off a url before logging it
mask_url("https://u:pw@api.x/v2?apiKey=SECRET&ip=8.8.8.8") # -> "https://u:pw@api.x/v2"
mask_url("https://api.x/v2") # -> "https://api.x/v2" (unchanged)
# partial, user-reportable proxy id: creds stripped, ipv4 middle octets masked
mask_proxy("172.58.32.1:8080") # -> "172.***.***.1:8080"
mask_proxy("user:supersecret@172.58.32.1:8080") # -> "172.***.***.1:8080" (creds gone)
mask_proxy("proxy.host.net:8080") # -> "proxy.host.net:8080" (hostname untouched)
# deterministic, regenerable id from ordered parts
stable_id("profit_lounge", "user_123") # -> "d1f4…" (same 16 hex chars every call)
``` ```
`mask_url` drops the query and fragment, keeping scheme/host/path. It does **not** parse
or hunt for sensitive params — a secret in a URL is the caller's bug, not this function's
to detect — so it structurally cannot leak a query param and cannot over-mask a legit one.
A URL with no query/fragment is returned unchanged. (Note: `user:pass@` userinfo is part of
the authority and is **not** stripped; keep credentials out of the URL you log.)
`mask_proxy` yields a partial identifier a user can safely report for debugging. Any
`user:pass@` userinfo (or a `scheme://` prefix) is dropped entirely, then an IPv4 host
`A.B.C.D` is masked to `A.***.***.D` with the port kept in full; a non-IPv4 host (hostname
or IPv6) passes through unchanged with only its credentials stripped. A spec with no
parseable host raises `ValueError`.
`stable_id` derives a fixed-length hex id from ordered string parts via SHA-256 — same
inputs always produce the same id, no state, so it's regenerable anywhere. Parts join with
a `\x00` separator so `("ab","c")` and `("a","bc")` differ; an empty or non-str part, or a
non-positive `length`, raises `ValueError`.
## retry ## retry
Exponential-backoff retry, sync (`retry`) and async (`aretry`). Each works as a **call Exponential-backoff retry, sync (`retry`) and async (`aretry`). Each works as a **call
+2 -2
View File
@@ -4,8 +4,8 @@ build-backend = "hatchling.build"
[project] [project]
name = "commons" name = "commons"
version = "0.2.0" version = "1.0.0"
description = "small stdlib-only sync helpers: time/timezone deltas, dotted-path dict access, display masking, ip/address tooling, and retry/backoff" description = "small stdlib-based sync helpers: time/timezone deltas, dotted-path dict access, display masking, ip/address tooling, and retry/backoff"
requires-python = ">=3.10" requires-python = ">=3.10"
dependencies = [] dependencies = []
+15 -17
View File
@@ -1,23 +1,13 @@
"""commons small sync helpers shared across projects. """commons - small sync helpers shared across projects: timing, paths, masking, addr, retry.
timing: unix-timestamp deltas + timezone-aware datetime conversions, bare
functions and a configurable Clock.
paths: nested dict/list access by dotted path (deep_get / deep_set).
masking: display masking for cards / cvv / tokens (cosmetic, not a security control).
addr: ip/address tooling — pure stdlib ip utils (`commons.addr.ip`) in base,
async geo lookups (`commons.addr.geo`) behind the `commons[addr]` extra.
retry: exponential-backoff retry, sync (`retry`) and async (`aretry`), call or
decorator form; re-raises the last exception (fail loud), never swallows.
base is stdlib only, no dependencies (the addr geo lookups add aiohttp via the base is stdlib only, no dependencies (the addr geo lookups add aiohttp via the
extra). to toggle the timing test mode for bare calls, set it on the module — `commons[addr]` extra). see each submodule's docstring for its api.
`from commons import timing; timing.FAST_MODE = True` — or use `Clock(fast=True)`
for instance-scoped control. addr is exposed as a submodule (`from commons import
addr`); its ip helpers live under `commons.addr.ip` to keep top-level uncluttered.
""" """
from importlib.metadata import PackageNotFoundError, version
from . import addr, masking, paths, timing from . import addr, masking, paths, timing
from .masking import credit, cvv, phantom, provider from .masking import credit, cvv, mask_proxy, mask_url, phantom, provider, stable_id
from .paths import deep_get, deep_set from .paths import deep_get, deep_gets, deep_set, deep_sets
from .retry import aretry, retry from .retry import aretry, retry
from .timing import ( from .timing import (
UTC, UTC,
@@ -55,12 +45,20 @@ __all__ = [
"Clock", "Clock",
"deep_get", "deep_get",
"deep_set", "deep_set",
"deep_gets",
"deep_sets",
"credit", "credit",
"cvv", "cvv",
"phantom", "phantom",
"provider", "provider",
"mask_url",
"mask_proxy",
"stable_id",
"retry", "retry",
"aretry", "aretry",
] ]
__version__ = "0.2.0" try:
__version__ = version("commons")
except PackageNotFoundError:
__version__ = "0.0.0+unknown"
+6 -9
View File
@@ -1,13 +1,10 @@
"""addr ip/address tooling for commons. """addr - ip/address tooling for commons.
two concerns: `ip`: pure stdlib ipaddress utilities, ships in the base install, no dependencies.
- `ip` — pure stdlib ipaddress utilities (validation, membership, cidr shape, `geo`: async network lookups (aiohttp), gated behind the `commons[addr]` extra;
conversions). ships in the base install, no dependencies. importing this package is fine without it, but calling a geo function raises until
- `geo` — async network lookups (public ip, ip->geo, reverse geocode). needs it is installed. the geo.ipify api_key is always injected by the caller, never
aiohttp, gated behind the `commons[addr]` extra; importing this package is fine hardcoded.
without it, but calling a geo function raises until it is installed.
the geo.ipify api_key is always injected by the caller — nothing is hardcoded.
""" """
from . import geo, ip from . import geo, ip
from .geo import fetch_ip, fetch_location, ip_location from .geo import fetch_ip, fetch_location, ip_location
+21 -13
View File
@@ -1,17 +1,13 @@
"""async ip/geo network lookups (aiohttp, gated behind the [addr] extra). """async ip/geo network lookups (aiohttp, gated behind the [addr] extra).
importing this module without aiohttp is fine; only calling a lookup without it importing this module without aiohttp is fine; only calling a lookup without it
raises a clear RuntimeError. each call may reuse a caller-supplied aiohttp raises a clear RuntimeError. every lookup returns None on any request/parse
ClientSession (`session=`) or create and close one internally. failure (logged, never raised, never printed). the only secret is geo.ipify's
`api_key`, a REQUIRED keyword on `ip_location` - the caller injects it, nothing
url-building and json->dict parsing are pulled into pure helpers (`_*_url`, is hardcoded here.
`_parse_*`) so they unit-test without network. every lookup returns None on any
request/parse failure (logged, never raised, never printed).
security: the only secret is geo.ipify's `api_key`, which is a REQUIRED keyword on
`ip_location` — the caller injects it. nothing is hardcoded here.
""" """
import logging import logging
import unicodedata
from typing import Optional from typing import Optional
from urllib.parse import urlencode from urllib.parse import urlencode
@@ -51,12 +47,23 @@ def _reverse_url(lat: float, lon: float) -> str:
def _parse_ipify(data: dict) -> Optional[str]: def _parse_ipify(data: dict) -> Optional[str]:
"""pull the ip string out of an ipify response""" """pull the ip string out of an ipify response"""
if not isinstance(data, dict):
return None
ip = data.get("ip") ip = data.get("ip")
return ip or None return ip or None
def _state_slug(state) -> str:
"""lowercase ascii-folded state slug with underscores (e.g. 'New York' -> 'new_york');
coerces to str first so a non-string `state` can't TypeError out of the parse contract"""
folded = unicodedata.normalize("NFKD", str(state)).encode("ascii", "ignore").decode("ascii")
return folded.lower().replace(" ", "_")
def _parse_reverse(data: dict) -> Optional[dict]: def _parse_reverse(data: dict) -> Optional[dict]:
"""parse a nominatim reverse response into {country: iso2 lower, state: slug|None}""" """parse a nominatim reverse response into {country: iso2 lower, state: slug|None}"""
if not isinstance(data, dict):
return None
address = data.get("address") address = data.get("address")
if not isinstance(address, dict): if not isinstance(address, dict):
return None return None
@@ -66,7 +73,7 @@ def _parse_reverse(data: dict) -> Optional[dict]:
state = address.get("state") state = address.get("state")
return { return {
"country": str(country).lower(), "country": str(country).lower(),
"state": state.lower().replace(" ", "-") if state else None, "state": _state_slug(state) if state else None,
} }
@@ -77,10 +84,11 @@ async def _get_json(
if not _HAVE_AIOHTTP: if not _HAVE_AIOHTTP:
raise RuntimeError(_MISSING) raise RuntimeError(_MISSING)
owns = session is None owns = session is None
request_timeout = aiohttp.ClientTimeout(total=timeout)
if owns: if owns:
session = aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=timeout)) session = aiohttp.ClientSession(timeout=request_timeout)
try: try:
async with session.get(url, headers=headers) as resp: async with session.get(url, headers=headers, timeout=request_timeout) as resp:
if resp.status != 200: if resp.status != 200:
log.warning("address lookup %s -> %s", url, resp.status) log.warning("address lookup %s -> %s", url, resp.status)
return None return None
@@ -105,7 +113,7 @@ async def ip_location(
"""look up geo data for an ip via geo.ipify; api_key is required and injected """look up geo data for an ip via geo.ipify; api_key is required and injected
returns the raw geo.ipify json dict, or None on failure. raises ValueError if returns the raw geo.ipify json dict, or None on failure. raises ValueError if
no api_key is supplied there is no default and nothing hardcoded. no api_key is supplied - there is no default and nothing hardcoded.
""" """
if not api_key: if not api_key:
raise ValueError("ip_location requires an api_key (inject it; never hardcode)") raise ValueError("ip_location requires an api_key (inject it; never hardcode)")
+6 -8
View File
@@ -1,12 +1,10 @@
"""pure ip/address utilities over stdlib ipaddress (no network, no deps). """pure ip/address utilities over stdlib ipaddress (no network, no deps).
every function accepts strings. validation/membership helpers validation/membership helpers (`is_valid`/`version`/`in_network`/`in_any`) never
(`is_valid`/`version`/`in_network`/`in_any`) never raise on bad input they return raise on bad input - they return a falsy value. functions that require a valid
a falsy value. functions that require a valid address or cidr (`to_int`/`set_bits`/ address or cidr (`to_int`/`set_bits`/the network-shape helpers) let `ValueError`
the network-shape helpers) let `ValueError` propagate so misuse is visible. propagate so misuse is visible. cidr parsing uses `ip_network(cidr, strict=False)`
throughout, so host bits set in the network string are tolerated (e.g. "10.0.0.5/24").
cidr parsing uses `ip_network(cidr, strict=False)` throughout, so host bits set in
the network string are tolerated (e.g. "10.0.0.5/24").
""" """
import ipaddress import ipaddress
from typing import List, Optional from typing import List, Optional
@@ -117,7 +115,7 @@ def hosts(cidr: str, *, limit: Optional[int] = None) -> List[str]:
gen = ipaddress.ip_network(cidr, strict=False).hosts() gen = ipaddress.ip_network(cidr, strict=False).hosts()
out: List[str] = [] out: List[str] = []
for host in gen: for host in gen:
out.append(str(host))
if limit is not None and len(out) >= limit: if limit is not None and len(out) >= limit:
break break
out.append(str(host))
return out return out
+95 -8
View File
@@ -1,14 +1,23 @@
"""display masking for sensitive-looking values. """display masking for sensitive-looking values (e.g. "•••• •••• •••• 1234").
these are DISPLAY helpers only — they format a value for showing in a UI or log DISPLAY only, not a security control: the underlying value is unchanged and still
(e.g. "•••• •••• •••• 1234"). they are not a security control: the underlying needs proper handling (encryption at rest, etc.).
value is unchanged and still needs proper handling (encryption at rest, etc.).
- ``credit``/``cvv``/``phantom``/``provider`` mask card-shaped values for display.
- ``mask_url`` strips the query and fragment off a url (keeping scheme/host/path),
so a url can be logged without leaking whatever a caller stuck in its params - it
does NOT hunt for tokens; a secret in a url is the caller's bug, not this to detect.
- ``mask_proxy`` gives a partial, user-reportable proxy identifier: credentials are
stripped and an ipv4 host's middle octets are masked (first/last octet + port kept).
- ``stable_id`` derives a deterministic, regenerable id from ordered string parts.
""" """
import hashlib
from urllib.parse import urlsplit, urlunsplit
def _digits(value: str) -> str: def _digits(value: str) -> str:
"""keep only the digit characters of a string""" """keep only ascii-decimal characters of a string (excludes unicode digit lookalikes)"""
return "".join(c for c in value if c.isdigit()) return "".join(c for c in value if c.isdecimal() and c.isascii())
def credit(card_number: str) -> str: def credit(card_number: str) -> str:
@@ -23,14 +32,20 @@ def cvv(value: str) -> str:
def phantom(value: str) -> str: def phantom(value: str) -> str:
"""show a long token as first-six...last-four (e.g. a hash or id)""" """show a long token as first-six...last-four (e.g. a hash or id)
values of length <= 10 fully mask instead - first6+last4 would otherwise
reveal (or double-reveal) every character while still looking masked.
"""
if len(value) <= 10:
return "" * len(value)
return f"{value[:6]}...{value[-4:]}" return f"{value[:6]}...{value[-4:]}"
def provider(card_number: str) -> str: def provider(card_number: str) -> str:
"""short card brand from the number's prefix, or UNKW if undetermined """short card brand from the number's prefix, or UNKW if undetermined
a BIN-prefix heuristic for display/labeling not authoritative validation. a BIN-prefix heuristic for display/labeling - not authoritative validation.
tolerates spaces/dashes and short or non-numeric input (returns UNKW). tolerates spaces/dashes and short or non-numeric input (returns UNKW).
""" """
n = _digits(card_number) n = _digits(card_number)
@@ -51,3 +66,75 @@ def provider(card_number: str) -> str:
if n.startswith(("30", "36", "38", "39")): if n.startswith(("30", "36", "38", "39")):
return "DNRS" return "DNRS"
return "UNKW" return "UNKW"
def mask_url(url: str) -> str:
"""strip query and fragment from a url, keeping scheme, host and path
does NOT parse or hunt for sensitive params - it simply drops everything after the
path, so it structurally cannot leak a query param and cannot over-mask a legit one.
a url with no query/fragment is returned unchanged; a url urlsplit cannot parse
(a malformed ipv6 literal) is also returned unchanged rather than crashing the
display/log call this feeds.
"""
try:
parts = urlsplit(url)
except ValueError:
return url
return urlunsplit((parts.scheme, parts.netloc, parts.path, "", ""))
def _is_ipv4(host: str) -> bool:
"""true if host is a dotted-quad ipv4 literal (four 0-255 octets)"""
octets = host.split(".")
if len(octets) != 4:
return False
for octet in octets:
if not (octet.isdigit() and octet.isascii()):
return False
if not 0 <= int(octet) <= 255:
return False
return True
def mask_proxy(spec: str) -> str:
"""mask the middle octets of a proxy ip, keeping first/last octet and full port; strips any credentials
drops a ``scheme://`` prefix and any ``user:pass@`` userinfo entirely (a credential is
removed, never partial-masked). an ipv4 host ``A.B.C.D`` becomes ``A.***.***.D`` with the
port kept as-is (``172.58.32.1:8080`` -> ``172.***.***.1:8080``); a non-ipv4 host (hostname,
ipv6) passes through unchanged with only its credentials stripped. raises ValueError on a
spec with no parseable host.
"""
rest = spec
if "://" in rest:
rest = rest.split("://", 1)[1]
if "@" in rest:
rest = rest.rsplit("@", 1)[1]
host, sep, port = rest.partition(":")
if not host:
raise ValueError(f"mask_proxy: no host in proxy spec {spec!r}")
if _is_ipv4(host):
octets = host.split(".")
host = f"{octets[0]}.***.***.{octets[3]}"
return f"{host}:{port}" if sep else host
def stable_id(*parts: str, length: int = 16) -> str:
"""deterministic id from ordered parts; same inputs always produce the same id
joins the parts with a ``\\x00`` separator (so ``("ab","c")`` != ``("a","bc")``) and
returns the leading ``length`` hex chars of their sha256. raises ValueError on no parts,
an empty part, a non-str part, or a non-positive length - fail loud rather than emit a
weak or constant id.
"""
if length <= 0:
raise ValueError(f"stable_id: length must be positive, got {length}")
if not parts:
raise ValueError("stable_id: needs at least one part")
for part in parts:
if not isinstance(part, str):
raise ValueError(f"stable_id: parts must be str, got {type(part).__name__}")
if part == "":
raise ValueError("stable_id: parts must be non-empty")
return hashlib.sha256("\x00".join(parts).encode()).hexdigest()[:length]
+144 -12
View File
@@ -1,12 +1,17 @@
"""nested access by dotted path. """nested dict/list access by dotted path
`deep_get(data, "in.this.old.notation")` walks dicts (and lists, when a segment is - deep_get/deep_set: single path, scalar in/out. numeric segment indexes a list;
a number) and returns a default instead of raising on a missing/!wrong path. setting into a tuple raises (immutable).
`deep_set` writes a nested value, creating intermediate dicts. - deep_gets/deep_sets: `*` segments iterate every element at a level (bulk get -> list,
bulk set -> value or fn(current)->new); multiple `*` fan out cartesian.
a `*` in the scalar verbs raises ValueError - use the plural verbs so the return type
is never ambiguous.
""" """
from typing import Any from typing import Any, Callable, List, Union
_MISSING = object() _MISSING = object()
_WILDCARD = "*"
def deep_get(data: Any, path: str, default: Any = None, *, sep: str = ".") -> Any: def deep_get(data: Any, path: str, default: Any = None, *, sep: str = ".") -> Any:
@@ -14,10 +19,14 @@ def deep_get(data: Any, path: str, default: Any = None, *, sep: str = ".") -> An
dict keys are matched by name; a numeric segment indexes a list/tuple dict keys are matched by name; a numeric segment indexes a list/tuple
(e.g. "items.0.id"). any missing key, out-of-range index, or non-container (e.g. "items.0.id"). any missing key, out-of-range index, or non-container
along the way yields `default` rather than raising. along the way yields `default` rather than raising. a `*` segment raises
ValueError - use deep_gets for wildcard paths.
""" """
segments = path.split(sep)
if _WILDCARD in segments:
raise ValueError(f"deep_get: wildcard '*' in path {path!r}; use deep_gets for wildcard paths")
cur = data cur = data
for seg in path.split(sep): for seg in segments:
if isinstance(cur, dict): if isinstance(cur, dict):
cur = cur.get(seg, _MISSING) cur = cur.get(seg, _MISSING)
if cur is _MISSING: if cur is _MISSING:
@@ -33,18 +42,141 @@ def deep_get(data: Any, path: str, default: Any = None, *, sep: str = ".") -> An
def deep_set(data: dict, path: str, value: Any, *, sep: str = ".") -> dict: def deep_set(data: dict, path: str, value: Any, *, sep: str = ".") -> dict:
"""set a nested value by dotted path, creating intermediate dicts """set a nested value by dotted path, creating intermediate dicts; returns data for chaining
returns the same top-level dict for chaining. overwrites any non-dict value a numeric segment updates a LIST element in place (out-of-range raises IndexError);
sitting where an intermediate dict is needed. setting through or into a tuple raises TypeError (immutable). a `*` raises ValueError -
use deep_sets for wildcard paths.
""" """
segments = path.split(sep) segments = path.split(sep)
if _WILDCARD in segments:
raise ValueError(f"deep_set: wildcard '*' in path {path!r}; use deep_sets for wildcard paths")
cur = data cur = data
for seg in segments[:-1]: for seg in segments[:-1]:
if isinstance(cur, tuple):
raise TypeError(f"deep_set: cannot set into an immutable tuple at {path!r}")
if isinstance(cur, list):
idx = int(seg)
if not -len(cur) <= idx < len(cur):
raise IndexError(f"deep_set: index {seg!r} out of range for list of length {len(cur)}")
nxt = cur[idx]
if not isinstance(nxt, (dict, list, tuple)):
nxt = {}
cur[idx] = nxt
cur = nxt
else:
nxt = cur.get(seg) nxt = cur.get(seg)
if not isinstance(nxt, dict): if not isinstance(nxt, (dict, list, tuple)):
nxt = {} nxt = {}
cur[seg] = nxt cur[seg] = nxt
cur = nxt cur = nxt
cur[segments[-1]] = value last = segments[-1]
if isinstance(cur, tuple):
raise TypeError(f"deep_set: cannot set into an immutable tuple at {path!r}")
if isinstance(cur, list):
idx = int(last)
if not -len(cur) <= idx < len(cur):
raise IndexError(f"deep_set: index {last!r} out of range for list of length {len(cur)}")
cur[idx] = value
else:
cur[last] = value
return data
def _iter_children(node: Any):
"""yield (key, child) pairs for a `*` step: dict items, or list/tuple index/item pairs"""
if isinstance(node, dict):
yield from node.items()
elif isinstance(node, (list, tuple)):
yield from enumerate(node)
def deep_gets(data: Any, path: str, *, sep: str = ".") -> List[Any]:
"""get every value matching a dotted path with `*` wildcards, as a list (may be empty)
a `*` iterates every element at that level; multiple `*` fan out cartesian. a branch
that dead-ends (missing key, out-of-range index, non-container) is skipped, not raised.
a path with no `*` returns a one- or zero-element list.
"""
segments = path.split(sep)
frontier = [data]
for seg in segments:
nxt: List[Any] = []
if seg == _WILDCARD:
for node in frontier:
for _, child in _iter_children(node):
nxt.append(child)
else:
for node in frontier:
if isinstance(node, dict):
got = node.get(seg, _MISSING)
if got is not _MISSING:
nxt.append(got)
elif isinstance(node, (list, tuple)):
try:
nxt.append(node[int(seg)])
except (ValueError, IndexError):
pass
frontier = nxt
return frontier
def deep_sets(data: Any, path: str, value: "Union[Any, Callable[[Any], Any]]", *, sep: str = ".") -> Any:
"""set every value matching a dotted path with `*` wildcards; returns data for chaining
`value` is a plain value written to every match, or a callable `fn(current) -> new`
(a bare callable is always invoked - to store one, wrap it `lambda _c, f=fn: f`). a `*`
iterates every element at that level; multiple `*` fan out cartesian. best-effort: any
dead-end (missing key, out-of-range index) is skipped, not raised. setting through or
into a tuple raises TypeError (immutable).
"""
compute: Callable[[Any], Any] = value if callable(value) else (lambda _cur, _v=value: _v)
segments = path.split(sep)
parents = segments[:-1]
last = segments[-1]
frontier = [data]
for seg in parents:
nxt: List[Any] = []
if seg == _WILDCARD:
for node in frontier:
if isinstance(node, tuple):
raise TypeError(f"deep_sets: cannot set through an immutable tuple at {path!r}")
for _, child in _iter_children(node):
nxt.append(child)
else:
for node in frontier:
if isinstance(node, tuple):
raise TypeError(f"deep_sets: cannot set through an immutable tuple at {path!r}")
if isinstance(node, dict):
got = node.get(seg, _MISSING)
if got is not _MISSING:
nxt.append(got)
elif isinstance(node, list):
try:
nxt.append(node[int(seg)])
except (ValueError, IndexError):
pass
frontier = nxt
for node in frontier:
if isinstance(node, tuple):
raise TypeError(f"deep_sets: cannot set into an immutable tuple at {path!r}")
if last == _WILDCARD:
if isinstance(node, dict):
for key in list(node.keys()):
node[key] = compute(node[key])
elif isinstance(node, list):
for idx in range(len(node)):
node[idx] = compute(node[idx])
elif isinstance(node, dict):
if node.get(last, _MISSING) is not _MISSING:
node[last] = compute(node[last])
elif isinstance(node, list):
try:
idx = int(last)
except ValueError:
continue
if -len(node) <= idx < len(node):
node[idx] = compute(node[idx])
return data return data
+11 -31
View File
@@ -1,25 +1,7 @@
"""retry with exponential backoff sync and async, one backoff engine. """retry with exponential backoff, sync (`retry`) and async (`aretry`), call or decorator form.
de-duplicates retry logic that was written divergently across several libs (HTTP
429/5xx caps, proxy burn/rotate caps, IMAP reconnects). both a call form and a
decorator form share one implementation per flavor; the backoff schedule is a pure
generator so it tests without real sleeps.
from commons import retry, aretry
# call form
rows = retry(lambda: db_read(), attempts=5, on=(IOError,))
data = await aretry(lambda: fetch(url), attempts=3, on=(TimeoutError,))
# decorator form (same kwargs)
@aretry(attempts=4, backoff=0.5, on=(ConnectionError,))
async def pull():
...
after `attempts` are exhausted the LAST exception is re-raised (fail loud, never after `attempts` are exhausted the LAST exception is re-raised (fail loud, never
swallowed). `on` narrows which exceptions retry; `give_up(exc) -> bool` stops early swallowed). see README for usage examples.
on a non-retryable error (e.g. a 400 vs a 429). each retry is logged (emit-only),
never printed.
""" """
import asyncio import asyncio
@@ -38,7 +20,7 @@ def _delays(attempts: int, backoff: float, factor: float, max_backoff: float):
"""yield the wait before each retry: min(backoff * factor**n, max_backoff) """yield the wait before each retry: min(backoff * factor**n, max_backoff)
yields `attempts - 1` delays (one before each retry after the first try). the yields `attempts - 1` delays (one before each retry after the first try). the
raw, un-jittered schedule jitter is applied at call time so the schedule stays raw, un-jittered schedule - jitter is applied at call time so the schedule stays
pure and testable. pure and testable.
""" """
for n in range(max(0, attempts - 1)): for n in range(max(0, attempts - 1)):
@@ -72,13 +54,12 @@ def retry(
sleep: Callable[[float], None] = time.sleep, sleep: Callable[[float], None] = time.sleep,
rand: Callable[[], float] = random.random, rand: Callable[[], float] = random.random,
): ):
"""retry a sync callable with exponential backoff; call form or decorator """retry a sync callable with exponential backoff; call form (`retry(fn, ...)`) or decorator
`retry(fn, ...)` runs immediately; `@retry(...)` wraps a function. retries on the `attempts` is floored at 1 so the callable always runs at least once.
`on` exceptions, stops early if `give_up(exc)` is true, re-raises the last
exception once `attempts` are exhausted. `sleep`/`rand` are injectable for tests.
""" """
types = _as_types(on) types = _as_types(on)
attempts = max(1, attempts)
def run(target: Callable, args, kwargs): def run(target: Callable, args, kwargs):
delays = list(_delays(attempts, backoff, factor, max_backoff)) delays = list(_delays(attempts, backoff, factor, max_backoff))
@@ -94,7 +75,7 @@ def retry(
wait = _jittered(delays[index], jitter, rand) wait = _jittered(delays[index], jitter, rand)
log.warning( log.warning(
"retry %d/%d after %s: %s", "retry %d/%d after %s: %s",
index + 1, last_index, type(exc).__name__, exc, index + 1, attempts, type(exc).__name__, exc,
) )
if wait > 0: if wait > 0:
sleep(wait) sleep(wait)
@@ -123,13 +104,12 @@ def aretry(
sleep: Callable[[float], "asyncio.Future"] = asyncio.sleep, sleep: Callable[[float], "asyncio.Future"] = asyncio.sleep,
rand: Callable[[], float] = random.random, rand: Callable[[], float] = random.random,
): ):
"""retry an async callable with exponential backoff; call form or decorator """async twin of `retry`; call form (`await aretry(coro_fn, ...)`) or decorator, same semantics
async twin of `retry`. `await aretry(coro_fn, ...)` runs immediately; `attempts` is floored at 1 so the callable always runs at least once.
`@aretry(...)` wraps a coroutine function. same semantics: retry on `on`, stop on
`give_up`, re-raise the last exception after `attempts`. `sleep`/`rand` injectable.
""" """
types = _as_types(on) types = _as_types(on)
attempts = max(1, attempts)
async def run(target: Callable, args, kwargs): async def run(target: Callable, args, kwargs):
delays = list(_delays(attempts, backoff, factor, max_backoff)) delays = list(_delays(attempts, backoff, factor, max_backoff))
@@ -145,7 +125,7 @@ def aretry(
wait = _jittered(delays[index], jitter, rand) wait = _jittered(delays[index], jitter, rand)
log.warning( log.warning(
"retry %d/%d after %s: %s", "retry %d/%d after %s: %s",
index + 1, last_index, type(exc).__name__, exc, index + 1, attempts, type(exc).__name__, exc,
) )
if wait > 0: if wait > 0:
await sleep(wait) await sleep(wait)
+6 -12
View File
@@ -1,22 +1,16 @@
"""time helpers built on unix timestamps with timezone-aware datetime support. """time helpers built on unix timestamps with timezone-aware datetime support.
one engine, two ergonomics: one engine, two ergonomics: bare module functions for stateless unix-int math
- bare module functions (now/add/ahead/ago/is_expired/to_dt/...) operate on unix (all delta math routes through `_delta_seconds`, the single source of truth), and
ints for quick, stateless use. all delta math routes through `_delta_seconds`. `Clock`, which holds a timezone + fast flag and delegates to the same functions.
- `Clock` holds a timezone + fast flag and delegates to the same functions, so a
configured clock and the bare functions never diverge.
unix ints stay the storable value; datetimes are produced on demand in whatever
timezone you ask for (stdlib zoneinfo, no pytz).
""" """
import time as _time import time as _time
from datetime import datetime, timezone from datetime import datetime, timezone
from typing import Optional, Union from typing import Optional, Union
from zoneinfo import ZoneInfo from zoneinfo import ZoneInfo
# default fast flag for bare module calls. when True, every unit (day/hour/minute) # default fast flag for bare calls (test-only: collapses each unit to 1s); set
# counts as one second so time-based flows run fast in tests. set # `commons.timing.FAST_MODE` from test setup, off by default, not a config import.
# `commons.timing.FAST_MODE` from test setup; off by default. not a config import.
FAST_MODE = False FAST_MODE = False
UTC = timezone.utc UTC = timezone.utc
@@ -42,7 +36,7 @@ def now() -> int:
def _delta_seconds(days: int, hours: int, minutes: int, seconds: int, fast: bool) -> int: def _delta_seconds(days: int, hours: int, minutes: int, seconds: int, fast: bool) -> int:
"""seconds for a unit combination; collapses to 1s/unit when fast is True """seconds for a unit combination; collapses to 1s/unit when fast is True
single source of truth for unit math every delta helper routes through here. single source of truth for unit math - every delta helper routes through here.
""" """
if fast: if fast:
return days + hours + minutes + seconds return days + hours + minutes + seconds