commons/README.md

# commons

Small sync helpers shared across projects. Base is stdlib only — **no dependencies**.

- `timing` — unix-timestamp deltas + timezone-aware datetime conversions
- `paths` — nested dict/list access by dotted path
- `masking` — display masking for cards / cvv / tokens
- `retry` — exponential-backoff retry, sync (`retry`) and async (`aretry`)
- `addr` — ip/address tooling: pure stdlib ip utils in base, async geo lookups
  behind the `commons[addr]` extra

## Install

```
commons @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v0.3.5
# async address/geo lookups (fetch_ip / ip_location / fetch_location) need the extra:
commons[addr] @ git+ssh://git@git.rethinkstudios.io/rethink-public/commons.git@v0.3.5
```

The base install pulls **nothing** (stdlib). Only `commons[addr]` adds `aiohttp`, and
only for the geo lookups — the pure `commons.addr.ip` utilities ship in base.

Drop the `@v0.3.5` suffix from the line above to install the latest unpinned.

## timing

Unix ints stay the storable value; datetimes are produced on demand in whatever
timezone you ask for. One engine (`_delta_seconds`) backs both the bare functions
and `Clock`.

### Deltas (bare functions)

```python
from commons import now, add, ahead, ago, is_expired

now()                       # current unix ts (int)
add(ts, days=1, hours=-3)   # shift a ts by signed units
ahead(days=7)               # now + delta  (replaces the old in_days/in_hours/...)
ago(hours=2)                # now - delta
is_expired(deadline)        # True if past; None/0 never expires
```

`add`/`ahead`/`ago` take `days/hours/minutes/seconds` as keyword units — one call,
any combination, signed.

### Datetime + timezone

```python
from commons import to_dt, to_unix, now_dt, convert, fmt, date

to_dt(ts)                       # aware datetime in UTC
to_dt(ts, "America/New_York")   # same instant, eastern wall clock
to_unix(some_datetime)          # datetime -> unix (naive read as UTC, or pass tz)
now_dt("Asia/Tokyo")            # current time as an aware datetime in a tz
convert(dt, "Asia/Tokyo")       # re-express any datetime in another tz (same instant)
fmt(ts, "America/New_York", "%H:%M")   # formatted string in a tz
date(ts, "America/New_York")    # "mm/dd/YYYY" in a tz (ts optional -> now)
```

Timezones accept an IANA name (`"America/New_York"`), a tzinfo, or `None` (UTC).

### Clock

A `Clock` binds a timezone + fast flag and delegates to the same functions, so you
don't repeat the tz on every call:

```python
from commons import Clock

clock = Clock("America/New_York")     # any IANA tz; defaults to UTC
clock.now()                            # unix ts
clock.add(ts, days=1)                  # delta (honors the clock's fast flag)
clock.ahead(days=7) / clock.ago(hours=2)
clock.to_dt(ts) / clock.now_dt()       # datetimes in the clock's tz
clock.convert(dt)                      # re-express dt in the clock's tz
clock.fmt(ts) / clock.date(ts)         # formatted in the clock's tz
```

### Test mode

`fast` collapses every unit to one second so time-based flows run quickly in tests.
Per-call (`add(ts, days=1, fast=True)`), per-clock (`Clock(fast=True)`), or as the
module default for bare calls:

```python
from commons import timing
timing.FAST_MODE = True     # in test setup
```

## paths

Two pairs of verbs: **single-path** (scalar in/out) and **wildcard** (bulk, always a list).

### single-path — `deep_get` / `deep_set`

```python
from commons import deep_get, deep_set

data = {"in": {"this": {"old": {"notation": 42}}}, "items": [{"id": "a"}, {"id": "b"}]}

deep_get(data, "in.this.old.notation")   # 42
deep_get(data, "items.1.id")             # "b"  (numeric segment indexes a list/tuple)
deep_get(data, "in.nope.here", "DEF")    # "DEF" (missing -> default, no raise)

deep_set({}, "a.b.c", 9)                 # {"a": {"b": {"c": 9}}}
deep_set(data, "items.1.id", "B")        # updates the list element in place
deep_get(data, "items.1.id")             # "B"  (get/set share the same grammar)
deep_set(data, "items.9.id", "X")        # raises IndexError (out of range, no
                                          # silent mis-store)
```

`deep_get` steps into both lists and tuples on a numeric segment. `deep_set` updates a
**list** element in place; setting into (or through) a **tuple** raises `TypeError` —
tuples are immutable, so "update in place" is impossible; flatten to a list first. An
out-of-range numeric segment raises `IndexError` instead of silently corrupting the
structure.

### wildcard — `deep_gets` / `deep_sets`

A `*` segment iterates every element at that level (dict values or list/tuple items);
multiple `*` fan out cartesian. These are **separate verbs** with a fixed list/bulk
return type — a `*` passed to `deep_get`/`deep_set` raises `ValueError` (use the plural
verb).

```python
from commons import deep_gets, deep_sets

data = {"users": [{"username": "al"}, {"username": "bo"}, {"username": "cy"}]}

deep_gets(data, "users.*.username")      # ["al", "bo", "cy"]  (ALWAYS a list, in order)
deep_gets(data, "users.*.nope")          # []  (no match -> empty list, no raise)

deep_sets(data, "users.*.username", "X")            # set every match to "X"
deep_sets(data, "users.*.username", str.upper)      # callable fn(current)->new per match
```

`deep_sets`'s second arg is either a plain value (write it to every match) or a callable
`fn(current) -> new` (compute each). It writes only matches that already exist (it does
not create missing keys). Setting through/into a tuple raises `TypeError`.

## masking

Display helpers only — they format a value for showing; they are **not a security
control** (the underlying value is unchanged and still needs proper handling).

```python
from commons import credit, cvv, phantom, provider, mask_url, mask_proxy, stable_id

credit("4111 1111 1111 1234")   # "•••• •••• •••• 1234"
cvv("123")                       # "•••"
phantom("abcdef1234567890")      # "abcdef...7890"
phantom("1234567890")            # "••••••••••"  (len <= 10 fully masks, never a false reveal)
provider("4111111111111111")     # "VISA"  (VISA/MC/AMEX/UPAY/DISC/JCB/DNRS/UNKW)
provider("4111²111111111234")    # "VISA"  (unicode digit lookalikes ignored, never raises)

# strip query + fragment off a url before logging it
mask_url("https://u:pw@api.x/v2?apiKey=SECRET&ip=8.8.8.8")  # -> "https://u:pw@api.x/v2"
mask_url("https://api.x/v2")                                # -> "https://api.x/v2"  (unchanged)

# partial, user-reportable proxy id: creds stripped, ipv4 middle octets masked
mask_proxy("172.58.32.1:8080")                 # -> "172.***.***.1:8080"
mask_proxy("user:supersecret@172.58.32.1:8080")  # -> "172.***.***.1:8080"  (creds gone)
mask_proxy("proxy.host.net:8080")              # -> "proxy.host.net:8080"  (hostname untouched)

# deterministic, regenerable id from ordered parts
stable_id("profit_lounge", "user_123")         # -> "d1f4…"  (same 16 hex chars every call)
```

`mask_url` drops the query and fragment, keeping scheme/host/path. It does **not** parse
or hunt for sensitive params — a secret in a URL is the caller's bug, not this function's
to detect — so it structurally cannot leak a query param and cannot over-mask a legit one.
A URL with no query/fragment is returned unchanged. (Note: `user:pass@` userinfo is part of
the authority and is **not** stripped; keep credentials out of the URL you log.)

`mask_proxy` yields a partial identifier a user can safely report for debugging. Any
`user:pass@` userinfo (or a `scheme://` prefix) is dropped entirely, then an IPv4 host
`A.B.C.D` is masked to `A.***.***.D` with the port kept in full; a non-IPv4 host (hostname
or IPv6) passes through unchanged with only its credentials stripped. A spec with no
parseable host raises `ValueError`.

`stable_id` derives a fixed-length hex id from ordered string parts via SHA-256 — same
inputs always produce the same id, no state, so it's regenerable anywhere. Parts join with
a `\x00` separator so `("ab","c")` and `("a","bc")` differ; an empty or non-str part, or a
non-positive `length`, raises `ValueError`.

## retry

Exponential-backoff retry, sync (`retry`) and async (`aretry`). Each works as a **call
form** or a **decorator**, with the same kwargs. After the attempts are exhausted the
**last exception is re-raised** — it never swallows or returns a default.

```python
from commons import retry, aretry

# call form
rows = retry(lambda: read_db(), attempts=5, on=(IOError,))
data = await aretry(lambda: fetch(url), attempts=3, backoff=0.5, on=(TimeoutError,))

# decorator form (same kwargs)
@aretry(attempts=4, backoff=0.5, factor=2.0, on=(ConnectionError,))
async def pull():
    ...
```

Knobs: `attempts` (total tries), `backoff` / `factor` / `max_backoff` (delay is
`min(backoff * factor**n, max_backoff)`), `jitter` (full jitter, on by default),
`on=` (tuple of retryable exception types), and `give_up=lambda exc: ...` to stop early
on a non-retryable error (e.g. a 400 vs a 429):

```python
# retry 429/5xx but give up immediately on a 4xx
await aretry(send, attempts=4, on=(HTTPError,),
             give_up=lambda e: 400 <= e.status < 500 and e.status != 429)
```

Each retry is logged (emit-only). `sleep=` and `rand=` are injectable for deterministic
tests (no real waits).

## addr

IP/address tooling, exposed as a submodule. The pure `ip` utilities ship in the base
install (stdlib `ipaddress`); the async `geo` lookups need `commons[addr]`.

### ip (pure, base install)

```python
from commons import addr

addr.ip.is_valid("8.8.8.8")            # True   (False on bad input, never raises)
addr.ip.version("2001:db8::1")         # 6      (None if invalid)
addr.ip.to_int("0.0.0.1")              # 1
addr.ip.from_int(1)                     # "0.0.0.1"   (version=6 for ipv6)
addr.ip.is_private("10.0.0.5")         # True
addr.ip.is_global("8.8.8.8")           # True

addr.ip.in_network("10.0.0.5", "10.0.0.0/24")          # True
addr.ip.in_any("10.0.0.5", ["1.2.3.0/24", "10.0.0.0/8"])  # True (allow/blocklists)
# in_network / in_any return False on bad address OR bad cidr — they never raise

addr.ip.network_address("10.0.0.5/24")  # "10.0.0.0"   (strict=False tolerates host bits)
addr.ip.broadcast_address("10.0.0.5/24")# "10.0.0.255"
addr.ip.netmask("10.0.0.0/24")          # "255.255.255.0"
addr.ip.prefix_bits("10.0.0.0/24")      # 24
addr.ip.host_bits("10.0.0.0/24")        # 8
addr.ip.num_addresses("10.0.0.0/24")    # 256
addr.ip.set_bits("0.0.0.3")             # 2   (popcount of the address int)
addr.ip.hosts("10.0.0.0/29")            # ["10.0.0.1", ... "10.0.0.6"]
addr.ip.hosts("10.0.0.0/8", limit=100)  # cap materialization on huge ranges
```

### geo (async, needs `commons[addr]`)

Each lookup is async, accepts an optional `session=` (reuse an aiohttp
`ClientSession`; one is created and closed internally if omitted) and `timeout=15`,
and returns `None` on any request/parse failure.

```python
from commons.addr import fetch_ip, ip_location, fetch_location

await fetch_ip()                                  # your public ip via ipify
await ip_location("8.8.8.8", api_key=cfg_key)     # geo.ipify; api_key REQUIRED, injected
await fetch_location(40.7, -74.0)                 # nominatim reverse -> {"country", "state"}
```

- `ip_location`'s `api_key` is a **required keyword you inject** — there is no default
  and nothing hardcoded. (The old code shipped a hardcoded key; it's gone.)
- `fetch_location` sets a `User-Agent` (Nominatim's terms require one); override via
  `user_agent="your-app/1.0"`.
- Without the `[addr]` extra installed, the package still imports — but calling a geo
  function raises a clear `RuntimeError` telling you to install `commons[addr]`.