revert: fingerprint_data back to canonical-json hash + stringify + callable guard
the custom normalization layer (_fingerprint_normalize / _fingerprint_key / _fingerprint_default + type-tagged markers + determinism allowlist) guarded against non-json inputs that never arrive by contract, and in exchange crashed on ordinary value types (uuid/decimal/path/objectid) and still collided (bytes vs ["bytes",hex]). worse than the four-liner on both axes, and it regressed across four iterations. fingerprint_data takes json-shaped record data, so json.dumps(sort_keys=True, separators, default=str) is the correct tool: order-independent, whitespace-stable, non-json scalars stringified rather than crashing. the one real footgun default=str introduces - a callable's str() embeds a memory address, non-deterministic across processes - is guarded with an explicit TypeError. non-str dict keys still raise from json natively (str-key contract, matching encrypt_data's own guard and the original). deletes the whole machinery. closes the fingerprint regressions R1/R2/R3 and the marker-encoding docstring nit at the root. digest VALUE changes vs the marker-based version - any persisted fingerprints must re-baseline. Signed-off-by: disqualifier <dev@disqualifier.me>
This commit is contained in:
@@ -630,77 +630,21 @@ def decrypt_record(crypto: EnvelopeCrypto, record, traversal_level: int = 2) ->
|
|||||||
return result
|
return result
|
||||||
|
|
||||||
|
|
||||||
_DETERMINISTIC_SCALARS = (str, int, float, bool, type(None))
|
|
||||||
|
|
||||||
|
|
||||||
def _fingerprint_default(value: Any) -> Any:
|
|
||||||
"""json.dumps default= handler for values fingerprint_data can't natively serialize
|
|
||||||
|
|
||||||
handles datetime (isoformat) and bytes (hex) deterministically, returning a 2-element
|
|
||||||
``["type", value]`` list so the encoding is a JSON ARRAY that no genuine str value can
|
|
||||||
ever equal (a str encodes as a JSON string) - collision-resistant by construction.
|
|
||||||
anything else - a callable, an arbitrary object, a generator - has no guaranteed stable
|
|
||||||
representation across processes (its repr usually embeds a memory address), so it fails
|
|
||||||
loud with a TypeError rather than yield an unstable digest.
|
|
||||||
"""
|
|
||||||
if hasattr(value, "isoformat"):
|
|
||||||
return ["datetime", value.isoformat()]
|
|
||||||
if isinstance(value, (bytes, bytearray)):
|
|
||||||
return ["bytes", value.hex()]
|
|
||||||
raise TypeError(
|
|
||||||
f"fingerprint_data: {type(value).__name__} has no deterministic representation; "
|
|
||||||
"serialize it to a stable str/bytes/datetime before fingerprinting"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _fingerprint_key(key: Any) -> str:
|
|
||||||
"""derive a deterministic, type-qualified tag for a dict key
|
|
||||||
|
|
||||||
scalars tag by type + value; set/frozenset keys canonically sort their members; any other
|
|
||||||
key type (a callable, an identity-repr object, a nested container) has no stable key form
|
|
||||||
and raises TypeError, mirroring the value-side determinism guard.
|
|
||||||
"""
|
|
||||||
if isinstance(key, _DETERMINISTIC_SCALARS):
|
|
||||||
return f"{type(key).__name__}:{key!r}"
|
|
||||||
if isinstance(key, (set, frozenset)):
|
|
||||||
normalized = _fingerprint_normalize(key)
|
|
||||||
return f"{type(key).__name__}:" + json.dumps(normalized, sort_keys=True, default=_fingerprint_default)
|
|
||||||
raise TypeError(
|
|
||||||
f"fingerprint_data: dict key of type {type(key).__name__} has no deterministic "
|
|
||||||
"representation; use a scalar or set/frozenset key"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _fingerprint_normalize(value: Any) -> Any:
|
|
||||||
"""recursively normalize dict keys and sets so the encoding is deterministic across processes"""
|
|
||||||
if isinstance(value, dict):
|
|
||||||
return {
|
|
||||||
_fingerprint_key(key): _fingerprint_normalize(sub_value)
|
|
||||||
for key, sub_value in value.items()
|
|
||||||
}
|
|
||||||
if isinstance(value, (set, frozenset)):
|
|
||||||
members = [_fingerprint_normalize(item) for item in value]
|
|
||||||
members.sort(key=lambda item: json.dumps(item, sort_keys=True, default=_fingerprint_default))
|
|
||||||
# keep the concrete type in the marker so a set value and a frozenset value with the
|
|
||||||
# same members stay distinct; the \x00 prefix keeps it uncollidable with a user str key
|
|
||||||
return {f"\x00{type(value).__name__}:members": members}
|
|
||||||
if isinstance(value, (list, tuple)):
|
|
||||||
return [_fingerprint_normalize(item) for item in value]
|
|
||||||
return value
|
|
||||||
|
|
||||||
|
|
||||||
def fingerprint_data(data: dict) -> str:
|
def fingerprint_data(data: dict) -> str:
|
||||||
"""return a deterministic, collision-resistant SHA-256 hex fingerprint of a dict
|
"""deterministic sha256 fingerprint of json-shaped dict data
|
||||||
|
|
||||||
keys of different types that would coerce to the same JSON string are kept distinct via
|
the input is a record sent over the wire, so it is json-serializable by contract:
|
||||||
a type-tagged pre-pass; set/frozenset keys and values are canonically sorted so their
|
keys are canonically sorted for an order-independent digest, separators are fixed for
|
||||||
unordered members hash stably; datetime/bytes go through a type-qualified default= handler
|
whitespace stability, and any non-json scalar (uuid/decimal/path/datetime/objectid) is
|
||||||
whose ``\\x00``-prefixed tags cannot collide with a genuine str value. any key or value
|
stringified rather than crashing the encode. a callable value raises - its str() embeds a
|
||||||
with no deterministic representation (a callable, an identity-repr object) raises TypeError
|
memory address, which would make the digest vary across processes. never logs the data.
|
||||||
rather than yield a digest that varies across processes. never logs the data.
|
|
||||||
"""
|
"""
|
||||||
normalized = _fingerprint_normalize(data)
|
def _stringify(value: object) -> str:
|
||||||
encoded = json.dumps(normalized, sort_keys=True, default=_fingerprint_default)
|
if callable(value):
|
||||||
|
raise TypeError(f"fingerprint_data: cannot fingerprint a callable: {value!r}")
|
||||||
|
return str(value)
|
||||||
|
|
||||||
|
encoded = json.dumps(data, sort_keys=True, separators=(",", ":"), default=_stringify)
|
||||||
return hashlib.sha256(encoded.encode()).hexdigest()
|
return hashlib.sha256(encoded.encode()).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user