feat(zarr-metadata): composition rules layer, shape-exact codec guards, typed builders

[d-v-b]

🤖 AI text below 🤖

Ships the reframed scope for the metadata package: in addition to code that models the structure of the literal JSON, the package now ships code that models the rules governing composition of the elements of a full metadata document. The package's layering becomes one contract per layer:

1. zarr_metadata.model — structure, element by element; never interprets extension points
2. zarr_metadata.rules (new) — composition across the document; rules are data, keyed by the keys they read
3. zarr_metadata.builder (new) — construction conveniences that apply layers 1 + 2

The composition layer

zarr_metadata.rules exports the engine (Rule, applicable, run_rules), per-document rule sets, and read-side validate_* / is_* / parse_* trios that mirror the model grammar with a stronger judgment: structure and composition, every problem reported together with loc-precise diagnostics. This is the front door for readers — "is this loaded zarr.json a document I should act on?" — which previously had no blessed path. The is_* functions deliberately return bool, not TypeIs: a composition-invalid document is still an instance of the TypedDict, so only the structural layer can narrow honestly.

ZARR_V3_ARRAY_RULES (twelve rules): fill value ↔ data type (every dtype family, drift-tested), codec pipeline kind ordering (AA* AB BB*, exactly one AB, inconclusive in the presence of unknown codecs), known-name shapes, dimension_names counts, chunk-grid values (positive extents) and geometry (regular rank; rectilinear rank + per-dimension chunk-size sums incl. RLE pairs), transpose orders (self-permutation at any depth, rank agreement with shape), and sharding — inner codecs/index_codecs are judged as pipelines recursively at every nesting depth, and inner chunk shapes must be positive, rank-matched, and evenly divide the enclosing chunk, recursively. New ZARR_V2_ARRAY_RULES (chunks/shape rank) and ZARR_V3_GROUP_RULES (inline consolidated metadata recurses, judging each embedded child by its own rules at its path).

Boundary change: two composition checks that had leaked into the structural validator (v3 dimension_names↔shape, v2 chunks↔shape) moved into the rules layer. model parsers and dataclasses now accept those documents — they are lossless, structurally well-formed representations of what a store may contain — and the move kills the bug where one dimension_names mismatch was reported twice in different words.

Strictness stance, now documented on the package: zarr_metadata models canonical documents and is deliberately stricter than any given implementation (it rejects fill_value: 5.0 for uint8; zarr-python coerces it). Implementations coerce ambiguous input as they see fit and then validate the canonical result — disagreement in that direction is the contract, not drift.

Shape-exact codec classification

zarr_metadata.v3.codec.kind ships branded per-kind unions and TypeIs guards. An adversarial review found the object-form guards unsound — is_array_array_codec({"name": "transpose"}) narrowed a dict with no configuration to TransposeCodecObject (reading the narrowed value raised KeyError), and is_known_codec({}) itself raised. TypeIs narrowing is two-sided, so the fix is exactness: a new v3._shape module holds one type-level validator per known codec and chunk grid, derived from the TypedDicts' __annotations__/__required_keys__ (never restated), and both the guards and the rules' spelling checks consume it. That single mechanism also closed the config-value hole: {"name": "gzip", "configuration": {"level": "high"}}, endian: "middle", blosc cname: "nope", and unexpected configuration keys are now rejected instead of passing a key-presence-only check. Drift tests tie every codec/grid/data-type module to its judgment registry, so an unregistered addition fails a test rather than silently suppressing the exactly-one-array→bytes check.

Construction layer

• create_* factories — one per public document TypedDict, **kwargs: Unpack[<TypedDict>], enforced by a drift test. At literal-keyword call sites missing required keys and wrong value types are static errors; the docs are explicit that **-splat bypasses that coverage, and the runtime pass (normalize → structure → composition, one combined raise) exists for exactly those callers. The strict on-disk .zarray/.zgroup factories reject attributes at runtime (previously a splatted attributes came back typed as a shape that forbids it), and the v2 consolidated factory validates the .zmetadata envelope instead of validating nothing.
• ZarrV3ArrayMetadataBuilder — immutable incremental accumulation for staged assembly: eager rule firing whenever a rule's dependencies are present, unconstrained field order, cross-call conflicts naming both fields, T | UNSET properties on the PEP 661 sentinel, no shared mutable state, no build_unchecked. Scoped honestly: v3 arrays only, the document type with the richest cross-field coupling.

Review provenance

The branch was attacked by four independent adversarial reviewers (premise, static-typing claims, validation soundness, maintenance cost) plus roborev. Everything they confirmed is fixed here: the TypeIs unsoundness, the drifted "regular" literal, malformed r<N> dtype names escaping judgment, the .zarray/.zgroup type-laundering, the do-nothing v2 consolidated factory, the double-reported dimension mismatch, the nine accepted-invalid documents, and the overclaimed "strictly stronger than runtime" static story. One notable non-finding: 30k mutated documents through the one-shot and incremental construction paths found zero verdict divergence — the eager rule engine is monotone by construction.

Verified: just check green throughout — ruff, pyright 1.1.404 strict (0 errors), 734 tests, strict docs build (new API pages for rules and builder).

Known follow-up: v2 fill-value/dtype consistency (NumPy dtype grammar) has no rule yet; front-door re-export of rules names is an open curation decision.

🤖 Generated with Claude Code

[d-v-b]

prefer tuples over lists for this return type

[d-v-b]

avoid falsy iterables. check explicitly if the length is 0

[d-v-b]

consider making this return type a tagged union wrapping either the collection of validation problems or the validated document

[d-v-b]

this is cool but I don't want to be creative here. is there prior art for this design that we can cite / lean on? we don't want a formal dependency, but we do want to avoid making mistakes that have been sanded off by experience in other projects.

[d-v-b]

this design has _v3_array.py owning a rule that depends on the details of two chunk grids. what happens if we add a third? a better design gives individual fields / variants of fields their own rules, which are collected / called in a central place. then adding a new codec / chunk grid etc doesn't require changes in this module.

[d-v-b]

manually constructing this variable here means we can easily define a rule and forget to register it. consider more robust alternatives.

[d-v-b]

is making these fields closed supported by the spec? if it's ambiguous in some cases (which I suspect it might be, e.g. for codecs) we should ensure that this library can still represent at some level metadata that uses the implicit openness, even if the core validation assumes closed.

[d-v-b]

why is this typed dict[str, Any]? The bar for using Any should be incredibly high in this codebase. object is preferred in 99% of cases. And we know it's a metadata document. We have a type for that.

[d-v-b]

prefer create_zarr_v2_zarray_json and create_zarr_v2_zgroup_json. ensure that no other instances of "z_array" and "z_group" exist.

[d-v-b]

this name is vague, and especially confusing in conjunction with without, which implies a with. we should find something expressive and symmetrical, like with_fields and without_fields