Conversation
github-actions
Bot
added
needs release notes
d-v-b
force-pushed
the
zarr-metadata/rules-wrap-blosc
branch
from
September 19, 2026 19:43
f39ce0d to
59a1288
Compare
Documentation build overview
12 files changed ·
|
d-v-b
added a commit
to d-v-b/zarr-python
that referenced
this pull request
Sep 20, 2026
`typesize` is the only conditionally-required configuration member of any entity this package models: "Positive integer specifying the stride in bytes over which shuffling is performed. Required unless `shuffle` is `"noshuffle"`, in which case the value is ignored." A TypedDict cannot express that, so `BloscCodecConfiguration` declares it `NotRequired` and nothing supplied the condition — leaving the member the spec singles out as required as the one member of a blosc configuration that could always be omitted, while `blocksize`, which the spec never marks required, was. A rule supplies it, alongside the value constraints the shape validator cannot state: `clevel` in [0, 9], `typesize` positive, `blocksize` non-negative. blosc leaves the deliberately-rule-free list. These rules were written for the stacked zarr-developers#4380, which is based on this PR's pre-redesign head and would not apply to it; they are brought down here because this is the PR that makes `typesize` optional, and zarr-developers#4380 should drop its blosc half when it rebases. Verified against the spec's own example document and against what zarr-python writes for blosc with and without shuffling. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
d-v-b
force-pushed
the
zarr-metadata/rules-wrap-blosc
branch
from
September 20, 2026 12:20
59a1288 to
bde5d5d
Compare
d-v-b
changed the title
d-v-b
force-pushed
the
zarr-metadata/rules-wrap-blosc
branch
from
September 20, 2026 12:26
bde5d5d to
7009a21
Compare
…dators, create_* factories Adds `zarr_metadata.rules`: cross-field judgments over whole documents (fill value vs data type, codec pipeline ordering, chunk-grid geometry, sharding and transpose against the array each codec actually receives, struct field rules, dimension-name counts), registered where they are defined and dispatched per entity; `validate_*` / `is_*` / `parse_*` / `check_*` front doors for readers; `v3._shape` validators derived from the entity TypedDicts; and one `create_*` factory per document TypedDict. The two rank checks (v3 `dimension_names` vs `shape`, v2 `chunks` vs `shape`) move from the structural validator into the rules layer. The pydantic field types now run the rules layer before normalizing, so they are strictly stronger than before rather than weaker. Relative to #296 this drops the incremental builder, the extension-point provenance table, `TypeIs` codec guards, fill-value propagation through codec chains (no rule read it), registry introspection helpers, and a duplicated v2 consolidated envelope check; renames the `.zarray`/`.zgroup` factories to `zarray`/`zgroup`; and requires an endianness on the `bytes` codec inside `index_codecs`. Split from #296 (part 2 of 3). Assisted-by: ClaudeCode:claude-fable-5-1
Assisted-by: ClaudeCode:claude-fable-5-1
Assisted-by: Codex:GPT-6
…eate_* factories Upstream enabled ruff S101 for runtime code (zarr-developers#4363). The six parse-then-raise factories share a _parsed_or_raise helper that narrows the parsed document, and the rule-registration import uses importlib.import_module instead of an assert to keep it referenced. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…4379 Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The builder is a construction feature in a read-side PR, and its factories re-implemented the structure-plus-composition combination this PR already exposes as `rules.validate_*` / `rules.parse_*`. Held back for a later PR alongside the incremental builder. `test_validator_and_factory_agree` becomes `test_validator_and_parser_agree`: the same property against the front door the package keeps. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A fourth read-side front door for a question `validate_*` and `parse_*` already answer, with two new public types (`Valid`, `Invalid`) and no consumer. Removed with its tests and changelog fragment. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The `is_*` counterparts were `bool`, not `TypeIs`, and each docstring had to explain that they do not do what the name promises; `model.is_*` remains for narrowing. Their removal leaves `validate_*` and `parse_*`, which now share `_judged` over an already-normalized document instead of walking it twice per call. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- `initial_spec` was a four-line tail of `chain_initial_spec` living in another module; folded into its only caller. - `STORAGE_TRANSFORMERS` was defined and exported without a reader. - `_engine`'s prior-art bibliography condensed to the claim it supports. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The struct spec states its field constraints in the bytes codec's terms rather than inventing its own: field types must have "a fixed encoded size", and "When a `struct` type contains multi-byte numeric fields, the `bytes` codec MUST be configured with an explicit `endian` setting". So the struct rule's fixed-size question and the bytes rule's endianness question are one classification, and the two hand-written tables of data-type sizes were the same table twice. `rules._storage_class` now owns it, keyed off the data-type modules' own name constants, and `test_registry_drift` fails if a new data type arrives without a class — previously it would have silently gone unjudged by both rules. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The strict reading was already the behavior; the prose hedged, telling callers they "can filter" a kind that every raising path rejects. State the decision instead: an unmodelled member of a known entity's configuration is almost always a typo or a setting meant for a different entity, so `parse_*` and the pydantic field types both refuse it, and the dedicated kind is there for triage via `validate_*`. Two tests pin what was untested: that the pydantic field types run the rules layer at all, and that they reject an unknown configuration member. Also syncs the README and docs feature lists, which still described the pydantic integration as delegating to the model parser and omitted the rules layer entirely, and notes that the generated JSON Schemas leave configurations open, so schema-valid input can still fail at runtime. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Drop a `pyright: ignore[reportPrivateUsage]` on a public name; pyright is configured for `src` only, so the comment was never evaluated. - Keep `Sequence` under TYPE_CHECKING alongside `Callable` and `Rule`. - Rewrap the changelog fragment and say "two validation layers", so it does not read as contradicting the README's three-layer overview. - "different entity", consistently, in the pydantic strictness test. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The previous wording — "every entry point that raises rejects it, the pydantic field types included" — is false. Three raising entry points accept an unmodelled configuration member: `model.parse_*`, `ZarrV3ArrayMetadata.from_json`, and the bare `ZarrV3MetadataField` pydantic type. Only the rules layer judges configurations, so only `rules.parse_*` and the whole-document pydantic field types reject it. Say that instead, in the README, the docs site, the `ProblemKind` docstring and the changelog fragment, and assert the `ZarrV3MetadataField` half of the boundary so it cannot move unnoticed. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found by adversarial review; each has a regression test.
- scale_offset registers the identity spec transition the spec describes
("MUST be performed using the arithmetic semantics of the input array's
data type"; astype was removed in v3 in favour of cast_value). Without
it a no-op codec stopped propagation and stood down every later rule.
The test exemption that recorded this is gone.
- chain_initial_spec keeps the rank when it cannot keep the extents: an
ArraySpec extent may now be None individually, so a rectilinear grid or
a zero extent no longer hides a rank-mismatched transpose or shard.
- Entity rules declare the configuration members they read, and
run_entity_rules stands down only the rules that read an unusable
member rather than the whole entity. This also makes the invariant the
configuration["member"] accesses rely on explicit and checked at
registration, where it was previously true only by inspection.
- A rule reporting at the entity itself keeps that location instead of
being re-based under a "configuration" node a bare-string entity does
not have.
- The endianness problem names the data type, so the shard-index case
reads as uint64 rather than appearing to contradict the document.
Assisted-by: ClaudeCode:claude-opus-5
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…sking Two defects in the per-member gate added in 08d335d, both found by re-running the adversarial review against it. `run_entity_rules` decided "the configuration is unreadable" from the *length* of a problem's location, so `("must_understand",)` qualified and a malformed envelope flag stood down every composition rule for that entity. Test the two locations that actually mean it instead. `reads` promised that declaring a member makes `configuration[member]` safe, but it was validated against every modelled member rather than the required ones. A rule declaring an optional member still raised KeyError out of validate_*, which must never raise. `reads` now accepts only required members; `reads_optional` covers the presence-tested case (the bytes codec's `endian`), and registration refuses the unsafe spelling with a message naming the alternative. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three sites derived "the shape of the chunk this pipeline encodes" by hand — the document's grid, a shard's inner chunk shape, and the shard index — and each got a different subset of the reasoning right. Patching them one at a time produced three near-identical fixes and missed two more cases, so model the thing itself. `rules._chunk_grid` answers the question per dimension instead of per grid: a `GovernedShape` has one entry per axis, `None` where the chunks differ or the metadata cannot be read, and is itself `None` only when not even the rank is known. Following zarrs, a grid is read from its metadata together with the array shape it partitions (neither determines a grid alone), its rank is always available, and its extents are reported dimension by dimension rather than as one shape. Two cases the previous derivations could not express now work. A rectilinear grid whose chunk shapes are uniform pins the shard shape, in all three spellings the spec allows, while one uniform on a single axis is judged there and declines elsewhere. A shard index is judged against chunks-per-shard plus a trailing dimension of 2, as the spec derives it, instead of against nothing. Also extracts entity configuration access into rules._entity so the grid module can read a grid's configuration without importing the dispatcher. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A codec pipeline encodes one chunk, but the same pipeline encodes every
chunk, so a chain rule is a statement about all of them. The propagated
value said otherwise: `ArraySpec` held one chunk's shape, so an axis whose
chunks differ collapsed to "unknown" and the rules that could have judged
it stood down.
`ArrayParts` carries the `ChunkGrid` instead. A grid holds its rank, its
metadata as written, and per dimension the set of lengths that dimension's
chunks take — singletons for a regular grid, `{30, 34}` for a rectilinear
axis, `None` only where nothing can be read. Divisibility quantifies over
that set and names the length that fails.
This catches a case no previous shape could express: a rectilinear grid,
a transpose moving its varying axis, and a sharding codec that has to
divide what arrives. Inner extents of 15, 30 and 34 against chunks of 30
and 34 were all accepted before; only a common divisor passes now.
Two things fall out of holding the grid rather than a projection of it.
Sharding stops being a special case — a shard is a nested array, so its
inner pipeline is built by the same constructor as the document's own —
and a third-party grid is carried verbatim instead of being flattened at
the first hop, so a future rule can read its own configuration.
`data_type` becomes non-optional: the only documents that cannot supply
one are documents the structural layer has already rejected, so rather
than a half-populated value, a codec that can no longer be described
receives nothing at all. `NOTHING_KNOWN` is gone in favour of `| None`,
and rules test one guard instead of a field at a time.
Named for what it is, after zarrs: a grid is built from metadata *and* the
array shape it divides (neither determines a grid alone), its
dimensionality is total, and its edge lengths are reported per dimension.
`ArrayParts` avoids `chunk`, which would be singular, and `ArraySpec`,
which is taken by zarr.core.array_spec for the runtime type.
Assisted-by: ClaudeCode:claude-opus-5
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`st.from_type` resolves the codec TypedDicts unaided — ReadOnly, closed, NotRequired and Literal all work — once one strategy is registered for the recursive JSONValue alias, without which sharding_indexed, cast_value and scale_offset cannot be resolved at all. The existing totality test feeds arbitrary JSON to the document validators. That is the right guard for the structural layer and no guard for this one: a random object never names a codec, so across 5,000 examples it dispatched no entity rule. Drawing codecs from their own types and assembling the chain by kind takes documents reaching a chain rule from 0% to 94%; ordering is what does it, because a misordered chain is rejected before any other rule runs, and a flat list is misordered most of the time. Adds the property nothing covered: documents valid by construction, with inner chunk shapes drawn from the divisors of the extents they must divide, over both regular and rectilinear grids. Every recent fix made this layer stricter and under 5% of generated documents are valid, so the accept side had no generated coverage at all. Each strategy's reach is asserted, not assumed — the witnesses it must produce, and why the three chain rules it cannot reach are out of reach by construction. Verified by sabotage: stubbing out entity dispatch fails both reach tests, where previously the property suite stayed green. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ck tests Differential testing against the pre-ArrayParts tree found 390 documents the old validator rejected and the new one accepted. All of them are one regression: gating the shard's inner and index pipelines on `incoming`. Both are determined by the sharding codec's own `chunk_shape` and by the spec — the index is a `uint64` array whatever reaches the codec — so neither should ever have waited on upstream. It was a consequence of making `data_type` non-optional: with nothing to pair the grid with, both starts collapsed. The reasoning behind that change held for documents and failed inside a shard, where the grid is known and the element type is not, so `data_type` is optional again while `ArrayParts | None` keeps its own meaning of "no array here". Also: an unusable `data_type` costs itself and no longer hides the geometry, and `ChunkGrid.permuted` declines on a non-permutation instead of raising `IndexError` one careless caller away from a validator. Tests are now type-checked. They were covered by nothing — pyright was configured for `src` alone and the repo's mypy hook resolves this package's imports as `Any` — which is how a `TYPE_CHECKING` import of a type deleted three commits ago survived. It also showed that `from_key_value(to_key_value())`, the round trip the models advertise, did not type-check: `Mapping` is invariant in its key type, so a mapping keyed by literal store keys is not a `Mapping[str, bytes]`. Widened. A mutation audit put the suite's kill rate at 67%, with the largest hole in the `reads` gate: `st.from_type` honours the TypedDicts, so no strategy produced an ill-typed configuration member, and four one-token changes to the gate made `validate_*` raise while the suite stayed green. `corrupted_chains` covers it; the three mutations were verified to fail now. Two redundant property tests are gone, and the reach claim in the strategies module is replaced with measured figures — the previous 39% to 94% and its short-circuit explanation did not reproduce. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`typesize` is the only conditionally-required configuration member of any entity this package models: "Positive integer specifying the stride in bytes over which shuffling is performed. Required unless `shuffle` is `"noshuffle"`, in which case the value is ignored." A TypedDict cannot express that, so `BloscCodecConfiguration` declares it `NotRequired` and nothing supplied the condition — leaving the member the spec singles out as required as the one member of a blosc configuration that could always be omitted, while `blocksize`, which the spec never marks required, was. A rule supplies it, alongside the value constraints the shape validator cannot state: `clevel` in [0, 9], `typesize` positive, `blocksize` non-negative. blosc leaves the deliberately-rule-free list. These rules were written for the stacked zarr-developers#4380, which is based on this PR's pre-redesign head and would not apply to it; they are brought down here because this is the PR that makes `typesize` optional, and zarr-developers#4380 should drop its blosc half when it rebases. Verified against the spec's own example document and against what zarr-python writes for blosc with and without shuffling. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"An integer from -131072 to 22 which controls the speed and level of compression", with 0 selecting the default. The shape validator can only say it is an integer, so the range is a rule, and zstd leaves the deliberately-rule-free list alongside blosc. `checksum` needs none: the spec marks it "(Optional)" and the TypedDict already declares it `NotRequired`. Its "Should be omitted if false" is a SHOULD, and this package reports violations of requirements rather than of advice. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
d-v-b
force-pushed
the
zarr-metadata/rules-wrap-blosc
branch
from
September 20, 2026 13:19
7009a21 to
fde974f
Compare
Most "rules" were not composition judgments at all. Of 23 entity rules, 15 read only the entity's own configuration: blosc's clevel range, gzip's and zstd's level ranges, a transpose order being a permutation of its own indices, a time scale factor's bounds. Those are refinements of the type — the TypedDict says int, the spec says int in [0, 9] — and they were in the rule layer because that is where the machinery happened to live, not because anything about them spans a document. They now sit where the member is typed. `_shape` gains a small value vocabulary (`_int_in_range`, positive and non-negative integers, a permutation checker) and, for the one constraint spanning two members of a single configuration, an entity invariant: blosc's "typesize is required unless shuffle is noshuffle". Seven rules become table entries, three rule modules are deleted outright, and `rules._entities` is left with the eight judgments that genuinely need the document or the chain. The boundary is the member, and the attempt to push past it is recorded because it failed usefully: moving per-element checks (every extent of a chunk_shape positive) into the shape layer cost real precision, because a shape verdict marks the whole member unusable. A zero on one axis stopped a rectilinear grid reporting the axis beside it and stood down a shard's inner pipeline. Those stay where they can be judged element by element. Document rules get the same layout: `_v3_array` is partitioned into field rules (codec pipeline ordering, known-entity shapes) and composition rules (fill value against data type, dimension names against shape), with a test asserting the partition so a new rule forces the decision the way `_RULE_FREE` already does for entities. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One function from a syntactically valid document to either that document in its simplest equivalent spelling or every reason it is not semantically valid. Canonical is defined per metadata variety as the simplest form with the same semantics. The generic half — an entity with nothing to configure collapsing to its bare name, a defaulted `must_understand` dropping while an explicit `false` survives — the model layer's round trip already performs exactly, so it is delegated rather than written twice. What is new is per-variety: blosc drops a `typesize` the spec says is ignored under `noshuffle`, and a rectilinear dimension run-length encodes, since `[size, count]` is the spelling that does not grow with the chunk count. `dimension_names` of nothing but nulls says what omitting it says. Each canonicalization lives with the entity it belongs to, in `v3/codec/blosc.py` and `v3/chunk_grid/rectilinear.py`, because they are pure transforms over already-valid metadata and need nothing from the rules layer. Two spellings that look collapsible are left alone on purpose, and are tested: a dimension-level bare integer repeats until it covers the extent rather than naming a fixed list, and a one-element list names exactly one chunk. They produce identical extents but different coverage verdicts, so collapsing either would change meaning under a resize. Asserted over the generated corpus: canonicalizing twice changes nothing further, and canonicalizing never changes a verdict — the second is what would catch a simplification that quietly says something else. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A codec's knowledge was spread over four modules: a member-type table in the shape layer, a value-rule module in the rules layer, a kind tuple, a canonicalizer. Four tables meant four chances to drift, and a drift test per table to notice. Put it in the class instead. `coerce` is type-space, `problems` is value-space, `to_json` is the canonical spelling, and the dataclass fields are the configuration -- so the constructor signature is the configuration TypedDict unpacked, and a test asserts it stays that way. `Context` is the scope a reading happens in: which identifiers are in play at each extension point, as either the core specification or the core plus the extensions this package models. It is passed to every `coerce` and most ignore it, but `struct` and `sharding_indexed` carry other entities inside their configuration and cannot coerce without it. blosc is the first entity through; the rest follow. Nothing is wired in yet, so the existing path is untouched. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`coerce` and `to_json` turned out to be the same for every entity that does not contain another one: check the name, walk the member table, construct. So they moved into the base, driven by two class variables, and a codec now declares what is particular to it -- its members, its pipeline kind, the values the spec disallows -- and nothing else. Three spellings of the same set of members now exist: the dataclass fields, the configuration TypedDict, and the member table. Two new tests hold all three together, and a third derives `configuration_required` from the TypedDict's required keys, because the spec ties the bare-name spelling to exactly that. `CodecKind` moved next to the entity base. It had to: `codec.kind` imports every codec module to build its tuples, so no codec module could import back from it. No slots on entities. `slots=True` rebuilds the class and leaves a subclass's zero-argument `super()` pointing at the class it replaced, and every entity calls `super()` to narrow `to_json` to its own object TypedDict. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ties Chunk-extent positivity moves onto the grid that declares the extents, where it can point at the offending axis. What needs the array's shape -- one extent per dimension, explicit specs summing to the extent -- stays in the rules layer, because a grid cannot answer it alone. Rectilinear's run-length encoding becomes its canonical `configuration`, and its dimension specs get a type check written out rather than composed from `sequence_of`: an entry is an extent or a `[size, count]` run, which `sequence_of` cannot say. Coercion normalizes JSON arrays to tuples at every depth before checking them, so a member never holds a list where its own type says tuple. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seventeen of the nineteen are a name and nothing else, so they are a name and nothing else here: three lines each, next to the constant that spells them. `r<N>` is the one entity whose identifier is not a name any document carries, because the family is a shape rather than a spelling. It keeps the spelling it was given rather than the bit count it means, so `r008` comes back out as `r008` -- that is a valid way of writing eight bits, and rewriting it is not this package's call. `RAW_BYTES_FAMILY` moves to `data_type.raw`, which owns the grammar it names; `_extension_points` imports it alongside the pattern it already imported, and the cycle that would otherwise close does not. `Context` moves to `_registry`, where the mapping it wraps already lived. Keeping it next to `MetadataEntity` meant `_entity` importing `_extension_points`, which imports `data_type.raw`, which every data type module now imports back. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`struct`, `sharding_indexed` and `cast_value` hold entities inside their own configuration, and `Context.coerce` is the primitive they are built from: it reads one nested entity in the scope the outer one was read in, returning the value untouched when its name is out of scope, and prefixing the problems with where in the configuration it sat. A nested codec's bad compression level now reports at `codecs.1.level`. Storage class becomes polymorphism. `_storage_class`'s three name sets said which data types are single-byte, multi-byte or variable-length, and hand-rolled the recursion for `struct`; now each data type declares its own and `StructDataType` folds its fields, which is the same recursion written once as a method call. The extension-point constants move to `_entity`. An entity that contains others has to name the point it reads them at, and `_extension_points` also folds `r<N>` names -- which means importing the data types, which import the entity base. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_check_fill_for_dtype` was a chain of name comparisons ending in a
table lookup, with the struct case recursing by hand. It becomes a
method: `DataTypeEntity.fill_value_problems`, defaulting to accepting
anything, because a type this package does not model is not ours to
judge.
The families are where the widths go. Every integer type differs from
every other only in its bounds, every float only in its hex parser,
every complex only in its component -- so each family is written once
and the type supplies the number. `struct` recurses through its fields
by calling the same method on each, which is the hand-rolled recursion
turned back into a method call.
`must_understand` becomes keyword-only. It belongs to the envelope, not
the configuration, and it was silently taking the first positional slot
of every entity -- `RawBytesDataType("r16")` set the flag and left the
name at its default.
Assisted-by: ClaudeCode:claude-opus-5
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two more name-keyed tables become methods. `ChunkGrid.of` dispatched on the grid's name to read its configuration; now each grid entity builds its own, and a grid that cannot be read falls back to the rank the array pins. `_TRANSITIONS` was a registry of "what this codec does to the array it receives", keyed by name and populated by a decorator; now it is `CodecEntity.transition`, defaulting to None so a modelled codec that forgets to say still fails closed. `ChunkGrid`, `ArrayParts` and the rest move from `rules` to `v3._parts`, because a codec cannot own its transition while the thing it transforms lives in the layer above it. Coercion also now normalizes a tuple's contents, not just a list's. Raw JSON arrives as lists all the way down, but a hand-written tuple holding a list was left half-normalized and failed its own type check. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e entities replaced Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every remaining rule was "this entity against some part of the document", which is a method that takes that part. A grid answers `shape_problems(array_shape)`, a codec answers `incoming_problems(incoming)`, and the rule registry that dispatched them by name will have nothing left to dispatch. The chain walk moves to `v3._chain` because `sharding_indexed` holds two pipelines and has to walk them to judge itself -- it cannot reach up into `rules` to do that. Ordering is read off each codec's own `kind` rather than three tuples of names. `ArrayParts` now carries the coerced data type rather than the metadata verbatim, so the `bytes` codec asks it for its storage class instead of classifying its name. `ChunkGrid` loses its `metadata` field. It was there so a rule for an unmodelled grid could read its own configuration, and an unmodelled grid no longer reaches a rule at all. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`array_problems_v3` is the document-level check the architecture was for: read every extension point in a scope, ask each entity what is wrong with its own values, then ask the questions that span fields by handing an entity the part of the document it needs. It knows nothing about `blosc` or `int32` or `rectilinear`; a new extension is a class and a registry entry, and this module does not change. Checked against the rule registry it replaces, over the existing Hypothesis strategies: identical problems on valid documents, and identical verdicts everywhere. The two differ only in how much they report about an already-invalid entity -- the old layer judged the values of members it could still read beside one it could not. That precision is kept where it was load-bearing. An optional member that fails its type check falls back to absent, because a bad `index_location` says nothing about whether a shard's pipelines are well formed; a required one stops the entity, because there is no honest reading of a `blosc` whose level is a string. `Context.resolve` now gives the entity the last word via `accepts`. Folding finds a candidate -- every `r<N>` spelling is tabled under one invented identifier -- and the candidate says whether the name is really one of its own. Without that, a document could write the identifier itself. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
3900 lines. The rule engine existed to dispatch a rule to the entity it was about, gate it on whether that entity's metadata was readable, and propagate what each codec did to the array. Entities now own all three, so there is nothing left to dispatch: - `rules/_registry.py`, `_engine.py`, `_entity.py`: the registry, the `reads`/`reads_optional` gate, `blocking_problems`, `run_rules`. - `rules/_entities/`: nine modules of rules keyed by entity name. - `rules/_v3_array.py`: the fill-value table and the four document rules. - `rules/_storage_class.py`: three frozensets of data type names and a hand-rolled recursion for `struct`. - `rules/_spec.py`, `_chunk_grid.py`, `_pipeline.py`: moved to `v3/_parts.py` and `v3/_chain.py`, minus the name lookups. - `v3/_shape.py`: four tables of per-member type checks, and the six functions that read them. The two remaining cross-field checks that belong to no entity -- v2's `chunks` against `shape`, and a group's inline consolidated children -- are now plain functions. `Rule`, `RuleCheck`, `run_rules`, `applicable` and the six rule-set constants leave the public API with them. Canonicalization collapses to asking each entity for its own `to_json`, plus the one field no entity owns. Three test modules go with the machinery they tested. The property tests over valid documents and corrupted chains stay, and they were what established that the replacement agrees. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found by adversarial review. An extra member, a `configuration` that is not an object, and a `must_understand` that is not a boolean are all rejected at the top level and were all *accepted* one level in -- inside a shard's pipelines, a struct field's data type, a cast_value target. The canonicalizer then deleted them, so an invalid document came back valid and smaller. The old shape layer checked the envelope wherever it ran, including inside a shard; `named_configuration` silently tolerates all three. The fix is to reuse the model layer's own `validate_metadata_field_v3` in `Context.coerce`, because a metadata field is a metadata field wherever it appears. The document's own fields pass `envelope_judged=True`, the structural layer having already said so. The differential over 2267 documents missed this: `st.from_type` honours the TypedDicts, so it cannot emit a malformed envelope. Two more from the same review. `ShardingIndexedCodec.problems()` was the one site composing a location by hand instead of with `within`, so a codec inside a shard reported at a path that does not exist in the document. And `CORE` and `CORE_AND_EXTENSIONS` shared one dict for `chunk_key_encoding` -- registering there would have reached both, and the subset test was vacuous for that field. A test now walks every reported location into the document. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four more from the adversarial review.
`None` meant absent in the entity layer, which contradicts the package's
own invariant that `None` is a JSON `null` the document wrote. It cost a
real distinction: `scale_offset` with `{"offset": null}` canonicalized to
the bare name, silently erasing what was written. Optional members now
default to `UNSET`, as the model layer's do, and `null` survives -- and
is reported, because no data type admits it as a scalar.
A stray configuration key on an `r<N>` data type returned no entity, so
its fill value went unjudged and the document looked valid to anyone
taking the documented tolerant reading (collect problems, filter
`unknown_key`). An unknown key is survivable here as everywhere else.
`blosc` rejected a `typesize` below 1 even under `noshuffle`, where the
spec says the value is ignored and `configuration` deletes it -- so
`to_json` turned an invalid codec into a valid document.
`canonicalize_array_metadata_v3` was not idempotent on `json.loads`
output: the per-field simplifications test for `tuple`, and the
validator was normalizing on a copy the canonicalizer never saw.
And a member that cannot be read no longer hides the values of the ones
beside it. The entity is built from what did read, its value problems
are reported, and anything the defaults would say about an unreadable
member is dropped. Measured against the rule registry over one shared
corpus of 800 documents: no verdict differs in the laxer direction, the
400 valid documents are identical, and documents losing a report fall
from 98 to 34 -- the residue being composition judgments, which need an
entity that could not be built.
Assisted-by: ClaudeCode:claude-opus-5
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- `member_types`'s base default was one shared mutable dict; every configuration-free entity had the same object. Now a read-only mapping. - An `unknown_key` problem named the key only in prose, so two of them on one configuration were indistinguishable by location. The location now names it, as the model layer's already did. - Float fill values were unbounded while integers were range-checked, so `float32` accepted `1e39` and stored an infinity. Each width states the largest magnitude it holds; float64 states none, because a Python float is one. - `storage_transformers` was a registerable extension point the document layer never resolved. Nothing models one yet, so nothing is judged there today -- but a registered one is now reached. - The `r<N>` grammar used `\d`, which is every Unicode decimal, so `r<FULLWIDTH ONE><FULLWIDTH SIX>` was read as sixteen bits and a third-party name spelled that way was folded into the family. ASCII only; such a name is now left unjudged, as any unmodelled name is. - `RECTILINEAR_CHUNK_GRID_KIND` is exported, as its siblings are. - Three docstrings that had stopped being true: `Coerced` claimed an entity and problems never come back together (a survivable problem does exactly that), `incoming_problems` misstated where its locations are relative to, and the no-slots comment overstated the breakage and did not say CPython fixed it in 3.13. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three independent reviews reached the same conclusion from different directions: the entity layer answers the questions a reader has and the questions an extension author has, and none of it was reachable. Thirty of the thirty-four names an extension needs were private, and the validators took no scope, so the only way to have a third-party entity judged was to mutate a module-level dict at import time. `zarr_metadata.v3.entity` re-exports the layer: the four base classes, `Context` and the two scopes, `ArrayParts`/`ChunkGrid`, the member checks an entity's `member_types` is built from, and the data-type families. No code moved; this is the door, not a new room. `validate_array_metadata_v3`, `parse_array_metadata_v3`, `validate_group_metadata_v3`, `parse_group_metadata_v3` and `canonicalize_array_metadata_v3` take `context=`, defaulting to what this package models. A group carries it down into its consolidated children. Two guards turn the silent failures the review found into errors where the mistake is: - a subclass that does not declare `identifier` (or `kind`, or `scalar_storage`) type-checks cleanly and then raises `AttributeError` from whichever method runs first. `__init_subclass__` says so at import. `base=True` opts out the classes that exist to add a class variable rather than to be an entity. - an optional member with a default other than `UNSET` is emitted for every instance, so the bare-name spelling becomes unreachable and a canonicalized document gains a member nobody wrote. Also refused at import -- the test written for this commit walked straight into it. - a registry key that is not the entity's `identifier` can never resolve, so the entity is registered, validation runs, and the verdict is clean. `Context` refuses to be built that way. `tests/v3/test_extension_api.py` is the proof: a third-party codec and data type, defined and registered and judged, importing nothing private. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Includes the mypy assignability question, which was investigated and closed rather than fixed: an entity's `to_json` returns its own object TypedDict, mypy will not accept that where the envelope type is wanted, and every widening that satisfies mypy costs more than it buys -- `object` stops the package's `st.from_type` strategies terminating, and `Any` stops the corrupting strategy generating the ill-typed members it exists to generate, which its own reach test caught. The narrow type is also the true one. `zarr_metadata.v3.entity` documents the `cast`. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…mpromise The previous wording justified keeping `Mapping[str, JSONValue]` by what widening did to this package's Hypothesis strategies. That is backwards: a test utility's behaviour is a fact about the utility, not an argument about a type. The real reason is that the conversion mypy refuses is sound here. Mypy's rule exists because an ordinary TypedDict may carry extra items of undeclared types, so the union of declared value types does not bound the mapping. Every TypedDict in this package is `closed` (PEP 728), which forbids that; pyright implements PEP 728 and accepts the assignment, and mypy has not implemented it yet (python/mypy#8994, python/mypy#18439). So the annotation is accurate and stays accurate, one checker needs a `cast` until it catches up, and widening would buy mypy's silence by making the type say something false. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`just typecheck` pins pyright to 1.1.404 to match CI; a newer one does not report `reportUnusedClass` here, which is why this passed locally and failed on the runner. Both classes exist to be refused while they are being created, so neither is ever bound and neither can be used. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…der's door `MetadataEntity | object` was fifteen annotations that said something to a reader and nothing to a checker: `X | object` is `object`. Consumers paid two isinstance calls per field, the first useless, and could not distinguish an extension this package does not model from one that was modelled and refused -- both arrived as the raw value. `Opaque` carries the JSON and says which it is. Every extension point is now an exhaustive two-case union naming its own kind, and `Context.coerce` is overloaded on the extension point so it returns that kind rather than the base. The extension-point constants lose their widening annotation to keep their `Literal` types, which is what makes the overloads fire at call sites written with them. `ArrayDocumentV3.from_json` is the fail-fast reader zarrs has and this did not: construct or raise, with every problem on the exception. An out-of-scope name does not raise -- refusing it would make extension openness unimplementable -- so what fails is metadata that is wrong, not metadata that is unfamiliar. The judgment moves onto `ArrayDocumentV3.problems()`, so a document is read once whichever door you come in by. Verdicts are unchanged: the 800-document differential against the rule registry reports exactly as before. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
d-v-b
force-pushed
the
zarr-metadata/rules-wrap-blosc
branch
from
September 20, 2026 17:31
fde974f to
9921b88
Compare
`to_json` was documented as "the simplest equivalent spelling" and two
entities disagreed about whether they meant it: `r008` came back as
`r008`, while `[32, 32, 32]` came back run-length encoded. So reading a
document and writing it rewrote it -- the same complaint the review made
about the model layer, reproduced here, on the `from_json` just added as
the reader's front door.
They are different operations and now have different methods.
`to_json` is faithful to every member the entity holds. `canonical()`
returns the simplest equivalent, and `canonicalize_array_metadata_v3` is
the only caller -- canonicalizing is something you ask for, not something
serialization does to you on the way past. Rectilinear's run-length
encoding and blosc's ignored `typesize` move there, and the three
entities that contain others canonicalize what they contain.
The envelope's spelling is still normalized, because the entity does not
model it: a bare name, `{"name": x}` and `{"name": x, "configuration":
{}}` all read to the same entity, so all three write back as the bare
name. Worth knowing, because zarrs deliberately keeps the object form for
readers older than Zarr 3.1.
`RectilinearChunkGrid.configuration` is gone with it. It called `super()`
to build a member it then discarded, which is what prompted the question.
Assisted-by: ClaudeCode:claude-opus-5
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`out_of_range: "wrap"` is defined only for integral targets with a two's complement representation. A modelled non-integer target -- `bool` included, which is not two's complement -- is rejected; a target out of scope declines, since it may be an integral extension type. Re-landed on the entity layer, which changes where the fact lives. The original carried two frozensets of data type names plus a registry test asserting they partition the modelled types, because a type in neither would be silently accepted. Now each data type states `twos_complement` about itself, and `DataTypeEntity` requires it: a data type that does not decide fails at import, so there is nothing for a test to keep in step. Entities also gain `name`, the spelling a document writes, distinct from `identifier`, the key they are tabled under. They differ only for the raw-bytes family, whose identifier is invented -- and which was leaking into this rule's message as `got 'r<N>'` for a document that said `r24`. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
d-v-b
force-pushed
the
zarr-metadata/rules-wrap-blosc
branch
from
September 20, 2026 18:52
9921b88 to
1c3cf93
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Depends on #4379, and stacked on it: this targets
main, so its diff includes #4379's commits. Review only the last commit, or the diff against #4379's branch: d-v-b/zarr-python@zarr-metadata/rules-layer...zarr-metadata/rules-wrap-bloscWhat this adds
cast_value'sout_of_range: "wrap"is rejected for targets it is not defined on.The spec permits wrapping only for integral targets with a two's complement representation. A modelled non-integer target reports
invalid_valueatout_of_range—boolincluded, since it is not two's complement. A target this reader does not model declines, because it may be an integral extension type:Where the fact lives
Whether a type wraps is something the data type says about itself,
twos_complement, andDataTypeEntityrequires it. A data type that does not decide fails at import:Required rather than defaulted on purpose: either default answers for a new data type silently, and one of them accepts a cast the spec does not define.
That replaces what this rule carried before #4379 was rebuilt — two frozensets of data type names, plus a registry test asserting they partition every modelled type so that none could be accepted by omission. There is no longer a table to partition, so there is nothing for a test to keep in step.
Also here
Entities gain
name, the spelling a document writes, as distinct fromidentifier, the key they are tabled under. The two differ only for the raw-bytes family, whose identifier is invented and belongs in no message a reader sees — this rule was reportinggot 'r<N>'for a document that saidr24.🤖 Generated with Claude Code