fix(zarr-metadata): reject cast_value wrap on non-integer targets

packages/zarr-metadata/README.md

@@ -6,7 +6,7 @@ Documentation: <https://zarr-metadata.readthedocs.io/>
 
 ## What this is
 
-Two layers and an optional integration:
+Three layers and an optional integration:
 
 - **Typed JSON shapes**: `TypedDict` definitions and `Literal` aliases for the
   JSON documents specified by the [Zarr v2](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html)
@@ -17,28 +17,34 @@ Two layers and an optional integration:
   models of whole metadata documents, with structural validators, loc-aware
   parsers, and store-key (de)serialization. A document produced by `to_json`
   shares no mutable state with the model that produced it.
+- **Composition rules** (`zarr_metadata.rules`): cross-field judgments over
+  whole documents — fill value against data type, codec pipeline ordering,
+  chunk geometry — with `validate_*` / `parse_*` entry points that apply
+  structure and composition together.
 - **Optional Pydantic integration** (`zarr_metadata.pydantic`, requires
-  Pydantic 2.13 or newer): each model as a Pydantic field type that validates
-  raw documents through the same strict parser.
+  Pydantic 2.13 or newer): each model as a Pydantic field type that runs raw
+  documents through the rules layer.
 
 ## What this is for
 
 The public `TypedDict` definitions describe the static JSON shape of Zarr
-metadata. For strict, loc-aware validation of JSON loaded from disk, use the
-model parser:
+metadata. To judge JSON loaded from disk, structure and composition together,
+use the rules layer; to get a normalized document model, use the model parser:
 
 ```python
 import json
 from zarr_metadata.model import ZarrV3ArrayMetadata
+from zarr_metadata.rules import parse_array_metadata_v3
 
 with open("zarr.json", "rb") as f:
     raw = json.load(f)
 
-metadata = ZarrV3ArrayMetadata.from_json(raw)
+document = parse_array_metadata_v3(raw)  # raises with every problem found
+metadata = ZarrV3ArrayMetadata.from_json(document)
 ```
 
-The optional Pydantic integration delegates raw input to the same strict
-parser and returns the same normalized model class:
+The optional Pydantic integration runs raw input through the rules layer
+and returns the same normalized model class:
 
 ```python
 from pydantic import TypeAdapter
@@ -56,19 +62,44 @@ members that the strict model parser rejects.
 
 The model validators enforce the declared document structure and a small set
 of context-free consistency rules, including fixed format literals, finite
-JSON numbers, non-negative dimensions, non-empty v3 codec pipelines, and one
-`dimension_names` entry per array dimension. They do not interpret extension
-names or configurations, resolve codec pipelines, or decide whether a data
-type, chunk grid, codec, or storage transformer is supported. Those decisions
-belong to consumer implementations.
+JSON numbers, non-negative dimensions, and non-empty v3 codec pipelines.
+They do not interpret extension names or configurations.
+
+The composition rules (`zarr_metadata.rules`) judge the document as a whole:
+fill values against data types, codec pipeline ordering, chunk-grid
+geometry against `shape`, one `dimension_names` entry per array dimension,
+and the canonical configuration shapes of the codecs, chunk grids, chunk
+key encodings, and data types this package defines. Unknown extension names
+are left unjudged. The rules model canonical documents and are deliberately
+stricter than any given implementation: an implementation may coerce
+ambiguous input as it sees fit and then validate the canonical result.
+Nothing here decides whether a data type, chunk grid, codec, or storage
+transformer is *supported*; that belongs to consumer implementations.
+
+An unmodelled member inside a *known* entity's `configuration` is an error
+under that strict reading: such a member is almost always a typo or a
+setting meant for a different entity, and accepting it silently means
+silently ignoring what the writer asked for. It carries its own
+`unknown_key` problem kind, so a consumer who wants the tolerant reading
+can collect problems with `validate_*` and filter that kind out.
+
+Judging it is the rules layer's job, so it is `zarr_metadata.rules` and
+the whole-document Pydantic field types (which run the rules layer) that
+reject it. The model layer never interpreted entity configurations and
+still does not, so `model.parse_*` and `from_json` accept such a
+document; so does the bare `ZarrV3MetadataField` Pydantic type, which
+judges one metadata field and carries no composition rules.
 
 The Pydantic integration's generated JSON Schemas express independently
 checkable document structure and field constraints, but they are not a
 replacement for runtime model validation. Standard JSON Schema treats a
 mathematically integral number such as `1.0` as an integer, while the runtime
 boundary requires Python `int` values, and it cannot express arbitrary
 same-length relations such as `dimension_names` versus `shape` or v2 `chunks`
-versus `shape`. Consumers should run the model parser after schema validation.
+versus `shape`. The generated schemas also leave every `configuration`
+open, so a schema-valid document can still be rejected at runtime for an
+unmodelled configuration member. Consumers should run the runtime
+validators after schema validation.
 
 ## Scope
 

packages/zarr-metadata/changes/4379.bugfix.1.md

@@ -0,0 +1,15 @@
+Attributes are user data, and a non-finite number in them is read,
+validated and written back. zarr-python writes attributes with the
+defaults of Python's `json` module, so an xarray `_FillValue` or a CF
+`missing_value` of NaN is stored as a bare `NaN`, which RFC 8259 lacks.
+This package refused such a document at every layer, and could not read
+the store at all. A node's `attributes`, a v2 `.zattrs`, and the
+attributes of the nodes an inline `consolidated_metadata` holds may now
+hold `NaN`, `Infinity` and `-Infinity`, in every `validate_*`,
+`parse_*` and `is_*`, and in `from_key_value` and `to_key_value`.
+
+Wherever the spec interprets a value it spells those numbers as strings,
+so anywhere else a non-finite number is still refused. The store reader
+now says where one is (`fill_value: non-finite float nan is not JSON`)
+instead of failing to decode the document, and the writer still refuses
+to write one, since a model built by hand is not validated.

packages/zarr-metadata/changes/4379.bugfix.2.md

@@ -0,0 +1,10 @@
+The `scale_offset` codec's `offset` and `scale` are judged. The registry
+says each is "encoded to JSON using the Zarr V3 fill value encoding for
+the input array's data type", so each is now a fill value of the data type
+that reaches the codec, which after a `cast_value` is the type cast to:
+the string `"0"` is neither a `float32` nor an `int32`, and `-1` is no
+`uint8`. Any JSON value was accepted before. The codec is defined for data
+types with arithmetic, and the registry lists the integer and
+floating-point ones, so on a `bool` or `complex64` array it is refused at
+the codec. Over the 40,000-document corpus, 19 documents that were valid
+are not, each for a scalar no data type could hold (an object as `scale`).

packages/zarr-metadata/changes/4379.bugfix.md

@@ -0,0 +1,57 @@
+Five gaps in the composition rules, found by adversarial review of the
+rules layer:
+
+- A modelled `array -> array` codec with no registered spec transition
+  stopped spec propagation, standing down every rule after it. The
+  `scale_offset` codec had none, so inserting a no-op `scale_offset`
+  switched off the `bytes` codec's endianness requirement and every
+  shard's geometry check. It now registers the identity transition the
+  spec describes, and no codec is exempt from having one.
+- A chunk grid whose extents this package cannot read (a rectilinear
+  grid, or a regular grid with a non-positive extent) discarded the
+  *rank* along with the extents, so a rank-mismatched `transpose` order
+  or inner chunk shape went unreported. Every chunk of an array has the
+  array's rank whatever the grid, so the rank is kept and only the
+  geometry checks stand down.
+- One unusable configuration member suppressed every judgment about the
+  entities nested inside the same entity — a misspelled `index_location`
+  hid the problems in a shard's inner pipelines. A nested entity is now
+  read whatever else was found, so its problems are reported alongside.
+- Problems about an entity as a whole pointed at a `configuration` node
+  that a bare-string entity does not have (`("codecs", 0,
+  "configuration")` for `"codecs": ["bytes"]`). They now point at the
+  entity.
+- The endianness message named no data type, so inside a shard's
+  `index_codecs` it appeared to be about the array's own type rather
+  than the shard index's `uint64`. It names the type.
+- A malformed `must_understand` — part of an entity's envelope, not of
+  its configuration — counted as "the configuration is unreadable" and
+  stood down every judgment for that entity. It no longer does: it is
+  not a configuration member, so it cannot make one unreadable.
+
+The three derivations of "what array does this pipeline encode?" are now
+one abstraction, `v3._parts`, which answers per dimension rather
+than per grid, and plurally: what a codec is handed is `ArrayParts`, every
+part of an array the pipeline will encode, carrying the chunk grid itself
+rather than a summary of one. Every chain rule is a statement about all of
+those parts — a shard's inner chunk shape must divide *every* chunk it
+will see, which under a rectilinear grid is several different lengths, and
+a transpose can move a varying axis into the position the shard has to
+divide. Following zarrs, a chunk grid is read from its metadata
+*and* the array shape it partitions, its rank is always available, and its
+extents are reported dimension by dimension. That resolves two more cases
+the ad-hoc derivations could not express: a rectilinear grid whose chunk
+shapes are uniform now pins the shard shape (so a shard that does not
+divide it is reported, in all three spellings the spec allows), while a
+grid uniform on one axis only is judged on that axis and declines on the
+other; and a shard index is judged against its own derived shape —
+chunks-per-shard plus a trailing dimension of 2 — rather than against no
+shape at all.
+
+An unmodelled codec in front of a sharding codec — every `numcodecs.*`
+filter is one — silenced the whole shard interior: its inner pipeline and
+its index pipeline went unjudged. Both are determined by the sharding
+codec's own `chunk_shape` and by the spec (the index is `uint64`), so
+neither waits on what reached the codec. Likewise an unusable `data_type`
+no longer hides the geometry, and a chunk grid declines to permute by
+anything that is not a permutation rather than raising `IndexError`.

packages/zarr-metadata/changes/4379.feature.10.md

@@ -0,0 +1,50 @@
+An entity is the value guarantee, not just the type one. Construction
+refuses values the spec disallows, so `BloscCodec(BloscOptions(clevel=99))`
+raises rather than serializing a document no reader will accept, and
+there is no such thing as an invalid entity to hold. Code that wants
+unvalidated metadata already has somewhere to put it: `Opaque`, which
+carries the JSON verbatim, and the configuration record itself, which is
+the options as written and not yet judged.
+
+One piece: the configuration record's own `problems`, which yields the
+values the spec disallows as it finds them. The entity's constructor
+stops at the first and raises `MetadataValidationError`; `coerce` asks
+the record before it builds anything and reports every problem in the
+document instead of raising; a reader with a record asks it too,
+`BloscOptions(...).problems()`, and stops or collects as it likes.
+
+`must_understand` moves to a class variable and out of the configuration
+entirely. It is a property of the *kind* of metadata -- a codec is
+something you must understand every time it appears, consolidated
+metadata is unconditionally skippable -- not a per-occurrence choice. The
+spec permits `must_understand: false` on a codec; this package reads that
+as an oversight and refuses it at every extension point, while leaving it
+where it earns its keep: an unknown top-level extension field a reader
+really can skip.
+
+The cost is reporting density on documents that are already invalid. An
+entity that cannot be built cannot be asked anything, so a grid with one
+bad extent no longer pins its good axes and a shard with one bad inner
+extent no longer judges its pipelines. The report that survives is the one
+that has to be fixed first, and the JSON is still there on the `Opaque`
+standing in for the entity.
+
+Measured against the rule registry this layer replaced, over a shared
+corpus of 40,000 documents through all four `validate_*` entry points:
+**nothing this layer accepts was rejected before**. Twenty-one verdicts
+change, all the other way: two are the `must_understand: false` refusal
+above -- one at the top level, one nested in a shard's pipelines -- and
+nineteen are `scale_offset` scalars no data type could hold. Every
+document valid under both reports identically. Counted per entry point,
+among those invalid under both, 6,382 report fewer problems -- a
+malformed envelope is reported once, and a document with no codec list
+is no longer also told it lacks an `array -> bytes` codec -- and 655
+report more, because a problem that used to stand down the rest of an
+entity no longer does.
+
+Some ways of writing an entity type-check cleanly and then fail
+somewhere that will not name the class, so registration refuses them: a
+field whose annotation is not a shape JSON takes, a `__post_init__` of
+the entity's own, and a class variable the entity owes and did not
+declare (read off the annotations, so a family adding one cannot forget
+to require it).

packages/zarr-metadata/changes/4379.feature.11.md

@@ -0,0 +1,28 @@
+Reading a v3 array document is three layers, ordered by what each needs,
+each handing the next a typed value and its problems. `well_formed_array_v3(value)`
+needs only the value: the JSON is refined -- arrays as tuples, string
+keys, finite floats -- and the document's shape judged by the model layer;
+a value that is not JSON gets that verdict and nothing else is judged.
+`read_array_v3(document, context)` needs a scope: each extension point's
+name is related to a class through the context and the class is handed
+the field. `refine_array_v3(array)` needs the array: the fill value
+against its type, the grid against the shape, and the codec pipeline
+walked with what reaches each codec.
+
+The third layer's value is the resolved pipeline. `RefinedArrayV3`
+carries the `ArrayParts` the pipeline is handed and a `Pipeline` of
+`PipelineStage`s, each a codec and the array that reaches it -- None past
+the array->bytes boundary or after a codec that could not say what it
+does -- with a shard's `codecs` and `index_codecs` refined inside its
+stage. This is what a codec pipeline is built from, and what zarr-python
+computes separately today when it evolves an array spec codec by codec;
+validating the composition is what the walk finds on the way. A codec
+that holds pipelines declares them through `inner_pipelines(incoming)`,
+each by the member that holds it with the parts it is handed, and
+judges nothing inside them itself.
+
+`resolve(data, kind, context)` is the first two layers for a field on
+its own. Nothing downstream of the first layer normalizes or checks
+JSON-ness again, so the reader no longer carries an `envelope_judged`
+flag or re-normalizes arrays inside `coerce`; `chain_problems` is gone
+from the door, replaced by the refinement that produces the pipeline.

packages/zarr-metadata/changes/4379.feature.12.md

@@ -0,0 +1,7 @@
+`ArrayDocumentV3.must_understand_fields` names the top-level fields
+outside the spec's that do not say `must_understand: false`, as
+`ZarrV3ArrayMetadata.must_understand_fields` does for the model. The spec
+has a reader refuse to open an array carrying such a field unless it
+recognizes it, and recognition is the reader's own knowledge, so the
+document keeps the field, writes it back, and leaves the refusal to its
+reader: zarr-python recognizes none.

packages/zarr-metadata/changes/4379.feature.6.md

@@ -0,0 +1,25 @@
+Unknown members inside a *known* entity's `configuration` (e.g. an extra
+key in a `blosc` configuration) now report as their own `unknown_key`
+problem kind, and no longer suppress the other judgments about that
+entity.
+
+Whether configurations are closed remains unspecified
+([zarr-specs#270](https://github.com/zarr-developers/zarr-specs/issues/270)),
+and this package retains its strict reading deliberately: in practice an
+unmodelled member is a typo or a setting meant for a different entity, and
+accepting it silently means silently ignoring what the writer asked for.
+Judging it is the semantic layer's job: `rules.parse_*` and the
+whole-document pydantic field types reject it, while `model.parse_*` and
+the bare `ZarrV3MetadataField` type — neither of which interprets entity
+configurations — accept it. The dedicated kind exists so that a consumer
+who wants the tolerant reading can collect problems with `validate_*` and
+filter, and so that an unknown key never masks the other findings about
+its entity.
+
+Model round-trips preserve unmodeled members. Shape-exact `TypeIs` guards
+still reject them because the corresponding TypedDicts are closed.
+
+An unknown member is reported, not held, so an entity written back does
+not carry it. Only a caller who took the problems as data and continued
+past that one can reach that: `from_json` raises on it, and the JSON the
+caller passed in is still the JSON it passed in.