diff --git a/packages/zarr-metadata/README.md b/packages/zarr-metadata/README.md index 6b6b172aec..91530e644a 100644 --- a/packages/zarr-metadata/README.md +++ b/packages/zarr-metadata/README.md @@ -6,7 +6,7 @@ Documentation: ## What this is -Two layers and an optional integration: +Three layers and an optional integration: - **Typed JSON shapes**: `TypedDict` definitions and `Literal` aliases for the JSON documents specified by the [Zarr v2](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html) @@ -17,28 +17,34 @@ Two layers and an optional integration: models of whole metadata documents, with structural validators, loc-aware parsers, and store-key (de)serialization. A document produced by `to_json` shares no mutable state with the model that produced it. +- **Composition rules** (`zarr_metadata.rules`): cross-field judgments over + whole documents — fill value against data type, codec pipeline ordering, + chunk geometry — with `validate_*` / `parse_*` entry points that apply + structure and composition together. - **Optional Pydantic integration** (`zarr_metadata.pydantic`, requires - Pydantic 2.13 or newer): each model as a Pydantic field type that validates - raw documents through the same strict parser. + Pydantic 2.13 or newer): each model as a Pydantic field type that runs raw + documents through the rules layer. ## What this is for The public `TypedDict` definitions describe the static JSON shape of Zarr -metadata. For strict, loc-aware validation of JSON loaded from disk, use the -model parser: +metadata. To judge JSON loaded from disk, structure and composition together, +use the rules layer; to get a normalized document model, use the model parser: ```python import json from zarr_metadata.model import ZarrV3ArrayMetadata +from zarr_metadata.rules import parse_array_metadata_v3 with open("zarr.json", "rb") as f: raw = json.load(f) -metadata = ZarrV3ArrayMetadata.from_json(raw) +document = parse_array_metadata_v3(raw) # raises with every problem found +metadata = ZarrV3ArrayMetadata.from_json(document) ``` -The optional Pydantic integration delegates raw input to the same strict -parser and returns the same normalized model class: +The optional Pydantic integration runs raw input through the rules layer +and returns the same normalized model class: ```python from pydantic import TypeAdapter @@ -56,11 +62,33 @@ members that the strict model parser rejects. The model validators enforce the declared document structure and a small set of context-free consistency rules, including fixed format literals, finite -JSON numbers, non-negative dimensions, non-empty v3 codec pipelines, and one -`dimension_names` entry per array dimension. They do not interpret extension -names or configurations, resolve codec pipelines, or decide whether a data -type, chunk grid, codec, or storage transformer is supported. Those decisions -belong to consumer implementations. +JSON numbers, non-negative dimensions, and non-empty v3 codec pipelines. +They do not interpret extension names or configurations. + +The composition rules (`zarr_metadata.rules`) judge the document as a whole: +fill values against data types, codec pipeline ordering, chunk-grid +geometry against `shape`, one `dimension_names` entry per array dimension, +and the canonical configuration shapes of the codecs, chunk grids, chunk +key encodings, and data types this package defines. Unknown extension names +are left unjudged. The rules model canonical documents and are deliberately +stricter than any given implementation: an implementation may coerce +ambiguous input as it sees fit and then validate the canonical result. +Nothing here decides whether a data type, chunk grid, codec, or storage +transformer is *supported*; that belongs to consumer implementations. + +An unmodelled member inside a *known* entity's `configuration` is an error +under that strict reading: such a member is almost always a typo or a +setting meant for a different entity, and accepting it silently means +silently ignoring what the writer asked for. It carries its own +`unknown_key` problem kind, so a consumer who wants the tolerant reading +can collect problems with `validate_*` and filter that kind out. + +Judging it is the rules layer's job, so it is `zarr_metadata.rules` and +the whole-document Pydantic field types (which run the rules layer) that +reject it. The model layer never interpreted entity configurations and +still does not, so `model.parse_*` and `from_json` accept such a +document; so does the bare `ZarrV3MetadataField` Pydantic type, which +judges one metadata field and carries no composition rules. The Pydantic integration's generated JSON Schemas express independently checkable document structure and field constraints, but they are not a @@ -68,7 +96,10 @@ replacement for runtime model validation. Standard JSON Schema treats a mathematically integral number such as `1.0` as an integer, while the runtime boundary requires Python `int` values, and it cannot express arbitrary same-length relations such as `dimension_names` versus `shape` or v2 `chunks` -versus `shape`. Consumers should run the model parser after schema validation. +versus `shape`. The generated schemas also leave every `configuration` +open, so a schema-valid document can still be rejected at runtime for an +unmodelled configuration member. Consumers should run the runtime +validators after schema validation. ## Scope diff --git a/packages/zarr-metadata/changes/4379.bugfix.1.md b/packages/zarr-metadata/changes/4379.bugfix.1.md new file mode 100644 index 0000000000..ecc0d6b9d7 --- /dev/null +++ b/packages/zarr-metadata/changes/4379.bugfix.1.md @@ -0,0 +1,15 @@ +Attributes are user data, and a non-finite number in them is read, +validated and written back. zarr-python writes attributes with the +defaults of Python's `json` module, so an xarray `_FillValue` or a CF +`missing_value` of NaN is stored as a bare `NaN`, which RFC 8259 lacks. +This package refused such a document at every layer, and could not read +the store at all. A node's `attributes`, a v2 `.zattrs`, and the +attributes of the nodes an inline `consolidated_metadata` holds may now +hold `NaN`, `Infinity` and `-Infinity`, in every `validate_*`, +`parse_*` and `is_*`, and in `from_key_value` and `to_key_value`. + +Wherever the spec interprets a value it spells those numbers as strings, +so anywhere else a non-finite number is still refused. The store reader +now says where one is (`fill_value: non-finite float nan is not JSON`) +instead of failing to decode the document, and the writer still refuses +to write one, since a model built by hand is not validated. diff --git a/packages/zarr-metadata/changes/4379.bugfix.2.md b/packages/zarr-metadata/changes/4379.bugfix.2.md new file mode 100644 index 0000000000..c7d0343fb3 --- /dev/null +++ b/packages/zarr-metadata/changes/4379.bugfix.2.md @@ -0,0 +1,10 @@ +The `scale_offset` codec's `offset` and `scale` are judged. The registry +says each is "encoded to JSON using the Zarr V3 fill value encoding for +the input array's data type", so each is now a fill value of the data type +that reaches the codec, which after a `cast_value` is the type cast to: +the string `"0"` is neither a `float32` nor an `int32`, and `-1` is no +`uint8`. Any JSON value was accepted before. The codec is defined for data +types with arithmetic, and the registry lists the integer and +floating-point ones, so on a `bool` or `complex64` array it is refused at +the codec. Over the 40,000-document corpus, 19 documents that were valid +are not, each for a scalar no data type could hold (an object as `scale`). diff --git a/packages/zarr-metadata/changes/4379.bugfix.md b/packages/zarr-metadata/changes/4379.bugfix.md new file mode 100644 index 0000000000..b302942838 --- /dev/null +++ b/packages/zarr-metadata/changes/4379.bugfix.md @@ -0,0 +1,57 @@ +Five gaps in the composition rules, found by adversarial review of the +rules layer: + +- A modelled `array -> array` codec with no registered spec transition + stopped spec propagation, standing down every rule after it. The + `scale_offset` codec had none, so inserting a no-op `scale_offset` + switched off the `bytes` codec's endianness requirement and every + shard's geometry check. It now registers the identity transition the + spec describes, and no codec is exempt from having one. +- A chunk grid whose extents this package cannot read (a rectilinear + grid, or a regular grid with a non-positive extent) discarded the + *rank* along with the extents, so a rank-mismatched `transpose` order + or inner chunk shape went unreported. Every chunk of an array has the + array's rank whatever the grid, so the rank is kept and only the + geometry checks stand down. +- One unusable configuration member suppressed every judgment about the + entities nested inside the same entity — a misspelled `index_location` + hid the problems in a shard's inner pipelines. A nested entity is now + read whatever else was found, so its problems are reported alongside. +- Problems about an entity as a whole pointed at a `configuration` node + that a bare-string entity does not have (`("codecs", 0, + "configuration")` for `"codecs": ["bytes"]`). They now point at the + entity. +- The endianness message named no data type, so inside a shard's + `index_codecs` it appeared to be about the array's own type rather + than the shard index's `uint64`. It names the type. +- A malformed `must_understand` — part of an entity's envelope, not of + its configuration — counted as "the configuration is unreadable" and + stood down every judgment for that entity. It no longer does: it is + not a configuration member, so it cannot make one unreadable. + +The three derivations of "what array does this pipeline encode?" are now +one abstraction, `v3._parts`, which answers per dimension rather +than per grid, and plurally: what a codec is handed is `ArrayParts`, every +part of an array the pipeline will encode, carrying the chunk grid itself +rather than a summary of one. Every chain rule is a statement about all of +those parts — a shard's inner chunk shape must divide *every* chunk it +will see, which under a rectilinear grid is several different lengths, and +a transpose can move a varying axis into the position the shard has to +divide. Following zarrs, a chunk grid is read from its metadata +*and* the array shape it partitions, its rank is always available, and its +extents are reported dimension by dimension. That resolves two more cases +the ad-hoc derivations could not express: a rectilinear grid whose chunk +shapes are uniform now pins the shard shape (so a shard that does not +divide it is reported, in all three spellings the spec allows), while a +grid uniform on one axis only is judged on that axis and declines on the +other; and a shard index is judged against its own derived shape — +chunks-per-shard plus a trailing dimension of 2 — rather than against no +shape at all. + +An unmodelled codec in front of a sharding codec — every `numcodecs.*` +filter is one — silenced the whole shard interior: its inner pipeline and +its index pipeline went unjudged. Both are determined by the sharding +codec's own `chunk_shape` and by the spec (the index is `uint64`), so +neither waits on what reached the codec. Likewise an unusable `data_type` +no longer hides the geometry, and a chunk grid declines to permute by +anything that is not a permutation rather than raising `IndexError`. diff --git a/packages/zarr-metadata/changes/4379.feature.10.md b/packages/zarr-metadata/changes/4379.feature.10.md new file mode 100644 index 0000000000..a25f5fdc87 --- /dev/null +++ b/packages/zarr-metadata/changes/4379.feature.10.md @@ -0,0 +1,50 @@ +An entity is the value guarantee, not just the type one. Construction +refuses values the spec disallows, so `BloscCodec(BloscOptions(clevel=99))` +raises rather than serializing a document no reader will accept, and +there is no such thing as an invalid entity to hold. Code that wants +unvalidated metadata already has somewhere to put it: `Opaque`, which +carries the JSON verbatim, and the configuration record itself, which is +the options as written and not yet judged. + +One piece: the configuration record's own `problems`, which yields the +values the spec disallows as it finds them. The entity's constructor +stops at the first and raises `MetadataValidationError`; `coerce` asks +the record before it builds anything and reports every problem in the +document instead of raising; a reader with a record asks it too, +`BloscOptions(...).problems()`, and stops or collects as it likes. + +`must_understand` moves to a class variable and out of the configuration +entirely. It is a property of the *kind* of metadata -- a codec is +something you must understand every time it appears, consolidated +metadata is unconditionally skippable -- not a per-occurrence choice. The +spec permits `must_understand: false` on a codec; this package reads that +as an oversight and refuses it at every extension point, while leaving it +where it earns its keep: an unknown top-level extension field a reader +really can skip. + +The cost is reporting density on documents that are already invalid. An +entity that cannot be built cannot be asked anything, so a grid with one +bad extent no longer pins its good axes and a shard with one bad inner +extent no longer judges its pipelines. The report that survives is the one +that has to be fixed first, and the JSON is still there on the `Opaque` +standing in for the entity. + +Measured against the rule registry this layer replaced, over a shared +corpus of 40,000 documents through all four `validate_*` entry points: +**nothing this layer accepts was rejected before**. Twenty-one verdicts +change, all the other way: two are the `must_understand: false` refusal +above -- one at the top level, one nested in a shard's pipelines -- and +nineteen are `scale_offset` scalars no data type could hold. Every +document valid under both reports identically. Counted per entry point, +among those invalid under both, 6,382 report fewer problems -- a +malformed envelope is reported once, and a document with no codec list +is no longer also told it lacks an `array -> bytes` codec -- and 655 +report more, because a problem that used to stand down the rest of an +entity no longer does. + +Some ways of writing an entity type-check cleanly and then fail +somewhere that will not name the class, so registration refuses them: a +field whose annotation is not a shape JSON takes, a `__post_init__` of +the entity's own, and a class variable the entity owes and did not +declare (read off the annotations, so a family adding one cannot forget +to require it). diff --git a/packages/zarr-metadata/changes/4379.feature.11.md b/packages/zarr-metadata/changes/4379.feature.11.md new file mode 100644 index 0000000000..7368ca5731 --- /dev/null +++ b/packages/zarr-metadata/changes/4379.feature.11.md @@ -0,0 +1,28 @@ +Reading a v3 array document is three layers, ordered by what each needs, +each handing the next a typed value and its problems. `well_formed_array_v3(value)` +needs only the value: the JSON is refined -- arrays as tuples, string +keys, finite floats -- and the document's shape judged by the model layer; +a value that is not JSON gets that verdict and nothing else is judged. +`read_array_v3(document, context)` needs a scope: each extension point's +name is related to a class through the context and the class is handed +the field. `refine_array_v3(array)` needs the array: the fill value +against its type, the grid against the shape, and the codec pipeline +walked with what reaches each codec. + +The third layer's value is the resolved pipeline. `RefinedArrayV3` +carries the `ArrayParts` the pipeline is handed and a `Pipeline` of +`PipelineStage`s, each a codec and the array that reaches it -- None past +the array->bytes boundary or after a codec that could not say what it +does -- with a shard's `codecs` and `index_codecs` refined inside its +stage. This is what a codec pipeline is built from, and what zarr-python +computes separately today when it evolves an array spec codec by codec; +validating the composition is what the walk finds on the way. A codec +that holds pipelines declares them through `inner_pipelines(incoming)`, +each by the member that holds it with the parts it is handed, and +judges nothing inside them itself. + +`resolve(data, kind, context)` is the first two layers for a field on +its own. Nothing downstream of the first layer normalizes or checks +JSON-ness again, so the reader no longer carries an `envelope_judged` +flag or re-normalizes arrays inside `coerce`; `chain_problems` is gone +from the door, replaced by the refinement that produces the pipeline. diff --git a/packages/zarr-metadata/changes/4379.feature.12.md b/packages/zarr-metadata/changes/4379.feature.12.md new file mode 100644 index 0000000000..3c9094391e --- /dev/null +++ b/packages/zarr-metadata/changes/4379.feature.12.md @@ -0,0 +1,7 @@ +`ArrayDocumentV3.must_understand_fields` names the top-level fields +outside the spec's that do not say `must_understand: false`, as +`ZarrV3ArrayMetadata.must_understand_fields` does for the model. The spec +has a reader refuse to open an array carrying such a field unless it +recognizes it, and recognition is the reader's own knowledge, so the +document keeps the field, writes it back, and leaves the refusal to its +reader: zarr-python recognizes none. diff --git a/packages/zarr-metadata/changes/4379.feature.6.md b/packages/zarr-metadata/changes/4379.feature.6.md new file mode 100644 index 0000000000..d17f261545 --- /dev/null +++ b/packages/zarr-metadata/changes/4379.feature.6.md @@ -0,0 +1,25 @@ +Unknown members inside a *known* entity's `configuration` (e.g. an extra +key in a `blosc` configuration) now report as their own `unknown_key` +problem kind, and no longer suppress the other judgments about that +entity. + +Whether configurations are closed remains unspecified +([zarr-specs#270](https://github.com/zarr-developers/zarr-specs/issues/270)), +and this package retains its strict reading deliberately: in practice an +unmodelled member is a typo or a setting meant for a different entity, and +accepting it silently means silently ignoring what the writer asked for. +Judging it is the semantic layer's job: `rules.parse_*` and the +whole-document pydantic field types reject it, while `model.parse_*` and +the bare `ZarrV3MetadataField` type — neither of which interprets entity +configurations — accept it. The dedicated kind exists so that a consumer +who wants the tolerant reading can collect problems with `validate_*` and +filter, and so that an unknown key never masks the other findings about +its entity. + +Model round-trips preserve unmodeled members. Shape-exact `TypeIs` guards +still reject them because the corresponding TypedDicts are closed. + +An unknown member is reported, not held, so an entity written back does +not carry it. Only a caller who took the problems as data and continued +past that one can reach that: `from_json` raises on it, and the JSON the +caller passed in is still the JSON it passed in. diff --git a/packages/zarr-metadata/changes/4379.feature.7.md b/packages/zarr-metadata/changes/4379.feature.7.md new file mode 100644 index 0000000000..aedf54391e --- /dev/null +++ b/packages/zarr-metadata/changes/4379.feature.7.md @@ -0,0 +1,101 @@ +Three judgments rather than two: **type** (is this an integer?), +**value** (is it an integer in `[0, 9]`?), and **composition** (does this +codec's rank match the array that reached it?). Each one is answered by +the entity it is about, because all three are facts about that entity. + +So `blosc`'s `clevel` range, `gzip`'s and `zstd`'s `level` ranges, +`typesize` being positive, `blocksize` being non-negative, a `transpose` +`order` being a permutation of its own indices, a time data type's +`scale_factor` range, and `blosc`'s "typesize is required unless shuffle +is `noshuffle`" are all stated on the class that has those members. So +are the composition answers, each taking the part of the document it +needs. + +Three name-keyed tables disappear with them, replaced by a method on the +class that knew the answer all along: which storage class a data type +has (and how a `struct` folds its fields), what a codec does to the +array it receives, and what a chunk grid divides an array into. + +What a fill value may be is the data type's answer too, and the families +are where the widths live: every integer type differs from every other +only in its bounds, every float only in its hex parser, every complex +only in its component type. + +The boundary is worth recording: a member that fails its type check +stops the entity, optional or not. There is no honest reading of a +`blosc` whose level is a string, and building one around the hole would +have its rules see the member as absent and add a second, contradictory +problem at the same location. What the entity contains is still read -- +a shard's inner codecs are judged whatever its `index_location` says -- +and only the judgments the shard itself would make wait on the fix. + +An entity has the shape of its metadata: its name is the class, and its +configuration is a frozen `Configuration` record named in the one field +`configuration` -- `class GzipCodec(BytesBytesCodec)` with `configuration: +GzipOptions`; an entity of a bare name defaults the field to the empty +record, since the spec makes an absent configuration and an empty one the +same, so there is one rule for every entity. The record's constructor +makes the same type judgment of a value built by hand, and the entity's +holds the record to the one it declares, so neither `replace` nor +`with_configuration` can smuggle in a member of the wrong type or a +record of another entity. A document read makes each check once: the +parser type-checks the members and `coerce` runs the rules, then both +build through `create_unchecked`, the one path around the constructors, +named for what it is. The type judgment is read off the record's fields rather +than written twice. Which members exist, which may be left out (the type +admits `UNSET`), and how each is type-checked all follow from the field +annotations: `int`, `float` for any number, `bool`, `str`, a `Literal` +of names, the JSON-value alias, an array homogeneous or fixed, a union +of those, a nested object described by a TypedDict or a record +dataclass, an object of undeclared keys as `Mapping[str, V]`, a +`NewType` as the type it names, and a nested metadata field, written as +its kind with `Opaque`. That covers +every member this package models; an annotation outside them is refused +at registration, and `Annotated[str, FROM_NAME]` marks the one field of +the entity itself, carried by the envelope's name rather than a +configuration key (`r`), which `coerce` fills from the envelope. +Whether a configuration is required follows too, since the spec ties the +bare-name spelling to whether any member is required. + +The record's parser and writer are compiled once per class from its +annotations, and `coerce` runs the parser over the configuration as it +reads it. Field annotations are resolved per class, skipping class +variables by text, so a `ClassVar` naming something imported only for +the type checker cannot fail registration. + +One diagnostic became more precise: a malformed `[value, count]` pair in +a rectilinear grid is now reported at the offending element inside the +pair rather than at the pair. No verdict changes. + +An entity that contains other entities declares the field -- +`data_type: DataTypeEntity | Opaque`, `codecs: tuple[CodecEntity | +Opaque, ...]`, a record holding one -- and the contained entity is read +through the scope the containing one is read in, at whatever depth the +annotation puts it. Writing it back is the base's, through the contained +entity's own `to_json`; putting it in canonical form is the containing +entity's one line, `self.with_configuration(inner=self.inner.canonical())` +in `canonical`, the same way it simplifies its own members. +`canonical` is the entity itself by default, overridden where two +spellings of its members mean the same -- a rectilinear dimension's +run-length encoding, a `typesize` that `noshuffle` ignores. A field +typed as bare `MetadataEntity`, or as an entity without `Opaque`, is +refused at registration: no scope could place the one, and the other +lies about what the field holds when the inner name is out of scope. + +Everything finer than a type -- a bound, a rule about one member, a +rule that reads two members together -- is the record's own `problems`, +in plain code, yielding each problem as it finds it, located relative +to the configuration. The entity's constructor stops at the first, so +`BloscCodec(BloscOptions(clevel=99))` raises `MetadataValidationError`; +`coerce` asks the record before it builds anything and reports every +problem in the document; a reader with a record asks it too, stopping +or collecting as it needs. The space of +refinements is too wide to capture statically, and reading them in one +place per entity is what will show where a shared form is worth having. + +The set of annotation shapes the parser reads is closed: the shapes +JSON takes, and no others. A field annotation outside them is refused +at registration, and the field is written as one of them instead, with +any finer rule in the record's `problems`. The parser is one module that knows +nothing of entities; the entity layer hands it the one shape of its own, +a field holding an entity, as a leaf it reads at any depth. diff --git a/packages/zarr-metadata/changes/4379.feature.8.md b/packages/zarr-metadata/changes/4379.feature.8.md new file mode 100644 index 0000000000..6b95730c13 --- /dev/null +++ b/packages/zarr-metadata/changes/4379.feature.8.md @@ -0,0 +1,40 @@ +Serialization and canonicalization are separate operations. +`entity.to_json()` writes back what was read, member for member, so a +reader that reads a document and writes it returns the bytes it was +given. `entity.canonical()` returns the simplest equivalent, and +`canonicalize_array_metadata_v3` is what asks for it. An entity on its +own spells the envelope its own way, since it does not model it -- a +bare name, `{"name": x}` and `{"name": x, "configuration": {}}` all read +to the same entity -- and `ArrayDocumentV3.to_json` puts back the +spelling the document used, `must_understand` included, so the document +round-trips whole and only `canonical` changes it. + +Added `canonicalize_array_metadata_v3`, which answers with +`Canonical[T] | Invalid`: a semantically valid document in its simplest +equivalent spelling, or every reason it is not valid. + +Canonical is decided per metadata variety, and always means the simplest +form expressing the same semantics. An entity whose configuration carries +nothing collapses to its bare name, and a `must_understand` of `true` — +the default — is dropped while an explicit `false` is kept. `blosc` drops +a `typesize` that `shuffle: "noshuffle"` renders ignored, the spec saying +of that case that "the value is ignored". A rectilinear dimension's chunk +sizes run-length encode, because `[size, count]` is the spelling that does +not grow with the number of chunks. `dimension_names` of nothing but nulls +says what omitting the field says. + +Two spellings that look collapsible are deliberately left alone: a +dimension-level bare integer is a *step* that repeats until it covers the +extent, so it is not equivalent to any fixed list, and a one-element list +declares exactly one chunk rather than as many as it takes. Expanding +either would pin a grid that currently adapts. + +Two properties are asserted over generated documents: canonicalizing twice +changes nothing further, and canonicalizing never changes a verdict. + +The document owns its canonical form. `ArrayDocumentV3.canonical()` +puts each of its five extension points in canonical form -- +`storage_transformers` included -- and applies the one rule that is the +document's own. `canonicalize_array_metadata_v3` is the door onto it, +and reads the document once: the entities that judge it are the entities +that are rewritten. diff --git a/packages/zarr-metadata/changes/4379.feature.9.md b/packages/zarr-metadata/changes/4379.feature.9.md new file mode 100644 index 0000000000..5070be2354 --- /dev/null +++ b/packages/zarr-metadata/changes/4379.feature.9.md @@ -0,0 +1,30 @@ +`ArrayDocumentV3.from_json` is the fail-fast front door for readers: one +call, and either every extension point is read into an entity or a single +`MetadataValidationError` carries every reason it is not, structural and +semantic together. `validate_array_metadata_v3` remains for callers who +want the problems as data. + +A name the scope does not model is not a failure. It arrives as an +`Opaque` carrying the JSON the document wrote and saying which kind of +not-an-entity it is -- `out_of_scope` for an extension this reader does +not know, which is the reader's cue to resolve it elsewhere, and +`invalid` for a name that was claimed and then refused. + +That distinction was previously impossible to make, because every +extension point was typed `MetadataEntity | object` -- which *is* +`object`, so the union narrowed to nothing and a consumer could not tell +a third-party codec from a malformed one. Each field now names its own +kind (`DataTypeEntity | Opaque`, `CodecEntity | Opaque`, ...), an +exhaustive two-case union that narrows. `resolve(data, kind, context)`, +the reader, is generic in the kind, so it returns the entity type for +that point rather than the base. It relates the name in `data` to a +class through the context, and that class owns the validation routine, +its `coerce`; the context itself is a value with one question, +`claimant(kind, name)`. + +A *family* -- one class covering a parameterized set of names, as the +raw-byte types cover every `r` -- is registered like anything else. +It takes an invented identifier no document can write and claims its own +names through `accepts`, which `claimant` asks of each class. There +is no table of spellings anywhere in the package, so a third party can +register a family without changing it. diff --git a/packages/zarr-metadata/changes/4379.feature.md b/packages/zarr-metadata/changes/4379.feature.md new file mode 100644 index 0000000000..fadf1ee2c1 --- /dev/null +++ b/packages/zarr-metadata/changes/4379.feature.md @@ -0,0 +1,67 @@ +Added `zarr_metadata.rules`: semantic validation for full metadata +documents. The package now validates in two layers with one contract +each — `model` checks structure element by element, and `rules` judges +what structure alone cannot: whether the values are in range and whether +the document's parts agree with one another. + +The second layer is not a rule engine. Each extension — every codec, +data type, chunk grid and chunk key encoding — is a class that answers +for itself: + +- `coerce` reads raw metadata into the entity, or says why it is not one; +- its configuration is a record of its own, `GzipOptions`, whose + `problems` yields the values the spec disallows as it finds them; the + entity's constructor stops at the first and raises, and `coerce` + reports every one instead; +- `to_json` writes it back as it was read, and `canonical` gives its + simplest equivalent spelling; +- `incoming_problems`, `shape_problems`, `fill_value_problems` and + `transition` answer the questions that need a piece of the document: + the array reaching a codec, the shape a grid divides, the fill value a + data type must accept, what the next codec sees. + +Validating a document is composing those answers. Nothing in the +document layer knows what `blosc` or `int32` or `rectilinear` is, so +adding an extension is a class and a scope that includes it +(`CORE_AND_EXTENSIONS.extended_with(...)`, from `zarr_metadata.v3.entity`). + +- **What is judged**: fill value against data type (recursively, through + `struct` fields and `cast_value` targets), codec pipeline kind + ordering, dimension-name counts, chunk-grid extents and geometry + (regular rank; rectilinear rank and per-dimension chunk-size sums, RLE + pairs included), transpose orders (self-permutation, and rank + agreement with the array that reaches them at any depth), and sharding + (inner `codecs` and `index_codecs` judged as pipelines recursively; + inner chunk shapes positive, rank-matched, and evenly dividing every + chunk they will be handed). v2 arrays get chunks/shape rank agreement; + v3 groups recurse into inline consolidated metadata, judging each + embedded child document at its path. +- **Scope**: `zarr_metadata.v3.entity` maps each extension point to the + identifiers in play, in two scopes — `CORE`, what the specification + defines, and `CORE_AND_EXTENSIONS`, which adds what `zarr-extensions` + registers and this package models. A name in neither is not rejected; + it is simply not judged. The scope is a `Context`, passed to every + `coerce` and ignored by every entity that does not contain another. +- **Read-side entry points**: `validate_*` / `parse_*` for array and + group documents in both format versions mirror the model layer's + grammar with a stronger judgment — structure *and* semantics, every + problem reported together, JSON arrays normalized to tuples before + judgment. There is no `is_*` counterpart: a semantically invalid + document is still an instance of the TypedDict, so only the structural + layer can narrow honestly — use `zarr_metadata.model.is_*` for that. +- **Boundary change**: two checks that lived in the structural validator + moved here — v3 `dimension_names` vs `shape` and v2 `chunks` vs + `shape` rank agreement. `zarr_metadata.model`'s validators, parsers + and dataclasses now accept those documents (they are lossless, + structurally well-formed representations of what a store may contain); + use the `rules` validators to judge them. This also removes the double + report the overlap used to produce. +- **Strictness stance**, now documented on the package: `zarr_metadata` + models canonical documents and is deliberately stricter than any given + implementation; implementations coerce ambiguous input as they see fit + and then validate the canonical result. +- **Pydantic field types** for array and group documents run the + semantic checks as well as structural validation. + +Known follow-up: v2 fill-value/dtype consistency (NumPy dtype grammar) +has no check yet. diff --git a/packages/zarr-metadata/changes/4379.misc.1.md b/packages/zarr-metadata/changes/4379.misc.1.md new file mode 100644 index 0000000000..7d7c6c34c0 --- /dev/null +++ b/packages/zarr-metadata/changes/4379.misc.1.md @@ -0,0 +1,32 @@ +The rule registry is gone, and 3900 lines with it. It existed to +dispatch a rule to the entity it was about, gate it on whether that +entity's metadata was readable, and propagate what each codec did to the +array — all three of which the entities now do themselves. + +Deleted: the registry and its `reads`/`reads_optional` gate, the rule +engine, the nine modules of rules keyed by entity name, the fill-value +dispatch table, three frozensets of data type names with a hand-rolled +recursion for `struct`, four tables of per-member type checks, and the +drift tests that existed to keep those tables in step with the +TypedDicts. The tables cannot drift now, because there is one of each +fact instead of four. + +What replaces the drift tests is a correspondence test: an entity's +configuration record and its configuration TypedDict are two spellings +of one set of members, with the same requiredness, and whether the +bare-name spelling is allowed follows from that. + +One more table went the same way: `zarr_metadata.v3.codec.kind`, which +sorted codec *names* into the spec's three pipeline kinds. A name does +not have a pipeline position, a codec does, and each one now says so by +its base class: `ArrayArrayCodec`, `ArrayBytesCodec` or +`BytesBytesCodec`. Removed with it: `codec_kind_of_name`, +`ARRAY_ARRAY_CODEC_NAMES`, `ARRAY_BYTES_CODEC_NAMES`, +`BYTES_BYTES_CODEC_NAMES`, and `codec.blosc.canonical_configuration`, +whose one job is `BloscCodec.canonical()`. + +`zarr_metadata.rules` is a door and nothing else: `validate_*`, +`parse_*` and `canonicalize_*` with the scope they ask in. The logic +that had accumulated beside them -- a group's consolidated-metadata +recursion, the one v2 rule, the canonicalizer's walk -- lives with the +documents it is about, in `v3._document` and `v2._document`. diff --git a/packages/zarr-metadata/changes/4379.misc.2.md b/packages/zarr-metadata/changes/4379.misc.2.md new file mode 100644 index 0000000000..0f0e2c7402 --- /dev/null +++ b/packages/zarr-metadata/changes/4379.misc.2.md @@ -0,0 +1,153 @@ +Adversarial review of the entity layer, from five directions: reading and +writing `zarr.json` the way zarr-python does, authoring a third-party +extension, hunting wrong verdicts, type-checking a strict consumer, and +v2/group/consolidated workloads. Eleven real defects, all reproduced, all +fixed here; the rest of what it found is recorded against the merged +model layer instead. + +The one that mattered most: a metadata field nested inside a +configuration -- a shard's pipelines, a struct field's data type, a +`cast_value` target -- was not getting the structural judgment a +top-level one gets, so an extra member, a non-object `configuration` or a +non-boolean `must_understand` was accepted there and then *deleted* by +the canonicalizer. The differential over 2267 documents could not see it, +because `st.from_type` honours the TypedDicts and cannot emit a malformed +envelope. + +Two results worth recording. Eleven `zarr.json` documents written by this +repository's own `create_array` -- float32+blosc, sharded, string, +datetime64, complex128, structured, transposed -- validate clean. And +against the rule registry this layer replaced, over one shared corpus of +40,000 documents, no verdict differs in the laxer direction and the valid +documents report identically. + +An entity's `to_json` names its own JSON type as its return type -- +`GzipCodecObject` for `GzipCodec`; `Crc32cCodecName` alone for `crc32c`, +which has no members and only ever writes the bare name -- and the +entity base is not generic. For the conformance test to use the +package's own parser as its oracle, it reads two shapes it did not: +`Mapping[str, V]`, an object of undeclared keys, and a `NewType`, as the +type it names -- which also closes a gap for a third-party field typed +with either. + +Registration is the one moment an entity is refused, with what to +write: no `@dataclass`, a codec subclassing `CodecEntity` instead of a +kind, a field whose annotation is not a shape JSON takes, a +`__post_init__` of the entity's own, a class variable a base annotates +and nothing sets, a kind's abstract method left undefined. Nothing +happens at class creation, so a family is a plain subclass and a +forward reference in a field annotation resolves. Everything else an +author could get wrong, pyright says in the editor -- a field shadowing +a class variable, a `Literal` class variable outside its values. +Nothing is kept on the class: what the layer needs of an entity's +fields, it reads off them when it reads a document. + +Four adversarial reviews -- two extension authors writing a codec and a +data type against the door alone, a design review, an onboarding review +-- drove the last round. `coerce` no longer builds an entity around a +member that could not be read, which had the entity's rules judging the +hole; nested entities are resolved whatever else was found, so their +problems come out in the same pass; `FROM_NAME` fields are filled by +the base; `to_json` on a document writes only the fields it had; `float` +is a shape. Class creation and registration refuse the mistakes both +authors made or nearly made -- no `@dataclass`, a nested field without +`Opaque`, an `array_array` codec without `transition`, a list of +`problem()` tuples -- with +messages that say what to write. The door's docstring is now the guide +its example claimed to be, runnable as written, with the composition +contract per kind and what comes back; it exports what its example +needs (`UNSET`, `MetadataValidationError`, `ValidationProblem`, +`ProblemKind`, `JSONValue`, `ZarrV3MetadataFieldJSON`) and no longer the +hand-written check vocabulary the compiler made obsolete. The two +reviewer-written extensions are kept as `tests/v3/test_acme_affine.py` +and `tests/v3/test_acme_decimal.py`, written against the door alone. + +From the review of that: the entity base is an ABC, and what a kind +must answer is abstract on it -- `transition` on `ArrayArrayCodec`, +`fill_value_problems` on `DataTypeEntity`, `grid` on `ChunkGridEntity` +-- so a codec's kind is its base class (`ArrayArrayCodec`, +`ArrayBytesCodec`, `BytesBytesCodec`) rather than a string it sets, and +an entity that leaves a hook undefined is refused at registration +rather than accepted with a silent default. Nothing is derived from an +entity's fields ahead of time: `coerce` parses a configuration against +them as it reads it. The parser is one module that knows nothing of +entities; the entity layer hands it the one shape of its own -- a field +holding an entity, `Kind | Opaque` -- as a leaf it reads at any depth, +so a struct's fields' data types and a shard's pipelines are read by +the same walk that type-checks them, and no second walk over the +annotations exists. + +`canonical` is the entity's own, as `to_json` is: the entity itself by +default, overridden where two spellings mean the same and, in an entity +that contains entities, to put those in canonical form; an `Opaque` +answers both, with the JSON it kept and with itself, so a field typed +`CodecEntity | Opaque` is written and simplified without asking which +it holds. One diagnostic is more complete: when one element of a +shard's pipeline is not a metadata field at all, the other elements are +still read and judged, where before the whole member stood down. Over +the 40,000-document corpus that adds 133 problems to 91 documents +already invalid, and changes no verdict. + +A scope holds entities by kind -- `DataTypeEntity`, `ChunkGridEntity`, +`ChunkKeyEncodingEntity`, `CodecEntity`, `StorageTransformerEntity` -- +and `extended_with` takes classes, reading each one's kind off its base +and its key off its `identifier`, so nothing can be misfiled and the two +registration checks for that are gone. Which of a document's fields +holds which kind is the document's own knowledge, in one place; the +entity layer no longer names a document field. + +`to_json` is written once, in the base, from the configuration record: `writer_for` +is the parser's inverse over the same annotation, so what `coerce` reads +from a document, `to_json` puts back -- the bare name when every member +is absent, the object otherwise, a contained entity through its own +`to_json`, a JSON-valued member copied. No entity writes its own, and a +third party writes nothing for it. What is given up is the per-entity +return type: every entity's `to_json` is typed as the metadata field +union rather than as its own TypedDict, and a consumer who wants a +member of the written form typed narrows it. + +From three reviews of the result, each from the vantage of another +implementation (zarrs, TensorStore, zarrita.js): every codec declares +`variable_size`, because a default in either direction is a verdict +and the door's own example compressor was passing as a shard-index +codec; a malformed envelope is one problem, judged once by the scope, +where the entity used to add a second at the same location (a +configuration that is not an object no longer also reports "requires a +configuration"; a value that is not a metadata field no longer also +reports "expected a metadata field"); `ArrayDocumentV3.from_json` +reports structural and semantic problems together instead of hiding +the second behind the first; a problem about the member the envelope's +name carries lands on the field, not under a `configuration` the +document does not have; a name resolves by asking each entity of the +kind whether it is its own, so the invented-identifier veto is gone; +`Literal[1]` refuses JSON `true`, a `Literal` of mixed types no longer +fails on a comparison, a fixed-length array accepts a list as a +homogeneous one does, a key inside a nested object is reported at the +key like a top-level one, and a record dataclass that refuses its own +values is reported rather than raised. + +A parser is compiled once per class and run many times. Each takes the +reading it runs in as an argument -- for the one shape the entity layer +adds, a field holding another entity, that is the scope to read it in +and where its problems go -- so nothing that varies between reads is +compiled into it, and the plan for a class, like its resolved field +annotations, is a pure function of the class and cached as one. A +document with a shard, six codecs and a nested pipeline reads in 85 µs +where it took 199, and one codec in 6 µs where it took 21. + +An entity has the shape of its metadata. Its name is the class, and its +configuration is a frozen `Configuration` record named in the one field +`configuration` -- `class GzipCodec(BytesBytesCodec)` with `configuration: +GzipOptions`, read as `codec.configuration.level`, the shape the +metadata has. The +record's fields are the members and its `problems` the rules on them, +so the lift of configuration keys to the entity, which nothing in Python +expressed, is gone with the per-member loop in `coerce` that did it: a +configuration is parsed and written by the parser's and the writer's +record shapes. `with_configuration(**changes)` is the entity with members +of its configuration replaced, checked as any construction is. A record +is the options as written, judged on request; the entity is the value +guarantee, and `coerce` asks the record before it builds one, so no +unchecked construction path exists. The rule a family has about its +name -- `r` a multiple of 8 -- is the entity's `name_problems`, since +a name is not configuration. diff --git a/packages/zarr-metadata/changes/4379.misc.md b/packages/zarr-metadata/changes/4379.misc.md new file mode 100644 index 0000000000..f6acce53dd --- /dev/null +++ b/packages/zarr-metadata/changes/4379.misc.md @@ -0,0 +1,40 @@ +Property tests over codec chains drawn from the codec `TypedDict`s with +`hypothesis.strategies.from_type`, which needs one registered strategy for +the recursive `JSONValue` alias and then resolves `ReadOnly`, `closed`, +`NotRequired` and `Literal` unaided. + +The pre-existing totality test feeds arbitrary JSON to the document +validators; a random object never names a codec, so no codec was ever +asked about itself. Drawing codecs from their own types and assembling +the chain by pipeline kind (`array->array`* `array->bytes` +`bytes->bytes`*) reaches a real codec in every document, where arbitrary +JSON reached none; 86% report a problem, against 78% for a flat +list of the same codecs. Ordering earns that by guaranteeing an +`array->bytes` codec is present, not by avoiding an early exit — nothing +short-circuits on a misordered chain. + +Two properties are now covered that were not: that a chain of arbitrary +configurations always produces a verdict rather than raising, and that +documents built valid by construction — extents drawn, then inner chunk +shapes drawn from their divisors, across regular and rectilinear grids — +are accepted. The second guards the direction every recent fix pushed +against, since fewer than 5% of generated documents are valid. + +The reach of each strategy is asserted rather than assumed, so a change +that silently stops dispatching fails loudly instead of leaving a green +property run that tests nothing. + +`st.from_type` honours the TypedDicts exactly, so it cannot produce a +configuration member of the wrong *type* — which is exactly the case +where an entity has to decide what it can still say about itself. A +corrupting strategy covers that, and it is what established that the +entity-based validator returns the same verdicts as the rule registry it +replaced: identical problems on valid documents, identical verdicts on +corrupted ones. + +The package's tests are now type-checked along with its sources. They were +checked by nothing: pyright was configured for `src` alone, and the repo's +mypy hook resolves this package's imports as `Any`. Turning it on found a +`TYPE_CHECKING` import of a type deleted three commits earlier, and that +`from_key_value(to_key_value())` — the round trip the models advertise — +did not type-check, because `Mapping` is invariant in its key type. diff --git a/packages/zarr-metadata/changes/4380.feature.md b/packages/zarr-metadata/changes/4380.feature.md new file mode 100644 index 0000000000..0ffec7cc4b --- /dev/null +++ b/packages/zarr-metadata/changes/4380.feature.md @@ -0,0 +1,17 @@ +`cast_value`'s `out_of_range: "wrap"` is rejected for targets it is not +defined on. The spec permits wrapping only for integral targets with a +two's complement representation, so a known non-integer target -- `bool` +included, which is not two's complement -- reports `invalid_value` at +`out_of_range`. A target this reader does not model declines, since it +may be an integral extension type. + +Whether a data type wraps is a fact the data type states, +`twos_complement`, and it is owed rather than defaulted: a data type added +later has to decide, because either default would answer for it silently +and one of them accepts a cast the spec does not define. Registration +refuses a data type that has not decided. + +Entities gain `name`, the spelling a document writes, as distinct from +`identifier`, the key they are tabled under. The two differ only for the +raw-bytes family, whose identifier is invented and belongs in no message +a reader sees; a message about an `r24` now says `r24`. diff --git a/packages/zarr-metadata/docs/api/index.md b/packages/zarr-metadata/docs/api/index.md index 5e230c7aa2..15026ff2c9 100644 --- a/packages/zarr-metadata/docs/api/index.md +++ b/packages/zarr-metadata/docs/api/index.md @@ -8,6 +8,10 @@ The package is organized to mirror the structure of the Zarr specifications: - [`zarr_metadata.model`](model.md) — frozen-dataclass document models, structural validators, loc-aware parsers, and the `UNSET` sentinel +- [`zarr_metadata.rules`](rules.md) — composition rules: cross-field + judgments over full documents (fill value vs. data type, codec pipeline + ordering, chunk geometry), plus whole-document `validate`/`parse` + entry points combining structure and composition - [`zarr_metadata.pydantic`](pydantic.md) — optional Pydantic field types over the models - [`zarr_metadata.v2`](v2.md) — `TypedDict` shapes for Zarr v2 documents @@ -16,6 +20,10 @@ The package is organized to mirror the structure of the Zarr specifications: documents, with subpackages for [chunk grids](v3/chunk_grid.md), [chunk key encodings](v3/chunk_key_encoding.md), [codecs](v3/codec.md), and [data types](v3/data_type.md) +- [`zarr_metadata.v3.entity`](v3/entity.md) — the extension layer: read a + document into entities that answer for themselves, and write your own + codec, data type or chunk grid and add it to a scope; its module + docstring is the guide The document types, models, and spec vocabulary — including the store keys — are re-exported at the top level, so diff --git a/packages/zarr-metadata/docs/api/rules.md b/packages/zarr-metadata/docs/api/rules.md new file mode 100644 index 0000000000..ef010b72e2 --- /dev/null +++ b/packages/zarr-metadata/docs/api/rules.md @@ -0,0 +1,5 @@ +--- +title: rules +--- + +::: zarr_metadata.rules diff --git a/packages/zarr-metadata/docs/api/v3/entity.md b/packages/zarr-metadata/docs/api/v3/entity.md new file mode 100644 index 0000000000..bd846b75c7 --- /dev/null +++ b/packages/zarr-metadata/docs/api/v3/entity.md @@ -0,0 +1,8 @@ +--- +title: entity +--- + +::: zarr_metadata.v3.entity + options: + inherited_members: false + show_if_no_docstring: false diff --git a/packages/zarr-metadata/docs/index.md b/packages/zarr-metadata/docs/index.md index 0163dda5c1..38a66dafd3 100644 --- a/packages/zarr-metadata/docs/index.md +++ b/packages/zarr-metadata/docs/index.md @@ -32,28 +32,34 @@ closely model the content of the Zarr specifications, such as: validators, loc-aware parsers, and store-key (de)serialization. A document produced by `to_json` shares no mutable state with the model that produced it. +- **Composition rules** ([`zarr_metadata.rules`](api/rules.md)): cross-field + judgments over whole documents — fill value against data type, codec + pipeline ordering, chunk geometry — with `validate_*` / `parse_*` entry + points that apply structure and composition together. - **Optional Pydantic integration** ([`zarr_metadata.pydantic`](api/pydantic.md), requires Pydantic 2.13 or newer): each model as a Pydantic field type that - validates raw documents through the same strict parser. + runs raw documents through the rules layer. ## What this is for The public `TypedDict` definitions describe the static JSON shape of Zarr -metadata. For strict, loc-aware validation of JSON loaded from disk, use the -model parser: +metadata. To judge JSON loaded from disk, structure and composition together, +use the rules layer; to get a normalized document model, use the model parser: ```python import json from zarr_metadata.model import ZarrV3ArrayMetadata +from zarr_metadata.rules import parse_array_metadata_v3 with open("zarr.json", "rb") as f: raw = json.load(f) -metadata = ZarrV3ArrayMetadata.from_json(raw) +document = parse_array_metadata_v3(raw) # raises with every problem found +metadata = ZarrV3ArrayMetadata.from_json(document) ``` -The optional Pydantic integration delegates raw input to the same strict -parser and returns the same normalized model class: +The optional Pydantic integration runs raw input through the rules layer +and returns the same normalized model class: ```python from pydantic import TypeAdapter @@ -71,11 +77,33 @@ members that the strict model parser rejects. The model validators enforce the declared document structure and a small set of context-free consistency rules, including fixed format literals, finite -JSON numbers, non-negative dimensions, non-empty v3 codec pipelines, and one -`dimension_names` entry per array dimension. They do not interpret extension -names or configurations, resolve codec pipelines, or decide whether a data -type, chunk grid, codec, or storage transformer is supported. Those decisions -belong to consumer implementations. +JSON numbers, non-negative dimensions, and non-empty v3 codec pipelines. +They do not interpret extension names or configurations. + +The composition rules (`zarr_metadata.rules`) judge the document as a whole: +fill values against data types, codec pipeline ordering, chunk-grid +geometry against `shape`, one `dimension_names` entry per array dimension, +and the canonical configuration shapes of the codecs, chunk grids, chunk +key encodings, and data types this package defines. Unknown extension names +are left unjudged. The rules model canonical documents and are deliberately +stricter than any given implementation: an implementation may coerce +ambiguous input as it sees fit and then validate the canonical result. +Nothing here decides whether a data type, chunk grid, codec, or storage +transformer is *supported*; that belongs to consumer implementations. + +An unmodelled member inside a *known* entity's `configuration` is an error +under that strict reading: such a member is almost always a typo or a +setting meant for a different entity, and accepting it silently means +silently ignoring what the writer asked for. It carries its own +`unknown_key` problem kind, so a consumer who wants the tolerant reading +can collect problems with `validate_*` and filter that kind out. + +Judging it is the rules layer's job, so it is `zarr_metadata.rules` and +the whole-document Pydantic field types (which run the rules layer) that +reject it. The model layer never interpreted entity configurations and +still does not, so `model.parse_*` and `from_json` accept such a +document; so does the bare `ZarrV3MetadataField` Pydantic type, which +judges one metadata field and carries no composition rules. ## Scope diff --git a/packages/zarr-metadata/justfile b/packages/zarr-metadata/justfile index 843286b89f..ef55a98f38 100644 --- a/packages/zarr-metadata/justfile +++ b/packages/zarr-metadata/justfile @@ -8,26 +8,54 @@ set positional-arguments default: @just --list -# Run the test suite; extra args are passed to pytest +# Run the test suite on the project interpreter; extra args are passed to pytest test *args: uv run --group test pytest tests "$@" +# The oldest and newest interpreters the package is declared for: the floor is +# `requires-python`, the ceiling the newest classifier, both read from +# pyproject.toml so this recipe cannot drift from what CI's matrix spans. +# uv fetches an interpreter it does not have. A difference in how a version +# materialises annotations or resolves a stub shows up here, not only in CI. +# Run the test suite on the oldest and newest supported interpreters +test-versions *args: + #!/usr/bin/env bash + set -euo pipefail + read -r floor ceiling < <(uv run python - <<'EOF' + import re + import tomllib + + project = tomllib.load(open("pyproject.toml", "rb"))["project"] + floor = re.fullmatch(r">=\s*(\d+\.\d+)", project["requires-python"]).group(1) + declared = [ + classifier.rsplit(" ", 1)[1] + for classifier in project["classifiers"] + if classifier.startswith("Programming Language :: Python :: 3.") + ] + print(floor, max(declared, key=lambda version: tuple(map(int, version.split("."))))) + EOF + ) + for version in "$floor" "$ceiling"; do + echo "== python $version ==" + uv run --python "$version" --group test pytest tests "$@" + done + # Lint with the same invocation CI uses lint: uvx ruff check . -# Pinned to the last pyright that types PEP 661 sentinels in class attributes -# correctly; 1.1.405+ regressed (microsoft/pyright#11115). Unpin when fixed. -pyright_version := "1.1.404" +# Pinned so a pyright release cannot turn CI red on its own schedule. Bump it +# deliberately: run the new version, read what it found, then move the pin. +pyright_version := "1.1.414" -# CI runs pyright on python 3.11; the pinned pyright predates 3.14, whose -# stdlib it cannot parse, so pin the interpreter to match CI. -# Type-check the package sources +# Run under the interpreter CI runs pyright under, so a version-dependent +# stdlib or stub difference shows up here rather than only in CI. +# Type-check the package, sources and tests alike typecheck: - uv run --python 3.11 --group test --with 'pyright=={{ pyright_version }}' pyright src + uv run --python 3.11 --group test --with 'pyright=={{ pyright_version }}' pyright -# Run everything CI runs for this package -check: lint typecheck test docs-check +# Run everything CI runs for this package, on both ends of the version range +check: lint typecheck test-versions docs-check # Preview the changelog that the next release would generate changelog-draft: diff --git a/packages/zarr-metadata/mkdocs.yml b/packages/zarr-metadata/mkdocs.yml index 98a51e3645..3f84c280bd 100644 --- a/packages/zarr-metadata/mkdocs.yml +++ b/packages/zarr-metadata/mkdocs.yml @@ -16,6 +16,7 @@ nav: - API Reference: - api/index.md - ' zarr_metadata.model': api/model.md + - ' zarr_metadata.rules': api/rules.md - ' zarr_metadata.pydantic': api/pydantic.md - ' zarr_metadata.v2': api/v2.md - ' zarr_metadata.v3': @@ -24,6 +25,7 @@ nav: - ' zarr_metadata.v3.chunk_key_encoding': api/v3/chunk_key_encoding.md - ' zarr_metadata.v3.codec': api/v3/codec.md - ' zarr_metadata.v3.data_type': api/v3/data_type.md + - ' zarr_metadata.v3.entity': api/v3/entity.md - Release notes: release-notes.md # This site is a Read the Docs subproject of zarr-python; give readers a way # back to the parent docs, which list every companion package. diff --git a/packages/zarr-metadata/pyproject.toml b/packages/zarr-metadata/pyproject.toml index 6e7b0e4f52..958d5fec77 100644 --- a/packages/zarr-metadata/pyproject.toml +++ b/packages/zarr-metadata/pyproject.toml @@ -47,7 +47,7 @@ Changelog = "https://github.com/zarr-developers/zarr-python/blob/main/packages/z Documentation = "https://zarr-metadata.readthedocs.io/" [dependency-groups] -test = ["pytest", "pydantic>=2.13", "jsonschema"] +test = ["pytest", "pydantic>=2.13", "jsonschema", "hypothesis"] docs = [ # Pins match the zarr-python docs environment in the repo-root # pyproject.toml so the two sites render with the same toolchain. @@ -125,15 +125,42 @@ checks = [ "PR06", ] -# CI pins pyright==1.1.404: later versions regress PEP 661 sentinel typing in -# class attributes (microsoft/pyright#11115), which zarr_metadata.model._sentinel -# relies on. Use the same pin locally; unpin when the fix lands. +# The pyright version lives in the justfile, which is what CI runs; pinning it +# keeps a pyright release from turning CI red on its own schedule. [tool.pyright] -include = ["src"] +include = ["src", "tests"] +# Pyright is the only checker that runs on this package (the root mypy +# config covers the root package alone), and it honours `# type: ignore` +# as a blanket suppression of every diagnostic on the line. So that +# spelling is switched off: a suppression is `# pyright: ignore[rule]`, +# names the rule, and is an error once it stops suppressing anything. +enableTypeIgnoreComments = false +reportUnnecessaryTypeIgnoreComment = "error" +# `tests` is a package imported as `tests.*`, which pytest resolves from the +# rootdir; pyright needs the same root on its search path. +extraPaths = ["."] enableExperimentalFeatures = true typeCheckingMode = "strict" pythonVersion = "3.11" +# Tests are checked, but not for the inference strictness the sources are held +# to: test bodies are full of untyped JSON literals whose inferred types are of +# no interest, and strict mode reports about ninety of those. What is kept is +# everything a rename or a signature change breaks -- a dangling import, a call +# that no longer type-checks, an argument of the wrong type -- which is the +# reason for checking tests at all. Tests also reach into private modules and +# define functions purely to be refused at registration, both deliberately. +[[tool.pyright.executionEnvironments]] +root = "tests" +reportMissingTypeArgument = false +reportPrivateUsage = false +reportUnknownArgumentType = false +reportUnknownLambdaType = false +reportUnknownMemberType = false +reportUnknownParameterType = false +reportUnknownVariableType = false +reportUnusedFunction = false + [tool.towncrier] # Fragments for this package live alongside the package source, separate # from the parent zarr-python `changes/` directory, so a PR touching only diff --git a/packages/zarr-metadata/src/zarr_metadata/model/_array.py b/packages/zarr-metadata/src/zarr_metadata/model/_array.py index 26a69a304a..804c531b52 100644 --- a/packages/zarr-metadata/src/zarr_metadata/model/_array.py +++ b/packages/zarr-metadata/src/zarr_metadata/model/_array.py @@ -31,16 +31,13 @@ from zarr_metadata.v2.array import ( ZarrV2ArrayDimensionSeparator, ZarrV2ArrayMetadataJSON, - ZarrV2ArrayMetadataStoreKey, ZarrV2ArrayOrder, ZarrV2DataTypeMetadata, ) - from zarr_metadata.v2.attributes import ZarrV2AttributesStoreKey from zarr_metadata.v2.codec import ZarrV2CodecMetadata from zarr_metadata.v3._common import ZarrV3MetadataFieldJSON from zarr_metadata.v3.array import ( ZarrV3ArrayMetadataJSON, - ZarrV3ArrayMetadataStoreKey, ZarrV3ExtensionField, ) @@ -325,10 +322,12 @@ def must_understand_fields(self) -> dict[str, ZarrV3ExtensionField]: def from_key_value(cls, mapping: Mapping[str, bytes]) -> ZarrV3ArrayMetadata: return cls.from_json(load_store_json(mapping, ZARR_V3_ARRAY_METADATA_STORE_KEY)) - def to_key_value( - self, *, indent: int | str | None = None - ) -> Mapping[ZarrV3ArrayMetadataStoreKey, bytes]: - return {ZARR_V3_ARRAY_METADATA_STORE_KEY: dump_store_json(self.to_json(), indent=indent)} + def to_key_value(self, *, indent: int | str | None = None) -> Mapping[str, bytes]: + return { + ZARR_V3_ARRAY_METADATA_STORE_KEY: dump_store_json( + ZARR_V3_ARRAY_METADATA_STORE_KEY, self.to_json(), indent=indent + ) + } class ZarrV2ArrayMetadataPartial(TypedDict, total=False): @@ -488,16 +487,18 @@ def from_key_value(cls, mapping: Mapping[str, bytes]) -> ZarrV2ArrayMetadata: return cls.from_json({**zarray, "attributes": zattrs}) return cls.from_json(zarray) - def to_key_value( - self, *, indent: int | str | None = None - ) -> Mapping[ZarrV2ArrayMetadataStoreKey | ZarrV2AttributesStoreKey, bytes]: + def to_key_value(self, *, indent: int | str | None = None) -> Mapping[str, bytes]: # Attributes live only in the sibling `.zattrs` file; the `.zarray` # document must exclude them. The `.zattrs` key is present exactly # when attributes are set (even empty) — UNSET emits no file. zarray = {k: v for k, v in self.to_json().items() if k != "attributes"} - out: dict[ZarrV2ArrayMetadataStoreKey | ZarrV2AttributesStoreKey, bytes] = { - ZARR_V2_ARRAY_METADATA_STORE_KEY: dump_store_json(zarray, indent=indent) + out: dict[str, bytes] = { + ZARR_V2_ARRAY_METADATA_STORE_KEY: dump_store_json( + ZARR_V2_ARRAY_METADATA_STORE_KEY, zarray, indent=indent + ) } if self.attributes is not UNSET: - out[ZARR_V2_ATTRIBUTES_STORE_KEY] = dump_store_json(self.attributes, indent=indent) + out[ZARR_V2_ATTRIBUTES_STORE_KEY] = dump_store_json( + ZARR_V2_ATTRIBUTES_STORE_KEY, self.attributes, indent=indent + ) return out diff --git a/packages/zarr-metadata/src/zarr_metadata/model/_group.py b/packages/zarr-metadata/src/zarr_metadata/model/_group.py index 5519e6fbb9..06d250bdc4 100644 --- a/packages/zarr-metadata/src/zarr_metadata/model/_group.py +++ b/packages/zarr-metadata/src/zarr_metadata/model/_group.py @@ -24,8 +24,8 @@ load_store_json, parse_group_metadata_v2, parse_group_metadata_v3, + stored_json_problems, validate_consolidated_metadata_v3, - validate_json, ) from zarr_metadata.v2.attributes import ZARR_V2_ATTRIBUTES_STORE_KEY from zarr_metadata.v2.consolidated import ZARR_V2_CONSOLIDATED_METADATA_STORE_KEY @@ -35,12 +35,10 @@ if TYPE_CHECKING: from zarr_metadata._common import JSONValue - from zarr_metadata.v2.attributes import ZarrV2AttributesStoreKey - from zarr_metadata.v2.consolidated import ZarrV2ConsolidatedMetadataStoreKey - from zarr_metadata.v2.group import ZarrV2GroupMetadataJSON, ZarrV2GroupMetadataStoreKey + from zarr_metadata.v2.group import ZarrV2GroupMetadataJSON from zarr_metadata.v3.array import ZarrV3ExtensionField from zarr_metadata.v3.consolidated import ZarrV3ConsolidatedMetadataJSON - from zarr_metadata.v3.group import ZarrV3GroupMetadataJSON, ZarrV3GroupMetadataStoreKey + from zarr_metadata.v3.group import ZarrV3GroupMetadataJSON class ZarrV3GroupMetadataPartial(TypedDict, total=False): @@ -179,10 +177,12 @@ def must_understand_fields(self) -> dict[str, ZarrV3ExtensionField]: def from_key_value(cls, mapping: Mapping[str, bytes]) -> ZarrV3GroupMetadata: return cls.from_json(load_store_json(mapping, ZARR_V3_GROUP_METADATA_STORE_KEY)) - def to_key_value( - self, *, indent: int | str | None = None - ) -> Mapping[ZarrV3GroupMetadataStoreKey, bytes]: - return {ZARR_V3_GROUP_METADATA_STORE_KEY: dump_store_json(self.to_json(), indent=indent)} + def to_key_value(self, *, indent: int | str | None = None) -> Mapping[str, bytes]: + return { + ZARR_V3_GROUP_METADATA_STORE_KEY: dump_store_json( + ZARR_V3_GROUP_METADATA_STORE_KEY, self.to_json(), indent=indent + ) + } @dataclass(frozen=True, slots=True, kw_only=True) @@ -334,18 +334,20 @@ def from_key_value(cls, mapping: Mapping[str, bytes]) -> ZarrV2GroupMetadata: return cls.from_json({**zgroup, "attributes": zattrs}) return cls.from_json(zgroup) - def to_key_value( - self, *, indent: int | str | None = None - ) -> Mapping[ZarrV2GroupMetadataStoreKey | ZarrV2AttributesStoreKey, bytes]: + def to_key_value(self, *, indent: int | str | None = None) -> Mapping[str, bytes]: # Attributes live only in the sibling `.zattrs` file; the `.zgroup` # document must exclude them. The `.zattrs` key is present exactly # when attributes are set (even empty) — UNSET emits no file. zgroup = {k: v for k, v in self.to_json().items() if k != "attributes"} - out: dict[ZarrV2GroupMetadataStoreKey | ZarrV2AttributesStoreKey, bytes] = { - ZARR_V2_GROUP_METADATA_STORE_KEY: dump_store_json(zgroup, indent=indent) + out: dict[str, bytes] = { + ZARR_V2_GROUP_METADATA_STORE_KEY: dump_store_json( + ZARR_V2_GROUP_METADATA_STORE_KEY, zgroup, indent=indent + ) } if self.attributes is not UNSET: - out[ZARR_V2_ATTRIBUTES_STORE_KEY] = dump_store_json(self.attributes, indent=indent) + out[ZARR_V2_ATTRIBUTES_STORE_KEY] = dump_store_json( + ZARR_V2_ATTRIBUTES_STORE_KEY, self.attributes, indent=indent + ) return out @@ -410,12 +412,14 @@ def from_json(cls, data: object) -> ZarrV2ConsolidatedMetadata: ) ) else: + # Each entry is the document its key names, so a `.zattrs` + # entry is user data (`stored_json_problems`). for key, value in cast("Mapping[str, object]", entries).items(): problems.extend( ValidationProblem( ("metadata", key, *problem.loc), problem.message, problem.kind ) - for problem in validate_json(value) + for problem in stored_json_problems(key, value) ) if len(problems) != 0: raise MetadataValidationError(problems) @@ -429,9 +433,9 @@ def from_json(cls, data: object) -> ZarrV2ConsolidatedMetadata: def from_key_value(cls, mapping: Mapping[str, bytes]) -> ZarrV2ConsolidatedMetadata: return cls.from_json(load_store_json(mapping, ZARR_V2_CONSOLIDATED_METADATA_STORE_KEY)) - def to_key_value( - self, *, indent: int | str | None = None - ) -> Mapping[ZarrV2ConsolidatedMetadataStoreKey, bytes]: + def to_key_value(self, *, indent: int | str | None = None) -> Mapping[str, bytes]: return { - ZARR_V2_CONSOLIDATED_METADATA_STORE_KEY: dump_store_json(self.to_json(), indent=indent) + ZARR_V2_CONSOLIDATED_METADATA_STORE_KEY: dump_store_json( + ZARR_V2_CONSOLIDATED_METADATA_STORE_KEY, self.to_json(), indent=indent + ) } diff --git a/packages/zarr-metadata/src/zarr_metadata/model/_validation.py b/packages/zarr-metadata/src/zarr_metadata/model/_validation.py index f927851e4c..15c6514b8c 100644 --- a/packages/zarr-metadata/src/zarr_metadata/model/_validation.py +++ b/packages/zarr-metadata/src/zarr_metadata/model/_validation.py @@ -7,17 +7,17 @@ Every `ValidationProblem` carries a machine-readable `kind` alongside its human-readable `message`, so consumers can dispatch on the failure mode -(`missing_key`, `invalid_type`, `invalid_value`, `invalid_json`) without -string-matching messages. +(`missing_key`, `invalid_type`, `invalid_value`, `invalid_json`, +`unknown_key`) without string-matching messages. """ from __future__ import annotations import json import math -from collections.abc import Mapping, Sequence +from collections.abc import Callable, Mapping, Sequence from dataclasses import dataclass -from typing import Final, Literal, NoReturn, cast +from typing import Final, Literal, cast from typing_extensions import TypeIs @@ -28,7 +28,7 @@ from zarr_metadata.v3.array import ZarrV3ArrayMetadataJSON from zarr_metadata.v3.group import ZarrV3GroupMetadataJSON -ProblemKind = Literal["missing_key", "invalid_type", "invalid_value", "invalid_json"] +ProblemKind = Literal["missing_key", "invalid_type", "invalid_value", "invalid_json", "unknown_key"] """Machine-readable classification of a `ValidationProblem`. - `missing_key`: a required key (document key or store key) is absent. @@ -37,6 +37,19 @@ - `invalid_value`: a value has an acceptable type but an invalid content (e.g. `zarr_format: 2` in a v3 document, `order: "Q"`). - `invalid_json`: bytes that do not decode as JSON. +- `unknown_key`: a member this package does not model appears inside an + entity whose shape it does model (e.g. an extra key in a `blosc` + configuration). Whether a `configuration` is closed is unspecified + (zarr-developers/zarr-specs#270 has been open since 2023), and this + package takes the strict reading: in practice such a member is a typo, + or a setting meant for a different entity, and accepting it silently + means silently ignoring what the writer asked for. Judging it belongs + to the rules layer, so `rules.parse_*` and the whole-document pydantic + field types reject it while `model.parse_*` — which never interpreted + configurations — accepts it. It gets a kind of its own so that a caller + who wants the tolerant reading can collect problems with + `rules.validate_*` and filter, and so that it never masks the other + findings about the same entity. """ @@ -44,8 +57,11 @@ class ValidationProblem: """A single structural problem found while validating a metadata document. - `loc` is the path from the document root to the offending value, e.g. - `("codecs", 0, "name")`. An empty `loc` refers to the document as a whole. + `loc` is the path from the root of what was judged to the offending + value, e.g. `("codecs", 0, "name")`, and an empty `loc` refers to that + root. The root is the document for the validators, the one field for + a scope's `coerce`, and the configuration for an entity's rules and + its constructor. `kind` classifies the failure mode for programmatic dispatch; `message` is the human-readable description. """ @@ -71,6 +87,18 @@ class MetadataValidationError(ValueError): def __init__(self, problems: Sequence[ValidationProblem]) -> None: self.problems = tuple(problems) + for entry in self.problems: + # The type says so; the check is for the trap the type cannot + # close: `problem()` returns a one-element tuple, and a list + # of those passes here and fails far away, where a `loc` is + # read off it. + if not isinstance(entry, ValidationProblem): # pyright: ignore[reportUnnecessaryIsInstance] + msg = ( + f"MetadataValidationError takes ValidationProblem values, got " + f"{type(entry).__name__}; `problem()` returns a tuple of them, so collect " + "with `extend`, not `append`" + ) + raise TypeError(msg) super().__init__("\n".join(str(problem) for problem in self.problems)) @@ -83,8 +111,16 @@ def _prefix( def validate_json(value: object) -> tuple[ValidationProblem, ...]: """Return every reason `value` is not JSON-serializable (recursively).""" + return _json_problems(value, finite=True) + + +def _json_problems(value: object, *, finite: bool) -> tuple[ValidationProblem, ...]: + """Every reason `value` is not JSON, a non-finite number being one only when `finite`. + + `finite` is false for user data alone; see `refine_node_json`. + """ if isinstance(value, float): - if math.isfinite(value): + if not finite or math.isfinite(value): return () return (ValidationProblem((), f"non-finite float {value!r} is not JSON", "invalid_value"),) if isinstance(value, (str, int, bool)) or value is None: @@ -97,32 +133,153 @@ def validate_json(value: object) -> tuple[ValidationProblem, ...]: ValidationProblem((), f"non-string key {key!r} in JSON object", "invalid_type") ) continue - problems.extend(_prefix(key, validate_json(item))) + problems.extend(_prefix(key, _json_problems(item, finite=finite))) return tuple(problems) if isinstance(value, Sequence) and not isinstance(value, (bytes, bytearray)): for index, item in enumerate(cast("Sequence[object]", value)): - problems.extend(_prefix(index, validate_json(item))) + problems.extend(_prefix(index, _json_problems(item, finite=finite))) return tuple(problems) return (ValidationProblem((), f"not a JSON-serializable value: {value!r}", "invalid_type"),) -def _is_canonical_json(value: object) -> TypeIs[JSONValue]: - """Whether `value` already uses the concrete containers in `JSONValue`.""" +def refine_json( + value: object, loc: tuple[str | int, ...] = () +) -> tuple[JSONValue | None, tuple[ValidationProblem, ...]]: + """`value` as JSON with arrays as tuples, or None with every reason it is not JSON. + + The first layer of reading, which needs nothing but the value. One + walk normalizes and judges: a mapping becomes a `dict` with string + keys, a sequence a tuple, a float must be finite. Everything after + it takes `JSONValue` and normalizes nothing. A value that is not + JSON is None, with the problems located at the leaves that are not: + not JSON is the first verdict, and there is nothing to read. + """ + return _refine(value, loc, finite=True) + + +def refine_node_json( + value: object, loc: tuple[str | int, ...] = () +) -> tuple[JSONValue | None, tuple[ValidationProblem, ...]]: + """A node document refined as `refine_json` refines it, except for the user data it holds. + + A node's `attributes` are user data. The spec asks only that each be + a JSON value, no layer interprets one, and zarr-python writes them + with the defaults of Python's `json` module, so an attribute can hold + `NaN`, `Infinity` or `-Infinity` -- xarray's `_FillValue`, a CF + `missing_value` -- which RFC 8259 lacks. Such a number is read here + as the float it is. Wherever the spec interprets a value, it spells + those numbers as strings, so anywhere else a non-finite number is not + JSON, as in `refine_json`. The documents a v3 group's inline + `consolidated_metadata` holds are node documents too; a v2 document + is judged merged, its `.zattrs` as `attributes`. A value that is not + an object is refined as `refine_json` refines it. + """ + if not isinstance(value, Mapping): + return refine_json(value, loc) + return _refine_members(cast("Mapping[object, object]", value), loc, _node_member) + + +_Refined = tuple[JSONValue | None, tuple[ValidationProblem, ...]] + + +def _refine(value: object, loc: tuple[str | int, ...], *, finite: bool) -> _Refined: + """`refine_json`, a non-finite number being JSON unless `finite`.""" if isinstance(value, float): - return math.isfinite(value) + if not finite or math.isfinite(value): + return value, () + return None, ( + ValidationProblem(loc, f"non-finite float {value!r} is not JSON", "invalid_value"), + ) + if isinstance(value, (str, int, bool)) or value is None: + return value, () + if isinstance(value, Mapping): + return _refine_members( + cast("Mapping[object, object]", value), + loc, + lambda _key, item, at: _refine(item, at, finite=finite), + ) + if isinstance(value, Sequence) and not isinstance(value, (bytes, bytearray)): + entries: list[JSONValue] = [] + found_in_entries: list[ValidationProblem] = [] + for index, item in enumerate(cast("Sequence[object]", value)): + entry, found = _refine(item, (*loc, index), finite=finite) + found_in_entries.extend(found) + if len(found) == 0: + entries.append(entry) + return (tuple(entries) if len(found_in_entries) == 0 else None), tuple(found_in_entries) + return None, ( + ValidationProblem(loc, f"not a JSON-serializable value: {value!r}", "invalid_type"), + ) + + +def _refine_members( + value: Mapping[object, object], + loc: tuple[str | int, ...], + member: Callable[[str, object, tuple[str | int, ...]], _Refined], +) -> _Refined: + """An object refined member by member, in the order it was written, with `member`.""" + members: dict[str, JSONValue] = {} + problems: list[ValidationProblem] = [] + for key, item in value.items(): + if not isinstance(key, str): + problems.append( + ValidationProblem(loc, f"non-string key {key!r} in JSON object", "invalid_type") + ) + continue + refined, found = member(key, item, (*loc, key)) + problems.extend(found) + if len(found) == 0: + members[key] = refined + return (members if len(problems) == 0 else None), tuple(problems) + + +def _node_member(key: str, item: object, loc: tuple[str | int, ...]) -> _Refined: + """One member of a node document: its attributes are user data.""" + if key == "attributes": + return _refine(item, loc, finite=False) + if key == "consolidated_metadata" and isinstance(item, Mapping): + return _refine_members(cast("Mapping[object, object]", item), loc, _consolidated_member) + return _refine(item, loc, finite=True) + + +def _consolidated_member(key: str, item: object, loc: tuple[str | int, ...]) -> _Refined: + """One member of an inline consolidated envelope: its `metadata` maps paths to nodes.""" + if key == "metadata" and isinstance(item, Mapping): + return _refine_members( + cast("Mapping[object, object]", item), + loc, + lambda _path, node, at: refine_node_json(node, at), + ) + return _refine(item, loc, finite=True) + + +def _is_canonical_json(value: object, *, finite: bool = True) -> TypeIs[JSONValue]: + """Whether `value` already uses the concrete containers in `JSONValue`. + + A non-finite number counts only when `finite` is false; see + `_is_canonical_node_json`. + """ + if isinstance(value, float): + return not finite or math.isfinite(value) if isinstance(value, (str, int, bool)) or value is None: return True if isinstance(value, (list, tuple)): sequence = cast("list[object] | tuple[object, ...]", value) - return all(_is_canonical_json(item) for item in sequence) + return all(_is_canonical_json(item, finite=finite) for item in sequence) if isinstance(value, dict): mapping = cast("dict[object, object]", value) return all( - isinstance(key, str) and _is_canonical_json(item) for key, item in mapping.items() + isinstance(key, str) and _is_canonical_json(item, finite=finite) + for key, item in mapping.items() ) return False +def _is_canonical_node_json(value: object) -> bool: + """Whether a node document already uses `JSONValue`'s containers, and is JSON as `refine_node_json` reads one.""" + return _is_canonical_json(value, finite=False) and len(refine_node_json(value)[1]) == 0 + + def is_json(value: object) -> TypeIs[JSONValue]: """Whether `value` is a canonical JSON structure (recursively).""" return _is_canonical_json(value) @@ -483,9 +640,11 @@ def _validate_attributes(value: object) -> tuple[ValidationProblem, ...]: ("attributes",), "expected a mapping with string keys", "invalid_type" ), ) + # User data: a non-finite number is an attribute value (see + # `refine_node_json`). problems: list[ValidationProblem] = [] for key, item in cast("Mapping[str, object]", value).items(): - problems.extend(_prefix("attributes", _prefix(key, validate_json(item)))) + problems.extend(_prefix("attributes", _prefix(key, _json_problems(item, finite=False)))) return tuple(problems) @@ -509,6 +668,13 @@ def validate_array_metadata_v3(value: object) -> tuple[ValidationProblem, ...]: problems.extend(_validate_dim_sequence(doc, "shape")) if "fill_value" in doc: problems.extend(_prefix("fill_value", validate_json(doc["fill_value"]))) + # Every extension *point* must be understood: ignoring a codec gives + # wrong bytes just as surely as ignoring a data type gives wrong + # values. The spec names only the first three + # (https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L1571-L1578), + # which this package reads as an oversight rather than a licence. + # `must_understand: false` keeps its meaning where it has one: an + # unknown top-level extension *field*, which a reader really can skip. for key in ("data_type", "chunk_grid", "chunk_key_encoding"): if key in doc: problems.extend( @@ -530,7 +696,17 @@ def validate_array_metadata_v3(value: object) -> tuple[ValidationProblem, ...]: ) ) for index, entry in enumerate(cast("Sequence[object]", entries)): - problems.extend(_prefix(key, _prefix(index, validate_metadata_field_v3(entry)))) + problems.extend( + _prefix( + key, + _prefix( + index, + validate_metadata_field_v3( + entry, allow_must_understand_false=False + ), + ), + ) + ) if "attributes" in doc: problems.extend(_validate_attributes(doc["attributes"])) if "dimension_names" in doc: @@ -550,23 +726,15 @@ def validate_array_metadata_v3(value: object) -> tuple[ValidationProblem, ...]: ("dimension_names",), "expected items of str or None", "invalid_type" ) ) - elif _is_int_sequence(doc.get("shape")) and len(cast("Sequence[object]", names)) != len( - cast("Sequence[int]", doc["shape"]) - ): - problems.append( - ValidationProblem( - ("dimension_names",), - "expected one name per dimension of shape", - "invalid_value", - ) - ) + # Whether the names count matches shape's dimensionality is a + # composition judgment, owned by zarr_metadata.rules. return tuple(problems) def is_array_metadata_v3(value: object) -> TypeIs[ZarrV3ArrayMetadataJSON]: """Whether `value` is a structurally-valid v3 array metadata document.""" return ( - _is_canonical_json(value) + _is_canonical_node_json(value) and not validate_array_metadata_v3(value) and _is_canonical_array_metadata_v3(value) ) @@ -599,26 +767,10 @@ def validate_array_metadata_v2(value: object) -> tuple[ValidationProblem, ...]: # ARRAY_METADATA_STANDARD_KEYS_V2 are not problems. problems: list[ValidationProblem] = list(_missing_keys(ARRAY_METADATA_REQUIRED_KEYS_V2, doc)) problems.extend(_check_literal(doc, "zarr_format", 2)) - shape_problems = _validate_dim_sequence(doc, "shape") - chunks_problems = _validate_dim_sequence(doc, "chunks") - problems.extend(shape_problems) - problems.extend(chunks_problems) - if ( - len(shape_problems) == 0 - and len(chunks_problems) == 0 - and _is_int_sequence(doc.get("shape")) - and _is_int_sequence(doc.get("chunks")) - ): - shape = cast("Sequence[int]", doc["shape"]) - chunks = cast("Sequence[int]", doc["chunks"]) - if len(shape) != len(chunks): - problems.append( - ValidationProblem( - ("chunks",), - "expected the same number of dimensions as shape", - "invalid_value", - ) - ) + problems.extend(_validate_dim_sequence(doc, "shape")) + problems.extend(_validate_dim_sequence(doc, "chunks")) + # Whether chunks matches shape's dimensionality is a composition + # judgment, owned by zarr_metadata.rules. if "dtype" in doc and not _is_dtype_v2(doc["dtype"]): problems.append( ValidationProblem( @@ -674,7 +826,7 @@ def validate_array_metadata_v2(value: object) -> tuple[ValidationProblem, ...]: def is_array_metadata_v2(value: object) -> TypeIs[ZarrV2ArrayMetadataJSON]: """Whether `value` is a structurally-valid v2 array metadata document.""" return ( - _is_canonical_json(value) + _is_canonical_node_json(value) and not validate_array_metadata_v2(value) and _is_canonical_array_metadata_v2(value) ) @@ -786,7 +938,7 @@ def validate_group_metadata_v3(value: object) -> tuple[ValidationProblem, ...]: def is_group_metadata_v3(value: object) -> TypeIs[ZarrV3GroupMetadataJSON]: """Whether `value` is a structurally-valid v3 group metadata document.""" - return _is_canonical_json(value) and not validate_group_metadata_v3(value) + return _is_canonical_node_json(value) and not validate_group_metadata_v3(value) def parse_group_metadata_v3(value: object) -> ZarrV3GroupMetadataJSON: @@ -819,7 +971,7 @@ def validate_group_metadata_v2(value: object) -> tuple[ValidationProblem, ...]: def is_group_metadata_v2(value: object) -> TypeIs[ZarrV2GroupMetadataJSON]: """Whether `value` is a structurally-valid v2 group metadata document.""" - return _is_canonical_json(value) and not validate_group_metadata_v2(value) + return _is_canonical_node_json(value) and not validate_group_metadata_v2(value) def parse_group_metadata_v2(value: object) -> ZarrV2GroupMetadataJSON: @@ -831,9 +983,47 @@ def parse_group_metadata_v2(value: object) -> ZarrV2GroupMetadataJSON: return cast(ZarrV2GroupMetadataJSON, normalized) -def _reject_json_constant(constant: str) -> NoReturn: - """Reject the JavaScript constants accepted by Python's JSON decoder.""" - raise ValueError(f"non-standard JSON constant {constant!r}") +def stored_json_problems(key: str, value: object) -> tuple[ValidationProblem, ...]: + """Every reason `value` is not JSON that may be stored at `key`. + + RFC 8259, except in user data, where a non-finite number is the value + it is (see `refine_node_json`). Which parts are user data follows + from what the key stores: a `zarr.json` is a node document, a + `.zattrs` is user data throughout, a `.zmetadata` holds documents + keyed the same way, and a `.zarray` or `.zgroup` holds none. + """ + name = key.rsplit("/", 1)[-1] + if name == "zarr.json": + return refine_node_json(value)[1] + if name == ".zattrs": + return _json_problems(value, finite=False) + if name == ".zmetadata" and isinstance(value, Mapping): + return _consolidated_v2_problems(cast("Mapping[object, object]", value)) + return validate_json(value) + + +def _consolidated_v2_problems( + consolidated: Mapping[object, object], +) -> tuple[ValidationProblem, ...]: + """A `.zmetadata` document: each entry judged as the document its key names.""" + entries = consolidated.get("metadata") + if not isinstance(entries, Mapping): + return validate_json(consolidated) + problems = list( + validate_json({key: item for key, item in consolidated.items() if key != "metadata"}) + ) + for entry_key, entry in cast("Mapping[object, object]", entries).items(): + if not isinstance(entry_key, str): + problems.append( + ValidationProblem( + ("metadata",), f"non-string key {entry_key!r} in JSON object", "invalid_type" + ) + ) + continue + problems.extend( + _prefix("metadata", _prefix(entry_key, stored_json_problems(entry_key, entry))) + ) + return tuple(problems) def load_store_json(mapping: Mapping[str, bytes], key: str) -> object: @@ -844,8 +1034,10 @@ def load_store_json(mapping: Mapping[str, bytes], key: str) -> object: into typed positions silently. Narrow the result with a `parse_*`. Every ingestion failure surfaces as `MetadataValidationError`: a missing - store key is a `missing_key` problem and undecodable bytes are an - `invalid_json` problem, rather than leaking `KeyError` / + store key is a `missing_key` problem, undecodable bytes are an + `invalid_json` problem, and a non-finite number outside user data -- + which Python's decoder reads as a float -- is located where it was + written (`stored_json_problems`), rather than leaking `KeyError` / `json.JSONDecodeError` to callers. """ if key not in mapping: @@ -853,16 +1045,30 @@ def load_store_json(mapping: Mapping[str, bytes], key: str) -> object: [ValidationProblem((key,), "missing store key", "missing_key")] ) try: - return json.loads(mapping[key], parse_constant=_reject_json_constant) + value: object = json.loads(mapping[key]) except (UnicodeDecodeError, ValueError) as exc: raise MetadataValidationError( [ValidationProblem((key,), f"invalid JSON: {exc}", "invalid_json")] ) from exc + problems = stored_json_problems(key, value) + if len(problems) != 0: + raise MetadataValidationError(problems) + return value -def dump_store_json(value: object, *, indent: int | str | None = None) -> bytes: - """Encode a metadata document as strict RFC 8259 JSON bytes.""" - return json.dumps(value, indent=indent, allow_nan=False).encode("utf-8") +def dump_store_json(key: str, value: object, *, indent: int | str | None = None) -> bytes: + """Encode the document stored at `key` as JSON bytes. + + RFC 8259, except that a non-finite number in user data is written as + Python's `json` writes it (`NaN`, `Infinity`, `-Infinity`), which is + how zarr-python writes attributes. Anywhere else one is refused, as + `load_store_json` refuses it: a model built by hand is not validated, + so this is where a non-finite fill value would otherwise be written. + """ + problems = stored_json_problems(key, value) + if len(problems) != 0: + raise MetadataValidationError(problems) + return json.dumps(value, indent=indent, allow_nan=True).encode("utf-8") def arrays_to_tuples(obj: object) -> object: diff --git a/packages/zarr-metadata/src/zarr_metadata/pydantic.py b/packages/zarr-metadata/src/zarr_metadata/pydantic.py index 5b0c9e5b57..6488b47547 100644 --- a/packages/zarr-metadata/src/zarr_metadata/pydantic.py +++ b/packages/zarr-metadata/src/zarr_metadata/pydantic.py @@ -6,13 +6,17 @@ Each exported name is an `Annotated` field type over the corresponding core model class — the instances ARE the core classes, so values interoperate freely with non-pydantic code (equality, isinstance, nesting). Validation -delegates to the library: a raw document routes through `from_json` (the -single source of truth for structural validation and normalization, so -pydantic's field-level coercion can never bypass it), an existing model -instance passes through unchanged, and serialization emits the canonical -document via `to_json`. `MetadataValidationError` subclasses `ValueError`, -so a failed parse surfaces as a pydantic `ValidationError` carrying the -loc-annotated problem messages. +delegates to the library: a raw document is judged by `zarr_metadata.rules` +(structure and composition together) and then normalized through +`from_json`, so pydantic's field-level coercion can never bypass either +layer; an existing model instance passes through unchanged, and +serialization emits the canonical document via `to_json`. +`MetadataValidationError` subclasses `ValueError`, so a failed parse +surfaces as a pydantic `ValidationError` carrying the loc-annotated +problem messages. + +The v2 consolidated field type and the bare metadata-field type carry no +composition rules and are validated structurally. Usage: @@ -33,6 +37,7 @@ class ArrayManifest(BaseModel): from pydantic import BeforeValidator, InstanceOf, PlainSerializer from zarr_metadata import model as _model +from zarr_metadata import rules as _rules from zarr_metadata._pydantic_schema import ( ZarrV2ArrayMetadataJSON as _ZarrV2ArrayMetadataSchema, ) @@ -54,6 +59,8 @@ class ArrayManifest(BaseModel): from zarr_metadata._pydantic_schema import ( ZarrV3MetadataFieldJSON as _ZarrV3MetadataFieldSchema, ) +from zarr_metadata.model._validation import arrays_to_tuples, validate_consolidated_metadata_v3 +from zarr_metadata.v3._document import consolidated_entries_problems if TYPE_CHECKING: from collections.abc import Callable @@ -72,10 +79,33 @@ def coerce(value: object) -> _M: return coerce +def _judged_by( + validate: Callable[[object], tuple[_model.ValidationProblem, ...]], + parse: Callable[[object], _M], +) -> Callable[[object], _M]: + """`parse`, preceded by a whole-document judgment that raises on any problem.""" + + def judged(value: object) -> _M: + problems = validate(value) + if len(problems) != 0: + raise _model.MetadataValidationError(problems) + return parse(value) + + return judged + + +def _validate_consolidated_v3(value: object) -> tuple[_model.ValidationProblem, ...]: + normalized = arrays_to_tuples(value) + return validate_consolidated_metadata_v3(normalized) + consolidated_entries_problems(normalized) + + ZarrV3ArrayMetadata = Annotated[ InstanceOf[_model.ZarrV3ArrayMetadata], BeforeValidator( - _coerce_to(_model.ZarrV3ArrayMetadata, _model.ZarrV3ArrayMetadata.from_json), + _coerce_to( + _model.ZarrV3ArrayMetadata, + _judged_by(_rules.validate_array_metadata_v3, _model.ZarrV3ArrayMetadata.from_json), + ), json_schema_input_type=_ZarrV3ArrayMetadataSchema, ), PlainSerializer(_model.ZarrV3ArrayMetadata.to_json, return_type=_ZarrV3ArrayMetadataSchema), @@ -85,7 +115,10 @@ def coerce(value: object) -> _M: ZarrV2ArrayMetadata = Annotated[ InstanceOf[_model.ZarrV2ArrayMetadata], BeforeValidator( - _coerce_to(_model.ZarrV2ArrayMetadata, _model.ZarrV2ArrayMetadata.from_json), + _coerce_to( + _model.ZarrV2ArrayMetadata, + _judged_by(_rules.validate_array_metadata_v2, _model.ZarrV2ArrayMetadata.from_json), + ), json_schema_input_type=_ZarrV2ArrayMetadataSchema, ), PlainSerializer(_model.ZarrV2ArrayMetadata.to_json, return_type=_ZarrV2ArrayMetadataSchema), @@ -95,7 +128,10 @@ def coerce(value: object) -> _M: ZarrV3GroupMetadata = Annotated[ InstanceOf[_model.ZarrV3GroupMetadata], BeforeValidator( - _coerce_to(_model.ZarrV3GroupMetadata, _model.ZarrV3GroupMetadata.from_json), + _coerce_to( + _model.ZarrV3GroupMetadata, + _judged_by(_rules.validate_group_metadata_v3, _model.ZarrV3GroupMetadata.from_json), + ), json_schema_input_type=_ZarrV3GroupMetadataSchema, ), PlainSerializer(_model.ZarrV3GroupMetadata.to_json, return_type=_ZarrV3GroupMetadataSchema), @@ -105,7 +141,10 @@ def coerce(value: object) -> _M: ZarrV2GroupMetadata = Annotated[ InstanceOf[_model.ZarrV2GroupMetadata], BeforeValidator( - _coerce_to(_model.ZarrV2GroupMetadata, _model.ZarrV2GroupMetadata.from_json), + _coerce_to( + _model.ZarrV2GroupMetadata, + _judged_by(_rules.validate_group_metadata_v2, _model.ZarrV2GroupMetadata.from_json), + ), json_schema_input_type=_ZarrV2GroupMetadataSchema, ), PlainSerializer(_model.ZarrV2GroupMetadata.to_json, return_type=_ZarrV2GroupMetadataSchema), @@ -117,7 +156,7 @@ def coerce(value: object) -> _M: BeforeValidator( _coerce_to( _model.ZarrV3ConsolidatedMetadata, - _model.ZarrV3ConsolidatedMetadata.from_json, + _judged_by(_validate_consolidated_v3, _model.ZarrV3ConsolidatedMetadata.from_json), ), json_schema_input_type=_ZarrV3ConsolidatedMetadataSchema, ), diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/__init__.py b/packages/zarr-metadata/src/zarr_metadata/rules/__init__.py new file mode 100644 index 0000000000..597c5c8783 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/__init__.py @@ -0,0 +1,43 @@ +"""Validate structure and composition of Zarr metadata documents. + +`zarr_metadata.model` checks JSON structure. This module also checks +cross-field constraints such as fill-value compatibility, codec +ordering, and dimension counts. Its `validate_*` and `parse_*` functions +mirror the model API, and `canonicalize_array_metadata_v3` answers with +either the document in its simplest equivalent spelling or every reason +it is not valid. + +Rules target canonical metadata and may be stricter than readers that +coerce inputs. Unknown entity names are left unjudged. Known entities +must match their modeled shape; extra configuration keys produce an +`unknown_key` problem without suppressing other checks. Model +round-trips preserve those unmodeled members. +""" + +from zarr_metadata.rules._documents import ( + Canonical, + Invalid, + canonicalize_array_metadata_v3, + parse_array_metadata_v2, + parse_array_metadata_v3, + parse_group_metadata_v2, + parse_group_metadata_v3, + validate_array_metadata_v2, + validate_array_metadata_v3, + validate_group_metadata_v2, + validate_group_metadata_v3, +) + +__all__ = [ + "Canonical", + "Invalid", + "canonicalize_array_metadata_v3", + "parse_array_metadata_v2", + "parse_array_metadata_v3", + "parse_group_metadata_v2", + "parse_group_metadata_v3", + "validate_array_metadata_v2", + "validate_array_metadata_v3", + "validate_group_metadata_v2", + "validate_group_metadata_v3", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_documents.py b/packages/zarr-metadata/src/zarr_metadata/rules/_documents.py new file mode 100644 index 0000000000..68b9a7ff0f --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_documents.py @@ -0,0 +1,276 @@ +"""Whole-document validation and canonicalization: the door. + +These `validate_*` and `parse_*` functions mirror the model API but apply +both validation layers, and `canonicalize_array_metadata_v3` answers with +`Canonical[T] | Invalid`: the document in its simplest equivalent +spelling, or every reason it is not valid. The work is done by the +documents themselves -- `zarr_metadata.v3._document` and +`zarr_metadata.v2._document` -- and what this module decides is which +entities are in scope while it asks. + +There is deliberately no `is_*` counterpart: composition validity is +stricter than TypedDict membership, so a guard here could not narrow +honestly. Use `zarr_metadata.model.is_*` for type narrowing. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from dataclasses import dataclass +from typing import TYPE_CHECKING, Generic, Literal, TypeVar, cast + +from zarr_metadata.model._array import ZarrV3ArrayMetadata +from zarr_metadata.model._validation import ( + MetadataValidationError, + refine_node_json, +) +from zarr_metadata.model._validation import ( + validate_array_metadata_v2 as _validate_structure_v2, +) +from zarr_metadata.model._validation import ( + validate_group_metadata_v2 as _validate_group_structure_v2, +) +from zarr_metadata.model._validation import ( + validate_group_metadata_v3 as _validate_group_structure_v3, +) +from zarr_metadata.v2._document import array_problems_v2 +from zarr_metadata.v3._document import ( + group_problems_v3, + read_array_v3, + refine_array_v3, + well_formed_array_v3, +) +from zarr_metadata.v3._registry import CORE_AND_EXTENSIONS, Context + +if TYPE_CHECKING: + from collections.abc import Callable + + from zarr_metadata._common import JSONValue + from zarr_metadata.model._validation import ValidationProblem + from zarr_metadata.v2.array import ZarrV2ArrayMetadataJSON + from zarr_metadata.v2.group import ZarrV2GroupMetadataJSON + from zarr_metadata.v3.array import ZarrV3ArrayMetadataJSON + from zarr_metadata.v3.group import ZarrV3GroupMetadataJSON + + _StructuralValidator = Callable[[object], tuple[ValidationProblem, ...]] + _SemanticValidator = Callable[[Mapping[str, JSONValue]], tuple[ValidationProblem, ...]] + +DocumentT = TypeVar("DocumentT") + + +@dataclass(frozen=True, slots=True) +class Canonical(Generic[DocumentT]): + """A semantically valid document, in its simplest equivalent spelling.""" + + document: DocumentT + valid: Literal[True] = True + + +@dataclass(frozen=True, slots=True) +class Invalid: + """Every reason a document is not semantically valid; never empty.""" + + problems: tuple[ValidationProblem, ...] + valid: Literal[False] = False + + def __post_init__(self) -> None: + if len(self.problems) == 0: + msg = "Invalid requires at least one validation problem" + raise ValueError(msg) + + +def _no_semantics(document: Mapping[str, JSONValue]) -> tuple[ValidationProblem, ...]: + """v2 group documents carry no cross-field constraints.""" + return () + + +def _group_semantics_v3(context: Context) -> _SemanticValidator: + """The v3 group semantics, asked in `context`. + + A group's only semantic content is its inline consolidated children, + and those are array and group documents judged in the same scope. + """ + + def judge(document: Mapping[str, JSONValue]) -> tuple[ValidationProblem, ...]: + return group_problems_v3(document, context) + + return judge + + +def _judged( + value: object, structure: _StructuralValidator, semantics: _SemanticValidator +) -> tuple[JSONValue | None, tuple[ValidationProblem, ...]]: + """`value` refined to JSON, and its structural and semantic problems. + + The layers for a document without a codec pipeline: JSON first, + then the shape, then whatever the semantics need of an object. A + value that is not JSON is None with only that verdict. + """ + refined, problems = refine_node_json(value) + if refined is None: + return None, problems + problems = structure(refined) + if isinstance(refined, Mapping): + problems = (*problems, *semantics(cast("Mapping[str, JSONValue]", refined))) + return refined, tuple(problems) + + +def validate_array_metadata_v3( + value: object, *, context: Context = CORE_AND_EXTENSIONS +) -> tuple[ValidationProblem, ...]: + """Why `value` is not a valid v3 array document. + + Every structural problem, and every semantic problem that can be + determined. One member that cannot be read costs the *composition* + judgments about the entity holding it -- whether a shard's inner + shape divides the array it is handed cannot be answered by a shard + that could not be built -- so a document with two defects in one + configuration may need a second pass. The verdict is never affected. + + The three layers of reading, in order, each handing the next what it + needs: the value refined to JSON and judged for shape, the extension + points read in `context`, the whole refined against the array. Their + problems are reported together; a value that is not JSON gets only + that verdict. List-spelled documents (fresh `json.loads` output) are + judged at the canonical data level rather than rejected for their + spelling. + """ + document, problems = well_formed_array_v3(value) + if document is None: + return problems + array, found = read_array_v3(document, context) + _, composed = refine_array_v3(array) + return (*problems, *found, *composed) + + +def parse_array_metadata_v3( + value: object, *, context: Context = CORE_AND_EXTENSIONS +) -> ZarrV3ArrayMetadataJSON: + """Return `value` as a valid `ZarrV3ArrayMetadataJSON`, or raise. + + Normalizes JSON arrays to tuples, then raises a single + `MetadataValidationError` carrying every structural and composition + problem found. + """ + document, problems = well_formed_array_v3(value) + if document is not None: + array, found = read_array_v3(document, context) + _, composed = refine_array_v3(array) + problems = (*problems, *found, *composed) + if document is None or len(problems) != 0: + raise MetadataValidationError(problems) + return cast("ZarrV3ArrayMetadataJSON", document) + + +def validate_array_metadata_v2(value: object) -> tuple[ValidationProblem, ...]: + """Every reason `value` is not a valid v2 array document (merged form). + + JSON arrays are normalized to tuples before judgment, as in + `validate_array_metadata_v3`. + """ + return _judged(value, _validate_structure_v2, array_problems_v2)[1] + + +def parse_array_metadata_v2(value: object) -> ZarrV2ArrayMetadataJSON: + """Return `value` as a valid `ZarrV2ArrayMetadataJSON`, or raise. + + Normalizes JSON arrays to tuples, then raises a single + `MetadataValidationError` carrying every structural and composition + problem found. + """ + refined, problems = _judged(value, _validate_structure_v2, array_problems_v2) + if len(problems) != 0: + raise MetadataValidationError(problems) + return cast("ZarrV2ArrayMetadataJSON", refined) + + +def validate_group_metadata_v3( + value: object, *, context: Context = CORE_AND_EXTENSIONS +) -> tuple[ValidationProblem, ...]: + """Every reason `value` is not a valid v3 group document. + + Composition rules recurse into inline consolidated metadata, so a + consolidated child document invalid under its own rules is reported + here, at its path. + """ + return _judged(value, _validate_group_structure_v3, _group_semantics_v3(context))[1] + + +def parse_group_metadata_v3( + value: object, *, context: Context = CORE_AND_EXTENSIONS +) -> ZarrV3GroupMetadataJSON: + """Return `value` as a valid `ZarrV3GroupMetadataJSON`, or raise.""" + refined, problems = _judged(value, _validate_group_structure_v3, _group_semantics_v3(context)) + if len(problems) != 0: + raise MetadataValidationError(problems) + return cast("ZarrV3GroupMetadataJSON", refined) + + +def validate_group_metadata_v2(value: object) -> tuple[ValidationProblem, ...]: + """Every reason `value` is not a valid v2 group document (merged form). + + v2 group documents carry no composition constraints today, so this is + the structural judgment, offered here for a uniform read-side API. + """ + return _judged(value, _validate_group_structure_v2, _no_semantics)[1] + + +def parse_group_metadata_v2(value: object) -> ZarrV2GroupMetadataJSON: + """Return `value` as a valid `ZarrV2GroupMetadataJSON`, or raise.""" + refined, problems = _judged(value, _validate_group_structure_v2, _no_semantics) + if len(problems) != 0: + raise MetadataValidationError(problems) + return cast("ZarrV2GroupMetadataJSON", refined) + + +def canonicalize_array_metadata_v3( + document: object, *, context: Context = CORE_AND_EXTENSIONS +) -> Canonical[ZarrV3ArrayMetadataJSON] | Invalid: + """`document` in canonical form, or every reason it is not valid. + + Canonical means the simplest spelling with the same meaning, and the + document decides that for itself in `ArrayDocumentV3.canonical`: each + entity in its own canonical form, and `dimension_names` of nothing + but nulls omitted. Two properties are worth holding on to, and + `tests/rules/test_canonical.py` asserts both: canonicalizing twice + changes nothing further, and canonicalizing never changes a verdict. + + Takes any value, like `validate_array_metadata_v3`: a document the + model layer has not accepted is not an error -- the structural + problems come back with the semantic ones, and the result is + `Invalid` rather than a canonical document. The document is read + once: the entities that judge it are the entities that are rewritten. + + Tell the two apart with `result.valid is True` or + `isinstance(result, Invalid)`; pyright narrows the literal on a + comparison, not on `if result.valid`. + + An entity whose canonical form breaks its own rules raises + `MetadataValidationError` from here, as its constructor does: that + is a bug in the entity, not a verdict on the document, which was + valid. + """ + refined, problems = well_formed_array_v3(document) + if refined is None: + return Invalid(problems) + array, found = read_array_v3(refined, context) + _, composed = refine_array_v3(array) + problems = (*problems, *found, *composed) + if len(problems) != 0: + return Invalid(problems) + return Canonical(ZarrV3ArrayMetadata.from_json(array.canonical().to_json()).to_json()) + + +__all__ = [ + "Canonical", + "Invalid", + "canonicalize_array_metadata_v3", + "parse_array_metadata_v2", + "parse_array_metadata_v3", + "parse_group_metadata_v2", + "parse_group_metadata_v3", + "validate_array_metadata_v2", + "validate_array_metadata_v3", + "validate_group_metadata_v2", + "validate_group_metadata_v3", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/v2/_document.py b/packages/zarr-metadata/src/zarr_metadata/v2/_document.py new file mode 100644 index 0000000000..05a62a1f4e --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/v2/_document.py @@ -0,0 +1,45 @@ +"""What a v2 array document can get wrong beyond its shape. + +Deliberately small: the one cross-field constraint the package interprets +is that `chunks` and `shape` agree on dimensionality. v2 has no extension +mechanism, so there are no entities to ask -- this is the whole of it. +Fill-value/dtype consistency for v2 (NumPy dtype strings, base64 fills +for bytes dtypes) is a known follow-up, tracked in the package docs. +""" + +from __future__ import annotations + +from collections.abc import Sequence +from typing import TYPE_CHECKING, cast + +from zarr_metadata.model._validation import ValidationProblem + +if TYPE_CHECKING: + from collections.abc import Mapping + + +def _as_sequence(value: object) -> tuple[object, ...] | None: + """`value` as a tuple if it is a JSON array, else None.""" + if isinstance(value, str) or not isinstance(value, Sequence): + return None + return tuple(cast("Sequence[object]", value)) + + +def array_problems_v2(document: Mapping[str, object]) -> tuple[ValidationProblem, ...]: + """Every semantic problem in a v2 array document.""" + shape = _as_sequence(document.get("shape")) + chunks = _as_sequence(document.get("chunks")) + if shape is None or chunks is None or len(shape) == len(chunks): + return () + return ( + ValidationProblem( + ("chunks",), + "expected the same number of dimensions as shape", + "invalid_value", + ), + ) + + +__all__ = [ + "array_problems_v2", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/_chain.py b/packages/zarr-metadata/src/zarr_metadata/v3/_chain.py new file mode 100644 index 0000000000..2281e94889 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/v3/_chain.py @@ -0,0 +1,158 @@ +"""Walking a codec pipeline, handing each codec what reaches it. + +The spec orders a pipeline `array->array`* `array->bytes` `bytes->bytes`*, +and each array-to-array codec transforms what the next one sees. So a +codec's configuration is judged against the array that *reaches* it, not +against the document's top-level fields: `transpose` permutes the grid, +`cast_value` changes the element type, and a shard that follows either +one sees the transformed array. + +The walk produces the resolved pipeline: at each position the codec and +the array that reaches it, and for a codec that holds pipelines of its +own -- a shard's inner chunks and its index -- those pipelines refined +the same way. Validation is what the walk finds on the way. + +Propagation stops -- every later codec receives `None` -- at the +array-to-bytes boundary, where there is no array left, and after any +codec that cannot say what it does to the array: one whose name is out of +scope, or a modelled one with no transition. An unknown codec might +change anything, so declining is the only honest answer. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import TYPE_CHECKING + +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.v3._entity import ArrayArrayCodec, ArrayBytesCodec, CodecEntity, within + +if TYPE_CHECKING: + from collections.abc import Mapping, Sequence + + from zarr_metadata.v3._entity import Loc, Opaque + from zarr_metadata.v3._parts import ArrayParts + + +@dataclass(frozen=True, slots=True) +class PipelineStage: + """One position of a refined pipeline: the codec, and the array that reaches it.""" + + codec: CodecEntity | Opaque + incoming: ArrayParts | None + """What reaches this codec. + + None past the array->bytes boundary, where there is no array, and + after a codec that could not say what it does to one. + """ + inner: Mapping[str, Pipeline] + """The pipelines this codec holds, refined, by the member that holds each; empty for most.""" + + +@dataclass(frozen=True, slots=True) +class Pipeline: + """A codec pipeline refined against the array handed to it: what reaches each codec.""" + + stages: tuple[PipelineStage, ...] + + +def _stage(codec: CodecEntity) -> tuple[int, str]: + """Where in the pipeline a codec stands, as a rank and as the spec names it.""" + if isinstance(codec, ArrayArrayCodec): + return 0, "array->array" + if isinstance(codec, ArrayBytesCodec): + return 1, "array->bytes" + return 2, "bytes->bytes" + + +def _label(codec: CodecEntity | Opaque) -> str: + if isinstance(codec, CodecEntity): + return repr(type(codec).identifier) + return repr(codec.json) + + +def order_problems( + codecs: Sequence[CodecEntity | Opaque], loc: Loc +) -> tuple[ValidationProblem, ...]: + """Whether the pipeline is shaped the way the spec orders it. + + A codec out of scope is skipped: it imposes no ordering constraint, + and it makes the exactly-one-`array->bytes` count inconclusive, + because it might be the pipeline's own `array->bytes` stage. So that + count is only checked when every codec is in scope. + """ + problems: list[ValidationProblem] = [] + latest = -1 + array_bytes = 0 + for index, codec in enumerate(codecs): + if not isinstance(codec, CodecEntity): + continue + rank, stage = _stage(codec) + if rank < latest: + problems.append( + ValidationProblem( + (*loc, index), + f"{stage} codec {_label(codec)} may not " + "follow a later-stage codec in the pipeline", + "invalid_value", + ) + ) + latest = max(latest, rank) + if isinstance(codec, ArrayBytesCodec): + array_bytes += 1 + if array_bytes > 1: + problems.append( + ValidationProblem( + (*loc, index), + f"extra array->bytes codec {_label(codec)}: a pipeline has exactly one", + "invalid_value", + ) + ) + if array_bytes == 0 and all(isinstance(codec, CodecEntity) for codec in codecs): + problems.append( + ValidationProblem(loc, "codec pipeline has no array->bytes codec", "invalid_value") + ) + return tuple(problems) + + +def refine_pipeline( + codecs: Sequence[CodecEntity | Opaque], start: ArrayParts | None, loc: Loc +) -> tuple[Pipeline, tuple[ValidationProblem, ...]]: + """The pipeline refined against `start`, with every problem found on the way. + + `start` is what the first codec receives: the document's own array, + or a shard's inner chunk, or its index. Each codec is asked what it + cannot take of what reaches it, then what it hands on; a codec that + holds pipelines has them refined in turn, located under the member + that holds each. Ordering is judged first, over the whole pipeline. + """ + problems = list(order_problems(codecs, loc)) + stages: list[PipelineStage] = [] + incoming = start + for index, codec in enumerate(codecs): + if not isinstance(codec, CodecEntity): + # Out of scope: unjudged, and everything after it is too. + stages.append(PipelineStage(codec, incoming, {})) + incoming = None + continue + problems.extend(within((*loc, index), codec.incoming_problems(incoming))) + inner: dict[str, Pipeline] = {} + for member, (held, handed) in codec.inner_pipelines(incoming).items(): + pipeline, found = refine_pipeline(held, handed, (*loc, index, "configuration", member)) + inner[member] = pipeline + problems.extend(found) + stages.append(PipelineStage(codec, incoming, inner)) + incoming = ( + codec.transition(incoming) + if incoming is not None and isinstance(codec, ArrayArrayCodec) + else None + ) + return Pipeline(tuple(stages)), tuple(problems) + + +__all__ = [ + "Pipeline", + "PipelineStage", + "order_problems", + "refine_pipeline", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/_document.py b/packages/zarr-metadata/src/zarr_metadata/v3/_document.py new file mode 100644 index 0000000000..bfebfdf6c7 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/v3/_document.py @@ -0,0 +1,555 @@ +"""A whole v3 array document, read in three layers, each with what it needs. + +1. `well_formed_array_v3`: the value alone. JSON syntax and the + document's shape -- arrays as tuples, string keys, floats finite + outside the user's attributes, the keys a v3 array has and the shapes + their values take, the envelope of each extension point. +2. `read_array_v3`: a scope. Each extension point's name related to a + class in the `Context`, and the class handed the field: the + configuration parsed against its record, the rules asked, nested + entities read the same way. +3. `refine_array_v3`: the array. The fill value against the data type, + the grid against the shape, and the codec pipeline walked with what + reaches each codec, which is the resolved pipeline -- the array each + codec is handed, and a shard's inner pipelines refined the same way. + Validation is what the walk finds. + +Each layer hands the next a typed value and its problems; the next reads +what it can and never repeats the work of the one before. Nothing here +knows what `blosc` or `int32` or `rectilinear` is. A new extension is a +class and a registry entry, and this module does not change. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from dataclasses import dataclass, replace +from typing import TYPE_CHECKING, TypeVar, cast + +from zarr_metadata.model._array import must_understand_subset +from zarr_metadata.model._validation import ( + ARRAY_METADATA_STANDARD_KEYS_V3, + MetadataValidationError, + ValidationProblem, + refine_node_json, +) +from zarr_metadata.model._validation import ( + validate_array_metadata_v3 as validate_array_metadata_v3_structure, +) +from zarr_metadata.v3._chain import Pipeline, refine_pipeline +from zarr_metadata.v3._entity import ( + ChunkGridEntity, + ChunkKeyEncodingEntity, + CodecEntity, + DataTypeEntity, + MetadataEntity, + Opaque, + StorageTransformerEntity, + held_problems, + problem, + read_field, + within, +) +from zarr_metadata.v3._parts import ArrayParts, ChunkGrid +from zarr_metadata.v3._registry import CORE_AND_EXTENSIONS, Context + +if TYPE_CHECKING: + from collections.abc import Sequence + + from zarr_metadata._common import JSONValue + from zarr_metadata.v3._entity import Loc + from zarr_metadata.v3.array import ZarrV3ExtensionField + + +_EntityT = TypeVar("_EntityT", bound=MetadataEntity) + + +@dataclass(frozen=True, slots=True) +class ArrayDocumentV3: + """A v3 array document with its extension points read as entities: the second layer's value. + + A field that could not be read holds an `Opaque`, which carries the + JSON the document wrote and says whether the name was out of scope -- + an extension this reader does not model, which is not an error -- or + claimed and refused. Both are narrowable: every field is an exhaustive + two-case union. `refine_array_v3` takes it on to the third layer. + """ + + document: Mapping[str, JSONValue] + data_type: DataTypeEntity | Opaque + chunk_grid: ChunkGridEntity | Opaque + chunk_key_encoding: ChunkKeyEncodingEntity | Opaque + codecs: tuple[CodecEntity | Opaque, ...] + storage_transformers: tuple[StorageTransformerEntity | Opaque, ...] + + def __post_init__(self) -> None: + """Refuse a field holding anything but an entity of its kind or an `Opaque`. + + `read_array_v3` builds a document that holds what it says by + construction; this is the same guarantee for one built by hand, + located at the field. + """ + found = ( + *_as_object(self.document), + *held_problems(self.data_type, DataTypeEntity, ("data_type",)), + *held_problems(self.chunk_grid, ChunkGridEntity, ("chunk_grid",)), + *held_problems( + self.chunk_key_encoding, ChunkKeyEncodingEntity, ("chunk_key_encoding",) + ), + *( + entry + for index, codec in enumerate(self.codecs) + for entry in held_problems(codec, CodecEntity, ("codecs", index)) + ), + *( + entry + for index, transformer in enumerate(self.storage_transformers) + for entry in held_problems( + transformer, StorageTransformerEntity, ("storage_transformers", index) + ) + ), + ) + if len(found) != 0: + raise MetadataValidationError(found) + + def canonical(self) -> ArrayDocumentV3: + """This document in the simplest form that means the same thing. + + Each entity in its own canonical form and in its own spelling of + the envelope -- the bare name when nothing is configured, no + `must_understand`, which means what absence means -- and the one + rule that is the document's own: `dimension_names` of nothing + but nulls says what omitting the field says. A *transformation*, + asked for by `canonicalize_array_metadata_v3`; `to_json` does + not apply it. What comes back is a document written that way, so + writing it changes nothing further. + """ + simplified = replace( + self, + data_type=self.data_type.canonical(), + chunk_grid=self.chunk_grid.canonical(), + chunk_key_encoding=self.chunk_key_encoding.canonical(), + codecs=tuple(codec.canonical() for codec in self.codecs), + storage_transformers=tuple(entry.canonical() for entry in self.storage_transformers), + ) + document = {**self.document, **_rendered(simplified)} + names = document.get("dimension_names") + if isinstance(names, tuple) and all( + entry is None for entry in cast("tuple[object, ...]", names) + ): + del document["dimension_names"] + return replace(simplified, document=document) + + def to_json(self) -> dict[str, JSONValue]: + """The document as it was written, with each entity's members as the entity has them. + + Faithful: a document read and written comes out as it went in, + an entity's envelope included -- `{"name": "crc32c"}` stays an + object, an empty `configuration` and a `must_understand` of + `true` stay written -- because the document knows the spelling + it read and puts it back around what the entity writes. What + changed is what changes: a member replaced through + `with_configuration` is written as the entity now has it, and an + entity put in by hand is written as it writes itself. A field + the document did not have is not invented, and one it wrote as + something no entity could be read from stands as written. Ask + `canonical` first for the simplest equivalent spelling. + """ + return { + **self.document, + **{ + key: cast("JSONValue", _as_written(self.document[key], value)) + for key, value in _rendered(self).items() + }, + } + + @property + def must_understand_fields(self) -> dict[str, ZarrV3ExtensionField]: + """The fields outside the spec's that do not say `must_understand: false`. + + A reader must refuse to open the array if this holds a field it + does not recognize. Recognition is the reader's own knowledge: a + top-level field is no extension point, so no scope claims one, + and the document partitions by obligation and leaves the verdict + to its reader, as `ZarrV3ArrayMetadata.must_understand_fields` + does. An extension point the scope does not claim is the same + question asked of an `Opaque`. + """ + extra = { + key: value + for key, value in self.document.items() + if key not in ARRAY_METADATA_STANDARD_KEYS_V3 + } + return must_understand_subset(cast("Mapping[str, ZarrV3ExtensionField]", extra)) + + @classmethod + def from_json(cls, value: object, *, context: Context = CORE_AND_EXTENSIONS) -> ArrayDocumentV3: + """A v3 array document read into entities, or raise. + + The reader's front door, and the one entry point that fails fast: + all three layers, and either every extension point is read and + the whole composes, or a single `MetadataValidationError` carries + every reason it does not -- structural and semantic together. Use + `validate_array_metadata_v3` instead when you want the problems + as data, and the layers themselves when you want to stop between + them. + + A name this `context` does not model is *not* a failure. It comes + back as an `Opaque` marked `out_of_scope`, because a document may + legitimately use an extension this reader does not know, and + refusing it would make openness unimplementable. What fails is + metadata that is wrong, not metadata that is unfamiliar. + """ + document, problems = well_formed_array_v3(value) + if document is not None: + # Read whatever the shape allowed, so a structural problem does + # not hide the semantic ones behind it. + array, found = read_array_v3(document, context) + _, composed = refine_array_v3(array) + problems = (*problems, *found, *composed) + if len(problems) == 0: + return array + raise MetadataValidationError(problems) + + +@dataclass(frozen=True, slots=True) +class RefinedArrayV3: + """A v3 array document refined against its own array: the third layer's value. + + `parts` is the array the codec pipeline is handed -- its chunks, + under its grid, of its data type -- and `pipeline` is that pipeline + resolved: at each position the codec and what reaches it, a shard's + inner pipelines refined inside it. What a codec pipeline is built + from, and what validating the composition finds on the way. + """ + + array: ArrayDocumentV3 + parts: ArrayParts + pipeline: Pipeline + + +def well_formed_array_v3( + value: object, +) -> tuple[Mapping[str, JSONValue] | None, tuple[ValidationProblem, ...]]: + """The first layer: `value` as a refined v3 array document, with every structural problem. + + Needs nothing but the value. The JSON is refined -- arrays as + tuples, string keys, floats finite except in the attributes, which + are user data (`refine_node_json`) -- and the document's shape is + judged by the model layer: the keys a v3 array has, the shapes their + values take, the envelope of each extension point. What comes back + is refined JSON that the next layer reads without normalizing or + judging JSON-ness again, and the structural problems beside it, which + do not stop the next layer from reading what it can. A value that is + not JSON, or not an object, is None with the reasons: not JSON is + the first verdict, and there is nothing to read. + """ + refined, problems = refine_node_json(value) + if refined is None: + return None, problems + if not isinstance(refined, Mapping): + return None, problem((), f"expected a v3 array document as an object, got {refined!r}") + document = cast("Mapping[str, JSONValue]", refined) + return document, validate_array_metadata_v3_structure(document) + + +def read_array_v3( + document: Mapping[str, JSONValue], context: Context +) -> tuple[ArrayDocumentV3, tuple[ValidationProblem, ...]]: + """The second layer: `document`'s extension points, read in `context`. + + Needs a scope. The one place that knows which of a document's fields + holds which kind of entity; each is handed to `read_field`, its + envelope having been judged with the document. Type-space only: what + comes back is well-typed by construction, and the problems are the + reasons some of it is not an entity. + """ + data_type, found_1 = _read_one(context, DataTypeEntity, document, "data_type") + chunk_grid, found_2 = _read_one(context, ChunkGridEntity, document, "chunk_grid") + encoding, found_3 = _read_one(context, ChunkKeyEncodingEntity, document, "chunk_key_encoding") + codecs, found_4 = _read_each(context, CodecEntity, document, "codecs") + transformers, found_5 = _read_each( + context, StorageTransformerEntity, document, "storage_transformers" + ) + return ( + ArrayDocumentV3( + document=document, + data_type=data_type, + chunk_grid=chunk_grid, + chunk_key_encoding=encoding, + codecs=codecs, + storage_transformers=transformers, + ), + (*found_1, *found_2, *found_3, *found_4, *found_5), + ) + + +def refine_array_v3(array: ArrayDocumentV3) -> tuple[RefinedArrayV3, tuple[ValidationProblem, ...]]: + """The third layer: `array` against its own array, and the pipeline resolved. + + Needs the array: the fill value is judged by the data type it fills, + the grid against the shape it divides, the dimension names counted + against it, and the codec pipeline walked from the parts the grid + and data type make, each codec handed what reaches it. A pipeline + the document did not write as an array was not read, and is not + judged as an empty one. + """ + parts = _parts(array) + pipeline, composed = ( + refine_pipeline(array.codecs, parts, ("codecs",)) + if _listed(array.document, "codecs") is not None + else (Pipeline(()), ()) + ) + problems = ( + *_fill_value_problems(array), + *_grid_problems(array), + *_dimension_names_problems(array), + *composed, + ) + return RefinedArrayV3(array, parts, pipeline), problems + + +def _parts(array: ArrayDocumentV3) -> ArrayParts: + """The array the codec pipeline is handed.""" + shape = array.document.get("shape") + # A grid out of scope still divides an array of some rank, and the + # shape is what pins it -- which is enough to catch a shard whose + # inner chunk has the wrong number of dimensions. + grid = ( + array.chunk_grid.grid(shape) + if isinstance(array.chunk_grid, ChunkGridEntity) + else ChunkGrid.unreadable(shape) + ) + return ArrayParts( + grid, array.data_type if isinstance(array.data_type, DataTypeEntity) else None + ) + + +def _as_object(value: object) -> tuple[ValidationProblem, ...]: + """Why `value` is not the document as an object, which is what the reader read it as.""" + if isinstance(value, Mapping): + return () + return problem((), f"expected the document as an object, got {value!r}") + + +def _rendered(array: ArrayDocumentV3) -> dict[str, JSONValue]: + """Each entity field the document has, as its entities write it; one nothing was read from is left out.""" + rendered: dict[str, JSONValue] = {} + for key, entity in ( + ("data_type", array.data_type), + ("chunk_grid", array.chunk_grid), + ("chunk_key_encoding", array.chunk_key_encoding), + ): + if key in array.document: + rendered[key] = entity.to_json() + for key, entities in ( + ("codecs", array.codecs), + ("storage_transformers", array.storage_transformers), + ): + if _listed(array.document, key) is not None: + rendered[key] = tuple(entity.to_json() for entity in entities) + return rendered + + +_ENVELOPE_KEYS = frozenset({"name", "configuration"}) +"""What an entity writes around its members; anything else around a name is the document's.""" + + +def _as_written(original: object, rendered: object) -> object: + """`rendered`, an entity's JSON, in the spelling `original`, the document's JSON at the same place, used. + + The envelope's writer, the counterpart of `named_configuration`. An + entity writes its members and its own spelling of the envelope, + since it has no document to be faithful to; the document has, so + the spellings that mean the same come back as they were written: + the object around a bare name, a `configuration` of nothing, a + `must_understand` of `true`. The two trees are walked together, so + a codec inside a shard is dressed as one in the pipeline is. Where + they disagree -- an entity put in or taken out by hand, a name + changed -- the rendered value stands, members and all. + """ + nothing: dict[str, object] = {} + if isinstance(original, Mapping): + before = cast("Mapping[str, object]", original) + if isinstance(rendered, str): + if before.get("name") == rendered: + return { + key: nothing if key == "configuration" else value + for key, value in before.items() + } + return rendered + if isinstance(rendered, Mapping): + after = cast("Mapping[str, object]", rendered) + written = {key: _as_written(before.get(key), value) for key, value in after.items()} + if ( + after.keys() <= _ENVELOPE_KEYS + and "name" in after + and before.get("name") == after["name"] + ): + for key, value in before.items(): + if key not in written: + written[key] = nothing if key == "configuration" else value + return written + if isinstance(original, (list, tuple)) and isinstance(rendered, (list, tuple)): + before = cast("Sequence[object]", original) + after = cast("Sequence[object]", rendered) + if len(before) != len(after): + return tuple(after) + return tuple(_as_written(entry, value) for entry, value in zip(before, after, strict=True)) + return rendered + + +def _listed(document: Mapping[str, JSONValue], key: str) -> Sequence[JSONValue] | None: + """What the document lists at `key`; None if it wrote no array there.""" + entries = document.get(key) + return cast("Sequence[JSONValue]", entries) if isinstance(entries, (list, tuple)) else None + + +def _read_one( + context: Context, kind: type[_EntityT], document: Mapping[str, JSONValue], key: str +) -> tuple[_EntityT | Opaque, tuple[ValidationProblem, ...]]: + """The entity of `kind` the document names at `key`; an `Opaque` if it names none.""" + value = document.get(key) + if value is None: + return Opaque.create_unchecked(None, "invalid"), () + return read_field(value, kind, context, (key,)) + + +def _read_each( + context: Context, kind: type[_EntityT], document: Mapping[str, JSONValue], key: str +) -> tuple[tuple[_EntityT | Opaque, ...], tuple[ValidationProblem, ...]]: + """The entities of `kind` the document lists at `key`, in order.""" + entries = _listed(document, key) + if entries is None: + return (), () + read: list[_EntityT | Opaque] = [] + problems: list[ValidationProblem] = [] + for index, entry in enumerate(entries): + entity, found = read_field(entry, kind, context, (key, index)) + read.append(entity) + problems.extend(found) + return tuple(read), tuple(problems) + + +def _fill_value_problems(array: ArrayDocumentV3) -> tuple[ValidationProblem, ...]: + """The fill value, judged by the data type it fills.""" + if not isinstance(array.data_type, DataTypeEntity) or "fill_value" not in array.document: + return () + return array.data_type.fill_value_problems(array.document["fill_value"], ("fill_value",)) + + +def _dimension_names_problems(array: ArrayDocumentV3) -> tuple[ValidationProblem, ...]: + """One name per dimension, if names are given at all.""" + names = array.document.get("dimension_names") + shape = array.document.get("shape") + if not isinstance(names, (list, tuple)) or not isinstance(shape, (list, tuple)): + return () + given = len(cast("Sequence[object]", names)) + rank = len(cast("Sequence[object]", shape)) + if given == rank: + return () + return ( + ValidationProblem( + ("dimension_names",), + f"dimension_names has {given} entries but shape has {rank} dimensions", + "invalid_value", + ), + ) + + +def _grid_problems(array: ArrayDocumentV3) -> tuple[ValidationProblem, ...]: + """The chunk grid, judged against the array it divides.""" + if not isinstance(array.chunk_grid, ChunkGridEntity): + return () + return within(("chunk_grid",), array.chunk_grid.shape_problems(array.document.get("shape"))) + + +def array_problems_v3( + document: Mapping[str, JSONValue], context: Context +) -> tuple[ValidationProblem, ...]: + """The second and third layers' problems for a refined document, together. + + For a document the first layer has refined; a consolidated child is + one. + """ + array, problems = read_array_v3(document, context) + _, composed = refine_array_v3(array) + return (*problems, *composed) + + +def _prefixed(loc: Loc, problems: Sequence[ValidationProblem]) -> tuple[ValidationProblem, ...]: + """Re-base every problem's `loc` under `loc`, for a nested document.""" + return tuple( + ValidationProblem((*loc, *found.loc), found.message, found.kind) for found in problems + ) + + +def _as_string_mapping(value: object) -> Mapping[str, JSONValue] | None: + """`value` as a string-keyed mapping, or None if it is not one.""" + if not isinstance(value, Mapping): + return None + mapping = cast("Mapping[object, object]", value) + if any(not isinstance(key, str) for key in mapping): + return None + return cast("Mapping[str, JSONValue]", mapping) + + +def group_problems_v3( + document: Mapping[str, JSONValue], context: Context = CORE_AND_EXTENSIONS +) -> tuple[ValidationProblem, ...]: + """Every semantic problem in a v3 group document. + + A group says almost nothing that can be wrong on its own. The one + thing it can carry is consolidated metadata -- the child documents of + a whole subtree, inline -- and each of those is judged exactly as it + would be standing alone, at its own path. `consolidated_metadata` is + not a declared member of the group TypedDict: the spec grandfathers + it as a convention that "lacks the name member required of extension + objects". + """ + if "consolidated_metadata" not in document: + return () + return consolidated_entries_problems( + document["consolidated_metadata"], ("consolidated_metadata",), context + ) + + +def consolidated_entries_problems( + value: object, loc: Loc = (), context: Context = CORE_AND_EXTENSIONS +) -> tuple[ValidationProblem, ...]: + """Semantic problems in an inline consolidated envelope's children. + + Structural validity of the envelope and its entries is the model + layer's job; an entry that is not interpretable as a node document + declines in its favour. + """ + consolidated = _as_string_mapping(value) + if consolidated is None: + return () + metadata = _as_string_mapping(consolidated.get("metadata")) + if metadata is None: + return () + problems: list[ValidationProblem] = [] + for path, entry in metadata.items(): + node = _as_string_mapping(entry) + if node is None: + continue + entry_loc = (*loc, "metadata", path) + node_type = node.get("node_type") + if node_type == "array": + problems.extend(_prefixed(entry_loc, array_problems_v3(node, context))) + elif node_type == "group": + problems.extend(_prefixed(entry_loc, group_problems_v3(node, context))) + return tuple(problems) + + +__all__ = [ + "ArrayDocumentV3", + "RefinedArrayV3", + "array_problems_v3", + "consolidated_entries_problems", + "group_problems_v3", + "read_array_v3", + "refine_array_v3", + "well_formed_array_v3", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/_entity.py b/packages/zarr-metadata/src/zarr_metadata/v3/_entity.py new file mode 100644 index 0000000000..3632740fd4 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/v3/_entity.py @@ -0,0 +1,1139 @@ +"""What every metadata entity can do for itself. + +A codec, data type, chunk grid, chunk key encoding or storage transformer +is one frozen dataclass whose fields are its configuration, and the +field annotations are its schema: `coerce` reads a document's +configuration against them, member by member, with `_typed_json`. A +field typed `CodecEntity | Opaque` holds another entity, read through +the scope the containing one is read in. Everything finer than a type -- +a bound, a rule about a member, members read together -- is the +record's own `problems`, which yields them as it finds them; the +constructor stops at the first, `coerce` reports every one, and a +reader may ask a record before building anything. + +What an entity writes follows from the same fields: `to_json` is +written once here, the parser's inverse over each field's annotation. +What it simplifies to is its own: `canonical` defaults to the entity +itself, and an entity that contains entities canonicalizes them there. + +Composition -- what needs the document or the codec chain -- is the +entity's to answer through `incoming_problems`, `shape_problems`, +`fill_value_problems`, `transition` and `grid`, each taking the part of +the document it needs. The document that composes those answers is +`_document`; which entities are in scope is `_registry`, which refuses, +at registration, an entity `coerce` could not read. +""" + +from __future__ import annotations + +import functools +from abc import ABC, abstractmethod +from collections.abc import Callable, Mapping +from dataclasses import dataclass, is_dataclass, replace +from typing import ( + TYPE_CHECKING, + ClassVar, + Final, + Literal, + TypeAlias, + TypeVar, + cast, + get_args, +) + +from zarr_metadata.model._sentinel import UNSET +from zarr_metadata.model._validation import ( + MetadataValidationError, + ValidationProblem, + is_json, + refine_json, + validate_metadata_field_v3, +) +from zarr_metadata.v3._typed_json import ( + Loc, + Parsed, + Parser, + RecordWriter, + Writer, + declared_class_vars, + field_hints, + is_integer, + is_optional, + is_union, + members_of, + parser, + parser_for, + problem, + record_of, + record_writer, + strip_annotation, + without_unset, +) + +if TYPE_CHECKING: + from collections.abc import Iterator, Sequence + from typing import Self + + from zarr_metadata._common import JSONValue + from zarr_metadata.v3._common import ZarrV3MetadataFieldJSON + from zarr_metadata.v3._parts import ArrayParts, ChunkGrid + from zarr_metadata.v3._registry import Context + +EntityT = TypeVar("EntityT", bound="MetadataEntity") + +# A real alias, not a string one, so a return annotation can subscript +# it -- `Coerced[Self]` -- without deferring annotation evaluation. +Coerced: TypeAlias = tuple[EntityT | None, tuple[ValidationProblem, ...]] +"""The entity if it could be built, and every problem found. + +One direction holds: no entity means at least one problem. The converse +does not -- a survivable problem, an unknown key, comes back *with* the +entity, because the entity is still readable and saying so is more +useful than refusing. A member of the wrong type is not survivable: +the entity's rules are written over a whole configuration, and an +entity is never built around a hole. + +So test `entity is None` to decide whether to go on reading, and test the +problems to decide the verdict. They are different questions. +""" + + +StorageClass = Literal["single_byte", "multi_byte", "variable_length"] +"""How one scalar of a data type occupies bytes. + +`single_byte` and `multi_byte` are both fixed-size; they differ only in +whether a byte order applies, which is what the `bytes` codec's `endian` +member is about. +""" + + +class _FromName: + """The marker behind `FROM_NAME`.""" + + __slots__ = () + + def __repr__(self) -> str: + return "FROM_NAME" + + +FROM_NAME: Final = _FromName() +"""Marks a field carried by the metadata envelope's `name`, not its configuration. + + data_type_name: Annotated[str, FROM_NAME] + +A member all the same -- `__post_init__` judges it -- but not a +configuration key, so it is neither read from nor written to a +`configuration` object. The raw-bytes family is the case: `r` keeps its +width in its name and has no configuration at all. +""" + + +def is_from_name(annotation: object) -> bool: + """Whether `FROM_NAME` marks the field: carried by the envelope's name, not a configuration key.""" + return any(entry is FROM_NAME for entry in strip_annotation(annotation)[1]) + + +def within(prefix: Loc, problems: Sequence[ValidationProblem]) -> tuple[ValidationProblem, ...]: + """One entity's problems, located in the document that holds it. + + An entity reports relative to its own `configuration`, so that is what + goes between the field and the member. A problem with an empty + location is about the entity itself -- a malformed `r` name, a + codec that cannot encode what reaches it -- and lands on the field. + """ + return tuple( + ValidationProblem( + (*prefix, *(("configuration", *found.loc) if len(found.loc) != 0 else ())), + found.message, + found.kind, + ) + for found in problems + ) + + +def named_configuration( + value: object, +) -> tuple[str | None, Mapping[str, object] | None, tuple[ValidationProblem, ...]]: + """Split metadata into `(name, configuration, problems)`. + + The shared shape every entity arrives in: a bare name, or an object + carrying one. A `None` name means the value is not a metadata field + at all; a `None` configuration means the bare spelling was used, or + the key was left out. A configuration that is present and not an + object is the one problem reported, at `("configuration",)`. + """ + if isinstance(value, str): + return value, None, () + if not isinstance(value, Mapping): + return None, None, () + entry = cast("Mapping[str, object]", value) + name = entry.get("name") + if not isinstance(name, str): + return None, None, () + if "configuration" not in entry: + return name, None, () + configuration = entry["configuration"] + if not isinstance(configuration, Mapping): + return name, None, problem(("configuration",), f"expected an object, got {configuration!r}") + return name, cast("Mapping[str, object]", configuration), () + + +def is_metadata_field(value: object, loc: Loc) -> tuple[ValidationProblem, ...]: + """A nested metadata field: a bare name or a named-configuration object. + + Only the envelope's shape. Which entity the name denotes, and whether + its configuration is well formed, is settled when the containing + entity reads it in scope. + """ + if not isinstance(value, (str, Mapping)): + return problem(loc, f"expected a metadata field, got {value!r}") + return () + + +@dataclass(frozen=True, slots=True) +class Opaque: + """A metadata field this reading did not turn into an entity. + + Carries the JSON the document wrote, so a reader holding a + `CodecEntity | Opaque` has everything the document said in either + case, and `reason` says which case it is. `out_of_scope` is a name no + entity in this `Context` claims -- an extension this reader does not + model, which is not an error and is the reader's cue to resolve it + elsewhere. `invalid` is a name that *was* claimed and then refused, + for the reasons reported alongside. + + Answers `to_json` and `canonical` as an entity does, so a field typed + `CodecEntity | Opaque` is written and simplified without asking + which case it holds. + + Built by the reader through `create_unchecked`, from JSON it has + refined; the constructor checks one built by hand. + """ + + json: JSONValue + reason: Literal["out_of_scope", "invalid"] + + def __post_init__(self) -> None: + """Refuse a reason that is not one of the reader's two, and a `json` that is not JSON.""" + reason: object = self.reason + found = ( + *( + () + if reason in ("out_of_scope", "invalid") + else problem(("reason",), f"expected 'out_of_scope' or 'invalid', got {reason!r}") + ), + *( + () + if is_json(self.json) + else problem(("json",), f"expected JSON, got {self.json!r}") + ), + ) + if len(found) != 0: + raise MetadataValidationError(found) + + @classmethod + def create_unchecked(cls, json: JSONValue, reason: Literal["out_of_scope", "invalid"]) -> Self: + """An `Opaque` built without the constructor's check, for the reader, whose JSON is refined.""" + opaque = object.__new__(cls) + object.__setattr__(opaque, "json", json) + object.__setattr__(opaque, "reason", reason) + return opaque + + def to_json(self) -> ZarrV3MetadataFieldJSON: + """The JSON the document wrote, as it wrote it. + + An `Opaque` inside a built entity is out of scope -- an inner + name no entity in scope claimed -- and its JSON passed the + envelope check as a metadata field, which is what the cast says. + """ + return cast("ZarrV3MetadataFieldJSON", self.json) + + def canonical(self) -> Self: + """Itself: what was not read cannot be simplified.""" + return self + + +def unreadable(cls: type[MetadataEntity]) -> str | None: + """Why `coerce` could not read an instance of `cls`; None if it can. + + An entity's fields are `configuration`, which an entity narrows to + its own `Configuration` record or defaults to the empty one, and at + most one field the envelope's name fills. Things that type-check + cleanly and then go wrong somewhere that will not name the class: a + field of any other name; a configuration that is not a + `Configuration`, or a member of it whose annotation is not a shape + JSON takes, or is itself a `Configuration`, whose rules nothing would + ask; a `__post_init__` of the entity's or its record's own, since the + rules go in `problems`; and a class variable a base annotates and + nothing sets -- `identifier` for every entity, `bounds` for an + integer type -- which the first lookup would fail. Registration asks, + and refuses the class with the answer. + """ + try: + hints = field_hints(cls) + except NameError as unresolved: + return _unresolved(cls, unresolved) + for name, annotation in hints.items(): + if name == "configuration" or is_from_name(annotation): + continue + return ( + f"{cls.__name__} declares a field {name!r}; an entity's fields are `configuration`, " + "a frozen dataclass of its members, and a name it carries marked FROM_NAME -- put " + f"{name!r} in the configuration record" + ) + record = hints["configuration"] + if not (isinstance(record, type) and issubclass(record, Configuration)): + return ( + f"{cls.__name__}: configuration is annotated {record!r}; annotate it with a frozen " + "dataclass subclassing Configuration, one field per configuration member" + ) + try: + members = field_hints(record) + except NameError as unresolved: + return _unresolved(record, unresolved) + if record.__post_init__ is not Configuration.__post_init__: + return ( + f"{record.__name__} defines __post_init__; write its rules as `problems`, " + "yielding each: the entity's constructor stops at the first, `coerce` reports " + "every one" + ) + unread: list[str] = [] + for name, annotation in members.items(): + inner = without_unset(strip_annotation(annotation)[0]) + if isinstance(inner, type) and issubclass(inner, Configuration): + return ( + f"{cls.__name__}: {name} is annotated {inner.__name__}, a Configuration, whose " + "rules nothing would ask; a member that is an object is a plain record " + "dataclass or a TypedDict, and its rules belong to the entity's configuration" + ) + try: + accepted = parser_for(annotation, _nested_field) is not None + except TypeError as refused: + return f"{cls.__name__}: {name} {refused}" + except NameError as unresolved: + return _unresolved(record, unresolved) + if not accepted: + unread.append(name) + if len(unread) != 0: + return ( + f"{cls.__name__}: " + f"{'; '.join(f'{name} is annotated {members[name]!r}' for name in unread)}" + ", which is not a shape JSON takes. A member is int, float, bool, str, JSONValue, " + "a Literal of names, tuple[T, ...] or tuple[T1, T2], a TypedDict or dataclass " + "record, Mapping[str, V], a NewType, or an entity kind with Opaque " + "(CodecEntity | Opaque); add | UNSET for an optional member, and put any finer " + "rule in the record's `problems`" + ) + if "__post_init__" in vars(cls): + return ( + f"{cls.__name__} defines __post_init__; write its rules as `problems` on its " + "Configuration record, yielding each: the constructor stops at the first, `coerce` " + "reports every one" + ) + annotated = declared_class_vars(cls) + missing = sorted(name for name in annotated if not hasattr(cls, name)) + if len(missing) != 0: + owed = ", ".join(f"{name} (annotated by {annotated[name].__name__})" for name in missing) + return f"{cls.__name__} does not declare {owed}; set each as a class variable" + return None + + +def _unresolved(cls: type, unresolved: NameError) -> str: + return ( + f"{cls.__name__}: a field annotation names {unresolved.name!r}, which is not " + "defined where the class is; define it at module level, or import it outside " + "`TYPE_CHECKING`" + ) + + +def nested_kind(annotation: object) -> type[MetadataEntity] | None: + """The kind an annotation of the form `Kind | Opaque` names; None if it names no entity. + + The one shape a field holding another entity takes, with `Opaque` + because that is what the field holds when the inner name is out of + scope, and a kind -- or a subclass of one, `GzipCodec` -- because a + scope resolves names by kind. An annotation naming an entity any + other way is a `TypeError` saying so, which registration reports + against the field. + """ + parts = get_args(annotation) if is_union(annotation) else (annotation,) + entities = [ + part for part in parts if isinstance(part, type) and issubclass(part, MetadataEntity) + ] + if len(entities) == 0: + return None + kind = entities[0] + if len(entities) == 1 and set(parts) == {kind, Opaque} and kind_of(kind) is not None: + return kind + msg = ( + "holds an entity but is not written as its kind | Opaque (CodecEntity | Opaque), " + "which is what the field holds when the inner name is out of scope; a kind is a " + "codec kind, DataTypeEntity, ChunkGridEntity, ChunkKeyEncodingEntity or " + "StorageTransformerEntity, or a subclass of one" + ) + raise TypeError(msg) + + +@dataclass(frozen=True, slots=True) +class _Reading: + """What one reading hands down into the fields that hold entities. + + The scope the inner entities are read in, and the problems found + inside them, kept apart from the containing entity's own: its rules + still run over its own members when only a contained entity is + wrong. + """ + + context: Context + nested: list[ValidationProblem] + + +def _nested_field(annotation: object) -> Parser[_Reading] | None: + """The parser for a field holding another entity: `Kind | Opaque`, read in the reading's scope. + + The entity layer's one shape of its own, asked by the parser at + every depth, so a struct's fields' data types and a shard's inner + pipelines are read the same way. The envelope's shape is the + containing field's own problem; what is found inside the entity + goes to the reading. + """ + kind = nested_kind(annotation) + if kind is None: + return None + + def parse(value: object, loc: Loc, reading: _Reading) -> Parsed: + problems = is_metadata_field(value, loc) + if len(problems) != 0: + return value, problems + # The parser hands refined JSON; its own type is `object` because + # the checker knows no JSON type. + entity, found = _resolve_field(cast("JSONValue", value), kind, reading.context, loc) + reading.nested.extend(found) + return entity, () + + return parse + + +def _nested_field_writer(annotation: object) -> Writer | None: + """The writer for a field holding another entity: what it holds, as it writes itself.""" + if nested_kind(annotation) is None: + return None + + def write(value: object) -> JSONValue: + if isinstance(value, (MetadataEntity, Opaque)): + return value.to_json() + msg = f"{value!r} is not an entity or an Opaque" + raise TypeError(msg) + + return write + + +def resolve( + data: object, + kind: type[EntityT], + context: Context, + loc: Loc = (), +) -> tuple[EntityT | Opaque, tuple[ValidationProblem, ...]]: + """`data`, one metadata field, read as an entity of `kind` in `context`. + + The reader for one field: the first two layers of reading a + document, applied to a field on its own. The first needs nothing but + the value -- `data` is refined to JSON, arrays as tuples, and judged + as a metadata field, an extra member, a `configuration` that is not + an object or a `must_understand` that is not a boolean or is `false` + each a problem. The second needs `context`: the identifier in the + field is related to a concrete class through it, and that class owns + the validation routine, its `coerce`, which is handed the field. + + What comes back is the entity, or an `Opaque` saying why not. A name + no class in `context` claims is `out_of_scope` -- an unmodelled + extension, left unjudged, which is what makes the format open. A + name claimed and refused, or of another kind than this position + takes, is `invalid`, for the reasons reported alongside; so is a + value that is not JSON, or names no entity. `loc` prefixes the + problems, so they point at where in the containing configuration the + field sat. + """ + refined, problems = refine_json(data, loc) + if refined is None: + return Opaque.create_unchecked(None, "invalid"), problems + return _resolve_field(refined, kind, context, loc) + + +def _resolve_field( + data: JSONValue, kind: type[EntityT], context: Context, loc: Loc +) -> tuple[EntityT | Opaque, tuple[ValidationProblem, ...]]: + """A refined field, its envelope judged, then read. + + What a nested field gets: a metadata field is a metadata field + wherever it appears, so the envelope gets the same structural + judgment here that the model layer gives a top-level one. + """ + problems = tuple( + ValidationProblem((*loc, *found.loc), found.message, found.kind) + for found in validate_metadata_field_v3(data, allow_must_understand_false=False) + ) + entity, found = read_field(data, kind, context, loc) + return entity, (*problems, *found) + + +def read_field( + data: JSONValue, kind: type[EntityT], context: Context, loc: Loc = () +) -> tuple[EntityT | Opaque, tuple[ValidationProblem, ...]]: + """The second layer for one field whose envelope the first has judged. + + What `resolve` does once the envelope is judged, and what the + document reader does for a top-level field, whose envelope the model + layer judged with the document: relate the identifier in `data` to a + class through `context`, and hand the class the field. The class is + asked whenever there is one to ask -- a stray member or a malformed + `must_understand` says nothing about the configuration -- and not + when the value names no entity or its configuration is not an + object, which the envelope judgment has said. + """ + name, _, malformed = named_configuration(data) + if name is None or len(malformed) != 0: + return Opaque.create_unchecked(data, "invalid"), () + # Asked with the kind's kind, so a class of the wrong kind for this + # position is found, and told apart from a name nothing claims. + registered = kind_of(kind) + entity_type = None if registered is None else context.claimant(registered, name) + if entity_type is None: + return Opaque.create_unchecked(data, "out_of_scope"), () + if not issubclass(entity_type, kind): + # In scope, so not for another reader to resolve: the name is an + # entity of the wrong kind for this position. + return Opaque.create_unchecked(data, "invalid"), ( + ValidationProblem( + loc, + f"expected {_an(kind.__name__)}, got {name!r}, " + f"{_an(_refinement(entity_type, kind).__name__)}", + "invalid_value", + ), + ) + entity, found = entity_type.coerce(data, context) + problems = tuple( + ValidationProblem((*loc, *entry.loc), entry.message, entry.kind) for entry in found + ) + if entity is None: + return Opaque.create_unchecked(data, "invalid"), problems + return entity, problems + + +def _refinement(entity: type[MetadataEntity], kind: type[MetadataEntity]) -> type[MetadataEntity]: + """The class just below `kind`'s kind that `entity` is: `ArrayArrayCodec` for a transpose codec.""" + mro = entity.__mro__ + return mro[mro.index(kind_of(kind) or MetadataEntity) - 1] + + +def _an(noun: str) -> str: + """`noun` with its indefinite article: `an ArrayArrayCodec`, `a BytesBytesCodec`.""" + return f"an {noun}" if noun[:1].upper() in "AEIOU" else f"a {noun}" + + +def held_problems( + value: object, kind: type[MetadataEntity], loc: Loc +) -> tuple[ValidationProblem, ...]: + """Why `value` is not what a field typed `kind | Opaque` holds: an entity of the kind, or an `Opaque`.""" + if isinstance(value, (kind, Opaque)): + return () + return problem(loc, f"expected an entity of {kind.__name__} or an Opaque, got {value!r}") + + +def _held_field(annotation: object) -> Parser[None] | None: + """The check on a value a record holds in a field typed as an entity or as a record. + + The leaf the record's constructor checks with, where `_nested_field` + is the one a document is read with: a field typed `Kind | Opaque` + holds an instance of the kind or an `Opaque`, and a field typed as a + record dataclass holds an instance of it, checked in turn. Every + other shape is one JSON takes, and a Python value of that shape + passes the same parser. + """ + kind = nested_kind(annotation) + if kind is not None: + + def holds_entity(value: object, loc: Loc, state: None) -> Parsed: + return value, held_problems(value, kind, loc) + + return holds_entity + inner = without_unset(strip_annotation(annotation)[0]) + if isinstance(inner, type) and is_dataclass(inner): + + def holds_record(value: object, loc: Loc, state: None) -> Parsed: + if not isinstance(value, inner): + return value, problem(loc, f"expected {inner.__name__}, got {value!r}") + return value, tuple( + ValidationProblem((*loc, *found.loc), found.message, found.kind) + for found in mistyped(value) + ) + + return holds_record + return None + + +@functools.cache +def _checks(record: type) -> dict[str, tuple[bool, Parser[None]]]: + """Each field of a record: whether it may be `UNSET`, and the check on a value held in it.""" + return { + name: (is_optional(annotation), parser(annotation, _held_field)) + for name, annotation in field_hints(record).items() + } + + +def mistyped(record: object) -> tuple[ValidationProblem, ...]: + """Every field of `record` holding a value not of its type, located at the field. + + The runtime half of a record's type. Pyright checks the values a + caller writes; this checks the ones it cannot see -- through + `**changes`, through `replace`, through anything typed `object` -- + with the parsers a document is read by, so a record is well-typed + however it was built. + """ + found: list[ValidationProblem] = [] + for name, (optional, check) in _checks(type(record)).items(): + value = getattr(record, name) + if optional and value is UNSET: + continue + # An unknown key in a mapping member is the reader's report, not + # a type: the record holds what the document said, as `coerce` + # lets it, and a hand-built one may say the same. + found.extend( + entry for entry in check(value, (name,), None)[1] if entry.kind != "unknown_key" + ) + return tuple(found) + + +@dataclass(frozen=True, slots=True) +class _Plan: + """How `coerce` reads and `to_json` writes one class, compiled once from its fields.""" + + from_name: str | None + """The field the envelope's name fills, for a family; None for every other entity.""" + record: type[Configuration] + """The record the entity declares, which its constructor holds it to.""" + read: Callable[ + [object, Loc, _Reading], tuple[Configuration | None, tuple[ValidationProblem, ...]] + ] + """The configuration record, built when every member read, and the problems found.""" + write: Callable[[MetadataEntity], dict[str, JSONValue]] + """The entity's configuration as the JSON object it writes; empty for a bare name.""" + requires_configuration: bool + """Whether the record has a member the document must write.""" + + +@functools.cache +def _plan(cls: type[MetadataEntity]) -> _Plan: + """The plan for `cls`, a pure function of the class: its fields are fixed once it exists. + + Each parser is a function of its annotation alone, taking the reading + it runs in as an argument. `TypeError` for a shape no parser reads, + which registration refuses first. The record is built through + `create_unchecked`: its members were type-checked by the parsers the + constructor would use, and a read does each check once. + """ + hints = field_hints(cls) + from_name = next((key for key, annotation in hints.items() if is_from_name(annotation)), None) + record = hints["configuration"] + if not (isinstance(record, type) and issubclass(record, Configuration)): # pragma: no cover + msg = f"{cls.__name__}: configuration is annotated {record!r}, not a Configuration" + raise TypeError(msg) + required = any(not is_optional(annotation) for annotation in field_hints(record).values()) + members = members_of(field_hints(record), _nested_field) + if members is None: # pragma: no cover - registration refused the member first + msg = f"{cls.__name__}: a configuration member is not a shape JSON takes" + raise TypeError(msg) + parse = record_of(record.create_unchecked, members) + writes: RecordWriter = record_writer(record, _nested_field_writer) + + def read( + value: object, loc: Loc, reading: _Reading + ) -> tuple[Configuration | None, tuple[ValidationProblem, ...]]: + typed, found = parse(value, loc, reading) + # The parser builds the record only from an object whose every + # member read; anything else comes back as it came. + return (typed if isinstance(typed, Configuration) else None), found + + def write(entity: MetadataEntity) -> dict[str, JSONValue]: + return writes(entity.configuration) + + return _Plan(from_name, record, read, write, required) + + +@dataclass(frozen=True) +class Configuration: + """What an entity is configured with: a record of its members, and the rules on them. + + A frozen dataclass whose fields are the configuration's members, + each a shape JSON takes; the entity names it in its `configuration` + field, and an entity of a bare name defaults it to this one, empty, + since the spec makes an absent configuration and an empty one the + same. + `problems` is where everything finer than a type goes -- a + bound, a rule about one member, members read together -- yielding + each problem as it is found, located relative to the configuration. + A reader stops at the first or collects them all, as it needs: the + entity's constructor stops at the first, `coerce` reports every one, + and `BloscOptions(...).problems()` answers without an entity at all. + + The constructor refuses a member of the wrong type, so a record is + well-typed however it was built -- by hand, through `replace`, + through an entity's `with_configuration` -- and the rules can trust + what they read. Values are the rules' business, and the entity's + constructor asks them. + """ + + def __post_init__(self) -> None: + """Refuse every member of the wrong type, so `GzipOptions(level="high")` raises.""" + found = mistyped(self) + if len(found) != 0: + raise MetadataValidationError(found) + + @classmethod + def create_unchecked(cls, **members: object) -> Self: + """This record with these members, built without the constructor's check. + + The one way around the check, for a caller that has just made + it: the parser, which type-checked every member against the same + annotations before building the record. Every field is given -- + the parser gives an absent optional member as `UNSET` -- since + nothing here applies a default. Anything that has not checked + the members goes through the constructor. + """ + record = object.__new__(cls) + for name, value in members.items(): + object.__setattr__(record, name, value) + return record + + def problems(self) -> Iterator[ValidationProblem]: + """Every reason these values are not allowed, yielded as found. Default: none.""" + yield from () + + +@dataclass(frozen=True) +class MetadataEntity(ABC): + """One named entity, coerced from its metadata. + + An entity is well-typed and allowed however it was built. `coerce` + builds one only from metadata it accepted, through `create_unchecked` + once it has; by hand, the record's constructor refuses a member of + the wrong type, the entity's refuses a record that is not its own + and then a value the rules disallow, and `replace` and + `with_configuration` go through both. An optional member is typed + `| UNSET` with a default of `UNSET`, so absence is representable -- + and distinct from a `null` the document wrote -- and a canonical + spelling can leave it out. + + Frozen, so an entity of hashable members is hashable. One holding a + value out of scope is not, because that value is the JSON the document + wrote and a JSON object is a `dict` -- the same way any frozen + dataclass holding a list is unhashable. It cannot be an immutable + mapping instead: `MappingProxyType` is unhashable too, and anything + else stops `json.dumps` from serializing what `to_json` returns. + + A subclass names its configuration record, a `Configuration` whose + `problems` holds what the spec says beyond the members' types -- so + `BloscCodec(BloscOptions(clevel=99))` raises on the first, and + `coerce` reports every one instead -- and writes `canonical` where + two spellings of its members mean the same. An entity of a bare + name defaults the field to the empty record. `coerce` and `to_json` + are written once here, against what the record says. + """ + + configuration: Configuration + """The record of this entity's members. + + An entity with members narrows it to its own record, `configuration: + GzipOptions`, its one positional argument. An entity of a bare name + defaults it to the empty record -- `configuration: Configuration = + field(default_factory=Configuration)` -- so that `Crc32cCodec()` + builds; `coerce` passes the record either way. + """ + + identifier: ClassVar[str] + """The name this entity is registered under. + + Usually the `name` the metadata carries. The raw-bytes data types are + the exception: every `r` spelling is one family, so the family gets + an invented identifier that no real name can collide with. + """ + + @classmethod + def name_problems(cls, name: str) -> Iterator[ValidationProblem]: + """Why `name`, which `accepts` claimed, is not a well-formed name of this family. + + For a family, whose names carry data -- `r` -- and which claims + a malformed member so that it is reported rather than waved + through as an unknown extension. Locations are relative to the + entity: `()`. Default: none, for an entity of one name. + """ + yield from () + + def __post_init__(self) -> None: + """Refuse a record that is not this entity's own, then the first problem the rules find. + + The runtime half of the entity's type, as the record's constructor + is of the record's: `GzipCodec(BloscOptions(...))` and a family + member carrying a name that is not a string are refused before + any rule reads them. Then `BloscCodec(BloscOptions(clevel=99))` + raises on the first problem the rules yield. + """ + plan = _plan(type(self)) + if not isinstance(self.configuration, plan.record): + raise MetadataValidationError( + problem( + (), + f"expected a {plan.record.__name__} configuration, got " + f"{type(self.configuration).__name__}", + ) + ) + name: object = self.identifier if plan.from_name is None else getattr(self, plan.from_name) + if not isinstance(name, str): + raise MetadataValidationError(problem((), f"expected a string name, got {name!r}")) + first = next(type(self).name_problems(name), None) + if first is None: + first = next(self.configuration.problems(), None) + if first is not None: + raise MetadataValidationError((first,)) + + @classmethod + def accepts(cls, name: str) -> bool: + """Whether `name` denotes this entity. + + Constant for all but the raw-bytes family, where one class covers + every `r`. + """ + return name == cls.identifier + + @property + def name(self) -> str: + """The name this entity carries, as a document writes it. + + The identifier, except for a family whose name carries a value: + the raw-bytes family's identifier is invented and belongs in no + message a reader sees, and its name is the `r24` the document + wrote. Anything a reader sees wants this; anything looking a + class up wants `identifier`. + """ + from_name = _plan(type(self)).from_name + return type(self).identifier if from_name is None else cast("str", getattr(self, from_name)) + + @classmethod + def create_unchecked(cls, **fields: object) -> Self: + """This entity with these fields, built without the constructor's checks. + + The one way around them, for a caller that has just made them: + `coerce`, which type-checked the record and ran the rules before + building. Every field is given -- the record, and the carried + name for a family -- since nothing here applies a default. + Anything that has not checked goes through the constructor. + """ + entity = object.__new__(cls) + for name, value in fields.items(): + object.__setattr__(entity, name, value) + return entity + + def with_configuration(self, **changes: object) -> Self: + """This entity with these configuration members changed. + + `codec.with_configuration(typesize=UNSET)` is the record rebuilt + through its constructor, which refuses a member of the wrong + type, and the entity rebuilt through its own, which refuses a + value the rules disallow -- the same checks as any construction, + since pyright cannot see the members through `**changes`. A + name that is not a member is refused the way `replace` refuses + it. + """ + return replace(self, configuration=replace(self.configuration, **changes)) + + @classmethod + def coerce(cls, value: JSONValue, context: Context) -> Coerced[Self]: + """`value` as this entity, or the reasons it is not one: the class's validation routine. + + `resolve` relates a field's name to this class and hands it the + field, refined JSON with arrays as tuples, which is what `value` + is; this is what the class does with it. The configuration is + parsed against the record the `configuration` field names, + member by member; a member holding another entity is read in + `context`, the scope this reading is happening in. An optional + member the document left out is `UNSET` in the record, so no + field's default decides what a document said. The entity is + built only when every member of its own read -- its rules are + written over a whole configuration -- and handed back only when + everything inside it read too. + + The envelope is the field's, not the class's, and `resolve` + judges it: a stray member or a `must_understand` of `false` is + not reported here. Called on a class no scope has registered, + this runs with none of registration's refusals having happened. + """ + name, given, envelope = named_configuration(value) + if name is None or not cls.accepts(name): + return None, problem((), f"expected the {cls.identifier!r} entity") + if len(envelope) != 0: + return None, envelope + plan = _plan(cls) + carried: dict[str, str] = {} if plan.from_name is None else {plan.from_name: name} + if given is None and plan.requires_configuration: + return None, problem( + ("configuration",), + f"{cls.identifier!r} requires a configuration", + "missing_key", + ) + reading = _Reading(context, []) + # A bare name's record has no members, so any key is an unknown one. + record, own = plan.read({} if given is None else given, ("configuration",), reading) + found = (*own, *reading.nested) + if record is None: + # An unknown key is survivable; a member that could not be + # read is a hole, and judging around it would be guessing. + return None, found + # The rules, asked of the name and of the record before anything + # is built: a name problem lands on the entity, a configuration + # problem under the configuration. + refused = (*cls.name_problems(name), *within((), tuple(record.problems()))) + if len(refused) != 0: + # Values the spec disallows: reported rather than raised, + # every one. + return None, (*found, *refused) + if any(entry.kind != "unknown_key" for entry in reading.nested): + # A contained entity could not be read. This entity's own + # rules ran -- an invalid inner is an `Opaque`, as an + # out-of-scope one is -- but what is handed back is not an + # entity that would be asked composition questions it cannot + # answer. + return None, found + return cls.create_unchecked(configuration=record, **carried), found + + def canonical(self) -> Self: + """This entity in the simplest form that means the same thing. + + A *transformation*, asked for by `canonicalize_array_metadata_v3` + and by nothing else. `to_json` does not apply it, because writing + a document back is not the same as asking for it to be rewritten: + a reader that reads and writes should not change bytes it was not + asked to change. + + The default is the entity itself. Override it where two spellings + of the entity's members mean the same -- a rectilinear + dimension's run-length encoding, a `typesize` that `noshuffle` + ignores -- and, in an entity that contains entities, to put those + in canonical form: `self.with_configuration(inner=self.inner.canonical())`. + """ + return self + + def to_json(self) -> ZarrV3MetadataFieldJSON: + """This entity as a document would write it. + + Written from the configuration record by the same declaration + `coerce` reads it by, each member by the writer its annotation + implies: the bare name when every member it holds is absent, the + object otherwise, a contained entity through its own `to_json`, + a JSON-valued member copied so the document is not a handle on + the entity. Faithful to every member: read a document, write it + back, and those come out as they went in. The envelope is + written the entity's way -- the bare name when nothing is + configured, the object otherwise, no `must_understand`, which + means what absence means -- because an entity alone has no + document to be faithful to; `ArrayDocumentV3.to_json` puts back + the spelling the document used. Ask `canonical` first if you + want the simplest equivalent spelling. + + An entity whose JSON is not its fields overrides this; none in + the package does. + """ + plan = _plan(type(self)) + name = self.identifier + if plan.from_name is not None: + carried = getattr(self, plan.from_name) + name = carried if isinstance(carried, str) else name + configuration = plan.write(self) + if len(configuration) == 0: + return name + return {"name": name, "configuration": configuration} + + +@dataclass(frozen=True) +class CodecEntity(MetadataEntity): + """An entity that occupies a position in the codec pipeline. + + Of one of three kinds, each a base class: `ArrayArrayCodec`, + `ArrayBytesCodec`, `BytesBytesCodec`. The kind fixes where in the + pipeline the codec may stand, and what it must answer. + """ + + variable_size: ClassVar[bool] + """Whether this codec's output size depends on the bytes it is given. + + A compressor's does, so a shard index encoded with one has no size + derivable from metadata alone, and the shard cannot be read. Every + codec says, because a default in either direction is a verdict. + """ + + def incoming_problems(self, incoming: ArrayParts | None) -> tuple[ValidationProblem, ...]: + """Why this codec cannot be applied to the array that reaches it. + + `incoming` is None once the chain can no longer say what reaches + here, and the default answer to that is nothing: declining beats + guessing. Locations are relative to this codec's `configuration`; + an empty one lands on the codec itself. + """ + return () + + def inner_pipelines( + self, incoming: ArrayParts | None + ) -> Mapping[str, tuple[Sequence[CodecEntity | Opaque], ArrayParts | None]]: + """The pipelines this codec holds, by the member holding each, with what each is handed. + + A shard holds two: its `codecs`, handed its inner chunk, and its + `index_codecs`, handed the shard index. Refinement walks them as + it walks the pipeline this codec stands in, locating what it + finds under the member, so a codec that holds pipelines says + which and what they receive, and judges nothing inside them + itself. Default: none. + """ + return {} + + +@dataclass(frozen=True) +class ArrayArrayCodec(CodecEntity): + """A codec that transforms the array: what reaches the next codec is its to say.""" + + @abstractmethod + def transition(self, incoming: ArrayParts) -> ArrayParts | None: + """What the next codec in the chain sees. + + `incoming` itself if this codec leaves the array's shape, grid and + data type alone; the parts it hands on if it changes one; None if + that cannot be determined from the metadata, which ends the + judgments downstream rather than inventing them. + """ + + +@dataclass(frozen=True) +class ArrayBytesCodec(CodecEntity): + """The one codec in a pipeline that turns the array into bytes.""" + + +@dataclass(frozen=True) +class BytesBytesCodec(CodecEntity): + """A codec that transforms bytes, after the array is gone.""" + + +@dataclass(frozen=True) +class ChunkGridEntity(MetadataEntity): + """An entity that divides an array into the parts a pipeline encodes.""" + + def shape_problems(self, array_shape: object) -> tuple[ValidationProblem, ...]: + """Why this grid does not divide an array of `array_shape`. + + Locations are relative to the grid's `configuration`. Default: + nothing, for a grid this package reads but has no such rule for. + """ + return () + + @abstractmethod + def grid(self, array_shape: object) -> ChunkGrid: + """What this grid divides an array of `array_shape` into. + + The array shape is a parameter because neither determines a grid + alone: a grid whose own metadata cannot be read still has the + array's rank, and rank is enough for several rules. + """ + + +@dataclass(frozen=True) +class DataTypeEntity(MetadataEntity): + """An entity that says how the array's scalars are stored. + + Only data types answer that, and every rule that turns on it -- a + `bytes` codec is pointless before a single-byte type, a struct field + cannot be variable-length -- asks a data type rather than consulting + a table of names. + """ + + scalar_storage: ClassVar[StorageClass] + + twos_complement: ClassVar[bool] + """Whether this type's scalars are two's complement integers. + + Asked by `cast_value`, whose `out_of_range: "wrap"` is defined only + for such a target. Owed rather than defaulted: a data type added + later must decide, because either default would answer for it + silently -- and getting it wrong in one direction accepts a cast the + spec does not define. + """ + + def storage_class(self) -> StorageClass | None: + """How one scalar occupies bytes, or None if undetermined. + + None only for a composite whose parts are not all in scope: an + answer would be a guess, and the rules that ask decline instead. + """ + return type(self).scalar_storage + + @abstractmethod + def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]: + """Why `value` is not a fill value of this type, if it is not. + + Every data type answers this; one that accepts any fill value + says so with `return ()`. + """ + + +@dataclass(frozen=True) +class ChunkKeyEncodingEntity(MetadataEntity): + """An entity that says how a chunk's coordinates become a store key.""" + + +@dataclass(frozen=True) +class StorageTransformerEntity(MetadataEntity): + """An entity that stands between the codec pipeline and the store.""" + + +KINDS: Final[tuple[type[MetadataEntity], ...]] = ( + DataTypeEntity, + ChunkGridEntity, + ChunkKeyEncodingEntity, + CodecEntity, + StorageTransformerEntity, +) +"""The kinds of entity: what a scope holds a table of, and what a document's fields are read as.""" + + +def kind_of(cls: type[MetadataEntity]) -> type[MetadataEntity] | None: + """The kind `cls` is of, or None for a class under none of them.""" + return next((kind for kind in cls.__mro__ if kind in KINDS), None) + + +__all__ = [ + "FROM_NAME", + "KINDS", + "ArrayArrayCodec", + "ArrayBytesCodec", + "BytesBytesCodec", + "ChunkGridEntity", + "ChunkKeyEncodingEntity", + "CodecEntity", + "Coerced", + "Configuration", + "DataTypeEntity", + "EntityT", + "Loc", + "MetadataEntity", + "Opaque", + "StorageClass", + "StorageTransformerEntity", + "held_problems", + "is_from_name", + "is_integer", + "is_metadata_field", + "kind_of", + "named_configuration", + "nested_kind", + "problem", + "read_field", + "resolve", + "unreadable", + "within", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/_parts.py b/packages/zarr-metadata/src/zarr_metadata/v3/_parts.py new file mode 100644 index 0000000000..0223c3119f --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/v3/_parts.py @@ -0,0 +1,211 @@ +"""How an array is divided, and what a codec chain does to it. + +A codec pipeline encodes one chunk, but the same pipeline encodes *every* +chunk, so a rule about a pipeline is a statement about all of them at +once: a shard's inner chunk shape must divide every chunk it will ever be +handed, not some representative one. `ChunkGrid` is what makes that +statement expressible, and `ArrayParts` is what one codec hands the next. + +Three pieces of metadata divide an array: a document's `chunk_grid`, a +`sharding_indexed` codec's `chunk_shape` (a regular grid over the chunk +that codec receives), and that codec's shard index, whose shape the spec +derives from the other two. Each grid entity builds its own, because what +a grid divides an array into is the grid's own knowledge. + +Per dimension, and plurally +--------------------------- +`extents` holds one entry per dimension: the set of lengths that +dimension's chunks take. A regular grid gives singletons; a rectilinear +grid gives `{30, 34}` on an axis whose chunks differ; `None` marks an axis +this package cannot read. `rank` survives even when no extent does, +because every chunk of an array has the array's rank whatever divides it. + +Collapsing any of that loses real judgments. A rectilinear grid uniform on +one axis still pins that axis, and a shard is judged there while declining +on the others. + +Prior art +--------- +zarrs builds its grid from metadata *and* the array shape +(`ChunkGrid::create(metadata, array_shape)`) because neither determines a +grid alone, keeps `dimensionality()` total rather than optional, and +reports `chunk_edge_lengths(dimension)` per dimension for the reason +above. Its codec chain distinguishes "no global grid, but one per chunk" +(`ChunkGridMapped::ChunkLocal`) from "nothing known" (`::None`); an +`extents` entry of `None` beside a known `rank` is the per-dimension form +of that distinction. + +- https://github.com/zarrs/zarrs/blob/main/zarrs_chunk_grid/src/lib.rs +- https://github.com/zarrs/zarrs/blob/main/zarrs_codec/src/lib.rs +""" + +from __future__ import annotations + +from dataclasses import dataclass, replace +from typing import TYPE_CHECKING, TypeAlias, cast + +if TYPE_CHECKING: + from collections.abc import Sequence + + from zarr_metadata.v3._entity import DataTypeEntity + + +Extents: TypeAlias = tuple[frozenset[int] | None, ...] +"""One entry per dimension: the lengths that dimension's chunks take. + +A singleton is a uniform axis. `None` is an axis whose lengths this +package cannot determine — distinct from an empty set, which would claim +the axis has no chunks at all. +""" + + +def _positive_int(value: object) -> int | None: + """`value` as a chunk length, or None if it is not a usable one.""" + if isinstance(value, bool) or not isinstance(value, int): + return None + return value if value >= 1 else None + + +def _rank_of(array_shape: object) -> int | None: + """The number of dimensions `array_shape` declares, if it declares any.""" + if not isinstance(array_shape, tuple): + return None + dimensions = cast("tuple[object, ...]", array_shape) + if not all(isinstance(v, int) and not isinstance(v, bool) for v in dimensions): + return None + return len(dimensions) + + +def _uniform(lengths: Sequence[object]) -> Extents: + """Extents for a grid whose chunks are the same everywhere.""" + return tuple( + None if (length := _positive_int(value)) is None else frozenset({length}) + for value in lengths + ) + + +@dataclass(frozen=True, slots=True) +class ChunkGrid: + """The division of an array into the parts a codec pipeline encodes. + + Nothing here is the metadata: a grid entity keeps its own, and what + reaches a codec is the division, not the spelling of it. A derived + grid -- the regular one a sharding codec imposes, or a transposed one + -- has no metadata to keep anyway. + """ + + rank: int | None + extents: Extents | None + + @classmethod + def unreadable(cls, array_shape: object) -> ChunkGrid: + """A grid nothing is known about but the rank the array pins. + + Every third-party grid, and every modelled one whose own metadata + could not be read: the array still has a rank, and a rule about + rank is still answerable. + """ + rank = _rank_of(array_shape) + return cls(rank, None if rank is None else (None,) * rank) + + @classmethod + def derived(cls, extents: Extents) -> ChunkGrid: + """A grid this package computed rather than read from a document.""" + return cls(len(extents), extents) + + @classmethod + def regular(cls, lengths: Sequence[object]) -> ChunkGrid: + """The regular grid a sharding codec's `chunk_shape` imposes.""" + return cls.derived(_uniform(lengths)) + + def permuted(self, order: Sequence[int]) -> ChunkGrid: + """This grid with its dimensions reordered by `order`. + + A transposed grid is still a grid — permuting a regular one gives + a regular one — but it is no longer the grid the document wrote, + so the metadata does not survive the trip. + + Declines on anything that is not a permutation of this grid's rank. + The caller checks that too and reports it, but an order is only + shape-validated as a tuple of integers, so this must not be the + thing that decides whether a validator raises `IndexError`. + """ + if self.extents is None or sorted(order) != list(range(len(self.extents))): + return ChunkGrid(self.rank, None) + return ChunkGrid.derived(tuple(self.extents[axis] for axis in order)) + + def axis(self, dimension: int) -> frozenset[int] | None: + """The lengths `dimension`'s chunks take, or None if undetermined.""" + if self.extents is None or dimension >= len(self.extents): + return None + return self.extents[dimension] + + +UNKNOWN_GRID: ChunkGrid = ChunkGrid(None, None) +"""A grid nothing is known about — not even how many dimensions it has.""" + + +def shard_index_grid(shard: ChunkGrid, inner: Sequence[object]) -> ChunkGrid: + """The grid of a shard's index array. + + The spec derives it from the two shapes around it: "The index is an + array with 64-bit unsigned integers with a shape that matches the + chunks per shard tuple with an appended dimension of size 2." The + index is one array rather than a divided one, so each axis holds a + single length — except that under a rectilinear grid the shard itself + varies, so the chunk count varies with it and the axis holds every + value it takes. + """ + inner_extents = _uniform(inner) + trailing: frozenset[int] | None = frozenset({2}) + if shard.extents is None or len(shard.extents) != len(inner_extents): + return ChunkGrid.derived((*(None,) * len(inner_extents), trailing)) + counts: list[frozenset[int] | None] = [] + for lengths, divisor in zip(shard.extents, inner_extents, strict=True): + if lengths is None or divisor is None: + counts.append(None) + continue + step = next(iter(divisor)) + quotients = {length // step for length in lengths if length % step == 0} + counts.append(frozenset(quotients) if len(quotients) == len(lengths) else None) + return ChunkGrid.derived((*counts, trailing)) + + +@dataclass(frozen=True, slots=True) +class ArrayParts: + """Every part of an array a codec will be handed, and their type. + + The parts an array is divided into, not the fields of its metadata. + Plural deliberately: one pipeline encodes every chunk, so a rule about + it quantifies over all of them — a shard's inner chunk shape must + divide *every* chunk, which under a rectilinear grid is several + different lengths. + + `data_type` is the coerced data type, so a rule asks it what it is + rather than comparing names, and it is `None` where the element type + is undetermined + while the array itself is not. That happens inside a shard: the inner + grid is the sharding codec's own `chunk_shape` whatever reached it, so + an unreadable codec upstream costs the type and not the parts. `None` + in place of the whole value means something else again — that there is + no array here at all, past the array->bytes boundary or beyond a codec + that could have changed anything. + """ + + grid: ChunkGrid + data_type: DataTypeEntity | None + + def with_grid(self, grid: ChunkGrid) -> ArrayParts: + return replace(self, grid=grid) + + def with_data_type(self, data_type: DataTypeEntity | None) -> ArrayParts: + return replace(self, data_type=data_type) + + +__all__ = [ + "UNKNOWN_GRID", + "ArrayParts", + "ChunkGrid", + "Extents", + "shard_index_grid", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/_registry.py b/packages/zarr-metadata/src/zarr_metadata/v3/_registry.py new file mode 100644 index 0000000000..6486984489 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/v3/_registry.py @@ -0,0 +1,233 @@ +"""Which entities are in scope when raw JSON is coerced. + +The only registry the package needs. A validator reading a document has +to decide, for each kind of entity, which identifier maps to which class +— and that decision is the *scope* it validates against, not a property +of the entities themselves. Which of a document's fields holds which +kind is the document's own knowledge, in `_document`. + +Two scopes, because the question "is this document valid?" has two useful +answers. `CORE` is what the Zarr v3 specification itself defines, so a +document validating against it uses nothing an implementation could +refuse for being optional. `CORE_AND_EXTENSIONS` adds what +`zarr-extensions` registers and this package models. A name in neither is +not rejected — extension openness — it is simply not judged. + +An entity is registered under its `identifier`, which is the `name` the +metadata carries -- except for a family, which covers many names with +one class (every `r` spelling is one data-type family) and registers +under an invented one. Resolution asks each entity of a kind whether a +name is its own, through `accepts`; the identifier is the key that +`extended_with` takes a name over with. +""" + +from __future__ import annotations + +import inspect +from collections.abc import Mapping +from dataclasses import dataclass +from types import MappingProxyType +from typing import Final + +from zarr_metadata.v3._entity import ( + KINDS, + ArrayArrayCodec, + ArrayBytesCodec, + BytesBytesCodec, + CodecEntity, + EntityT, + MetadataEntity, + kind_of, + unreadable, +) +from zarr_metadata.v3._typed_json import is_class_var, own_annotations +from zarr_metadata.v3.chunk_grid.rectilinear import RectilinearChunkGrid +from zarr_metadata.v3.chunk_grid.regular import RegularChunkGrid +from zarr_metadata.v3.chunk_key_encoding.default import DefaultChunkKeyEncoding +from zarr_metadata.v3.chunk_key_encoding.v2 import V2ChunkKeyEncoding +from zarr_metadata.v3.codec.blosc import BloscCodec +from zarr_metadata.v3.codec.bytes import BytesCodec +from zarr_metadata.v3.codec.cast_value import CastValueCodec +from zarr_metadata.v3.codec.crc32c import Crc32cCodec +from zarr_metadata.v3.codec.gzip import GzipCodec +from zarr_metadata.v3.codec.scale_offset import ScaleOffsetCodec +from zarr_metadata.v3.codec.sharding_indexed import ShardingIndexedCodec +from zarr_metadata.v3.codec.transpose import TransposeCodec +from zarr_metadata.v3.codec.zstd import ZstdCodec +from zarr_metadata.v3.data_type.bool import BoolDataType +from zarr_metadata.v3.data_type.bytes import BytesDataType +from zarr_metadata.v3.data_type.complex64 import Complex64DataType +from zarr_metadata.v3.data_type.complex128 import Complex128DataType +from zarr_metadata.v3.data_type.float16 import Float16DataType +from zarr_metadata.v3.data_type.float32 import Float32DataType +from zarr_metadata.v3.data_type.float64 import Float64DataType +from zarr_metadata.v3.data_type.int8 import Int8DataType +from zarr_metadata.v3.data_type.int16 import Int16DataType +from zarr_metadata.v3.data_type.int32 import Int32DataType +from zarr_metadata.v3.data_type.int64 import Int64DataType +from zarr_metadata.v3.data_type.numpy_datetime64 import NumpyDatetime64DataType +from zarr_metadata.v3.data_type.numpy_timedelta64 import NumpyTimedelta64DataType +from zarr_metadata.v3.data_type.raw import RawBytesDataType +from zarr_metadata.v3.data_type.string import StringDataType +from zarr_metadata.v3.data_type.struct import StructDataType +from zarr_metadata.v3.data_type.uint8 import Uint8DataType +from zarr_metadata.v3.data_type.uint16 import Uint16DataType +from zarr_metadata.v3.data_type.uint32 import Uint32DataType +from zarr_metadata.v3.data_type.uint64 import Uint64DataType + +Tables = Mapping[type[MetadataEntity], Mapping[str, type[MetadataEntity]]] +"""By kind, then by the identifier each entity is registered under.""" + + +@dataclass(frozen=True, slots=True) +class Context: + """The entities in scope while metadata is being read. + + A value, with no reading of its own: `resolve` reads a field in it, + and `claimant` is the one question it answers, which class a name + belongs to. Built from classes with `Context.of`; extended with more + by `extended_with`. What each class is registered as is read off it + -- its kind is its base class, its key is its `identifier` -- so + there is nothing to misfile. + """ + + tables: Tables + + @classmethod + def of(cls, *entities: type[MetadataEntity]) -> Context: + """A scope of exactly these entities; a later one takes over an identifier from an earlier.""" + tables: dict[type[MetadataEntity], dict[str, type[MetadataEntity]]] = { + kind: {} for kind in KINDS + } + for entity in entities: + kind = _registrable(entity) + tables[kind][entity.identifier] = entity + return cls( + MappingProxyType({kind: MappingProxyType(table) for kind, table in tables.items()}) + ) + + def extended_with(self, *entities: type[MetadataEntity]) -> Context: + """This scope, plus entities of your own. + + A name already registered under the same kind is taken over by + what is passed here, which is how a reader substitutes its own + reading of a codec the package already models. + """ + return Context.of(*self.entities(), *entities) + + def entities(self) -> tuple[type[MetadataEntity], ...]: + """Every entity in scope, kind by kind.""" + return tuple(entity for table in self.tables.values() for entity in table.values()) + + def claimant(self, kind: type[EntityT], name: str) -> type[EntityT] | None: + """The class in scope that claims `name` as an entity of `kind`; None if none does. + + Asks each class registered under the kind's kind whether the name + is its own -- a family claims every `r` -- rather than looking + a key up, so the identifier keys exist for `extended_with` to + take a name over, not for lookup. A class that claims the name + but is not a `kind` -- `transpose` asked for as a + `BytesBytesCodec` -- is none; `resolve` asks with the kind's kind + to tell that case from a name nothing claims. + """ + registered = kind_of(kind) + if registered is None: + return None + table = self.tables.get(registered, {}) + entity = next((candidate for candidate in table.values() if candidate.accepts(name)), None) + if entity is None or not issubclass(entity, kind): + return None + return entity + + +def _registrable(entity: type[MetadataEntity]) -> type[MetadataEntity]: + """The kind `entity` is registered under; `TypeError` for a class no scope can use. + + The one moment an entity is refused: every way of writing one that + type-checks cleanly and then fails somewhere that will not name the + class -- no kind, a codec skipping the kind classes, no `@dataclass`, + a field `coerce` could not parse, a `__post_init__` `coerce` would + never ask, a class variable owed and unset, what a kind leaves + abstract -- with a message that says what to write. + """ + kind = kind_of(entity) + if kind is None: + msg = ( + f"{entity.__name__} is of no kind; subclass a codec kind, DataTypeEntity, " + "ChunkGridEntity, ChunkKeyEncodingEntity or StorageTransformerEntity" + ) + raise TypeError(msg) + if issubclass(entity, CodecEntity) and not issubclass( + entity, (ArrayArrayCodec, ArrayBytesCodec, BytesBytesCodec) + ): + # The kind classes say what a codec does to the array, and what + # each must answer is abstract on them. + msg = ( + f"{entity.__name__} subclasses CodecEntity directly; subclass ArrayArrayCodec, " + "ArrayBytesCodec or BytesBytesCodec, which says what the codec does to the array" + ) + raise TypeError(msg) + if "__dataclass_fields__" not in vars(entity) and any( + not is_class_var(annotation) for annotation in own_annotations(entity).values() + ): + msg = ( + f"{entity.__name__} declares fields but is not a dataclass; decorate it with " + "@dataclass(frozen=True), which is what makes its fields the configuration" + ) + raise TypeError(msg) + refused = unreadable(entity) + if refused is not None: + raise TypeError(refused) + if inspect.isabstract(entity): + left = ", ".join(sorted(entity.__abstractmethods__)) + msg = f"{entity.__name__} does not define {left}, which its base leaves abstract" + raise TypeError(msg) + return kind + + +_CORE: Final[tuple[type[MetadataEntity], ...]] = ( + BloscCodec, + BytesCodec, + Crc32cCodec, + GzipCodec, + ShardingIndexedCodec, + TransposeCodec, + BoolDataType, + Int8DataType, + Int16DataType, + Int32DataType, + Int64DataType, + Uint8DataType, + Uint16DataType, + Uint32DataType, + Uint64DataType, + Float16DataType, + Float32DataType, + Float64DataType, + Complex64DataType, + Complex128DataType, + RawBytesDataType, + RegularChunkGrid, + DefaultChunkKeyEncoding, + V2ChunkKeyEncoding, +) +"""What the Zarr v3 specification itself defines.""" + +_EXTENSIONS: Final[tuple[type[MetadataEntity], ...]] = ( + CastValueCodec, + ScaleOffsetCodec, + ZstdCodec, + BytesDataType, + StringDataType, + NumpyDatetime64DataType, + NumpyTimedelta64DataType, + StructDataType, + RectilinearChunkGrid, +) +"""What `zarr-extensions` registers and this package models.""" + +CORE: Final = Context.of(*_CORE) +"""Only what the Zarr v3 specification defines.""" + +CORE_AND_EXTENSIONS: Final = Context.of(*_CORE, *_EXTENSIONS) +"""What the specification defines, plus what `zarr-extensions` registers.""" diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/_typed_json.py b/packages/zarr-metadata/src/zarr_metadata/v3/_typed_json.py new file mode 100644 index 0000000000..95fb8668e9 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/v3/_typed_json.py @@ -0,0 +1,885 @@ +"""JSON values parsed by type annotation. + +A dataclass's fields are its schema, and this module reads that schema +in both directions. `parser_for` turns a field annotation into a parser: +a function of a JSON value and its location that returns the typed +value and every problem found with it. `writer_for` turns the same +annotation into the parser's inverse: a function of the typed value that +returns the JSON a document writes for it. The annotations they read are the shapes JSON +takes and no others -- `int`, `float` for any number, `bool`, `str`, +`JSONValue`, a `Literal` of names, `tuple[T, ...]` and `tuple[T1, T2]`, +a union of those, an object described by a TypedDict or a record +dataclass, `Mapping[str, V]`, a `NewType` as the type it names -- which +is what keeps it small. `UNSET` in a union says the member may be +absent; `is_optional` reads that, and a parser only ever sees a present +value. + +Nothing here knows what a metadata entity is. A caller with a shape of +its own -- a field that holds another entity, read through a scope -- +passes a `leaf`, which is asked first for every annotation at every +depth; the parser it returns is used as it is. A parser is compiled +once per annotation and takes the reading it runs in as an argument, +so what varies between reads never has to be compiled into it. +""" + +from __future__ import annotations + +import functools +import sys +import types +from collections.abc import Callable, Mapping, Sequence +from copy import deepcopy +from dataclasses import is_dataclass +from typing import ( + TYPE_CHECKING, + Annotated, + ClassVar, + Literal, + NewType, + NotRequired, + Required, + TypeAlias, + TypeVar, + Union, + cast, + get_args, + get_origin, + get_type_hints, +) + +from typing_extensions import ReadOnly, TypeIs, is_typeddict + +from zarr_metadata._common import JSONValue +from zarr_metadata.model._sentinel import UNSET +from zarr_metadata.model._validation import ValidationProblem, is_json + +if TYPE_CHECKING: + from zarr_metadata.model._validation import ProblemKind + + +S = TypeVar("S") +"""The reading a parser runs in: whatever the caller hands through, untouched here.""" + +Loc: TypeAlias = tuple[str | int, ...] +"""Where in a document a value sits: the keys and indices down to it.""" + +Parsed: TypeAlias = tuple[object, tuple[ValidationProblem, ...]] +"""What a parser returns: the typed value, and every problem found with it.""" + +Parser: TypeAlias = Callable[[object, Loc, S], Parsed] +"""One value against one annotation, located at `loc`, in a reading `S`. + +A parser is compiled once from the annotation and run many times; what +varies between runs -- for a caller whose leaf reads a nested entity, +the scope to read it in -- is the reading, an argument every parser +hands down to the parsers it is built from and reads nothing of itself. +""" + +Leaf: TypeAlias = Callable[[object], "Parser[S] | None"] +"""A caller's own shapes: asked first for every annotation, None to decline.""" + +Writer: TypeAlias = Callable[[object], JSONValue] +"""A typed value as the JSON a document writes for it: a parser's inverse over the same annotation.""" + +WriterLeaf: TypeAlias = Callable[[object], "Writer | None"] +"""A caller's own shapes, for writing: asked first for every annotation, None to decline.""" + +RecordWriter: TypeAlias = Callable[[object], dict[str, JSONValue]] +"""A record dataclass as the JSON object a document writes for it.""" + + +def problem( + loc: Loc, message: str, kind: ProblemKind = "invalid_type" +) -> tuple[ValidationProblem, ...]: + """One problem, as the one-element tuple every parser returns. + + A tuple so that a parser can return it directly and a rule can + `found.extend(problem(...))` and raise `MetadataValidationError(found)` + once. The default `kind` names a type mismatch; a value rule passes + `"invalid_value"`. + """ + return (ValidationProblem(loc, message, kind),) + + +def is_integer(value: object) -> TypeIs[int]: + """A JSON integer: an `int`, and not a `bool`. + + `True` is an `int` in Python and `true` is not a number in JSON, so + the two have to be told apart everywhere a number is expected. + """ + return not isinstance(value, bool) and isinstance(value, int) + + +# --- annotations --------------------------------------------------------- + + +def strip_annotation(annotation: object) -> tuple[object, tuple[object, ...]]: + """An annotation's type, and the metadata `Annotated` wrapped it in. + + `NotRequired`, `Required` and `ReadOnly` are qualifiers on a TypedDict + key, not part of the value's type; peeled with the `Annotated` layers, + in whatever order they were written. + """ + metadata: list[object] = [] + while True: + origin = get_origin(annotation) + if origin is Annotated: + inner, *extras = get_args(annotation) + metadata.extend(extras) + annotation = inner + elif origin in (NotRequired, Required, ReadOnly): + (annotation,) = get_args(annotation) + else: + return annotation, tuple(metadata) + + +def is_not_required(annotation: object) -> bool: + """Whether a TypedDict key is marked `NotRequired`, under whatever qualifiers.""" + while True: + origin = get_origin(annotation) + if origin is NotRequired: + return True + if origin is Annotated: + annotation = get_args(annotation)[0] + elif origin in (Required, ReadOnly): + (annotation,) = get_args(annotation) + else: + return False + + +def is_union(annotation: object) -> bool: + return get_origin(annotation) in (Union, types.UnionType) + + +def is_optional(annotation: object) -> bool: + """Whether a field may be absent: its type admits `UNSET`.""" + inner, _ = strip_annotation(annotation) + return is_union(inner) and any(arg is UNSET for arg in get_args(inner)) + + +def without_unset(inner: object) -> object: + """The type of a present value: the union less `UNSET`.""" + if not is_union(inner): + return inner + arguments = get_args(inner) + present = [argument for argument in arguments if argument is not UNSET] + if len(present) == len(arguments): + return inner + if len(present) == 1: + return present[0] + return Union[tuple(present)] # noqa: UP007 - built from a tuple, which `|` cannot be + + +def own_annotations(klass: type) -> dict[str, object]: + """A class's own annotations, unevaluated. + + From 3.14 a class does not carry an `__annotations__` dict until it is + asked for one, and asking evaluates every annotation at once -- so a + `ClassVar` naming something imported only for the type checker would + fail the whole class. `annotationlib` can hand them back as the text + they were written as, which is what the callers here want anyway: + class variables are skipped by text before anything is evaluated. + Earlier versions leave the dict on the class, strings or values as + the module chose. + """ + if sys.version_info >= (3, 14): + import annotationlib + + return dict(annotationlib.get_annotations(klass, format=annotationlib.Format.STRING)) + return dict(vars(klass).get("__annotations__", {})) + + +def is_class_var(annotation: object) -> bool: + """Whether an annotation says `ClassVar`. + + `from __future__ import annotations` leaves them as strings, so this + reads the text when it gets one -- the same thing `dataclasses` does, + and for the same reason: resolving the name needs a module namespace + that is not available while the class is still being built. + """ + if isinstance(annotation, str): + head = annotation.strip().split("[", 1)[0].strip() + return head.rsplit(".", 1)[-1] == "ClassVar" + return annotation is ClassVar or get_origin(annotation) is ClassVar + + +@functools.cache +def field_hints(cls: type) -> Mapping[str, object]: + """The dataclass fields of `cls`, resolved, base first. + + Each class's own annotations are resolved in that class's module, + and class variables are skipped *before* resolving, by text -- so a + `ClassVar` whose annotation names something imported only for the + type checker cannot fail registration. `@dataclass` sees the same + set, in the same order. + + Cached per class: a class's annotations are fixed once it exists, + and resolving them costs a third of a read. A name that does not + resolve raises, and a raise is not cached. Read-only, since every + caller shares the one mapping. + """ + hints: dict[str, object] = {} + for ancestor in reversed(cls.__mro__): + raw = { + name: annotation + for name, annotation in own_annotations(ancestor).items() + if not is_class_var(annotation) + } + if len(raw) == 0: + continue + shell = type("_Fields", (), {"__annotations__": raw, "__module__": ancestor.__module__}) + hints.update(get_type_hints(shell, include_extras=True)) + return types.MappingProxyType(hints) + + +def declared_class_vars(cls: type) -> dict[str, type]: + """Every class variable annotated anywhere in `cls`'s ancestry. + + Mapped to the class that annotated it, so a message can say where the + requirement comes from. Base first, so a redeclaration names the + nearest ancestor. + """ + found: dict[str, type] = {} + for ancestor in reversed(cls.__mro__): + for name, annotation in own_annotations(ancestor).items(): + if is_class_var(annotation): + found[name] = ancestor + return found + + +# --- what a message calls a shape, and which shape a value has ------------- + + +def describe(annotation: object) -> str: + """The annotation as a message would name it: "an integer", "an object".""" + inner = without_unset(strip_annotation(annotation)[0]) + if inner is int: + return "an integer" + if inner is float: + return "a number" + if inner is bool: + return "a boolean" + if inner is str: + return "a string" + if inner is JSONValue: + return "a JSON value" + origin = get_origin(inner) + if origin is Literal: + return f"one of {tuple(sorted(get_args(inner), key=repr))!r}" + if is_union(inner): + return " or ".join(describe(branch) for branch in get_args(inner)) + if origin is tuple: + arguments = get_args(inner) + if len(arguments) == 2 and arguments[1] is Ellipsis: + return f"an array of {describe(arguments[0])} elements" + if len(arguments) == 2: + return f"a [{describe(arguments[0])}, {describe(arguments[1])}] pair" + return f"an array of {len(arguments)} elements" + if origin in (Mapping, dict): + return "an object" + if isinstance(inner, NewType): + return describe(inner.__supertype__) + if is_typeddict(inner) or is_dataclass(inner): + return "an object" + return "a value" + + +def shape_of(annotation: object) -> str | None: + """The top-level JSON shape an annotation admits, for choosing a union branch. + + None means any shape -- a JSON value, a union that mixes them, or a + shape the caller's `leaf` reads. + """ + inner = without_unset(strip_annotation(annotation)[0]) + if inner is int: + return "int" + if inner is float: + return "number" + if inner is bool: + return "bool" + if inner is str: + return "str" + origin = get_origin(inner) + if origin is Literal: + # `True` is a bool before it is an int, as `is_integer` says. + values: tuple[object, ...] = get_args(inner) + shapes = {shape_of(type(value)) for value in values} + return shapes.pop() if len(shapes) == 1 else None + if origin is tuple: + return "tuple" + if origin in (Mapping, dict): + return "mapping" + if isinstance(inner, NewType): + return shape_of(inner.__supertype__) + if is_typeddict(inner) or is_dataclass(inner): + return "mapping" + return None + + +def has_shape(shape: str | None, value: object) -> bool: + if shape is None: + return True + if shape == "int": + return is_integer(value) + if shape == "number": + return not isinstance(value, bool) and isinstance(value, (int, float)) + if shape == "bool": + return isinstance(value, bool) + if shape == "str": + return isinstance(value, str) + if shape == "tuple": + return isinstance(value, tuple) + return isinstance(value, Mapping) # "mapping" + + +# --- the parsers --------------------------------------------------------- + + +def _scalar(description: str, admits: Callable[[object], bool]) -> Parser[object]: + def parse(value: object, loc: Loc, state: object) -> Parsed: + if admits(value): + return value, () + return value, problem(loc, f"expected {description}, got {value!r}") + + return parse + + +_INTEGER: Parser[object] = _scalar("an integer", is_integer) +_NUMBER: Parser[object] = _scalar( + "a number", lambda value: not isinstance(value, bool) and isinstance(value, (int, float)) +) +_BOOLEAN: Parser[object] = _scalar("a boolean", lambda value: isinstance(value, bool)) +_STRING: Parser[object] = _scalar("a string", lambda value: isinstance(value, str)) +_JSON: Parser[object] = _scalar("a JSON value", is_json) + + +def one_of(allowed: tuple[object, ...]) -> Parser[object]: + """A member whose type is a closed set of values. + + Equal and of the same type: JSON `true` is not the integer 1, though + Python says `True == 1`. + """ + + def parse(value: object, loc: Loc, state: object) -> Parsed: + if not any(value == entry and type(value) is type(entry) for entry in allowed): + return value, problem( + loc, f"expected one of {allowed!r}, got {value!r}", "invalid_value" + ) + return value, () + + return parse + + +def sequence_of(element: Parser[S]) -> Parser[S]: + """A member whose type is an array of one element type, parsed element by element.""" + + def parse(value: object, loc: Loc, state: S) -> Parsed: + if not isinstance(value, (list, tuple)): + return value, problem(loc, f"expected a sequence, got {value!r}") + entries = cast("list[object] | tuple[object, ...]", value) + parsed: list[object] = [] + found: list[ValidationProblem] = [] + for index, entry in enumerate(entries): + item, problems = element(entry, (*loc, index), state) + parsed.append(item) + found.extend(problems) + return tuple(parsed), tuple(found) + + return parse + + +def fixed_tuple(elements: Sequence[Parser[S]], description: str) -> Parser[S]: + """A member whose type is an array of a fixed length, parsed position by position.""" + + def parse(value: object, loc: Loc, state: S) -> Parsed: + if not isinstance(value, (list, tuple)): + return value, problem(loc, f"expected {description}, got {value!r}") + entries = tuple(cast("list[object] | tuple[object, ...]", value)) + if len(entries) != len(elements): + return entries, problem(loc, f"expected {description}, got {entries!r}") + parsed: list[object] = [] + found: list[ValidationProblem] = [] + for position, (element, entry) in enumerate(zip(elements, entries, strict=True)): + item, problems = element(entry, (*loc, position), state) + parsed.append(item) + found.extend(problems) + return tuple(parsed), tuple(found) + + return parse + + +def any_of(branches: Sequence[tuple[object, Parser[S]]], description: str) -> Parser[S]: + """A member whose type is a union of shapes, parsed by the branch it fits. + + The branch whose top-level shape the value has is the one that + reports -- so an element inside a malformed array is located inside + the array, rather than the whole array being called wrong. A value + fitting no branch's shape is reported once, by what was expected; + one fitting several is parsed by the first that accepts it, and + reported by the first that does not. + """ + + def parse(value: object, loc: Loc, state: S) -> Parsed: + first: Parsed | None = None + for annotation, branch in branches: + if not has_shape(shape_of(annotation), value): + continue + result = branch(value, loc, state) + if len(result[1]) == 0: + return result + if first is None: + first = result + if first is None: + return value, problem(loc, f"expected {description}, got {value!r}") + return first + + return parse + + +Members: TypeAlias = Mapping[str, tuple[bool, "Parser[S]"]] +"""An object's declared keys: whether each is required, and its parser.""" + + +def _keys( + members: Members[S], entries: Mapping[str, object], loc: Loc, state: S +) -> tuple[dict[str, object], tuple[ValidationProblem, ...]]: + """The declared keys of one object, each parsed at its own key. + + Closed, like every configuration in this package: a key the type does + not declare is `unknown_key`, a required one missing is `missing_key`, + both located at the key. An optional key left out is parsed as + `UNSET`, so a record never depends on a default for it. + """ + parsed: dict[str, object] = {} + found: list[ValidationProblem] = [] + for key in entries: + if key not in members: + found.extend(problem((*loc, key), f"unexpected key {key!r}", "unknown_key")) + for key, (required, member) in members.items(): + if key not in entries: + if required: + found.extend(problem((*loc, key), f"missing required key {key!r}", "missing_key")) + else: + parsed[key] = UNSET + continue + item, problems = member(entries[key], (*loc, key), state) + parsed[key] = item + found.extend(problems) + return parsed, tuple(found) + + +def object_of(members: Members[S]) -> Parser[S]: + """A member that is itself an object with declared keys, kept as the mapping it came as. + + A key the type does not declare is reported and kept: the member + still says what the document said. + """ + + def parse(value: object, loc: Loc, state: S) -> Parsed: + if not isinstance(value, Mapping): + return value, problem(loc, f"expected an object, got {value!r}") + entries = cast("Mapping[str, object]", value) + parsed, found = _keys(members, entries, loc, state) + return {**entries, **{key: item for key, item in parsed.items() if key in entries}}, found + + return parse + + +def record_of(record: Callable[..., object], members: Members[S]) -> Parser[S]: + """A member that is itself an object with declared keys, built as a dataclass. + + Built only from an object whose every key read; otherwise the value + comes back as it came, with the reasons. A record has no rules of + its own, and every member it is built from has just been checked, + so building it cannot fail. + """ + + def parse(value: object, loc: Loc, state: S) -> Parsed: + if not isinstance(value, Mapping): + return value, problem(loc, f"expected an object, got {value!r}") + entries = cast("Mapping[str, object]", value) + parsed, found = _keys(members, entries, loc, state) + if any(entry.kind != "unknown_key" for entry in found): + return entries, found + return record(**parsed), found + + return parse + + +def mapping_of(value: Parser[S]) -> Parser[S]: + """A member whose type is an object with any keys, parsed value by value. + + The open counterpart of `object_of`: a `Mapping[str, V]` says nothing + about which keys there are, only what each value must be. + """ + + def parse(candidate: object, loc: Loc, state: S) -> Parsed: + if not isinstance(candidate, Mapping): + return candidate, problem(loc, f"expected an object, got {candidate!r}") + entries = cast("Mapping[str, object]", candidate) + parsed: dict[str, object] = {} + found: list[ValidationProblem] = [] + for key, entry in entries.items(): + item, problems = value(entry, (*loc, key), state) + parsed[key] = item + found.extend(problems) + return parsed, tuple(found) + + return parse + + +# --- the compiler -------------------------------------------------------- + + +def members_of(annotations: Mapping[str, object], leaf: Leaf[S]) -> Members[S] | None: + """A member table for an object's keys; None if any key's type has no parser. + + For a caller with a record class of its own to hand to `record_of`; + `parser_for` builds the same table for a record it meets as an + annotation. + """ + members: dict[str, tuple[bool, Parser[S]]] = {} + for key, annotation in annotations.items(): + member = parser_for(annotation, leaf) + if member is None: + return None + required = not is_not_required(annotation) and not is_optional(annotation) + members[key] = (required, member) + return members + + +def _union(inner: object, leaf: Leaf[S]) -> Parser[S] | None: + compiled = [(branch, parser_for(branch, leaf)) for branch in get_args(inner)] + branches = [(branch, member) for branch, member in compiled if member is not None] + if len(branches) != len(compiled): + return None + return any_of(branches, describe(inner)) + + +def _tuple(inner: object, leaf: Leaf[S]) -> Parser[S] | None: + arguments = get_args(inner) + if len(arguments) == 2 and arguments[1] is Ellipsis: + element = parser_for(arguments[0], leaf) + return None if element is None else sequence_of(element) + compiled = [parser_for(argument, leaf) for argument in arguments] + elements = [element for element in compiled if element is not None] + if len(elements) != len(compiled): + return None + return fixed_tuple(elements, describe(inner)) + + +def _mapping(inner: object, leaf: Leaf[S]) -> Parser[S] | None: + arguments = get_args(inner) + if len(arguments) != 2 or arguments[0] is not str: + return None + value = parser_for(arguments[1], leaf) + return None if value is None else mapping_of(value) + + +def no_leaf(annotation: object) -> Parser[object] | None: + """The leaf of a caller with no shapes of its own.""" + return None + + +def parser_for(annotation: object, leaf: Leaf[S]) -> Parser[S] | None: + """The parser a field annotation implies, or None if it implies none. + + A small compiler over the shapes JSON takes and no others, listed in + the module docstring. `leaf` is asked first, here and at every depth + -- inside a union, an array, an object -- and what it returns is + used as it is. Closed: an annotation outside these implies no + parser, and an entity declaring one is refused at registration. The + field is written as one of these shapes instead, with any finer + rule in the configuration's `problems`. + """ + inner = without_unset(strip_annotation(annotation)[0]) + found = leaf(inner) + if found is not None: + return found + if inner is int: + return _INTEGER + if inner is float: + return _NUMBER + if inner is bool: + return _BOOLEAN + if inner is str: + return _STRING + if inner is JSONValue: + return _JSON + if get_origin(inner) is Literal: + # Sorted, because the order `get_args` reports is not the order + # the `Literal` was written in: two `Literal`s over the same + # values compare and hash equal, so the first one built anywhere + # in the process is the one every later one resolves to. The + # parse is a membership test either way; this is so the message + # listing the values does not depend on import order. + return one_of(tuple(sorted(get_args(inner), key=repr))) + if is_union(inner): + return _union(inner, leaf) + if get_origin(inner) is tuple: + return _tuple(inner, leaf) + if is_typeddict(inner): + members = members_of(get_type_hints(inner, include_extras=True), leaf) + return None if members is None else object_of(members) + if get_origin(inner) in (Mapping, dict): + return _mapping(inner, leaf) + if isinstance(inner, NewType): + # A `NewType` is its supertype to a document; the distinction is + # the code's, for a value it has vouched for. + return parser_for(inner.__supertype__, leaf) + # Last, because `is_dataclass` narrows what pyright knows of `inner` + # for every line after it. + if isinstance(inner, type) and is_dataclass(inner): + if "__post_init__" in vars(inner): + # A record is plain data, built whenever its keys read; a + # rule about it belongs in the configuration's `problems`, + # which is asked for every problem rather than the first. + msg = ( + f"{inner.__name__} defines __post_init__; a record is plain data, and a rule " + "about it belongs in the configuration's `problems`" + ) + raise TypeError(msg) + members = members_of(field_hints(inner), leaf) + return None if members is None else record_of(inner, members) + return None + + +def parser(annotation: object, leaf: Leaf[S]) -> Parser[S]: + """The parser a field annotation implies; `TypeError` if it implies none.""" + found = parser_for(annotation, leaf) + if found is None: + msg = f"{annotation!r} is not a shape JSON takes" + raise TypeError(msg) + return found + + +# --- the writers --------------------------------------------------------- +# +# The inverse of each parser, over the same annotation. A writer is asked +# for a value the entity holds, which its field's type vouches for; the +# checks here are what stand between a hand-built entity holding +# something that is not JSON and a document that is not JSON. + + +def _not_json(value: object) -> TypeError: + return TypeError( + f"{value!r} is not a JSON value; an entity's members are the JSON the document writes" + ) + + +def _as_json(value: object) -> JSONValue: + """A scalar or a JSON value as it is.""" + if is_json(value): + return value + raise _not_json(value) + + +def _copied(value: object) -> JSONValue: + """A JSON value, copied: the document handed out is not a handle on a frozen entity.""" + if is_json(value): + return deepcopy(value) + raise _not_json(value) + + +def each_of(element: Writer) -> Writer: + """An array, each element written by its type.""" + + def write(value: object) -> JSONValue: + if not isinstance(value, (list, tuple)): + raise _not_json(value) + return tuple(element(entry) for entry in cast("list[object] | tuple[object, ...]", value)) + + return write + + +def positions_of(elements: Sequence[Writer]) -> Writer: + """An array of a fixed length, each position written by its type.""" + + def write(value: object) -> JSONValue: + if not isinstance(value, (list, tuple)): + raise _not_json(value) + entries = cast("list[object] | tuple[object, ...]", value) + if len(entries) != len(elements): + raise _not_json(entries) + return tuple(element(entry) for element, entry in zip(elements, entries, strict=True)) + + return write + + +def one_of_writers(branches: Sequence[tuple[object, Writer]]) -> Writer: + """A union, written by the branch whose shape the value has, as the parser chose it.""" + + def write(value: object) -> JSONValue: + for annotation, branch in branches: + if has_shape(shape_of(annotation), value): + return branch(value) + return _as_json(value) + + return write + + +def keys_of(members: Mapping[str, Writer]) -> Writer: + """An object kept as a mapping: declared keys written by their types, the rest copied.""" + + def write(value: object) -> JSONValue: + if not isinstance(value, Mapping): + raise _not_json(value) + entries = cast("Mapping[str, object]", value) + return { + key: members[key](entry) if key in members else _copied(entry) + for key, entry in entries.items() + } + + return write + + +def fields_of(members: Mapping[str, Writer]) -> RecordWriter: + """A record dataclass as an object: each field written by its type, an absent optional one left out.""" + + def write(value: object) -> dict[str, JSONValue]: + written: dict[str, JSONValue] = {} + for key, member in members.items(): + entry = getattr(value, key) + if entry is not UNSET: + written[key] = member(entry) + return written + + return write + + +def values_of(value: Writer) -> Writer: + """An object of any keys, each value written by its type.""" + + def write(candidate: object) -> JSONValue: + if not isinstance(candidate, Mapping): + raise _not_json(candidate) + entries = cast("Mapping[str, object]", candidate) + return {key: value(entry) for key, entry in entries.items()} + + return write + + +def _writers_of(annotations: Mapping[str, object], leaf: WriterLeaf) -> dict[str, Writer] | None: + members: dict[str, Writer] = {} + for key, annotation in annotations.items(): + member = writer_for(annotation, leaf) + if member is None: + return None + members[key] = member + return members + + +def no_writer_leaf(annotation: object) -> Writer | None: + """The leaf of a caller with no shapes of its own.""" + return None + + +def writer_for(annotation: object, leaf: WriterLeaf) -> Writer | None: + """The writer a field annotation implies, or None if it implies none. + + The inverse of `parser_for` over the same shapes: what the parser + reads from a document, the writer puts back. `leaf` is asked first, + here and at every depth, as the parser's is. + """ + inner = without_unset(strip_annotation(annotation)[0]) + found = leaf(inner) + if found is not None: + return found + if inner is int or inner is float or inner is bool or inner is str: + return _as_json + if get_origin(inner) is Literal: + return _as_json + if inner is JSONValue: + return _copied + if is_union(inner): + compiled = [(branch, writer_for(branch, leaf)) for branch in get_args(inner)] + branches = [(branch, member) for branch, member in compiled if member is not None] + return one_of_writers(branches) if len(branches) == len(compiled) else None + if get_origin(inner) is tuple: + arguments = get_args(inner) + if len(arguments) == 2 and arguments[1] is Ellipsis: + element = writer_for(arguments[0], leaf) + return None if element is None else each_of(element) + compiled = [writer_for(argument, leaf) for argument in arguments] + elements = [element for element in compiled if element is not None] + return positions_of(elements) if len(elements) == len(compiled) else None + if is_typeddict(inner): + members = _writers_of(get_type_hints(inner, include_extras=True), leaf) + return None if members is None else keys_of(members) + if get_origin(inner) in (Mapping, dict): + arguments = get_args(inner) + if len(arguments) != 2 or arguments[0] is not str: + return None + value = writer_for(arguments[1], leaf) + return None if value is None else values_of(value) + if isinstance(inner, NewType): + return writer_for(inner.__supertype__, leaf) + if isinstance(inner, type) and is_dataclass(inner): + members = _writers_of(field_hints(inner), leaf) + return None if members is None else fields_of(members) + return None + + +def writer(annotation: object, leaf: WriterLeaf) -> Writer: + """The writer a field annotation implies; `TypeError` if it implies none.""" + found = writer_for(annotation, leaf) + if found is None: + msg = f"{annotation!r} is not a shape JSON takes" + raise TypeError(msg) + return found + + +def record_writer(record: type, leaf: WriterLeaf) -> RecordWriter: + """The writer of a record dataclass, as the object it writes; `TypeError` for a field no writer reads.""" + members = _writers_of(field_hints(record), leaf) + if members is None: + msg = f"{record.__name__} has a field that is not a shape JSON takes" + raise TypeError(msg) + return fields_of(members) + + +__all__ = [ + "Leaf", + "Loc", + "Members", + "Parsed", + "Parser", + "RecordWriter", + "Writer", + "WriterLeaf", + "any_of", + "declared_class_vars", + "describe", + "each_of", + "field_hints", + "fields_of", + "fixed_tuple", + "has_shape", + "is_class_var", + "is_integer", + "is_not_required", + "is_optional", + "is_union", + "keys_of", + "mapping_of", + "members_of", + "no_leaf", + "no_writer_leaf", + "object_of", + "one_of", + "one_of_writers", + "own_annotations", + "parser", + "parser_for", + "positions_of", + "problem", + "record_of", + "record_writer", + "sequence_of", + "shape_of", + "strip_annotation", + "values_of", + "without_unset", + "writer", + "writer_for", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/chunk_grid/rectilinear.py b/packages/zarr-metadata/src/zarr_metadata/v3/chunk_grid/rectilinear.py index 78c38b702a..c039e7524c 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/chunk_grid/rectilinear.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/chunk_grid/rectilinear.py @@ -4,13 +4,35 @@ See https://github.com/zarr-developers/zarr-extensions/blob/4da7b37a84f76e660902f6d3de3eaef0e0febae6/chunk-grids/rectilinear/README.md """ -from typing import Final, Literal, NotRequired +from dataclasses import dataclass +from typing import TYPE_CHECKING, ClassVar, Final, Literal, NotRequired, Self, cast from typing_extensions import TypedDict +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.v3._entity import ( + ChunkGridEntity, + Configuration, + Loc, + is_integer, + problem, +) +from zarr_metadata.v3._parts import ChunkGrid + +if TYPE_CHECKING: + from collections.abc import Iterator, Sequence + + RECTILINEAR_CHUNK_GRID_NAME: Final = "rectilinear" """The `name` field value of the rectilinear chunk grid.""" +RECTILINEAR_CHUNK_GRID_KIND: Final = ("inline",) +"""The `kind` values the rectilinear grid defines. + +Only `inline` so far: the extents are written into the metadata. The +member exists so a later kind can put them somewhere else. +""" + RectilinearChunkGridName = Literal["rectilinear"] """Literal type of the `name` field of the rectilinear chunk grid.""" @@ -47,11 +69,188 @@ class RectilinearChunkGridObject(TypedDict, closed=True): https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L1562-L1564 """ + +def _not_positive(loc: Loc, value: int) -> ValidationProblem: + return ValidationProblem(loc, f"expected an integer >= 1, got {value}", "invalid_value") + + +def canonical_dim_spec(spec: RectilinearDimSpec) -> RectilinearDimSpec: + """One dimension's chunk sizes in their simplest equivalent form. + + Runs of equal sizes collapse to `[size, count]` pairs, because that is + the spelling that does not grow with the number of chunks: a million + equal chunks is two numbers, not a million. A run of one stays a bare + size, and `[size, 1]` collapses to one, since a pair says nothing extra + there. Adjacent spellings of the same size merge, which is what makes + this idempotent: `[[32, 2], 32]` and `[32, [32, 2]]` both become + `[[32, 3]]`. + + A dimension-level bare integer is left alone. It is a *step* that + repeats until it covers the extent, so it is not equivalent to any + fixed list — expanding it would pin a grid that currently adapts, and + the two would diverge the moment the array were resized. For the same + reason a one-element list is never collapsed to a bare integer: + `[32]` declares exactly one chunk and `32` declares as many as it takes. + + Assumes a spec the shape validator has already accepted. + """ + if not isinstance(spec, tuple): + return spec + runs: list[tuple[int, int]] = [] + for entry in spec: + size, count = entry if isinstance(entry, tuple) else (entry, 1) + if len(runs) != 0 and runs[-1][0] == size: + runs[-1] = (size, runs[-1][1] + count) + else: + runs.append((size, count)) + return tuple(size if count == 1 else (size, count) for size, count in runs) + + +def canonical_chunk_shapes( + chunk_shapes: tuple[RectilinearDimSpec, ...], +) -> tuple[RectilinearDimSpec, ...]: + """Every dimension's chunk sizes in their simplest equivalent form.""" + return tuple(canonical_dim_spec(spec) for spec in chunk_shapes) + + __all__ = [ + "RECTILINEAR_CHUNK_GRID_KIND", "RECTILINEAR_CHUNK_GRID_NAME", + "RectilinearChunkGrid", "RectilinearChunkGridConfiguration", "RectilinearChunkGridMetadata", "RectilinearChunkGridName", "RectilinearChunkGridObject", + "RectilinearChunkGridOptions", "RectilinearDimSpec", + "canonical_chunk_shapes", + "canonical_dim_spec", ] + + +def _covered_extent(spec: tuple[int | tuple[int, int], ...]) -> int | None: + """How much of a dimension an explicit spec covers, or None. + + None when any entry is non-positive: `problems` reports that, and a + total computed from a nonsense entry would be nonsense too. + """ + total = 0 + for item in spec: + if isinstance(item, int): + if item < 1: + return None + total += item + continue + size, count = item + if size < 1 or count < 1: + return None + total += size * count + return total + + +def _axis_lengths(spec: RectilinearDimSpec) -> frozenset[int] | None: + """The lengths one dimension's chunks take, or None if undetermined. + + A bare integer is a regular step, so every chunk is that long. An + explicit list names them, with `[size, count]` pairs standing for + repeats; the distinct sizes are what any divisibility question needs. + None for a non-positive length, which `problems` reports -- a grid + that does not tile answers nothing about what divides it. + """ + if isinstance(spec, int): + return frozenset({spec}) if spec > 0 else None + lengths: set[int] = set() + for item in spec: + size = item if isinstance(item, int) else item[0] + if size < 1 or (not isinstance(item, int) and item[1] < 1): + return None + lengths.add(size) + return frozenset(lengths) if len(lengths) != 0 else None + + +@dataclass(frozen=True) +class RectilinearChunkGridOptions(Configuration): + """What a `rectilinear` grid is configured with.""" + + kind: Literal["inline"] + chunk_shapes: tuple[RectilinearDimSpec, ...] + + def problems(self) -> "Iterator[ValidationProblem]": + """Every extent, and every run's length and count, is at least 1.""" + for axis, spec in enumerate(self.chunk_shapes): + if isinstance(spec, int): + if spec < 1: + yield _not_positive(("chunk_shapes", axis), spec) + continue + for index, entry in enumerate(spec): + if isinstance(entry, int): + if entry < 1: + yield _not_positive(("chunk_shapes", axis, index), entry) + continue + for position, value in enumerate(entry): + if value < 1: + yield _not_positive(("chunk_shapes", axis, index, position), value) + + +@dataclass(frozen=True) +class RectilinearChunkGrid(ChunkGridEntity): + """The `rectilinear` chunk grid, coerced from its metadata.""" + + configuration: RectilinearChunkGridOptions + + identifier: ClassVar[str] = RECTILINEAR_CHUNK_GRID_NAME + + def shape_problems(self, array_shape: object) -> tuple[ValidationProblem, ...]: + """One spec per dimension, and explicit specs must cover it. + + A bare integer is uniform shorthand, so it covers whatever the + dimension turns out to be and imposes no sum; an explicit list + names every chunk, so the names have to add up. + """ + if not isinstance(array_shape, (list, tuple)): + return () + extents = tuple(cast("Sequence[object]", array_shape)) + if len(self.configuration.chunk_shapes) != len(extents): + return problem( + ("chunk_shapes",), + f"chunk_shapes has {len(self.configuration.chunk_shapes)} entries but shape has " + f"{len(extents)} dimensions", + "invalid_value", + ) + found: list[ValidationProblem] = [] + for dim, (spec, extent) in enumerate( + zip(self.configuration.chunk_shapes, extents, strict=True) + ): + if isinstance(spec, int) or not is_integer(extent): + continue + total = _covered_extent(spec) + if total is not None and total < extent: + found.extend( + problem( + ("chunk_shapes", dim), + f"chunk sizes sum to {total} but must cover shape[{dim}] extent {extent}", + "invalid_value", + ) + ) + return tuple(found) + + def grid(self, array_shape: object) -> ChunkGrid: + """The distinct lengths each axis's chunks take. + + Plural per axis, which is the point of a rectilinear grid: an + axis of `[30, 34]` gives `{30, 34}`, and anything asking about + divisibility has to hold for both. + """ + return ChunkGrid.derived( + tuple(_axis_lengths(spec) for spec in self.configuration.chunk_shapes) + ) + + def canonical(self) -> Self: + """Run-length encoded, which is the spelling that does not grow. + + Two dimension specs listing the same extents describe the same + grid, and the encoded one stays the same size as the array grows. + """ + return self.with_configuration( + chunk_shapes=canonical_chunk_shapes(self.configuration.chunk_shapes) + ) diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/chunk_grid/regular.py b/packages/zarr-metadata/src/zarr_metadata/v3/chunk_grid/regular.py index bdcd9e06c5..a34d65f6a6 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/chunk_grid/regular.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/chunk_grid/regular.py @@ -4,10 +4,23 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#regular-grids """ -from typing import Final, Literal, NotRequired +from dataclasses import dataclass +from typing import TYPE_CHECKING, ClassVar, Final, Literal, NotRequired, cast from typing_extensions import TypedDict +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.v3._entity import ( + ChunkGridEntity, + Configuration, + problem, +) +from zarr_metadata.v3._parts import ChunkGrid + +if TYPE_CHECKING: + from collections.abc import Iterator, Sequence + + REGULAR_CHUNK_GRID_NAME: Final = "regular" """The `name` field value of the regular chunk grid.""" @@ -40,8 +53,53 @@ class RegularChunkGridObject(TypedDict, closed=True): __all__ = [ "REGULAR_CHUNK_GRID_NAME", + "RegularChunkGrid", "RegularChunkGridConfiguration", "RegularChunkGridMetadata", "RegularChunkGridName", "RegularChunkGridObject", + "RegularChunkGridOptions", ] + + +@dataclass(frozen=True) +class RegularChunkGridOptions(Configuration): + """What a `regular` grid is configured with.""" + + chunk_shape: tuple[int, ...] + + def problems(self) -> "Iterator[ValidationProblem]": + for index, extent in enumerate(self.chunk_shape): + if extent < 1: + yield ValidationProblem( + ("chunk_shape", index), + f"expected an integer >= 1, got {extent}", + "invalid_value", + ) + + +@dataclass(frozen=True) +class RegularChunkGrid(ChunkGridEntity): + """The `regular` chunk grid, coerced from its metadata.""" + + configuration: RegularChunkGridOptions + + identifier: ClassVar[str] = REGULAR_CHUNK_GRID_NAME + + def shape_problems(self, array_shape: object) -> tuple[ValidationProblem, ...]: + """A regular grid must chunk every array dimension.""" + if not isinstance(array_shape, (list, tuple)): + return () + extents = tuple(cast("Sequence[object]", array_shape)) + if len(self.configuration.chunk_shape) == len(extents): + return () + return problem( + ("chunk_shape",), + f"chunk_shape has {len(self.configuration.chunk_shape)} entries but shape has " + f"{len(extents)} dimensions", + "invalid_value", + ) + + def grid(self, array_shape: object) -> ChunkGrid: + """One extent per axis, the same for every chunk on that axis.""" + return ChunkGrid.regular(self.configuration.chunk_shape) diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/default.py b/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/default.py index 10a7c0cb55..057d9a9083 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/default.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/default.py @@ -7,10 +7,17 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#chunk-key-encoding """ -from typing import Final, Literal, NotRequired +from dataclasses import dataclass +from typing import ClassVar, Final, Literal, NotRequired from typing_extensions import TypedDict +from zarr_metadata.model._sentinel import UNSET +from zarr_metadata.v3._entity import ( + ChunkKeyEncodingEntity, + Configuration, +) + DEFAULT_CHUNK_KEY_ENCODING_NAME: Final = "default" """The `name` field value of the default chunk key encoding.""" @@ -55,9 +62,27 @@ class DefaultChunkKeyEncodingObject(TypedDict, closed=True): __all__ = [ "DEFAULT_CHUNK_KEY_ENCODING_NAME", "DEFAULT_CHUNK_KEY_ENCODING_SEPARATOR", + "DefaultChunkKeyEncoding", "DefaultChunkKeyEncodingConfiguration", "DefaultChunkKeyEncodingMetadata", "DefaultChunkKeyEncodingName", "DefaultChunkKeyEncodingObject", + "DefaultChunkKeyEncodingOptions", "DefaultChunkKeyEncodingSeparator", ] + + +@dataclass(frozen=True) +class DefaultChunkKeyEncodingOptions(Configuration): + """What the `default` encoding is configured with.""" + + separator: DefaultChunkKeyEncodingSeparator | UNSET = UNSET + + +@dataclass(frozen=True) +class DefaultChunkKeyEncoding(ChunkKeyEncodingEntity): + """The `default` chunk key encoding, coerced from its metadata.""" + + configuration: DefaultChunkKeyEncodingOptions + + identifier: ClassVar[str] = DEFAULT_CHUNK_KEY_ENCODING_NAME diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/v2.py b/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/v2.py index 63cdf26783..783c02726c 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/v2.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/v2.py @@ -13,10 +13,17 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#chunk-key-encoding """ -from typing import Final, Literal, NotRequired +from dataclasses import dataclass +from typing import ClassVar, Final, Literal, NotRequired from typing_extensions import TypedDict +from zarr_metadata.model._sentinel import UNSET +from zarr_metadata.v3._entity import ( + ChunkKeyEncodingEntity, + Configuration, +) + V2_CHUNK_KEY_ENCODING_NAME: Final = "v2" """The `name` field value of the v2 chunk key encoding.""" @@ -61,9 +68,27 @@ class V2ChunkKeyEncodingObject(TypedDict, closed=True): __all__ = [ "V2_CHUNK_KEY_ENCODING_NAME", "V2_CHUNK_KEY_ENCODING_SEPARATOR", + "V2ChunkKeyEncoding", "V2ChunkKeyEncodingConfiguration", "V2ChunkKeyEncodingMetadata", "V2ChunkKeyEncodingName", "V2ChunkKeyEncodingObject", + "V2ChunkKeyEncodingOptions", "V2ChunkKeyEncodingSeparator", ] + + +@dataclass(frozen=True) +class V2ChunkKeyEncodingOptions(Configuration): + """What the `v2` encoding is configured with.""" + + separator: V2ChunkKeyEncodingSeparator | UNSET = UNSET + + +@dataclass(frozen=True) +class V2ChunkKeyEncoding(ChunkKeyEncodingEntity): + """The `v2` chunk key encoding, coerced from its metadata.""" + + configuration: V2ChunkKeyEncodingOptions + + identifier: ClassVar[str] = V2_CHUNK_KEY_ENCODING_NAME diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/codec/__init__.py b/packages/zarr-metadata/src/zarr_metadata/v3/codec/__init__.py index c8a9a150fc..b3c4553daf 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/codec/__init__.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/codec/__init__.py @@ -14,6 +14,10 @@ `codecs` list and in sharding's inner pipelines), import `ZarrV3MetadataFieldJSON` from `zarr_metadata.v3`. +Each codec's pipeline position (`array -> array`, `array -> bytes`, +`bytes -> bytes`) is the kind class its entity subclasses, in +`zarr_metadata.v3.entity`. + See https://zarr-specs.readthedocs.io/en/latest/v3/codecs/index.html """ diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/codec/blosc.py b/packages/zarr-metadata/src/zarr_metadata/v3/codec/blosc.py index 3387a7c285..5675663f89 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/codec/blosc.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/codec/blosc.py @@ -4,10 +4,21 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/codecs/blosc/index.html """ -from typing import Final, Literal, NotRequired +from dataclasses import dataclass +from typing import TYPE_CHECKING, ClassVar, Final, Literal, NotRequired, Self from typing_extensions import TypedDict +from zarr_metadata.model._sentinel import UNSET +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.v3._entity import ( + BytesBytesCodec, + Configuration, +) + +if TYPE_CHECKING: + from collections.abc import Iterator + BLOSC_CODEC_NAME: Final = "blosc" """The `name` field value of the `blosc` codec.""" @@ -20,6 +31,14 @@ BLOSC_SHUFFLE: Final = ("noshuffle", "shuffle", "bitshuffle") """Tuple of permitted values for the `shuffle` field of the `blosc` codec.""" +BLOSC_NO_SHUFFLE: Final = "noshuffle" +"""The `shuffle` value under which `typesize` carries no information. + +The spec requires `typesize` "unless `shuffle` is `"noshuffle"`, in which +case the value is ignored", so this is the one value that changes whether +another member is required. +""" + BloscCName = Literal["lz4", "lz4hc", "blosclz", "snappy", "zlib", "zstd"] """Literal type of blosc compressor identifiers.""" @@ -55,14 +74,86 @@ class BloscCodecObject(TypedDict, closed=True): https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L1562-L1564 (short-hand names only "if no configuration metadata is required") """ + __all__ = [ "BLOSC_CNAME", "BLOSC_CODEC_NAME", + "BLOSC_NO_SHUFFLE", "BLOSC_SHUFFLE", "BloscCName", + "BloscCodec", "BloscCodecConfiguration", "BloscCodecMetadata", "BloscCodecName", "BloscCodecObject", + "BloscOptions", "BloscShuffle", ] + + +@dataclass(frozen=True) +class BloscOptions(Configuration): + """What `blosc` is configured with.""" + + cname: BloscCName + clevel: int + shuffle: BloscShuffle + blocksize: int + typesize: int | UNSET = UNSET + + def problems(self) -> "Iterator[ValidationProblem]": + """Bounds on `clevel` and `blocksize`; `typesize` against `shuffle`. + + Under `noshuffle` the spec says of `typesize` that "the value is + ignored", and `canonical` drops it; under either shuffle it is + required, and positive. + """ + if not 0 <= self.clevel <= 9: + yield ValidationProblem( + ("clevel",), f"expected an integer in [0, 9], got {self.clevel}", "invalid_value" + ) + if self.blocksize < 0: + yield ValidationProblem( + ("blocksize",), f"expected an integer >= 0, got {self.blocksize}", "invalid_value" + ) + if self.shuffle != BLOSC_NO_SHUFFLE: + if self.typesize is UNSET: + yield ValidationProblem( + ("typesize",), + f"typesize is required when shuffle is {self.shuffle!r}", + "missing_key", + ) + elif self.typesize < 1: + yield ValidationProblem( + ("typesize",), + f"expected a positive integer, got {self.typesize}", + "invalid_value", + ) + + +@dataclass(frozen=True) +class BloscCodec(BytesBytesCodec): + """The `blosc` codec, coerced from its metadata. + + Everything blosc knows about itself: the shape its metadata takes, the + values the spec allows in it, and the simplest spelling of an + equivalent document. + """ + + configuration: BloscOptions + + identifier: ClassVar[str] = BLOSC_CODEC_NAME + variable_size: ClassVar[bool] = True + + # Every member is required but `typesize`, which only means something + # when shuffling; `BloscOptions.problems` is where that conditional lives. + + def canonical(self) -> Self: + """Without a `typesize` that `noshuffle` renders meaningless. + + The spec says of that case that "the value is ignored", so two + documents differing only there describe the same codec. + """ + if self.configuration.shuffle != BLOSC_NO_SHUFFLE or self.configuration.typesize is UNSET: + return self + return self.with_configuration(typesize=UNSET) diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/codec/bytes.py b/packages/zarr-metadata/src/zarr_metadata/v3/codec/bytes.py index 4feb6b8c1c..4b676c5202 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/codec/bytes.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/codec/bytes.py @@ -4,10 +4,21 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/codecs/bytes/index.html """ -from typing import Final, Literal, NotRequired +from dataclasses import dataclass +from typing import ClassVar, Final, Literal, NotRequired from typing_extensions import TypedDict +from zarr_metadata.model._sentinel import UNSET +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.v3._entity import ( + ArrayBytesCodec, + Configuration, + DataTypeEntity, + problem, +) +from zarr_metadata.v3._parts import ArrayParts + BYTES_CODEC_NAME: Final = "bytes" """The `name` field value of the `bytes` codec.""" @@ -59,9 +70,60 @@ class BytesCodecObject(TypedDict, closed=True): __all__ = [ "BYTES_CODEC_NAME", "ENDIANNESS", + "BytesCodec", "BytesCodecConfiguration", "BytesCodecMetadata", "BytesCodecName", "BytesCodecObject", + "BytesOptions", "Endianness", ] + + +@dataclass(frozen=True) +class BytesOptions(Configuration): + """What `bytes` is configured with.""" + + endian: Endianness | UNSET = UNSET + + +@dataclass(frozen=True) +class BytesCodec(ArrayBytesCodec): + """The `bytes` codec, coerced from its metadata. + + `endian` is optional and absent means something: a one-byte data type + has no byte order to state, and the spec lets such an array omit it. + """ + + configuration: BytesOptions + + identifier: ClassVar[str] = BYTES_CODEC_NAME + variable_size: ClassVar[bool] = False + + def incoming_problems(self, incoming: ArrayParts | None) -> tuple[ValidationProblem, ...]: + """The data type reaching here must have a raw byte representation. + + A variable-length type has no fixed one, so this codec cannot + encode it. A multi-byte one has several orderings, so `endian` is + required -- and the message names the type, because inside a + shard's `index_codecs` the array is the shard index, whose + `uint64` type appears nowhere in the document. + """ + data_type = incoming.data_type if incoming is not None else None + if not isinstance(data_type, DataTypeEntity): + return () + storage = data_type.storage_class() + name = data_type.name + if storage == "variable_length": + return problem( + (), + f"bytes codec is not compatible with variable-length data_type {name!r}", + "invalid_value", + ) + if storage == "multi_byte" and self.configuration.endian is UNSET: + return problem( + ("endian",), + f"endian is required for data type {name!r}, which contains multi-byte values", + "missing_key", + ) + return () diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/codec/cast_value.py b/packages/zarr-metadata/src/zarr_metadata/v3/codec/cast_value.py index 656a509ed7..5571b2cc2a 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/codec/cast_value.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/codec/cast_value.py @@ -4,12 +4,25 @@ See https://github.com/zarr-developers/zarr-extensions/blob/4da7b37a84f76e660902f6d3de3eaef0e0febae6/codecs/cast_value/README.md """ -from typing import Final, Literal, NotRequired +from dataclasses import dataclass +from typing import TYPE_CHECKING, ClassVar, Final, Literal, NotRequired, Self from typing_extensions import TypedDict from zarr_metadata._common import JSONValue +from zarr_metadata.model._sentinel import UNSET +from zarr_metadata.model._validation import ValidationProblem from zarr_metadata.v3._common import ZarrV3MetadataFieldJSON +from zarr_metadata.v3._entity import ( + ArrayArrayCodec, + Configuration, + DataTypeEntity, + Opaque, +) +from zarr_metadata.v3._parts import ArrayParts + +if TYPE_CHECKING: + from collections.abc import Iterator CAST_VALUE_CODEC_NAME: Final = "cast_value" """The `name` field value of the `cast_value` codec.""" @@ -99,12 +112,73 @@ class CastValueCodecObject(TypedDict, closed=True): "CAST_OUT_OF_RANGE_MODE", "CAST_ROUNDING_MODE", "CAST_VALUE_CODEC_NAME", + "SCALAR_MAP_KEYS", "CastOutOfRangeMode", "CastRoundingMode", + "CastValueCodec", "CastValueCodecConfiguration", "CastValueCodecMetadata", "CastValueCodecName", "CastValueCodecObject", + "CastValueOptions", "ScalarMap", "ScalarMapEntry", ] + + +SCALAR_MAP_KEYS: Final = ("encode", "decode") +"""The two directions a `scalar_map` can override, both optional.""" + + +@dataclass(frozen=True) +class CastValueOptions(Configuration): + """What `cast_value` is configured with.""" + + data_type: DataTypeEntity | Opaque + rounding: CastRoundingMode | UNSET = UNSET + out_of_range: CastOutOfRangeMode | UNSET = UNSET + scalar_map: ScalarMap | UNSET = UNSET + + def problems(self) -> "Iterator[ValidationProblem]": + """`out_of_range: "wrap"` needs a target that wraps. + + The spec defines wrapping only for integral targets with a two's + complement representation, which is a fact each data type states + about itself. A target out of scope is not judged: it may well be + an integral extension type, and judging it here would be guessing. + """ + target = self.data_type + if ( + self.out_of_range == "wrap" + and isinstance(target, DataTypeEntity) + and not type(target).twos_complement + ): + yield ValidationProblem( + ("out_of_range",), + "out_of_range 'wrap' requires a two's complement integer data_type, " + f"got {target.name!r}", + "invalid_value", + ) + + +@dataclass(frozen=True) +class CastValueCodec(ArrayArrayCodec): + """The `cast_value` codec, coerced from its metadata. + + Holds the data type it casts to, so like `sharding_indexed` it is + read in a scope rather than on its own. + """ + + configuration: CastValueOptions + + identifier: ClassVar[str] = CAST_VALUE_CODEC_NAME + variable_size: ClassVar[bool] = False + + def canonical(self) -> Self: + """The target data type in its own canonical form.""" + return self.with_configuration(data_type=self.configuration.data_type.canonical()) + + def transition(self, incoming: ArrayParts) -> ArrayParts | None: + """The same parts, holding the type this codec casts to.""" + data_type = self.configuration.data_type + return incoming.with_data_type(data_type if isinstance(data_type, DataTypeEntity) else None) diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/codec/crc32c.py b/packages/zarr-metadata/src/zarr_metadata/v3/codec/crc32c.py index 05661d0b59..99574d897e 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/codec/crc32c.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/codec/crc32c.py @@ -7,10 +7,16 @@ key is absent from the metadata. """ -from typing import Final, Literal, NotRequired +from dataclasses import dataclass, field +from typing import ClassVar, Final, Literal, NotRequired from typing_extensions import TypedDict +from zarr_metadata.v3._entity import ( + BytesBytesCodec, + Configuration, +) + CRC32C_CODEC_NAME: Final = "crc32c" """The `name` field value of the `crc32c` codec.""" @@ -47,7 +53,21 @@ class Crc32cCodecObject(TypedDict, closed=True): __all__ = [ "CRC32C_CODEC_NAME", + "Crc32cCodec", "Crc32cCodecMetadata", "Crc32cCodecName", "Crc32cCodecObject", ] + + +@dataclass(frozen=True) +class Crc32cCodec(BytesBytesCodec): + """The `crc32c` codec, coerced from its metadata. + + The name says everything: a checksum has nothing to configure. + """ + + configuration: Configuration = field(default_factory=Configuration) + + identifier: ClassVar[str] = CRC32C_CODEC_NAME + variable_size: ClassVar[bool] = False diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/codec/gzip.py b/packages/zarr-metadata/src/zarr_metadata/v3/codec/gzip.py index d516088f99..9745a6697b 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/codec/gzip.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/codec/gzip.py @@ -4,10 +4,20 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/codecs/gzip/index.html """ -from typing import Final, Literal, NotRequired +from dataclasses import dataclass +from typing import TYPE_CHECKING, ClassVar, Final, Literal, NotRequired from typing_extensions import TypedDict +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.v3._entity import ( + BytesBytesCodec, + Configuration, +) + +if TYPE_CHECKING: + from collections.abc import Iterator + GZIP_CODEC_NAME: Final = "gzip" """The `name` field value of the `gzip` codec.""" @@ -50,8 +60,33 @@ class GzipCodecObject(TypedDict, closed=True): __all__ = [ "GZIP_CODEC_NAME", + "GzipCodec", "GzipCodecConfiguration", "GzipCodecMetadata", "GzipCodecName", "GzipCodecObject", + "GzipOptions", ] + + +@dataclass(frozen=True) +class GzipOptions(Configuration): + """What `gzip` is configured with.""" + + level: int + + def problems(self) -> "Iterator[ValidationProblem]": + if not 0 <= self.level <= 9: + yield ValidationProblem( + ("level",), f"expected an integer in [0, 9], got {self.level}", "invalid_value" + ) + + +@dataclass(frozen=True) +class GzipCodec(BytesBytesCodec): + """The `gzip` codec, coerced from its metadata.""" + + configuration: GzipOptions + + identifier: ClassVar[str] = GZIP_CODEC_NAME + variable_size: ClassVar[bool] = True diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/codec/scale_offset.py b/packages/zarr-metadata/src/zarr_metadata/v3/codec/scale_offset.py index 344a4c435c..b67ce0eaab 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/codec/scale_offset.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/codec/scale_offset.py @@ -4,11 +4,25 @@ See https://github.com/zarr-developers/zarr-extensions/blob/4da7b37a84f76e660902f6d3de3eaef0e0febae6/codecs/scale_offset/README.md """ -from typing import Final, Literal, NotRequired +from dataclasses import dataclass +from typing import TYPE_CHECKING, ClassVar, Final, Literal, NotRequired from typing_extensions import TypedDict from zarr_metadata._common import JSONValue +from zarr_metadata.model._sentinel import UNSET +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.v3._entity import ( + ArrayArrayCodec, + Configuration, + DataTypeEntity, + problem, +) +from zarr_metadata.v3._parts import ArrayParts +from zarr_metadata.v3.data_type._families import FloatDataType, IntegerDataType + +if TYPE_CHECKING: + from collections.abc import Iterator SCALE_OFFSET_CODEC_NAME: Final = "scale_offset" """The `name` field value of the `scale_offset` codec.""" @@ -56,8 +70,85 @@ class ScaleOffsetCodecObject(TypedDict, closed=True): __all__ = [ "SCALE_OFFSET_CODEC_NAME", + "ScaleOffsetCodec", "ScaleOffsetCodecConfiguration", "ScaleOffsetCodecMetadata", "ScaleOffsetCodecName", "ScaleOffsetCodecObject", + "ScaleOffsetOptions", ] + + +@dataclass(frozen=True) +class ScaleOffsetOptions(Configuration): + """What `scale_offset` is configured with.""" + + offset: JSONValue | UNSET = UNSET + scale: JSONValue | UNSET = UNSET + + def problems(self) -> "Iterator[ValidationProblem]": + """Each value is a scalar of the array's type, so neither is null. + + The registry says each is "JSON-encoded per the input array's + fill-value rules", and no data type admits `null` as a fill value. + Which scalar it should be needs the data type, so that part is the + document's question, not this codec's. + """ + if self.offset is None: + yield ValidationProblem(("offset",), "expected a scalar, got null", "invalid_value") + if self.scale is None: + yield ValidationProblem(("scale",), "expected a scalar, got null", "invalid_value") + + +@dataclass(frozen=True) +class ScaleOffsetCodec(ArrayArrayCodec): + """The `scale_offset` codec, coerced from its metadata. + + Both members are optional and any JSON scalar is well-typed here; what + a given value means depends on the data type it is applied to, which + `incoming_problems` asks of the type that reaches the codec. + """ + + configuration: ScaleOffsetOptions + + identifier: ClassVar[str] = SCALE_OFFSET_CODEC_NAME + variable_size: ClassVar[bool] = False + + def incoming_problems(self, incoming: ArrayParts | None) -> tuple[ValidationProblem, ...]: + """What the array handed to this codec must be, and what its members must be for it. + + The registry defines the codec for data types with arithmetic and + lists the integer and floating-point ones. `offset` and `scale` + are each "encoded to JSON using the Zarr V3 fill value encoding + for the input array's data type" -- the type that reaches this + codec, which after a `cast_value` is not the array's own -- so + each is a fill value of that type, and that type judges it: the + string `"0"` is no `float32` and no `int32`. + """ + data_type = incoming.data_type if incoming is not None else None + if not isinstance(data_type, DataTypeEntity): + return () + if not isinstance(data_type, (IntegerDataType, FloatDataType)): + return problem( + (), + "scale_offset is defined for integer and floating-point data types, not " + f"{data_type.name!r}", + "invalid_value", + ) + return tuple( + found + for member, value in ( + ("offset", self.configuration.offset), + ("scale", self.configuration.scale), + ) + if value is not UNSET + for found in data_type.fill_value_problems(value, (member,)) + ) + + def transition(self, incoming: ArrayParts) -> ArrayParts | None: + """The same array, element for element. + + The registry entry removed the `astype` field, so this codec no + longer changes the element type -- only the values. + """ + return incoming diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/codec/sharding_indexed.py b/packages/zarr-metadata/src/zarr_metadata/v3/codec/sharding_indexed.py index 0ce466b5cf..2a3c50726d 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/codec/sharding_indexed.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/codec/sharding_indexed.py @@ -4,11 +4,31 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/codecs/sharding-indexed/index.html """ -from typing import Final, Literal, NotRequired +from dataclasses import dataclass +from typing import TYPE_CHECKING, ClassVar, Final, Literal, NotRequired, Self from typing_extensions import TypedDict +from zarr_metadata.model._sentinel import UNSET +from zarr_metadata.model._validation import ValidationProblem from zarr_metadata.v3._common import ZarrV3MetadataFieldJSON +from zarr_metadata.v3._entity import ( + ArrayBytesCodec, + CodecEntity, + Configuration, + Opaque, + problem, +) +from zarr_metadata.v3._parts import ( + UNKNOWN_GRID, + ArrayParts, + ChunkGrid, + shard_index_grid, +) +from zarr_metadata.v3.data_type.uint64 import Uint64DataType + +if TYPE_CHECKING: + from collections.abc import Iterator, Mapping, Sequence SHARDING_INDEXED_CODEC_NAME: Final = "sharding_indexed" """The `name` field value of the `sharding_indexed` codec.""" @@ -69,8 +89,132 @@ class ShardingIndexedCodecObject(TypedDict, closed=True): "SHARDING_INDEXED_CODEC_NAME", "SHARDING_INDEX_LOCATION", "ShardingIndexLocation", + "ShardingIndexedCodec", "ShardingIndexedCodecConfiguration", "ShardingIndexedCodecMetadata", "ShardingIndexedCodecName", "ShardingIndexedCodecObject", + "ShardingIndexedOptions", ] + + +@dataclass(frozen=True) +class ShardingIndexedOptions(Configuration): + """What `sharding_indexed` is configured with.""" + + chunk_shape: tuple[int, ...] + codecs: tuple[CodecEntity | Opaque, ...] + index_codecs: tuple[CodecEntity | Opaque, ...] + index_location: ShardingIndexLocation | UNSET = UNSET + + def problems(self) -> "Iterator[ValidationProblem]": + for index, extent in enumerate(self.chunk_shape): + if extent < 1: + yield ValidationProblem( + ("chunk_shape", index), + f"expected an integer >= 1, got {extent}", + "invalid_value", + ) + + +@dataclass(frozen=True) +class ShardingIndexedCodec(ArrayBytesCodec): + """The `sharding_indexed` codec, coerced from its metadata. + + Holds two codec pipelines, so it is one of the few entities that + needs the scope it is being read in: an entry of either pipeline is + itself an entity, read the same way this one was. + """ + + configuration: ShardingIndexedOptions + + identifier: ClassVar[str] = SHARDING_INDEXED_CODEC_NAME + variable_size: ClassVar[bool] = True + + def canonical(self) -> Self: + """Each pipeline's codecs in their own canonical form.""" + return self.with_configuration( + codecs=tuple(codec.canonical() for codec in self.configuration.codecs), + index_codecs=tuple(codec.canonical() for codec in self.configuration.index_codecs), + ) + + def incoming_problems(self, incoming: ArrayParts | None) -> tuple[ValidationProblem, ...]: + """This shard against the array reaching it. + + One sharding configuration encodes every chunk, so its inner + shape has to divide all of them. Under a rectilinear grid an axis + has several lengths and the inner extent must divide each; an axis + whose lengths are unknown declines while the others are judged. + The index must be readable from metadata alone, so no codec of + variable output size may encode it. The two pipelines are + `inner_pipelines`, refined by the walk. + """ + return ( + *self._inner_chunk_problems(incoming), + *( + ValidationProblem( + ("index_codecs", index), + f"{type(codec).identifier!r} produces variable-size output; " + "index_codecs must be fixed-size", + "invalid_value", + ) + for index, codec in enumerate(self.configuration.index_codecs) + if isinstance(codec, CodecEntity) and type(codec).variable_size + ), + ) + + def inner_pipelines( + self, incoming: ArrayParts | None + ) -> "Mapping[str, tuple[Sequence[CodecEntity | Opaque], ArrayParts | None]]": + """The inner chunk pipeline and the index pipeline, with what each is handed. + + Both start from this codec's own configuration and from the + spec, so neither waits on what reached the codec. An unreadable + codec upstream costs the element type and the enclosing extents; + it does not make the inner chunk shape unknown, and the index is + a `uint64` array whatever precedes it. + """ + outer = incoming.grid if incoming is not None else UNKNOWN_GRID + return { + "codecs": ( + self.configuration.codecs, + ArrayParts( + ChunkGrid.regular(self.configuration.chunk_shape), + incoming.data_type if incoming is not None else None, + ), + ), + "index_codecs": ( + self.configuration.index_codecs, + ArrayParts( + shard_index_grid(outer, self.configuration.chunk_shape), Uint64DataType() + ), + ), + } + + def _inner_chunk_problems(self, incoming: ArrayParts | None) -> tuple[ValidationProblem, ...]: + """Whether the inner chunk divides every chunk this shard receives.""" + if incoming is None or incoming.grid.rank is None: + return () + if len(self.configuration.chunk_shape) != incoming.grid.rank: + return problem( + ("chunk_shape",), + f"chunk_shape has {len(self.configuration.chunk_shape)} entries but the incoming array " + f"has {incoming.grid.rank} dimensions", + "invalid_value", + ) + found: list[ValidationProblem] = [] + for position, extent in enumerate(self.configuration.chunk_shape): + lengths = incoming.grid.axis(position) + if lengths is None or extent < 1: + continue + indivisible = sorted(length for length in lengths if length % extent != 0) + if len(indivisible) != 0: + found.extend( + problem( + ("chunk_shape", position), + f"inner chunk extent {extent} does not evenly divide the incoming " + f"extent {indivisible[0]}", + "invalid_value", + ) + ) + return tuple(found) diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/codec/transpose.py b/packages/zarr-metadata/src/zarr_metadata/v3/codec/transpose.py index 8024174605..0ee986cafc 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/codec/transpose.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/codec/transpose.py @@ -4,10 +4,22 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/codecs/transpose/index.html """ -from typing import Final, Literal, NotRequired +from dataclasses import dataclass +from typing import TYPE_CHECKING, ClassVar, Final, Literal, NotRequired from typing_extensions import TypedDict +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.v3._entity import ( + ArrayArrayCodec, + Configuration, + problem, +) +from zarr_metadata.v3._parts import ArrayParts + +if TYPE_CHECKING: + from collections.abc import Iterator + TRANSPOSE_CODEC_NAME: Final = "transpose" """The `name` field value of the `transpose` codec.""" @@ -45,8 +57,65 @@ class TransposeCodecObject(TypedDict, closed=True): __all__ = [ "TRANSPOSE_CODEC_NAME", + "TransposeCodec", "TransposeCodecConfiguration", "TransposeCodecMetadata", "TransposeCodecName", "TransposeCodecObject", + "TransposeOptions", ] + + +@dataclass(frozen=True) +class TransposeOptions(Configuration): + """What `transpose` is configured with.""" + + order: tuple[int, ...] + + def problems(self) -> "Iterator[ValidationProblem]": + """`order` must permute its own axes. + + Whether it permutes the *array's* axes is a different question -- it + needs the array's rank -- and the rules layer asks that one. + """ + if sorted(self.order) != list(range(len(self.order))): + yield ValidationProblem( + ("order",), + f"expected a permutation of 0..{len(self.order) - 1}, got {self.order!r}", + "invalid_value", + ) + + +@dataclass(frozen=True) +class TransposeCodec(ArrayArrayCodec): + """The `transpose` codec, coerced from its metadata.""" + + configuration: TransposeOptions + + identifier: ClassVar[str] = TRANSPOSE_CODEC_NAME + variable_size: ClassVar[bool] = False + + def incoming_problems(self, incoming: ArrayParts | None) -> tuple[ValidationProblem, ...]: + """A transpose permutes the array it receives, so ranks must agree. + + Judged against what actually reaches this codec: inside a shard + that is the inner chunk, and after another transpose it is that + transpose's output. + """ + rank = incoming.grid.rank if incoming is not None else None + if rank is None or len(self.configuration.order) == rank: + return () + return problem( + ("order",), + f"order has {len(self.configuration.order)} entries but the incoming array has {rank} dimensions", + "invalid_value", + ) + + def transition(self, incoming: ArrayParts) -> ArrayParts | None: + """The same array with its axes reordered. + + A transposed regular grid is still a regular grid, so the parts + survive the trip; the grid metadata does not, because it is no + longer the grid the document wrote. + """ + return incoming.with_grid(incoming.grid.permuted(self.configuration.order)) diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/codec/zstd.py b/packages/zarr-metadata/src/zarr_metadata/v3/codec/zstd.py index b7ee3f5685..9fc0c76804 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/codec/zstd.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/codec/zstd.py @@ -6,16 +6,33 @@ proposed the codec, was never merged). """ -from typing import Final, Literal, NotRequired +from dataclasses import dataclass +from typing import TYPE_CHECKING, ClassVar, Final, Literal, NotRequired from typing_extensions import TypedDict +from zarr_metadata.model._sentinel import UNSET +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.v3._entity import ( + BytesBytesCodec, + Configuration, +) + +if TYPE_CHECKING: + from collections.abc import Iterator + ZSTD_CODEC_NAME: Final = "zstd" """The `name` field value of the `zstd` codec.""" ZstdCodecName = Literal["zstd"] """Literal type of the `name` field of the `zstd` codec.""" +ZSTD_MIN_LEVEL: Final = -131072 +"""The lowest `level` zstd accepts: ZSTD_minCLevel(), -(1 << 17).""" + +ZSTD_MAX_LEVEL: Final = 22 +"""The highest `level` zstd accepts: ZSTD_maxCLevel().""" + class ZstdCodecConfiguration(TypedDict, closed=True): """ @@ -49,8 +66,38 @@ class ZstdCodecObject(TypedDict, closed=True): __all__ = [ "ZSTD_CODEC_NAME", + "ZSTD_MAX_LEVEL", + "ZSTD_MIN_LEVEL", + "ZstdCodec", "ZstdCodecConfiguration", "ZstdCodecMetadata", "ZstdCodecName", "ZstdCodecObject", + "ZstdOptions", ] + + +@dataclass(frozen=True) +class ZstdOptions(Configuration): + """What `zstd` is configured with.""" + + level: int + checksum: bool | UNSET = UNSET + + def problems(self) -> "Iterator[ValidationProblem]": + if not ZSTD_MIN_LEVEL <= self.level <= ZSTD_MAX_LEVEL: + yield ValidationProblem( + ("level",), + f"expected an integer in [{ZSTD_MIN_LEVEL}, {ZSTD_MAX_LEVEL}], got {self.level}", + "invalid_value", + ) + + +@dataclass(frozen=True) +class ZstdCodec(BytesBytesCodec): + """The `zstd` codec, coerced from its metadata.""" + + configuration: ZstdOptions + + identifier: ClassVar[str] = ZSTD_CODEC_NAME + variable_size: ClassVar[bool] = True diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/_families.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/_families.py new file mode 100644 index 0000000000..c65421c506 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/_families.py @@ -0,0 +1,237 @@ +"""What each family of data types accepts as a fill value. + +A fill value is judged against the data type, which makes it a question +the data type answers: `DataTypeEntity.fill_value_problems`. The families +here exist because the answer is the same for every width in a family +apart from one number -- the range, the hex parser, the component type -- +so each family is written once and parameterised by that number. + +Written per family rather than kept in a table keyed by name, so that +what `int32` accepts is stated where `int32` is, with nothing elsewhere +to keep in step with it. +""" + +from __future__ import annotations + +from collections.abc import Sequence +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, ClassVar, Final, Literal, cast + +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.v3._entity import ( + Configuration, + DataTypeEntity, + StorageClass, + is_integer, + problem, +) + +if TYPE_CHECKING: + from collections.abc import Callable, Iterator + + from zarr_metadata.v3._entity import Loc + +FLOAT_SPECIALS: Final = ("NaN", "Infinity", "-Infinity") +"""The three non-finite floats the spec spells as strings.""" + + +def as_sequence(value: object) -> tuple[object, ...] | None: + """`value` as a tuple if it is a JSON array, else None. + + A string is a sequence in Python and never a JSON array, so it is + excluded. + """ + if isinstance(value, str) or not isinstance(value, Sequence): + return None + # `isinstance` narrows to `Sequence[Unknown]`; its elements are objects. + return tuple(cast("Sequence[object]", value)) + + +def byte_values(value: object, expected: int | None, loc: Loc) -> tuple[ValidationProblem, ...]: + """An array of `expected` integers in [0, 255], or any length if None.""" + items = as_sequence(value) + if items is None: + return problem(loc, f"expected an array of byte values, got {value!r}", "invalid_value") + if expected is not None and len(items) != expected: + return problem(loc, f"expected {expected} byte values, got {len(items)}", "invalid_value") + return tuple( + found + for index, item in enumerate(items) + if not (is_integer(item) and 0 <= item <= 255) + for found in problem( + (*loc, index), f"expected integers in [0, 255], got {item!r}", "invalid_value" + ) + ) + + +@dataclass(frozen=True) +class IntegerDataType(DataTypeEntity): + """A fixed-width integer. The width is the whole difference.""" + + configuration: Configuration = field(default_factory=Configuration) + + bounds: ClassVar[tuple[int, int]] + twos_complement: ClassVar[bool] = True + + def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]: + low, high = type(self).bounds + if not is_integer(value): + return problem(loc, f"expected an integer, got {value!r}", "invalid_value") + if not low <= value <= high: + return problem( + loc, f"expected an integer in [{low}, {high}], got {value!r}", "invalid_value" + ) + return () + + +@dataclass(frozen=True) +class FloatDataType(DataTypeEntity): + """A binary float. A fill value may be a number, a named non-finite, or hex.""" + + configuration: Configuration = field(default_factory=Configuration) + + scalar_storage: ClassVar[StorageClass] = "multi_byte" + twos_complement: ClassVar[bool] = False + hex_parser: ClassVar[Callable[[str], object]] + + largest: ClassVar[float | None] + """The largest finite magnitude this width holds, or None for float64. + + None because a Python float *is* a float64, so no literal that reaches + here can exceed it. + """ + + def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]: + if is_integer(value) or isinstance(value, float): + largest = type(self).largest + if largest is not None and abs(value) > largest: + return problem( + loc, + f"expected a {type(self).identifier} value, got {value!r}", + "invalid_value", + ) + return () + if not isinstance(value, str): + return problem(loc, f"expected a number or string, got {value!r}", "invalid_value") + if value in FLOAT_SPECIALS: + return () + try: + type(self).hex_parser(value) + except ValueError: + return problem( + loc, + f"expected a number, one of 'NaN'/'Infinity'/'-Infinity', or a " + f"{type(self).identifier} hex string, got {value!r}", + "invalid_value", + ) + return () + + +@dataclass(frozen=True) +class ComplexDataType(DataTypeEntity): + """A complex number: a `[real, imag]` pair of the component float type.""" + + configuration: Configuration = field(default_factory=Configuration) + + scalar_storage: ClassVar[StorageClass] = "multi_byte" + twos_complement: ClassVar[bool] = False + component: ClassVar[type[FloatDataType]] + + def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]: + pair = as_sequence(value) + if pair is None or len(pair) != 2: + return problem(loc, f"expected a [real, imag] pair, got {value!r}", "invalid_value") + component = type(self).component() + return tuple( + ValidationProblem(found.loc, f"invalid component: {found.message}", found.kind) + for index, part in enumerate(pair) + for found in component.fill_value_problems(part, (*loc, index)) + ) + + +NumpyTimeUnit = Literal[ + "Y", "M", "W", "D", "h", "m", "s", "ms", "us", "μs", "ns", "ps", "fs", "as", "generic" +] +"""Time unit codes shared by `numpy.datetime64` and `numpy.timedelta64`.""" + +NUMPY_TIME_UNIT: Final = ( + "Y", + "M", + "W", + "D", + "h", + "m", + "s", + "ms", + "us", + "μs", + "ns", + "ps", + "fs", + "as", + "generic", +) +"""Tuple of the permitted `unit` values, in numpy's order from coarse to fine.""" + +NUMPY_TIME_MAX_SCALE_FACTOR: Final = 2**31 - 1 +"""The largest `scale_factor` numpy stores: the field is a signed int32.""" + + +@dataclass(frozen=True) +class NumpyTimeOptions(Configuration): + """What a numpy time type is configured with: a unit, and how many of it one tick is.""" + + unit: NumpyTimeUnit + scale_factor: int + + def problems(self) -> Iterator[ValidationProblem]: + if not 1 <= self.scale_factor <= NUMPY_TIME_MAX_SCALE_FACTOR: + yield ValidationProblem( + ("scale_factor",), + f"expected an integer in [1, {NUMPY_TIME_MAX_SCALE_FACTOR}], " + f"got {self.scale_factor}", + "invalid_value", + ) + + +@dataclass(frozen=True) +class NumpyTimeDataType(DataTypeEntity): + """A numpy time scalar: a signed 64-bit count of units, or `NaT`. + + The two time types share their configuration -- a unit and a scale + factor -- and the rule on it, so both live here with the family and + neither sibling imports them from the other. + """ + + configuration: NumpyTimeOptions + + scalar_storage: ClassVar[StorageClass] = "multi_byte" + # Stored as a signed integer, but it denotes an instant or a + # duration; wrapping one is not a defined cast. + twos_complement: ClassVar[bool] = False + + def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]: + if value == "NaT": + return () + if not is_integer(value): + return problem( + loc, f"expected a signed 64-bit integer or 'NaT', got {value!r}", "invalid_value" + ) + if not -(2**63) <= value <= 2**63 - 1: + return problem(loc, f"expected a signed 64-bit integer, got {value!r}", "invalid_value") + return () + + +__all__ = [ + "FLOAT_SPECIALS", + "NUMPY_TIME_MAX_SCALE_FACTOR", + "NUMPY_TIME_UNIT", + "ComplexDataType", + "FloatDataType", + "IntegerDataType", + "NumpyTimeDataType", + "NumpyTimeOptions", + "NumpyTimeUnit", + "as_sequence", + "byte_values", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bool.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bool.py index e36613a154..958fc0c795 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bool.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bool.py @@ -4,7 +4,17 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/data-types/index.html """ -from typing import Final, Literal +from dataclasses import dataclass, field +from typing import ClassVar, Final, Literal + +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.v3._entity import ( + Configuration, + DataTypeEntity, + Loc, + StorageClass, + problem, +) BOOL_DATA_TYPE_NAME: Final = "bool" """The `data_type` value for the `bool` type.""" @@ -18,6 +28,23 @@ __all__ = [ "BOOL_DATA_TYPE_NAME", + "BoolDataType", "BoolDataTypeName", "BoolFillValue", ] + + +@dataclass(frozen=True) +class BoolDataType(DataTypeEntity): + """The `bool` data type. The name says everything.""" + + configuration: Configuration = field(default_factory=Configuration) + + scalar_storage: ClassVar[StorageClass] = "single_byte" + twos_complement: ClassVar[bool] = False + identifier: ClassVar[str] = BOOL_DATA_TYPE_NAME + + def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]: + if not isinstance(value, bool): + return problem(loc, f"expected a boolean, got {value!r}", "invalid_value") + return () diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bytes.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bytes.py index 5892a1bdaa..3f06346d02 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bytes.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bytes.py @@ -5,7 +5,18 @@ """ import re -from typing import Final, Literal, NewType +from dataclasses import dataclass, field +from typing import ClassVar, Final, Literal, NewType + +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.v3._entity import ( + Configuration, + DataTypeEntity, + Loc, + StorageClass, + problem, +) +from zarr_metadata.v3.data_type._families import byte_values BYTES_DATA_TYPE_NAME: Final = "bytes" """The `data_type` value for the variable-length `bytes` type.""" @@ -42,7 +53,31 @@ def base64_bytes(value: str) -> Base64Bytes: __all__ = [ "BYTES_DATA_TYPE_NAME", "Base64Bytes", + "BytesDataType", "BytesDataTypeName", "BytesFillValue", "base64_bytes", ] + + +@dataclass(frozen=True) +class BytesDataType(DataTypeEntity): + """The `bytes` data type. The name says everything.""" + + configuration: Configuration = field(default_factory=Configuration) + + scalar_storage: ClassVar[StorageClass] = "variable_length" + twos_complement: ClassVar[bool] = False + identifier: ClassVar[str] = BYTES_DATA_TYPE_NAME + + def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]: + """Base64, or an array of byte values of any length.""" + if isinstance(value, str): + try: + base64_bytes(value) + except ValueError: + return problem( + loc, f"expected standard-alphabet base64, got {value!r}", "invalid_value" + ) + return () + return byte_values(value, None, loc) diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/complex128.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/complex128.py index 780bbbb02f..12ff217385 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/complex128.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/complex128.py @@ -4,9 +4,12 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/data-types/index.html """ -from typing import Final, Literal +from dataclasses import dataclass +from typing import ClassVar, Final, Literal -from zarr_metadata.v3.data_type.float64 import Float64FillValue +from zarr_metadata.v3._entity import StorageClass +from zarr_metadata.v3.data_type._families import ComplexDataType, FloatDataType +from zarr_metadata.v3.data_type.float64 import Float64DataType, Float64FillValue COMPLEX128_DATA_TYPE_NAME: Final = "complex128" """The `data_type` value for the `complex128` type.""" @@ -32,6 +35,16 @@ __all__ = [ "COMPLEX128_DATA_TYPE_NAME", "Complex128Component", + "Complex128DataType", "Complex128DataTypeName", "Complex128FillValue", ] + + +@dataclass(frozen=True) +class Complex128DataType(ComplexDataType): + """The `complex128` data type. The name says everything.""" + + scalar_storage: ClassVar[StorageClass] = "multi_byte" + component: ClassVar[type[FloatDataType]] = Float64DataType + identifier: ClassVar[str] = COMPLEX128_DATA_TYPE_NAME diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/complex64.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/complex64.py index 4aca608899..cf06a9214d 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/complex64.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/complex64.py @@ -4,9 +4,12 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/data-types/index.html """ -from typing import Final, Literal +from dataclasses import dataclass +from typing import ClassVar, Final, Literal -from zarr_metadata.v3.data_type.float32 import Float32FillValue +from zarr_metadata.v3._entity import StorageClass +from zarr_metadata.v3.data_type._families import ComplexDataType, FloatDataType +from zarr_metadata.v3.data_type.float32 import Float32DataType, Float32FillValue COMPLEX64_DATA_TYPE_NAME: Final = "complex64" """The `data_type` value for the `complex64` type.""" @@ -32,6 +35,16 @@ __all__ = [ "COMPLEX64_DATA_TYPE_NAME", "Complex64Component", + "Complex64DataType", "Complex64DataTypeName", "Complex64FillValue", ] + + +@dataclass(frozen=True) +class Complex64DataType(ComplexDataType): + """The `complex64` data type. The name says everything.""" + + scalar_storage: ClassVar[StorageClass] = "multi_byte" + component: ClassVar[type[FloatDataType]] = Float32DataType + identifier: ClassVar[str] = COMPLEX64_DATA_TYPE_NAME diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/float16.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/float16.py index 264b2c262c..d6504868d4 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/float16.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/float16.py @@ -5,7 +5,12 @@ """ import re -from typing import Final, Literal, NewType +from collections.abc import Callable +from dataclasses import dataclass +from typing import ClassVar, Final, Literal, NewType + +from zarr_metadata.v3._entity import StorageClass +from zarr_metadata.v3.data_type._families import FloatDataType FLOAT16_DATA_TYPE_NAME: Final = "float16" """The `data_type` value for the `float16` type.""" @@ -66,9 +71,20 @@ def hex_float16(value: str) -> HexFloat16: "CANONICAL_NEGATIVE_INFINITY_HEX_FLOAT16", "CANONICAL_POSITIVE_INFINITY_HEX_FLOAT16", "FLOAT16_DATA_TYPE_NAME", + "Float16DataType", "Float16DataTypeName", "Float16FillValue", "Float16SpecialFillValue", "HexFloat16", "hex_float16", ] + + +@dataclass(frozen=True) +class Float16DataType(FloatDataType): + """The `float16` data type. The name says everything.""" + + scalar_storage: ClassVar[StorageClass] = "multi_byte" + hex_parser: ClassVar[Callable[[str], object]] = staticmethod(hex_float16) + largest: ClassVar[float | None] = 65504.0 + identifier: ClassVar[str] = FLOAT16_DATA_TYPE_NAME diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/float32.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/float32.py index 3b2e786f07..5e2b287d52 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/float32.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/float32.py @@ -5,7 +5,12 @@ """ import re -from typing import Final, Literal, NewType +from collections.abc import Callable +from dataclasses import dataclass +from typing import ClassVar, Final, Literal, NewType + +from zarr_metadata.v3._entity import StorageClass +from zarr_metadata.v3.data_type._families import FloatDataType FLOAT32_DATA_TYPE_NAME: Final = "float32" """The `data_type` value for the `float32` type.""" @@ -66,9 +71,20 @@ def hex_float32(value: str) -> HexFloat32: "CANONICAL_NEGATIVE_INFINITY_HEX_FLOAT32", "CANONICAL_POSITIVE_INFINITY_HEX_FLOAT32", "FLOAT32_DATA_TYPE_NAME", + "Float32DataType", "Float32DataTypeName", "Float32FillValue", "Float32SpecialFillValue", "HexFloat32", "hex_float32", ] + + +@dataclass(frozen=True) +class Float32DataType(FloatDataType): + """The `float32` data type. The name says everything.""" + + scalar_storage: ClassVar[StorageClass] = "multi_byte" + hex_parser: ClassVar[Callable[[str], object]] = staticmethod(hex_float32) + largest: ClassVar[float | None] = 3.4028235e38 + identifier: ClassVar[str] = FLOAT32_DATA_TYPE_NAME diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/float64.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/float64.py index 21373d63f6..b52b9205bd 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/float64.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/float64.py @@ -5,7 +5,12 @@ """ import re -from typing import Final, Literal, NewType +from collections.abc import Callable +from dataclasses import dataclass +from typing import ClassVar, Final, Literal, NewType + +from zarr_metadata.v3._entity import StorageClass +from zarr_metadata.v3.data_type._families import FloatDataType FLOAT64_DATA_TYPE_NAME: Final = "float64" """The `data_type` value for the `float64` type.""" @@ -67,9 +72,20 @@ def hex_float64(value: str) -> HexFloat64: "CANONICAL_NEGATIVE_INFINITY_HEX_FLOAT64", "CANONICAL_POSITIVE_INFINITY_HEX_FLOAT64", "FLOAT64_DATA_TYPE_NAME", + "Float64DataType", "Float64DataTypeName", "Float64FillValue", "Float64SpecialFillValue", "HexFloat64", "hex_float64", ] + + +@dataclass(frozen=True) +class Float64DataType(FloatDataType): + """The `float64` data type. The name says everything.""" + + scalar_storage: ClassVar[StorageClass] = "multi_byte" + hex_parser: ClassVar[Callable[[str], object]] = staticmethod(hex_float64) + largest: ClassVar[float | None] = None + identifier: ClassVar[str] = FLOAT64_DATA_TYPE_NAME diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/int16.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/int16.py index b76f06761a..fb295e3dfc 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/int16.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/int16.py @@ -4,7 +4,11 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/data-types/index.html """ -from typing import Final, Literal +from dataclasses import dataclass +from typing import ClassVar, Final, Literal + +from zarr_metadata.v3._entity import StorageClass +from zarr_metadata.v3.data_type._families import IntegerDataType INT16_DATA_TYPE_NAME: Final = "int16" """The `data_type` value for the `int16` type.""" @@ -18,6 +22,16 @@ __all__ = [ "INT16_DATA_TYPE_NAME", + "Int16DataType", "Int16DataTypeName", "Int16FillValue", ] + + +@dataclass(frozen=True) +class Int16DataType(IntegerDataType): + """The `int16` data type. The name says everything.""" + + scalar_storage: ClassVar[StorageClass] = "multi_byte" + bounds: ClassVar[tuple[int, int]] = (-32768, 32767) + identifier: ClassVar[str] = INT16_DATA_TYPE_NAME diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/int32.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/int32.py index 7b41ec6c54..7d35f033cd 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/int32.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/int32.py @@ -4,7 +4,11 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/data-types/index.html """ -from typing import Final, Literal +from dataclasses import dataclass +from typing import ClassVar, Final, Literal + +from zarr_metadata.v3._entity import StorageClass +from zarr_metadata.v3.data_type._families import IntegerDataType INT32_DATA_TYPE_NAME: Final = "int32" """The `data_type` value for the `int32` type.""" @@ -18,6 +22,16 @@ __all__ = [ "INT32_DATA_TYPE_NAME", + "Int32DataType", "Int32DataTypeName", "Int32FillValue", ] + + +@dataclass(frozen=True) +class Int32DataType(IntegerDataType): + """The `int32` data type. The name says everything.""" + + scalar_storage: ClassVar[StorageClass] = "multi_byte" + bounds: ClassVar[tuple[int, int]] = (-2147483648, 2147483647) + identifier: ClassVar[str] = INT32_DATA_TYPE_NAME diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/int64.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/int64.py index 0005675c66..365370a9dc 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/int64.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/int64.py @@ -4,7 +4,11 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/data-types/index.html """ -from typing import Final, Literal +from dataclasses import dataclass +from typing import ClassVar, Final, Literal + +from zarr_metadata.v3._entity import StorageClass +from zarr_metadata.v3.data_type._families import IntegerDataType INT64_DATA_TYPE_NAME: Final = "int64" """The `data_type` value for the `int64` type.""" @@ -18,6 +22,16 @@ __all__ = [ "INT64_DATA_TYPE_NAME", + "Int64DataType", "Int64DataTypeName", "Int64FillValue", ] + + +@dataclass(frozen=True) +class Int64DataType(IntegerDataType): + """The `int64` data type. The name says everything.""" + + scalar_storage: ClassVar[StorageClass] = "multi_byte" + bounds: ClassVar[tuple[int, int]] = (-9223372036854775808, 9223372036854775807) + identifier: ClassVar[str] = INT64_DATA_TYPE_NAME diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/int8.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/int8.py index a5a16de761..5a2bf185ca 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/int8.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/int8.py @@ -4,7 +4,11 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/data-types/index.html """ -from typing import Final, Literal +from dataclasses import dataclass +from typing import ClassVar, Final, Literal + +from zarr_metadata.v3._entity import StorageClass +from zarr_metadata.v3.data_type._families import IntegerDataType INT8_DATA_TYPE_NAME: Final = "int8" """The `data_type` value for the `int8` type.""" @@ -18,6 +22,16 @@ __all__ = [ "INT8_DATA_TYPE_NAME", + "Int8DataType", "Int8DataTypeName", "Int8FillValue", ] + + +@dataclass(frozen=True) +class Int8DataType(IntegerDataType): + """The `int8` data type. The name says everything.""" + + scalar_storage: ClassVar[StorageClass] = "single_byte" + bounds: ClassVar[tuple[int, int]] = (-128, 127) + identifier: ClassVar[str] = INT8_DATA_TYPE_NAME diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/numpy_datetime64.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/numpy_datetime64.py index 4f9a6415c5..cd6f75786e 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/numpy_datetime64.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/numpy_datetime64.py @@ -4,21 +4,25 @@ See https://github.com/zarr-developers/zarr-extensions/blob/4da7b37a84f76e660902f6d3de3eaef0e0febae6/data-types/numpy.datetime64/README.md """ -from typing import Final, Literal, NotRequired +from dataclasses import dataclass +from typing import ClassVar, Final, Literal, NotRequired from typing_extensions import ReadOnly, TypedDict +from zarr_metadata.v3._entity import ( + StorageClass, +) +from zarr_metadata.v3.data_type._families import ( + NumpyTimeDataType, + NumpyTimeUnit, +) + NUMPY_DATETIME64_DATA_TYPE_NAME: Final = "numpy.datetime64" """The `name` field value of the `numpy.datetime64` data type.""" NumpyDatetime64DataTypeName = Literal["numpy.datetime64"] """Literal type of the `name` field of the `numpy.datetime64` data type.""" -NumpyTimeUnit = Literal[ - "Y", "M", "W", "D", "h", "m", "s", "ms", "us", "μs", "ns", "ps", "fs", "as", "generic" -] -"""Time unit codes used by numpy.datetime64.""" - class NumpyDatetime64Configuration(TypedDict, closed=True): """ @@ -55,7 +59,16 @@ class NumpyDatetime64(TypedDict, closed=True): "NUMPY_DATETIME64_DATA_TYPE_NAME", "NumpyDatetime64", "NumpyDatetime64Configuration", + "NumpyDatetime64DataType", "NumpyDatetime64DataTypeName", "NumpyDatetime64FillValue", "NumpyTimeUnit", ] + + +@dataclass(frozen=True) +class NumpyDatetime64DataType(NumpyTimeDataType): + """The `numpy.datetime64` data type, coerced from its metadata.""" + + scalar_storage: ClassVar[StorageClass] = "multi_byte" + identifier: ClassVar[str] = NUMPY_DATETIME64_DATA_TYPE_NAME diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/numpy_timedelta64.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/numpy_timedelta64.py index a1c4fef772..46b50b003e 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/numpy_timedelta64.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/numpy_timedelta64.py @@ -4,40 +4,27 @@ See https://github.com/zarr-developers/zarr-extensions/blob/4da7b37a84f76e660902f6d3de3eaef0e0febae6/data-types/numpy.timedelta64/README.md """ -from typing import Final, Literal, NotRequired +from dataclasses import dataclass +from typing import ClassVar, Final, Literal, NotRequired from typing_extensions import ReadOnly, TypedDict +from zarr_metadata.v3._entity import ( + StorageClass, +) +from zarr_metadata.v3.data_type._families import ( + NUMPY_TIME_MAX_SCALE_FACTOR, + NUMPY_TIME_UNIT, + NumpyTimeDataType, + NumpyTimeUnit, +) + NUMPY_TIMEDELTA64_DATA_TYPE_NAME: Final = "numpy.timedelta64" """The `name` field value of the `numpy.timedelta64` data type.""" NumpyTimedelta64DataTypeName = Literal["numpy.timedelta64"] """Literal type of the `name` field of the `numpy.timedelta64` data type.""" -NumpyTimeUnit = Literal[ - "Y", "M", "W", "D", "h", "m", "s", "ms", "us", "μs", "ns", "ps", "fs", "as", "generic" -] -"""Time unit codes used by numpy.timedelta64.""" - -NUMPY_TIME_UNIT: Final = ( - "Y", - "M", - "W", - "D", - "h", - "m", - "s", - "ms", - "us", - "μs", - "ns", - "ps", - "fs", - "as", - "generic", -) -"""Runtime tuple of the permitted `numpy.timedelta64`/`numpy.datetime64` unit strings.""" - class NumpyTimedelta64Configuration(TypedDict, closed=True): """ @@ -72,10 +59,20 @@ class NumpyTimedelta64(TypedDict, closed=True): __all__ = [ "NUMPY_TIMEDELTA64_DATA_TYPE_NAME", + "NUMPY_TIME_MAX_SCALE_FACTOR", "NUMPY_TIME_UNIT", "NumpyTimeUnit", "NumpyTimedelta64", "NumpyTimedelta64Configuration", + "NumpyTimedelta64DataType", "NumpyTimedelta64DataTypeName", "NumpyTimedelta64FillValue", ] + + +@dataclass(frozen=True) +class NumpyTimedelta64DataType(NumpyTimeDataType): + """The `numpy.timedelta64` data type, coerced from its metadata.""" + + scalar_storage: ClassVar[StorageClass] = "multi_byte" + identifier: ClassVar[str] = NUMPY_TIMEDELTA64_DATA_TYPE_NAME diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/raw.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/raw.py index 66e69c9b53..f7f89d03c8 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/raw.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/raw.py @@ -9,7 +9,21 @@ """ import re -from typing import Final, NewType +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Annotated, ClassVar, Final, NewType + +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.v3._entity import ( + FROM_NAME, + Configuration, + DataTypeEntity, + Loc, + StorageClass, +) +from zarr_metadata.v3.data_type._families import byte_values + +if TYPE_CHECKING: + from collections.abc import Iterator RawBytesDataTypeName = NewType("RawBytesDataTypeName", str) """A spec-conformant `r` raw-bytes name (e.g. `"r8"`, `"r16"`). @@ -18,7 +32,25 @@ https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/data-types/index.rst#L46-L47 """ -_RAW_BYTES_RE: Final = re.compile(r"^r(\d+)$") +RAW_BYTES_FAMILY: Final = "r" +"""Canonical key for the parameterized raw-bytes data type family. + +Spelled as the spec writes the family; the angle brackets keep it +unforgeable by a real name. +""" + +RAW_BYTES_NAME_PATTERN: Final = re.compile(r"^r([0-9]+)$") +"""The *shape* of a raw-bytes data type name, not its validity. + +ASCII digits only: `\\d` would also match every other Unicode decimal, so +`r\uff11\uff16` would be read as sixteen bits and a genuine third-party +name spelled that way would be folded into this family. + +Matches every `r` spelling including malformed ones (`r0`, `r12`), so +that a misspelled member of this family is recognized as belonging to it +and reported as a misspelling, rather than passing as an unknown +third-party extension. `raw_bytes_dtype_name` applies the validity rule +on top. Sole owner of this grammar: other modules match through it.""" def raw_bytes_dtype_name(value: str) -> RawBytesDataTypeName: @@ -27,7 +59,7 @@ def raw_bytes_dtype_name(value: str) -> RawBytesDataTypeName: Raises ValueError if `value` is not `r` followed by a positive multiple of 8. """ - match = _RAW_BYTES_RE.fullmatch(value) + match = RAW_BYTES_NAME_PATTERN.fullmatch(value) if match is None: raise ValueError(f"Expected 'r' followed by a positive integer, got {value!r}") bits = int(match.group(1)) @@ -44,7 +76,69 @@ def raw_bytes_dtype_name(value: str) -> RawBytesDataTypeName: __all__ = [ + "RAW_BYTES_FAMILY", + "RAW_BYTES_NAME_PATTERN", + "RawBytesDataType", "RawBytesDataTypeName", "RawBytesFillValue", "raw_bytes_dtype_name", ] + + +@dataclass(frozen=True) +class RawBytesDataType(DataTypeEntity): + """An `r` raw-bytes data type, coerced from its metadata. + + One class for the whole family, because `r8` and `r4096` differ only + in a number. That is why this is the one entity whose `identifier` is + not a name any document carries: `r` is a shape, not a spelling, + and no real name can collide with it. + + The spelling is kept rather than the bit count, so a document comes + back out as it went in. `r008` is a valid and distinct way of writing + `r8`, and canonicalizing it away is not this package's call. + """ + + # Keyword-only, so the carried name stays the one positional argument. + configuration: Configuration = field(default_factory=Configuration, kw_only=True) + + data_type_name: Annotated[str, FROM_NAME] + """The spelling as written -- `r8`, `r008` -- which is where the width lives.""" + + scalar_storage: ClassVar[StorageClass] = "single_byte" + twos_complement: ClassVar[bool] = False + identifier: ClassVar[str] = RAW_BYTES_FAMILY + + @classmethod + def accepts(cls, name: str) -> bool: + """Every `r` spelling, valid or not. + + A malformed member of the family is recognized as belonging to it + and reported as malformed, rather than passing unjudged as some + third party's extension. + """ + return RAW_BYTES_NAME_PATTERN.fullmatch(name) is not None + + @classmethod + def name_problems(cls, name: str) -> "Iterator[ValidationProblem]": + """This family's validity is in its name, not in a configuration. + + "raw bits, variable size given by *, limited to be a multiple of 8" + -- and zero bits is not a data type. + """ + try: + raw_bytes_dtype_name(name) + except ValueError as error: + yield ValidationProblem((), str(error), "invalid_value") + + def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]: + """One byte value per byte of the scalar. + + A malformed name says nothing about how wide the scalar is, so + there is no length to check against; `name_problems` reports the name. + """ + try: + raw_bytes_dtype_name(self.data_type_name) + except ValueError: + return () + return byte_values(value, int(self.data_type_name[1:]) // 8, loc) diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/string.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/string.py index 1e93a95d50..79979410e4 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/string.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/string.py @@ -4,7 +4,17 @@ See https://github.com/zarr-developers/zarr-extensions/blob/4da7b37a84f76e660902f6d3de3eaef0e0febae6/data-types/string/README.md """ -from typing import Final, Literal +from dataclasses import dataclass, field +from typing import ClassVar, Final, Literal + +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.v3._entity import ( + Configuration, + DataTypeEntity, + Loc, + StorageClass, + problem, +) STRING_DATA_TYPE_NAME: Final = "string" """The `data_type` value for the `string` type.""" @@ -18,6 +28,23 @@ __all__ = [ "STRING_DATA_TYPE_NAME", + "StringDataType", "StringDataTypeName", "StringFillValue", ] + + +@dataclass(frozen=True) +class StringDataType(DataTypeEntity): + """The `string` data type. The name says everything.""" + + configuration: Configuration = field(default_factory=Configuration) + + scalar_storage: ClassVar[StorageClass] = "variable_length" + twos_complement: ClassVar[bool] = False + identifier: ClassVar[str] = STRING_DATA_TYPE_NAME + + def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]: + if not isinstance(value, str): + return problem(loc, f"expected a string, got {value!r}", "invalid_value") + return () diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/struct.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/struct.py index 87a437951d..977a12d22f 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/struct.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/struct.py @@ -5,12 +5,25 @@ """ from collections.abc import Mapping -from typing import Final, Literal, NotRequired +from dataclasses import dataclass, replace +from typing import TYPE_CHECKING, ClassVar, Final, Literal, NotRequired, Self, cast from typing_extensions import ReadOnly, TypedDict from zarr_metadata._common import JSONValue +from zarr_metadata.model._validation import ValidationProblem from zarr_metadata.v3._common import ZarrV3MetadataFieldJSON +from zarr_metadata.v3._entity import ( + Configuration, + DataTypeEntity, + Loc, + Opaque, + StorageClass, + problem, +) + +if TYPE_CHECKING: + from collections.abc import Iterator STRUCT_DATA_TYPE_NAME: Final = "struct" """The `name` field value of the `struct` data type.""" @@ -19,6 +32,10 @@ """Literal type of the `name` field of the `struct` data type.""" +STRUCT_FIELD_KEYS: Final = ("name", "data_type") +"""The members a struct field entry carries, both required.""" + + class StructField(TypedDict, closed=True): """ A single field entry inside a structured dtype. @@ -59,9 +76,146 @@ class Struct(TypedDict, closed=True): __all__ = [ "STRUCT_DATA_TYPE_NAME", + "STRUCT_FIELD_KEYS", "Struct", "StructConfiguration", + "StructDataType", "StructDataTypeName", "StructField", + "StructFieldComponent", "StructFillValue", + "StructOptions", ] + + +@dataclass(frozen=True) +class StructFieldComponent: + """One field of a struct: a name, and the type of its values. + + `data_type` is the coerced entity when the field's type is in scope, + and the metadata untouched when it is not. + """ + + name: str + data_type: DataTypeEntity | Opaque + + +@dataclass(frozen=True) +class StructOptions(Configuration): + """What `struct` is configured with.""" + + fields: tuple[StructFieldComponent, ...] + + def problems(self) -> "Iterator[ValidationProblem]": + """Names exist, are non-empty and distinct; types are fixed-size. + + A fill value addresses fields by name, and a record's layout is not + determined by a variable-length field. Nothing about a field type's + own values: it is an entity, so it exists only if those are allowed. + """ + if len(self.fields) == 0: + yield ValidationProblem( + ("fields",), "expected at least one struct field", "invalid_value" + ) + seen: dict[str, int] = {} + for index, field in enumerate(self.fields): + if field.name == "": + yield ValidationProblem( + ("fields", index, "name"), "expected a non-empty field name", "invalid_value" + ) + first = seen.setdefault(field.name, index) + if first != index: + yield ValidationProblem( + ("fields", index, "name"), + f"duplicate field name {field.name!r}, already used by field {first}", + "invalid_value", + ) + if ( + isinstance(field.data_type, DataTypeEntity) + and field.data_type.storage_class() == "variable_length" + ): + yield ValidationProblem( + ("fields", index, "data_type"), + "struct fields must use fixed-size data types", + "invalid_value", + ) + + +@dataclass(frozen=True) +class StructDataType(DataTypeEntity): + """The `struct` data type, coerced from its metadata. + + A record of named fields, each with a data type of its own -- so this + is a data type that contains data types, and needs the scope it is + read in to make sense of them. + """ + + configuration: StructOptions + + identifier: ClassVar[str] = STRUCT_DATA_TYPE_NAME + scalar_storage: ClassVar[StorageClass] = "single_byte" + twos_complement: ClassVar[bool] = False + + def canonical(self) -> Self: + """Each field's data type in its own canonical form.""" + return self.with_configuration( + fields=tuple( + replace(field, data_type=field.data_type.canonical()) + for field in self.configuration.fields + ), + ) + + def storage_class(self) -> StorageClass | None: + """The widest class among the fields. + + A struct of `uint8` and `int32` is `multi_byte`, and one holding + a `string` anywhere inside it is `variable_length`. None when any + field's type is out of scope: the answer would be a guess. + """ + widest: StorageClass = "single_byte" + for field in self.configuration.fields: + if not isinstance(field.data_type, DataTypeEntity): + return None + found = field.data_type.storage_class() + if found is None: + return None + if found == "variable_length": + return "variable_length" + if found == "multi_byte": + widest = "multi_byte" + return widest + + def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]: + """A fill value per field, addressed by name. + + Every field needs one and nothing else may appear, because a + record's value is not determined otherwise. A field whose type is + out of scope still needs an entry -- that much is structural -- + but what the entry holds is left unjudged. + """ + if not isinstance(value, Mapping): + return problem( + loc, + f"expected an object of per-field fill values, got {value!r}", + "invalid_value", + ) + fills = cast("Mapping[str, object]", value) + found: list[ValidationProblem] = [] + for field in self.configuration.fields: + at: Loc = (*loc, field.name) + if field.name not in fills: + found.extend( + problem( + at, f"missing fill value for struct field {field.name!r}", "missing_key" + ) + ) + continue + if not isinstance(field.data_type, DataTypeEntity): + continue + found.extend(field.data_type.fill_value_problems(fills[field.name], at)) + declared = {field.name for field in self.configuration.fields} + found.extend( + ValidationProblem((*loc, key), f"unknown struct fill field {key!r}", "unknown_key") + for key in sorted(fills.keys() - declared) + ) + return tuple(found) diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/uint16.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/uint16.py index 37e35ec436..d87340123c 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/uint16.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/uint16.py @@ -4,7 +4,11 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/data-types/index.html """ -from typing import Final, Literal +from dataclasses import dataclass +from typing import ClassVar, Final, Literal + +from zarr_metadata.v3._entity import StorageClass +from zarr_metadata.v3.data_type._families import IntegerDataType UINT16_DATA_TYPE_NAME: Final = "uint16" """The `data_type` value for the `uint16` type.""" @@ -18,6 +22,16 @@ __all__ = [ "UINT16_DATA_TYPE_NAME", + "Uint16DataType", "Uint16DataTypeName", "Uint16FillValue", ] + + +@dataclass(frozen=True) +class Uint16DataType(IntegerDataType): + """The `uint16` data type. The name says everything.""" + + scalar_storage: ClassVar[StorageClass] = "multi_byte" + bounds: ClassVar[tuple[int, int]] = (0, 65535) + identifier: ClassVar[str] = UINT16_DATA_TYPE_NAME diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/uint32.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/uint32.py index f6cd4d447e..ab0c7b2cce 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/uint32.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/uint32.py @@ -4,7 +4,11 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/data-types/index.html """ -from typing import Final, Literal +from dataclasses import dataclass +from typing import ClassVar, Final, Literal + +from zarr_metadata.v3._entity import StorageClass +from zarr_metadata.v3.data_type._families import IntegerDataType UINT32_DATA_TYPE_NAME: Final = "uint32" """The `data_type` value for the `uint32` type.""" @@ -18,6 +22,16 @@ __all__ = [ "UINT32_DATA_TYPE_NAME", + "Uint32DataType", "Uint32DataTypeName", "Uint32FillValue", ] + + +@dataclass(frozen=True) +class Uint32DataType(IntegerDataType): + """The `uint32` data type. The name says everything.""" + + scalar_storage: ClassVar[StorageClass] = "multi_byte" + bounds: ClassVar[tuple[int, int]] = (0, 4294967295) + identifier: ClassVar[str] = UINT32_DATA_TYPE_NAME diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/uint64.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/uint64.py index 7151d2395a..241f4eb819 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/uint64.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/uint64.py @@ -4,7 +4,11 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/data-types/index.html """ -from typing import Final, Literal +from dataclasses import dataclass +from typing import ClassVar, Final, Literal + +from zarr_metadata.v3._entity import StorageClass +from zarr_metadata.v3.data_type._families import IntegerDataType UINT64_DATA_TYPE_NAME: Final = "uint64" """The `data_type` value for the `uint64` type.""" @@ -18,6 +22,16 @@ __all__ = [ "UINT64_DATA_TYPE_NAME", + "Uint64DataType", "Uint64DataTypeName", "Uint64FillValue", ] + + +@dataclass(frozen=True) +class Uint64DataType(IntegerDataType): + """The `uint64` data type. The name says everything.""" + + scalar_storage: ClassVar[StorageClass] = "multi_byte" + bounds: ClassVar[tuple[int, int]] = (0, 18446744073709551615) + identifier: ClassVar[str] = UINT64_DATA_TYPE_NAME diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/uint8.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/uint8.py index 787f1b7866..fb5fabbf5e 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/uint8.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/uint8.py @@ -4,7 +4,11 @@ See https://zarr-specs.readthedocs.io/en/latest/v3/data-types/index.html """ -from typing import Final, Literal +from dataclasses import dataclass +from typing import ClassVar, Final, Literal + +from zarr_metadata.v3._entity import StorageClass +from zarr_metadata.v3.data_type._families import IntegerDataType UINT8_DATA_TYPE_NAME: Final = "uint8" """The `data_type` value for the `uint8` type.""" @@ -18,6 +22,16 @@ __all__ = [ "UINT8_DATA_TYPE_NAME", + "Uint8DataType", "Uint8DataTypeName", "Uint8FillValue", ] + + +@dataclass(frozen=True) +class Uint8DataType(IntegerDataType): + """The `uint8` data type. The name says everything.""" + + scalar_storage: ClassVar[StorageClass] = "single_byte" + bounds: ClassVar[tuple[int, int]] = (0, 255) + identifier: ClassVar[str] = UINT8_DATA_TYPE_NAME diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/entity.py b/packages/zarr-metadata/src/zarr_metadata/v3/entity.py new file mode 100644 index 0000000000..c3f61676d5 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/v3/entity.py @@ -0,0 +1,341 @@ +"""The extension layer: what an entity is, and what is in scope. + +Every Zarr v3 extension point -- codecs, data types, chunk grids, chunk +key encodings, storage transformers -- is modelled as a class that +answers for itself. This module is the public door to that layer, for two +kinds of caller, and everything either needs is exported from it. + +**Reading metadata.** `ArrayDocumentV3.from_json` is the fail-fast front +door: one call, and either every extension point is read or a single +`MetadataValidationError` carries every reason it is not, in +`.problems`. The entities it yields know things the document does not +spell out -- what a data type's scalars are, which position a codec +occupies, what a grid divides an array into. A name the scope does not +model is not a failure: it arrives as an `Opaque` marked `out_of_scope`, +for the reader to resolve elsewhere. Nor is a top-level field outside the +spec's: `must_understand_fields` names the ones the reader must refuse +to open the array without recognizing. + + from zarr_metadata.v3.entity import ArrayBytesCodec, ArrayDocumentV3, CodecEntity + + array = ArrayDocumentV3.from_json(json.loads(raw)) # or raises + array.must_understand_fields # a field here you do not know: refuse + for codec in array.codecs: + if isinstance(codec, CodecEntity): + isinstance(codec, ArrayBytesCodec) # its pipeline position is its base class + else: + codec.json, codec.reason # 'out_of_scope': resolve it yourself + +**Three layers**, ordered by what each needs, each handing the next a +typed value and its problems. `well_formed_array_v3(value)` needs only +the value: JSON refined, arrays as tuples, and the document's shape +judged. `read_array_v3(document, context)` needs a scope: each +extension point's name related to a class and the class handed the +field. `refine_array_v3(array)` needs the array: the fill value against +the type, the grid against the shape, and the codec pipeline walked with +what reaches each codec. Its value, `RefinedArrayV3`, is the resolved +pipeline -- `Pipeline` of `PipelineStage`, each a codec and the +`ArrayParts` it is handed, a shard's inner pipelines refined inside it +-- which is what a codec pipeline is built from; validation is what the +walk finds. `from_json` and `validate_array_metadata_v3` run all three. + +**What comes back.** Problems, not exceptions, wherever a document is +being judged rather than demanded. `zarr_metadata.rules.validate_array_metadata_v3(document, context=...)` +returns a tuple of `ValidationProblem(loc, message, kind)`, each `loc` +indexing into the document: +`("codecs", 1, "configuration", "level")`, and `kind` one of +`invalid_type`, `invalid_value`, `missing_key`, `unknown_key` and +`invalid_json`. `resolve(entry, CodecEntity, SCOPE)` reads one metadata +field as an entity of that kind, the first two layers for a field on its +own: it refines and judges the field, relates the name in it to a class +in the scope, and hands that class the field, since the class owns its +validation routine. It returns `(entity, problems)` where `entity` +is the entity or an `Opaque` -- never `None` -- with `loc` relative to +the entry: `("configuration", "level")`. The class's routine, +`coerce(value, context)`, returns `(entity or None, problems)`, that is +`Coerced`, and judges the configuration; the envelope is the field's, +and `resolve` judges it. Constructing an entity by hand raises +`MetadataValidationError` with `loc` relative to the configuration: +`("level",)`. + +**Writing an extension.** Subclass the kind of thing it is -- a codec's +kind (`ArrayArrayCodec`, `ArrayBytesCodec`, `BytesBytesCodec`), +`DataTypeEntity`, `ChunkGridEntity`, `ChunkKeyEncodingEntity` or +`StorageTransformerEntity`; declare its configuration as a frozen +`Configuration` of its members, with every rule finer than a type in +its `problems`, and name it in the entity's one field, `configuration`; +add the class to a scope. An entity of a bare name defaults the field +to the empty record: `configuration: Configuration = +field(default_factory=Configuration)`. Complete, and runnable as written: + + from collections.abc import Iterator + from dataclasses import dataclass + from typing import ClassVar + + from zarr_metadata.rules import validate_array_metadata_v3 + from zarr_metadata.v3.entity import ( + CORE_AND_EXTENSIONS, + UNSET, + BytesBytesCodec, + Configuration, + ValidationProblem, + ) + + @dataclass(frozen=True) # the fields are the schema; frozen, so a configuration is a value + class AcmeLz4Options(Configuration): + acceleration: int | UNSET = UNSET # optional: absent reads as UNSET + + def problems(self) -> Iterator[ValidationProblem]: + if self.acceleration is not UNSET and not 1 <= self.acceleration <= 65537: + yield ValidationProblem( + ("acceleration",), + f"expected an integer in [1, 65537], got {self.acceleration}", + "invalid_value", + ) + + @dataclass(frozen=True) + class AcmeLz4Codec(BytesBytesCodec): + configuration: AcmeLz4Options # the shape of the metadata: a name, and a configuration + + identifier: ClassVar[str] = "acme.lz4" + variable_size: ClassVar[bool] = True # a compressor: its output length is not fixed + + SCOPE = CORE_AND_EXTENSIONS.extended_with(AcmeLz4Codec) + document = { + "zarr_format": 3, "node_type": "array", "shape": [8], "data_type": "uint8", + "fill_value": 0, "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": [8]}}, + "chunk_key_encoding": "default", + "codecs": ["bytes", {"name": "acme.lz4", "configuration": {"acceleration": 3}}], + } + assert validate_array_metadata_v3(document, context=SCOPE) == () + +An entity has the shape of its metadata: a name, which is the class, +and a configuration, which is a record dataclass named in the one field +`configuration`; an entity of a bare name defaults it to the empty +`Configuration`, since the spec makes an absent configuration and an +empty one the same. The record's fields are the +one place the entity's members are declared; the public `*Configuration` +TypedDict beside it declares the JSON, and a test holds the two to the +same keys. Which members exist, which may be left out (the type admits +`UNSET`), how each one is type-checked, and how each is written back +are all read off the annotations, and the shapes are the ones JSON +takes: `int`, `float` (any JSON number), `bool`, `str`, +`JSONValue`, a `Literal` of names, `tuple[T, ...]` or `tuple[T1, T2]`, a +TypedDict or dataclass record, `Mapping[str, V]`, a `NewType`, and a +nested entity, always as `inner: CodecEntity | Opaque`, because that is +what the field holds when the inner name is out of scope -- at any +depth, as an array's element or a record's field, and read in the scope +the containing entity is read in. Anything else is refused at +registration. A required member +has no default; an optional one is `| UNSET = UNSET`, so absence stays +distinct from a JSON `null`, and a member that means something when +absent is read that way where it is used, not defaulted. Only `| UNSET` +makes a member optional to a document: a plain default serves hand +construction, and a document must still write the member. A member is +read as `codec.configuration.acceleration`, the shape the metadata has; +nothing lifts it to the entity. `with_configuration(**changes)` +is the entity with members of its configuration replaced, checked as +any construction is. + +Everything finer than a type -- a bound, a rule about one member, members +read together -- is the record's `problems`, which yields +`ValidationProblem(loc, message, kind)` as it finds each, in plain +code. Locations are relative to the configuration, and `kind` is +`"invalid_value"` for a value rule. The entity's constructor stops at +the first problem it yields, so `AcmeLz4Codec(AcmeLz4Options(acceleration=0))` +raises `MetadataValidationError`, and the record's constructor refuses +a member of the wrong type, so `AcmeLz4Options(acceleration="fast")` +raises too, whether written by hand, through `replace` or through +`with_configuration`. `create_unchecked(**fields)`, on a record and on +an entity, is the one way around the constructors, for a reader that +has just made their checks: `coerce` is that reader, and a document +read makes each check once. `coerce` runs it to the end and +reports every problem in the document; a reader with a record asks +`options.problems()` directly and stops or collects. It runs only on a +configuration whose members all read: a member of the wrong type is +reported and the entity is not built. A family's rule about its name -- +`r` a multiple of 8 -- is the entity's `name_problems(name)`, a +classmethod, located on the entity. + +**What an entity answers for itself**, beyond its configuration. `to_json` is +written once in the base, from the record: the bare name when every +member is absent, the object otherwise, a contained entity through its +own `to_json`; an entity whose JSON is not its fields overrides it, and +none in the package does. `ArrayDocumentV3.to_json` puts each envelope +back as the document spelled it, so a document read and written comes +out as it went in. `canonical`, the entity in its simplest equivalent form: +the entity itself by default, overridden where two spellings of its +members mean the same, and in an entity that contains entities to put +those in canonical form -- `self.with_configuration(inner=self.inner.canonical())`. +An `Opaque` answers both as well, with the JSON it kept and with itself, +so a field typed `CodecEntity | Opaque` is written and simplified without +asking which it holds. `coerce` is written once in the base. Then, by +kind: + +- Every entity: `identifier`, the name it is registered under. A family + -- one class for every `acme.fixedN` -- overrides `accepts(name)` and + keeps the name in a field marked `Annotated[str, FROM_NAME]`, which + `coerce` fills from the envelope. +- A codec: its kind is its base class. One that holds pipelines of its + own, as a shard does, declares them through `inner_pipelines(incoming)` + -- each by the member that holds it, with the parts it is handed -- + and refinement walks them; it judges nothing inside them itself. An + `ArrayArrayCodec` defines + `transition(incoming: ArrayParts) -> ArrayParts | None` -- abstract: + return `incoming` if it leaves the array's shape, grid and data type + alone, the parts it hands the next codec, or None when the metadata + cannot say -- and any codec may define `incoming_problems(incoming: + ArrayParts | None) -> tuple[ValidationProblem, ...]` for what it + cannot take, where None is an array the chain lost track of and the + answer to it is nothing. Every codec declares `variable_size`, whether + its output length depends on its input, which is what keeps a + compressor out of a shard's index. +- A data type: `scalar_storage`, one of `StorageClass` -- + `"single_byte"`, `"multi_byte"`, `"variable_length"` -- which the + method `storage_class()` answers (the `bytes` codec asks it whether an + endianness is needed), and `fill_value_problems(value, loc) -> + tuple[ValidationProblem, ...]`, abstract: it judges a document's + `fill_value`, and a type that accepts any says so with `return ()`. + It also declares `twos_complement`, whether its scalars are two's + complement integers, which `cast_value` asks before it lets a cast + wrap. These composition hooks return tuples; a record's `problems` + yields. The families `IntegerDataType`, `FloatDataType`, + `ComplexDataType` and `NumpyTimeDataType` carry all three for the + types they cover; a family of + your own is a plain subclass that is never registered itself, and + passes its class variables down. +- A chunk grid: `grid(array_shape)`, abstract, and `shape_problems`; see + `ChunkGridEntity`. +- A family, one class for many names: `identifier` is an invented key no + document writes, `accepts(name)` says which names are its own, a field + marked `Annotated[str, FROM_NAME]` keeps the name as written, and + `name_problems(name)`, a classmethod, holds any rule about it. + +Registration is the one moment an entity is refused, with a message +that says what to write: a class without `@dataclass`, a codec +subclassing `CodecEntity` instead of a kind, a field other than +`configuration` and a carried name, a configuration that is not a +`Configuration` record, a +member whose annotation is not a shape JSON takes +-- a nested entity without `Opaque` among them -- +a `__post_init__` of the entity's own, a class variable a base +annotates and nothing sets, and what a kind leaves abstract. What is +left, pyright says in the editor and the constructors say at runtime: a +member of the wrong type, a record that is not the entity's own, a +value the rules disallow, a `canonical` returning something else, a +hook with the wrong signature. +A scope reads what a class is off the class: its kind is its base, its +key is its `identifier`, so `extended_with` takes the classes and +nothing can be misfiled -- and a class whose `identifier` the scope +already has takes the name over, so registering your own `"gzip"` +replaces the package's reading of it. `Context.of(*classes)` is a scope +of exactly those, and a scope is a value: `resolve` reads in it, and +`claimant(kind, name)` says which class a name belongs to. + +Two complete extensions written against this module alone, as tests in +the repository: +`tests/v3/test_acme_affine.py` (an `array_array` codec with a number, an +optional member and a nested data type) and +`tests/v3/test_acme_decimal.py` (a configured data type with a +fill-value rule). + +A name in no scope is not rejected -- that is what extension openness +means -- so registering yours is how you get it judged rather than waved +through. `CORE` is what the specification defines; `CORE_AND_EXTENSIONS` +adds the `zarr-extensions` registry; `extended_with(*classes)` adds yours. +""" + +from __future__ import annotations + +from zarr_metadata._common import JSONValue +from zarr_metadata.model._sentinel import UNSET +from zarr_metadata.model._validation import ( + MetadataValidationError, + ProblemKind, + ValidationProblem, +) +from zarr_metadata.v3._chain import Pipeline, PipelineStage +from zarr_metadata.v3._common import ZarrV3MetadataFieldJSON +from zarr_metadata.v3._document import ( + ArrayDocumentV3, + RefinedArrayV3, + read_array_v3, + refine_array_v3, + well_formed_array_v3, +) +from zarr_metadata.v3._entity import ( + FROM_NAME, + ArrayArrayCodec, + ArrayBytesCodec, + BytesBytesCodec, + ChunkGridEntity, + ChunkKeyEncodingEntity, + CodecEntity, + Coerced, + Configuration, + DataTypeEntity, + Loc, + MetadataEntity, + Opaque, + StorageClass, + StorageTransformerEntity, + is_integer, + named_configuration, + problem, + resolve, + within, +) +from zarr_metadata.v3._parts import ArrayParts, ChunkGrid, Extents +from zarr_metadata.v3._registry import CORE, CORE_AND_EXTENSIONS, Context +from zarr_metadata.v3.data_type._families import ( + ComplexDataType, + FloatDataType, + IntegerDataType, + NumpyTimeDataType, +) + +__all__ = [ + "CORE", + "CORE_AND_EXTENSIONS", + "FROM_NAME", + "UNSET", + "ArrayArrayCodec", + "ArrayBytesCodec", + "ArrayDocumentV3", + "ArrayParts", + "BytesBytesCodec", + "ChunkGrid", + "ChunkGridEntity", + "ChunkKeyEncodingEntity", + "CodecEntity", + "Coerced", + "ComplexDataType", + "Configuration", + "Context", + "DataTypeEntity", + "Extents", + "FloatDataType", + "IntegerDataType", + "JSONValue", + "Loc", + "MetadataEntity", + "MetadataValidationError", + "NumpyTimeDataType", + "Opaque", + "Pipeline", + "PipelineStage", + "ProblemKind", + "RefinedArrayV3", + "StorageClass", + "StorageTransformerEntity", + "ValidationProblem", + "ZarrV3MetadataFieldJSON", + "is_integer", + "named_configuration", + "problem", + "read_array_v3", + "refine_array_v3", + "resolve", + "well_formed_array_v3", + "within", +] diff --git a/packages/zarr-metadata/tests/helpers.py b/packages/zarr-metadata/tests/helpers.py new file mode 100644 index 0000000000..ccc6294790 --- /dev/null +++ b/packages/zarr-metadata/tests/helpers.py @@ -0,0 +1,28 @@ +"""Ways for a test to look into JSON it has just been handed, without lying about its type.""" + +from __future__ import annotations + +from collections.abc import Mapping, Sequence +from typing import cast + + +def entry_at(document: object, *steps: str | int) -> object: + """The value `steps` lead to in `document`: a key into an object, an index into an array.""" + node = document + for step in steps: + if isinstance(step, str): + assert isinstance(node, Mapping), (steps, node) + node = cast("Mapping[str, object]", node)[step] + else: + assert isinstance(node, Sequence), (steps, node) + assert not isinstance(node, str), (steps, node) + node = cast("Sequence[object]", node)[step] + return node + + +def configuration_of(field: object) -> Mapping[str, object]: + """The configuration of a metadata field spelled as an object; empty if it has none.""" + assert isinstance(field, Mapping), field + configuration = cast("Mapping[str, object]", field).get("configuration", {}) + assert isinstance(configuration, Mapping), configuration + return cast("Mapping[str, object]", configuration) diff --git a/packages/zarr-metadata/tests/model/test_array.py b/packages/zarr-metadata/tests/model/test_array.py index 69a3823b6f..9c84b5250c 100644 --- a/packages/zarr-metadata/tests/model/test_array.py +++ b/packages/zarr-metadata/tests/model/test_array.py @@ -5,7 +5,7 @@ import json from collections import UserDict from collections.abc import Callable -from typing import TYPE_CHECKING, get_args +from typing import TYPE_CHECKING, cast, get_args import pytest from typing_extensions import Unpack @@ -40,6 +40,8 @@ from zarr_metadata.model._validation import _prefix, arrays_to_tuples if TYPE_CHECKING: + from collections.abc import Mapping + from zarr_metadata._common import JSONValue from zarr_metadata.v2 import ZarrV2CodecMetadata @@ -779,11 +781,19 @@ def test_metadata_field_rejects_unknown_envelope_member() -> None: @pytest.mark.parametrize("field", ["codecs", "storage_transformers"]) -def test_optional_extension_points_allow_must_understand_false(field: str) -> None: - """Codecs and storage transformers may be explicitly ignorable.""" +def test_every_extension_point_rejects_must_understand_false(field: str) -> None: + """No extension point may be declared ignorable. + + Ignoring a codec gives wrong bytes as surely as ignoring a data type + gives wrong values, so `must_understand` is a property of the kind of + metadata rather than a per-occurrence choice. The spec names only the + three required points; this package reads that as an oversight. + """ doc: dict[str, object] = dict(ZarrV3ArrayMetadata.create_default().to_json()) doc[field] = ({"name": "optional", "must_understand": False},) - assert validate_array_metadata_v3(doc) == () + assert [problem.loc for problem in validate_array_metadata_v3(doc)] == [ + (field, 0, "must_understand") + ] @pytest.mark.parametrize("field", ["data_type", "chunk_grid", "chunk_key_encoding"]) @@ -977,7 +987,7 @@ def test_parse_metadata_field_materializes_abstract_containers() -> None: assert isinstance(parsed, dict) assert parsed == {"name": "example", "configuration": {"values": (0, 1)}} - assert type(parsed["configuration"]) is dict + assert type(cast("Mapping[str, object]", parsed)["configuration"]) is dict def test_metadata_field_type_guard_rejects_abstract_mapping() -> None: @@ -1325,16 +1335,23 @@ def test_v2_filters_must_be_codec_sequence_or_none() -> None: assert [(p.loc, p.kind) for p in problems] == [(("filters",), "invalid_type")], bad -def test_v2_shape_and_chunks_must_have_equal_rank() -> None: - """Raw v2 metadata requires one chunk length per array dimension.""" +def test_v2_shape_chunks_rank_agreement_is_not_structural() -> None: + """Whether chunks matches shape's dimensionality is a composition + judgment owned by zarr_metadata.rules; the structural validator and the + model classes deliberately accept the document (it is a lossless, + structurally well-formed representation of what a store may contain).""" doc = dict(ZarrV2ArrayMetadata.create_default(shape=(2, 3)).to_json()) doc["chunks"] = (1,) - assert [(p.loc, p.kind) for p in validate_array_metadata_v2(doc)] == [ + assert validate_array_metadata_v2(doc) == () + parsed = ZarrV2ArrayMetadata.from_key_value({".zarray": json.dumps(doc).encode()}) + assert parsed.chunks == (1,) + + from zarr_metadata import rules + + assert [(p.loc, p.kind) for p in rules.validate_array_metadata_v2(doc)] == [ (("chunks",), "invalid_value") ] - with pytest.raises(MetadataValidationError, match="same number of dimensions"): - ZarrV2ArrayMetadata.from_key_value({".zarray": json.dumps(doc).encode()}) def test_v2_filters_may_be_empty() -> None: @@ -1415,12 +1432,12 @@ def test_array_v3_from_json_materializes_abstract_containers() -> None: def test_from_key_value_rejects_non_standard_json_constant() -> None: - """Store JSON decoding rejects JavaScript NaN/Infinity constants.""" + """A JavaScript NaN constant outside the attributes is located, not decoded as a fill value.""" doc = dict(ZarrV3ArrayMetadata.create_default().to_json()) doc["fill_value"] = float("nan") raw = json.dumps(doc) - with pytest.raises(MetadataValidationError, match="invalid JSON"): + with pytest.raises(MetadataValidationError, match="fill_value: non-finite float nan"): ZarrV3ArrayMetadata.from_key_value({"zarr.json": raw.encode()}) @@ -1428,7 +1445,7 @@ def test_to_key_value_rejects_non_finite_model_value() -> None: """Strict encoding prevents directly-constructed models from writing invalid JSON.""" model = ZarrV3ArrayMetadata.create_default(fill_value=float("nan")) - with pytest.raises(ValueError, match="JSON compliant"): + with pytest.raises(MetadataValidationError, match="fill_value: non-finite float nan"): model.to_key_value() @@ -1527,12 +1544,18 @@ def test_shape_rejects_negative_dimensions() -> None: ] -def test_dimension_names_length_must_match_shape() -> None: - """dimension_names must have one entry per dimension of shape.""" +def test_dimension_names_length_is_not_structural() -> None: + """Whether dimension_names matches shape's dimensionality is a + composition judgment owned by zarr_metadata.rules; the structural + validator deliberately accepts the document.""" doc = dict(ZarrV3ArrayMetadata.create_default(shape=(10,)).to_json()) | { "dimension_names": ("x", "y", "z") } - assert [(p.loc, p.kind) for p in validate_array_metadata_v3(doc)] == [ + assert validate_array_metadata_v3(doc) == () + + from zarr_metadata import rules + + assert [(p.loc, p.kind) for p in rules.validate_array_metadata_v3(doc)] == [ (("dimension_names",), "invalid_value") ] @@ -1559,7 +1582,7 @@ def test_configuration_values_must_be_json() -> None: def test_v3_extension_keys_must_be_strings() -> None: """A non-string top-level key cannot be represented by a v3 document type.""" - doc: dict[object, object] = dict(ZarrV3ArrayMetadata.create_default().to_json()) + doc: dict[object, object] = {**ZarrV3ArrayMetadata.create_default().to_json()} doc[1] = {"must_understand": False} assert [(problem.loc, problem.kind) for problem in validate_array_metadata_v3(doc)] == [ ((), "invalid_type") @@ -1758,7 +1781,7 @@ def test_v2_absent_dimension_separator_means_dot() -> None: del doc["dimension_separator"] model = ZarrV2ArrayMetadata.from_json(doc) assert model.dimension_separator == "." - assert model.to_json()["dimension_separator"] == "." + assert cast("Mapping[str, object]", model.to_json())["dimension_separator"] == "." def test_v2_from_key_value_without_separator_means_dot() -> None: diff --git a/packages/zarr-metadata/tests/model/test_group.py b/packages/zarr-metadata/tests/model/test_group.py index b0e451b0a0..a526731f00 100644 --- a/packages/zarr-metadata/tests/model/test_group.py +++ b/packages/zarr-metadata/tests/model/test_group.py @@ -5,6 +5,7 @@ import json from collections import UserDict from collections.abc import Callable +from typing import TYPE_CHECKING, cast import pytest @@ -33,6 +34,10 @@ # --- ZarrV3GroupMetadata --------------------------------------------------- +if TYPE_CHECKING: + from collections.abc import Mapping + + def test_group_v3_roundtrip() -> None: """A v3 group document round-trips through the model unchanged.""" doc = {"zarr_format": 3, "node_type": "group", "attributes": {"a": (1, 2)}} @@ -163,7 +168,9 @@ def test_group_guards_reject_noncanonical_nested_json() -> None: assert not is_group_metadata_v3(v3) assert not is_group_metadata_v2(v2) assert parse_group_metadata_v3(v3)["extension"] == (0, 1) - assert parse_group_metadata_v2(v2)["attributes"] == {"values": (0, 1)} + assert cast("Mapping[str, object]", parse_group_metadata_v2(v2))["attributes"] == { + "values": (0, 1) + } def test_group_v3_extension_fields_are_validated() -> None: diff --git a/packages/zarr-metadata/tests/model/test_pydantic.py b/packages/zarr-metadata/tests/model/test_pydantic.py index e5714bb8fb..90502803b3 100644 --- a/packages/zarr-metadata/tests/model/test_pydantic.py +++ b/packages/zarr-metadata/tests/model/test_pydantic.py @@ -24,8 +24,8 @@ messages intact. """ -from collections.abc import Mapping -from typing import Annotated, Generic, TypeVar +from collections.abc import Mapping, Sequence +from typing import Annotated, Generic, TypeVar, cast import pytest from pydantic import ( @@ -234,8 +234,9 @@ def _canonicalize(cls, data: object) -> object: if isinstance(doc[key], str): doc[key] = {"name": doc[key]} for key in ("codecs", "storage_transformers"): + entries = cast("Sequence[object]", doc.get(key, ())) doc[key] = tuple( - {"name": item} if isinstance(item, str) else item for item in doc.get(key, ()) + {"name": item} if isinstance(item, str) else item for item in entries ) doc.setdefault("attributes", {}) return doc diff --git a/packages/zarr-metadata/tests/model/test_pydantic_module.py b/packages/zarr-metadata/tests/model/test_pydantic_module.py index 80067e81ae..eaf870ea74 100644 --- a/packages/zarr-metadata/tests/model/test_pydantic_module.py +++ b/packages/zarr-metadata/tests/model/test_pydantic_module.py @@ -7,12 +7,14 @@ import json import warnings +from collections.abc import Mapping import pytest from jsonschema import Draft202012Validator from pydantic import BaseModel, TypeAdapter, ValidationError import zarr_metadata.pydantic as zmp +from zarr_metadata._common import JSONValue from zarr_metadata.model import ( ZarrV2ArrayMetadata, ZarrV2ConsolidatedMetadata, @@ -156,7 +158,9 @@ def test_v2_recursive_structured_dtype_is_in_pydantic_schema() -> None: assert Draft202012Validator(adapter.json_schema()).is_valid(doc) -def _assert_runtime_and_schema_reject(field_type: object, document: dict[str, object]) -> None: +def _assert_runtime_and_schema_reject( + field_type: object, document: Mapping[str, JSONValue] +) -> None: adapter = TypeAdapter(field_type) with pytest.raises(ValidationError): adapter.validate_python(document) @@ -265,6 +269,46 @@ def test_metadata_field_serializes_shorthand_and_false_object() -> None: ) == {"name": "optional", "must_understand": False} +def test_field_types_apply_composition_rules() -> None: + """The field types judge composition, not just structure. + + `fill_value: 300` is a well-formed JSON integer, so only the rules + layer can reject it for a `uint8` array. + """ + document = {**V3_ARRAY_DOC, "data_type": "uint8", "fill_value": 300} + with pytest.raises(ValidationError, match="fill_value"): + TypeAdapter(zmp.ZarrV3ArrayMetadata).validate_python(document) + + +def test_field_types_reject_unknown_configuration_members() -> None: + """An unmodelled member of a known entity's configuration is an error. + + Whether a `configuration` is closed is unspecified + (zarr-developers/zarr-specs#270), and this integration takes the + strict reading deliberately: in practice such a member is a typo or a + setting meant for a different entity, and silently accepting it means + silently ignoring what the writer asked for. Callers who want the + tolerant reading use `rules.validate_*` and filter `unknown_key`. + + Scope: the whole-document field types run the rules layer and so + reject it. `ZarrV3MetadataField` judges one metadata field, carries no + composition rules, and accepts it — asserted below so the boundary + cannot move silently. + """ + document = { + **V3_ARRAY_DOC, + "codecs": ({"name": "bytes", "configuration": {"endian": "little", "endain": "big"}},), + } + with pytest.raises(ValidationError, match="unexpected key 'endain'"): + TypeAdapter(zmp.ZarrV3ArrayMetadata).validate_python(document) + + codec = {"name": "bytes", "configuration": {"endian": "little", "endain": "big"}} + assert TypeAdapter(zmp.ZarrV3MetadataField).validate_python(codec).configuration == { + "endian": "little", + "endain": "big", + } + + def test_core_package_does_not_import_pydantic() -> None: """Importing zarr_metadata (in a fresh interpreter) must not import pydantic: the integration is opt-in via zarr_metadata.pydantic.""" diff --git a/packages/zarr-metadata/tests/model/test_refine_json.py b/packages/zarr-metadata/tests/model/test_refine_json.py new file mode 100644 index 0000000000..c87aaee7cc --- /dev/null +++ b/packages/zarr-metadata/tests/model/test_refine_json.py @@ -0,0 +1,163 @@ +"""The first layer of reading: a value refined to JSON, or the reasons it is not.""" + +from __future__ import annotations + +import math +from collections import OrderedDict +from typing import TYPE_CHECKING, cast + +import pytest + +from zarr_metadata.model import MetadataValidationError, ValidationProblem +from zarr_metadata.model._validation import refine_json, refine_node_json, stored_json_problems + +if TYPE_CHECKING: + from collections.abc import Mapping, Sequence + + +@pytest.mark.parametrize( + ("value", "refined"), + [ + (1, 1), + (2.5, 2.5), + (True, True), + ("x", "x"), + (None, None), + ([1, [2, 3]], (1, (2, 3))), + ((1, 2), (1, 2)), + ({"a": [1], "b": {"c": None}}, {"a": (1,), "b": {"c": None}}), + (OrderedDict(k=[0]), {"k": (0,)}), + ], + ids=["int", "float", "bool", "str", "null", "nested-lists", "tuple", "object", "any-mapping"], +) +def test_json_is_refined_to_tuples_and_dicts(value: object, refined: object) -> None: + # One walk normalizes and judges; what comes back is what every later + # layer takes, and nothing later normalizes again. + assert refine_json(value) == (refined, ()) + + +@pytest.mark.parametrize( + ("value", "loc", "kind"), + [ + ({"a": object()}, ("a",), "invalid_type"), + ([1, [2, {3: 4}]], (1, 1), "invalid_type"), + ({"x": math.inf}, ("x",), "invalid_value"), + ({"x": [math.nan]}, ("x", 0), "invalid_value"), + (b"bytes", (), "invalid_type"), + ], + ids=["not-json-leaf", "non-string-key", "infinite", "nan-in-array", "bytes"], +) +def test_error_a_value_that_is_not_json_is_none_with_the_leaf_located( + value: object, loc: tuple[str | int, ...], kind: str +) -> None: + refined, problems = refine_json(value) + assert refined is None + assert [(problem.loc, problem.kind) for problem in problems] == [(loc, kind)] + + +def test_error_every_leaf_that_is_not_json_is_reported() -> None: + refined, problems = refine_json({"a": object(), "b": [1, object()]}) + assert refined is None + assert [problem.loc for problem in problems] == [("a",), ("b", 1)] + + +def test_error_the_error_refuses_what_is_not_a_problem() -> None: + # `problem()` in the entity layer returns a one-element tuple; a list + # of those passes the type at the call and fails far away otherwise. + with pytest.raises(TypeError, match="collect with `extend`, not `append`"): + MetadataValidationError([(ValidationProblem(("a",), "bad a", "invalid_value"),)]) # pyright: ignore[reportArgumentType] + + +# A node's attributes are user data: Python's `json` writes a non-finite +# number there as `NaN`/`Infinity`, and zarr-python writes attributes that +# way, so the model reads, validates, and writes one there -- and nowhere +# else, where the spec spells those numbers as strings. + +_INLINE: dict[str, object] = {"kind": "inline", "must_understand": False} + + +def _at(value: object, path: tuple[str | int, ...]) -> object: + for part in path: + if isinstance(part, str): + value = cast("Mapping[str, object]", value)[part] + else: + value = cast("Sequence[object]", value)[part] + return value + + +@pytest.mark.parametrize( + ("document", "path"), + [ + ({"attributes": {"_FillValue": math.nan}}, ("attributes", "_FillValue")), + ({"attributes": {"valid_range": [-math.inf, math.inf]}}, ("attributes", "valid_range", 1)), + ( + {"attributes": {"cf": {"missing_value": -math.inf}}}, + ("attributes", "cf", "missing_value"), + ), + ( + { + "consolidated_metadata": { + **_INLINE, + "metadata": {"a": {"attributes": {"x": math.nan}}}, + } + }, + ("consolidated_metadata", "metadata", "a", "attributes", "x"), + ), + ], + ids=["nan", "infinities-in-an-array", "nested-object", "consolidated-node"], +) +def test_a_node_documents_attributes_hold_non_finite_numbers( + document: dict[str, object], path: tuple[str | int, ...] +) -> None: + refined, problems = refine_node_json(document) + assert problems == () + held = _at(refined, path) + assert isinstance(held, float) + assert not math.isfinite(held) + # What the model reads, it writes back, in the spelling Python's + # `json` module and zarr-python use. + assert stored_json_problems("zarr.json", document) == () + assert stored_json_problems(".zattrs", _at(document, path[:-1])) == () + + +@pytest.mark.parametrize( + ("document", "loc"), + [ + ({"fill_value": math.nan}, ("fill_value",)), + ( + {"codecs": ({"name": "scale_offset", "configuration": {"scale": math.inf}},)}, + ("codecs", 0, "configuration", "scale"), + ), + ({"extension": {"must_understand": False, "x": math.nan}}, ("extension", "x")), + ( + {"consolidated_metadata": {**_INLINE, "metadata": {"a": {"fill_value": math.nan}}}}, + ("consolidated_metadata", "metadata", "a", "fill_value"), + ), + ], + ids=["fill-value", "codec-configuration", "extension-field", "consolidated-node"], +) +def test_error_a_non_finite_number_outside_attributes_is_not_json( + document: dict[str, object], loc: tuple[str | int, ...] +) -> None: + refined, problems = refine_node_json(document) + assert refined is None + assert [(problem.loc, problem.kind) for problem in problems] == [(loc, "invalid_value")] + assert [problem.loc for problem in stored_json_problems("zarr.json", document)] == [loc] + + +def test_error_a_zarray_holds_no_user_data() -> None: + # A v2 array's attributes live in `.zattrs`; its `.zarray` is RFC 8259 + # throughout -- which is where zarr-python 3.0 once wrote a bare `NaN`. + problems = stored_json_problems("a/.zarray", {"fill_value": math.nan}) + assert [(problem.loc, problem.kind) for problem in problems] == [ + (("fill_value",), "invalid_value") + ] + + +def test_error_a_zmetadata_entry_is_judged_as_the_document_its_key_names() -> None: + consolidated = { + "zarr_consolidated_format": 1, + "metadata": {"a/.zattrs": {"x": math.nan}, "a/.zarray": {"fill_value": math.nan}}, + } + problems = stored_json_problems(".zmetadata", consolidated) + assert [problem.loc for problem in problems] == [("metadata", "a/.zarray", "fill_value")] diff --git a/packages/zarr-metadata/tests/model/test_store_json.py b/packages/zarr-metadata/tests/model/test_store_json.py new file mode 100644 index 0000000000..34ac2ca63a --- /dev/null +++ b/packages/zarr-metadata/tests/model/test_store_json.py @@ -0,0 +1,151 @@ +"""What the model reads from and writes to a store, key by key. + +A node's attributes are user data, and zarr-python writes them with the +defaults of Python's `json` module, so an attribute may hold `NaN`, +`Infinity` or `-Infinity`. The model reads such a store, validates it, and +writes it back the same way; anywhere else a non-finite number is refused. +""" + +from __future__ import annotations + +import json +import math +from typing import TYPE_CHECKING, Any, Protocol, cast + +import pytest + +from zarr_metadata.model import ( + MetadataValidationError, + ZarrV2ArrayMetadata, + ZarrV2ConsolidatedMetadata, + ZarrV2GroupMetadata, + ZarrV3ArrayMetadata, + ZarrV3GroupMetadata, + is_array_metadata_v2, + is_array_metadata_v3, + is_group_metadata_v2, + is_group_metadata_v3, +) + +if TYPE_CHECKING: + from collections.abc import Callable, Mapping + + +class _Stored(Protocol): + """A model that writes itself to a store.""" + + def to_json(self) -> Mapping[str, object]: ... + + def to_key_value(self, *, indent: int | str | None = None) -> Mapping[str, bytes]: ... + + +ATTRIBUTES: dict[str, object] = {"_FillValue": math.nan, "valid_range": [-math.inf, math.inf]} + + +def _held(attributes: object) -> tuple[bool, bool, bool]: + held = cast("Mapping[str, object]", attributes) + fill = held["_FillValue"] + low, high = cast("tuple[float, float]", held["valid_range"]) + return (isinstance(fill, float) and math.isnan(fill), low == -math.inf, high == math.inf) + + +def _v3_array() -> Mapping[str, bytes]: + document = {**ZarrV3ArrayMetadata.create_default().to_json(), "attributes": ATTRIBUTES} + return {"zarr.json": json.dumps(document).encode()} + + +def _v3_group() -> Mapping[str, bytes]: + document = {"zarr_format": 3, "node_type": "group", "attributes": ATTRIBUTES} + return {"zarr.json": json.dumps(document).encode()} + + +def _v2_array() -> Mapping[str, bytes]: + zarray = dict(ZarrV2ArrayMetadata.create_default().to_json()) + return {".zarray": json.dumps(zarray).encode(), ".zattrs": json.dumps(ATTRIBUTES).encode()} + + +def _v2_group() -> Mapping[str, bytes]: + return { + ".zgroup": json.dumps({"zarr_format": 2}).encode(), + ".zattrs": json.dumps(ATTRIBUTES).encode(), + } + + +def _v2_consolidated() -> Mapping[str, bytes]: + document = { + "zarr_consolidated_format": 1, + "metadata": {".zgroup": {"zarr_format": 2}, ".zattrs": ATTRIBUTES}, + } + return {".zmetadata": json.dumps(document).encode()} + + +@pytest.mark.parametrize( + ("store", "read", "attributes_of", "guard"), + [ + ( + _v3_array, + ZarrV3ArrayMetadata.from_key_value, + lambda model: model.attributes, + is_array_metadata_v3, + ), + ( + _v3_group, + ZarrV3GroupMetadata.from_key_value, + lambda model: model.attributes, + is_group_metadata_v3, + ), + ( + _v2_array, + ZarrV2ArrayMetadata.from_key_value, + lambda model: model.attributes, + is_array_metadata_v2, + ), + ( + _v2_group, + ZarrV2GroupMetadata.from_key_value, + lambda model: model.attributes, + is_group_metadata_v2, + ), + ( + _v2_consolidated, + ZarrV2ConsolidatedMetadata.from_key_value, + lambda model: model.metadata[".zattrs"], + None, + ), + ], + ids=["v3-array", "v3-group", "v2-array", "v2-group", "v2-consolidated"], +) +def test_attributes_holding_non_finite_numbers_round_trip_through_the_store( + store: Callable[[], Mapping[str, bytes]], + read: Callable[[Mapping[str, bytes]], _Stored], + attributes_of: Callable[[Any], object], + guard: Callable[[object], bool] | None, +) -> None: + model = read(store()) + assert _held(attributes_of(model)) == (True, True, True) + if guard is not None: + assert guard(model.to_json()) + written = model.to_key_value() + assert b"NaN" in b"".join(written.values()) + assert _held(attributes_of(read(written))) == (True, True, True) + + +def test_error_a_non_finite_number_outside_attributes_is_located_when_read() -> None: + # Python's decoder reads a bare `NaN` as a float; where it may not be, + # the store says where it is. + zarray = dict(ZarrV2ArrayMetadata.create_default().to_json()) + zarray["fill_value"] = math.nan + with pytest.raises(MetadataValidationError) as raised: + ZarrV2ArrayMetadata.from_key_value({".zarray": json.dumps(zarray).encode()}) + assert [(problem.loc, problem.kind) for problem in raised.value.problems] == [ + (("fill_value",), "invalid_value") + ] + + +def test_error_a_non_finite_number_outside_attributes_is_not_written() -> None: + # A model built by hand is not validated; the writer is what stands + # between a non-finite fill value and a document no reader accepts. + model = ZarrV3ArrayMetadata.create_default(fill_value=math.nan, attributes={"x": math.nan}) + with pytest.raises(MetadataValidationError) as raised: + model.to_key_value() + assert [problem.loc for problem in raised.value.problems] == [("fill_value",)] diff --git a/packages/zarr-metadata/tests/rules/__init__.py b/packages/zarr-metadata/tests/rules/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/packages/zarr-metadata/tests/rules/strategies.py b/packages/zarr-metadata/tests/rules/strategies.py new file mode 100644 index 0000000000..7bd9750b5a --- /dev/null +++ b/packages/zarr-metadata/tests/rules/strategies.py @@ -0,0 +1,222 @@ +"""Hypothesis strategies over codec chains, derived from the codec types. + +Two kinds of strategy, because the two halves of "is this validator right?" +need opposite inputs. + +`codec_chains` draws each codec with `st.from_type` over its own +`TypedDict`, so the values are structurally plausible and almost always +semantically wrong. That is what the rejection side wants. The chain is +assembled by *kind* — `array->array`* `array->bytes` `bytes->bytes`* — +which guarantees an `array->bytes` codec is present: measured over 3000 +documents, an entity rule is dispatched for 100% of ordered chains and one +reports a problem for 86%, against 78% for a flat list of the same codecs. +Nothing short-circuits on a misordered chain, so the gain is coverage of +the array->bytes codecs rather than avoided early exit. + +What this cannot reach, because `st.from_type` honours the TypedDicts +exactly, is a configuration member of the wrong *type* — +`corrupted_chains` exists for that. + +`valid_documents` builds documents that must be accepted, by construction: +extents are drawn, then inner chunk shapes are drawn from their divisors. +Nothing else here can test the accept side, because fewer than 5% of +`from_type` documents are valid. + +`st.from_type` handles `ReadOnly`, `closed=True`, `NotRequired` and +`Literal` unaided. It cannot resolve the recursive `JSONValue` alias, so +this module registers one for it; without that, every codec whose +configuration admits arbitrary JSON (`sharding_indexed`, `cast_value`, +`scale_offset`) fails to resolve. +""" + +from __future__ import annotations + +from math import gcd +from typing import TYPE_CHECKING, Any, cast + +from hypothesis import strategies as st + +from zarr_metadata._common import JSONValue +from zarr_metadata.v3.codec.blosc import BloscCodecObject +from zarr_metadata.v3.codec.bytes import BytesCodecObject +from zarr_metadata.v3.codec.cast_value import CastValueCodecObject +from zarr_metadata.v3.codec.crc32c import Crc32cCodecObject +from zarr_metadata.v3.codec.gzip import GzipCodecObject +from zarr_metadata.v3.codec.scale_offset import ScaleOffsetCodecObject +from zarr_metadata.v3.codec.sharding_indexed import ShardingIndexedCodecObject +from zarr_metadata.v3.codec.transpose import TransposeCodecObject +from zarr_metadata.v3.codec.zstd import ZstdCodecObject + +if TYPE_CHECKING: + from collections.abc import Sequence + +JSON_VALUES = st.recursive( + st.none() | st.booleans() | st.integers() | st.text(max_size=8), + lambda children: ( + st.lists(children, max_size=3) | st.dictionaries(st.text(max_size=6), children, max_size=3) + ), + max_leaves=5, +) +# `JSONValue` is a `TypeAliasType`, which `register_type_strategy` does not +# accept in its signature but does resolve at runtime — it is exactly the +# forward reference `from_type` fails on. +st.register_type_strategy(JSONValue, JSON_VALUES) # pyright: ignore[reportArgumentType] + +# The codec TypedDicts, by pipeline kind. Hand-written because there is no +# name-to-type table to derive it from; `test_chain_properties.py` asserts it +# covers every codec the package models. +ARRAY_ARRAY = (TransposeCodecObject, CastValueCodecObject, ScaleOffsetCodecObject) +ARRAY_BYTES = (BytesCodecObject, ShardingIndexedCodecObject) +BYTES_BYTES = (BloscCodecObject, Crc32cCodecObject, GzipCodecObject, ZstdCodecObject) + + +_LITTLE: dict[str, object] = {"name": "bytes", "configuration": {"endian": "little"}} + + +def _any_of(types: Sequence[type]) -> st.SearchStrategy[Any]: + return st.one_of(*[st.from_type(entry) for entry in types]) + + +def codec_chains() -> st.SearchStrategy[tuple[object, ...]]: + """Chains in the shape the spec requires, with arbitrary configurations.""" + return st.tuples( + st.lists(_any_of(ARRAY_ARRAY), max_size=2), + _any_of(ARRAY_BYTES), + st.lists(_any_of(BYTES_BYTES), max_size=2), + ).map(lambda parts: (*parts[0], parts[1], *parts[2])) + + +@st.composite +def rank_matched_shards(draw: st.DrawFn) -> tuple[object, ...]: + """Chains whose shard has the document's rank, with arbitrary extents. + + `codec_chains` draws a shard's `chunk_shape` freely, so it almost never + has the right rank and the rank check short-circuits before geometry is + reached. Matching the rank is what exposes divisibility to the fuzzer. + """ + extents = tuple(draw(st.lists(st.integers(min_value=1, max_value=48), min_size=2, max_size=2))) + return ( + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": extents, + "codecs": (_LITTLE,), + "index_codecs": (_LITTLE,), + }, + }, + ) + + +def document(codecs: object, **overrides: object) -> dict[str, object]: + """A v3 array document around `codecs`, valid apart from what is passed.""" + return { + "zarr_format": 3, + "node_type": "array", + "shape": (64, 64), + "data_type": "uint16", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (32, 32)}}, + "chunk_key_encoding": "default", + "codecs": codecs, + **overrides, + } + + +def _divisors(value: int) -> list[int]: + return [candidate for candidate in range(1, value + 1) if value % candidate == 0] + + +@st.composite +def valid_documents(draw: st.DrawFn) -> dict[str, object]: + """Documents that must validate clean, correlated by construction. + + A transpose permutes the grid and the shard's inner shape follows it, + so the inner extents are checked against the axis they actually meet — + including a rectilinear axis whose chunks differ, where only a common + divisor of every length will do. + """ + rank = draw(st.integers(min_value=1, max_value=3)) + rectilinear = draw(st.booleans()) + # Per axis: the lengths its chunks take, and an extent dividing all of them. + lengths: list[tuple[int, ...]] = [] + for _ in range(rank): + if rectilinear: + axis = tuple(draw(st.lists(st.sampled_from([8, 12, 16, 24]), min_size=1, max_size=3))) + else: + axis = (draw(st.sampled_from([8, 12, 16, 24])),) + lengths.append(axis) + inner = [draw(st.sampled_from(_divisors(gcd(*axis, axis[0])))) for axis in lengths] + order = tuple(draw(st.permutations(range(rank)))) + + if rectilinear: + grid: dict[str, object] = { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": tuple(lengths)}, + } + shape = tuple(sum(axis) for axis in lengths) + else: + grid = { + "name": "regular", + "configuration": {"chunk_shape": tuple(axis[0] for axis in lengths)}, + } + shape = tuple(axis[0] * 2 for axis in lengths) + + shard = { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": tuple(inner[axis] for axis in order), + "codecs": (_LITTLE,), + "index_codecs": (_LITTLE,), + }, + } + return { + "zarr_format": 3, + "node_type": "array", + "shape": shape, + "data_type": "uint16", + "fill_value": 0, + "chunk_grid": grid, + "chunk_key_encoding": "default", + "codecs": ({"name": "transpose", "configuration": {"order": order}}, shard), + } + + +@st.composite +def corrupted_chains(draw: st.DrawFn) -> tuple[object, ...]: + """A well-typed chain with one configuration member replaced by any JSON. + + `st.from_type` honours the TypedDicts, so it never produces an ill-typed + member — and an ill-typed member is exactly what the `reads` gate exists + to stand rules down for. Without this, the gate that makes + `configuration["level"]` safe inside a rule has no generated coverage, + and a validator that raises `TypeError` on ordinary malformed metadata + looks identical to one that does not. + """ + chain = list(draw(codec_chains())) + index = draw(st.integers(min_value=0, max_value=len(chain) - 1)) + codec = chain[index] + if not isinstance(codec, dict): + return tuple(chain) + entry = cast("dict[str, object]", dict(codec)) + configuration = entry.get("configuration") + if not isinstance(configuration, dict) or len(cast("dict[str, object]", configuration)) == 0: + return tuple(chain) + members = cast("dict[str, object]", dict(configuration)) + member = draw(st.sampled_from(sorted(members))) + members[member] = draw(JSON_VALUES) + entry["configuration"] = members + chain[index] = entry + return tuple(chain) + + +__all__ = [ + "ARRAY_ARRAY", + "ARRAY_BYTES", + "BYTES_BYTES", + "JSON_VALUES", + "codec_chains", + "corrupted_chains", + "document", + "rank_matched_shards", + "valid_documents", +] diff --git a/packages/zarr-metadata/tests/rules/test_canonical.py b/packages/zarr-metadata/tests/rules/test_canonical.py new file mode 100644 index 0000000000..78b8efaf2a --- /dev/null +++ b/packages/zarr-metadata/tests/rules/test_canonical.py @@ -0,0 +1,167 @@ +"""Canonicalization: the simplest spelling with the same meaning. + +Two properties carry the weight. Canonicalizing twice must change nothing +further, or the form is not canonical. And canonicalizing must never +change a verdict, or it is not meaning-preserving — that one is what +catches a "simplification" that quietly says something else. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +import pytest +from hypothesis import HealthCheck, given, settings + +from tests.helpers import configuration_of, entry_at +from tests.rules.strategies import valid_documents +from zarr_metadata.rules import ( + Canonical, + Invalid, + canonicalize_array_metadata_v3, + validate_array_metadata_v3, +) + +if TYPE_CHECKING: + from collections.abc import Mapping + +_SLOW = settings(max_examples=300, deadline=None, suppress_health_check=(HealthCheck.too_slow,)) + +BASE: Mapping[str, object] = { + "zarr_format": 3, + "node_type": "array", + "shape": (64,), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (32,)}}, + "chunk_key_encoding": "default", + "codecs": ("bytes",), +} + + +def _canonical(**overrides: object) -> Mapping[str, object]: + result = canonicalize_array_metadata_v3({**BASE, **overrides}) + assert isinstance(result, Canonical), result + return result.document + + +# (what it should simplify, the field it lands in, the expected canonical value) +SIMPLIFICATIONS: dict[str, tuple[dict[str, object], str, object]] = { + "object-with-no-configuration": ({"codecs": ({"name": "bytes"},)}, "codecs", ("bytes",)), + "object-with-empty-configuration": ( + {"codecs": ({"name": "bytes", "configuration": {}},)}, + "codecs", + ("bytes",), + ), + "defaulted-must-understand": ( + {"codecs": ({"name": "bytes", "must_understand": True},)}, + "codecs", + ("bytes",), + ), + "chunk-key-encoding-shorthand": ( + {"chunk_key_encoding": {"name": "default"}}, + "chunk_key_encoding", + "default", + ), + "rectilinear-runs-encode": ( + { + "chunk_grid": { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": ((32, 32),)}, + } + }, + "chunk_grid", + { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": (((32, 2),),)}, + }, + ), +} + + +@pytest.mark.parametrize( + ("overrides", "field", "expected"), SIMPLIFICATIONS.values(), ids=list(SIMPLIFICATIONS) +) +def test_simplifies(overrides: dict[str, object], field: str, expected: object) -> None: + assert _canonical(**overrides)[field] == expected + + +def test_error_an_extension_point_may_not_be_declared_ignorable() -> None: + # `must_understand` belongs to the kind of metadata, not to a use of + # it, and no extension point is skippable -- so there is nothing for + # canonicalization to keep. + result = canonicalize_array_metadata_v3( + {**BASE, "codecs": ({"name": "bytes", "must_understand": False},)} + ) + assert isinstance(result, Invalid) + assert [problem.loc for problem in result.problems] == [("codecs", 0, "must_understand")] + + +def test_blosc_drops_a_typesize_that_shuffle_renders_ignored() -> None: + configuration = {"cname": "zstd", "clevel": 5, "blocksize": 0, "typesize": 4} + dropped = _canonical( + codecs=( + "bytes", + {"name": "blosc", "configuration": {**configuration, "shuffle": "noshuffle"}}, + ) + ) + assert "typesize" not in configuration_of(entry_at(dropped, "codecs", 1)) + kept = _canonical( + codecs=( + "bytes", + {"name": "blosc", "configuration": {**configuration, "shuffle": "shuffle"}}, + ) + ) + assert configuration_of(entry_at(kept, "codecs", 1))["typesize"] == 4 + + +def test_dimension_names_of_nothing_but_nulls_are_dropped() -> None: + assert "dimension_names" not in _canonical(dimension_names=(None,)) + assert _canonical(dimension_names=("x",))["dimension_names"] == ("x",) + + +def test_a_rectilinear_step_is_not_expanded() -> None: + # A bare integer repeats to cover the extent, so it is not equivalent + # to any fixed list -- and a one-element list is not equivalent to it. + for spec in (32, (32,)): + grid = { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": (spec,)}, + } + shape = (64,) if spec == 32 else (32,) + result = canonicalize_array_metadata_v3({**BASE, "shape": shape, "chunk_grid": grid}) + assert isinstance(result, Canonical), result + assert entry_at(result.document, "chunk_grid", "configuration", "chunk_shapes") == (spec,) + + +def test_error_a_semantically_invalid_document_reports_instead() -> None: + result = canonicalize_array_metadata_v3({**BASE, "fill_value": 999}) + assert isinstance(result, Invalid) + # The field is the location, not part of the message: the data type + # says what it accepts, and the document says where it was asked. + assert [problem.loc for problem in result.problems] == [("fill_value",)] + + +def test_error_invalid_cannot_be_empty() -> None: + with pytest.raises(ValueError, match="at least one"): + Invalid(()) + + +@given(valid_documents()) +@_SLOW +def test_canonicalizing_twice_changes_nothing_further(doc: Mapping[str, object]) -> None: + once = canonicalize_array_metadata_v3(doc) + assert isinstance(once, Canonical), once + twice = canonicalize_array_metadata_v3(once.document) + assert isinstance(twice, Canonical), twice + assert twice.document == once.document + + +@given(valid_documents()) +@_SLOW +def test_canonicalizing_never_changes_the_verdict(doc: Mapping[str, object]) -> None: + # A simplification that changed meaning would show up here as a + # document that validated before and does not after. + result = canonicalize_array_metadata_v3(doc) + assert isinstance(result, Canonical), result + assert validate_array_metadata_v3(result.document) == () diff --git a/packages/zarr-metadata/tests/rules/test_chain_properties.py b/packages/zarr-metadata/tests/rules/test_chain_properties.py new file mode 100644 index 0000000000..77da33530b --- /dev/null +++ b/packages/zarr-metadata/tests/rules/test_chain_properties.py @@ -0,0 +1,175 @@ +"""Property tests over codec chains drawn from the codec types themselves. + +The existing totality test feeds arbitrary JSON to the document +validators, which is the right guard for the structural layer and no guard +at all for this one: a random object never names a codec, so it never +dispatches an entity rule. `test_the_chain_strategy_reaches_the_rules` +exists so that stays visible — a refactor that silently stops dispatching +fails here instead of staying green. +""" + +from __future__ import annotations + +import pkgutil +from typing import TYPE_CHECKING + +from hypothesis import HealthCheck, given, settings + +import zarr_metadata.v3.codec +from tests.rules.strategies import ( + ARRAY_ARRAY, + ARRAY_BYTES, + BYTES_BYTES, + codec_chains, + corrupted_chains, + document, + rank_matched_shards, + valid_documents, +) +from zarr_metadata.rules import validate_array_metadata_v3 +from zarr_metadata.v3.entity import ( + CORE_AND_EXTENSIONS, + ArrayArrayCodec, + ArrayBytesCodec, + CodecEntity, +) + +if TYPE_CHECKING: + from collections.abc import Mapping + +# Drawing a codec chain and validating a document is slow per example and +# the outer functions below run a whole @given loop of their own, so the two +# timing checks are suppressed; nothing else is. +_SLOW = settings( + max_examples=300, + deadline=None, + suppress_health_check=(HealthCheck.too_slow, HealthCheck.data_too_large), +) + + +def test_the_strategies_cover_every_codec_the_package_models() -> None: + # The kind tuples are hand-written; a new codec module must join one. + drawn = { + entry.__annotations__["name"].__args__[0] + for entry in (*ARRAY_ARRAY, *ARRAY_BYTES, *BYTES_BYTES) + } + modelled = { + value + for info in pkgutil.iter_modules(zarr_metadata.v3.codec.__path__) + if not info.name.startswith("_") + for attribute, value in vars( + __import__(f"zarr_metadata.v3.codec.{info.name}", fromlist=["_"]) + ).items() + if attribute.endswith("_CODEC_NAME") and isinstance(value, str) + } + assert modelled == drawn + for kinds, expected in ((ARRAY_ARRAY, ArrayArrayCodec), (ARRAY_BYTES, ArrayBytesCodec)): + for entry in kinds: + name = entry.__annotations__["name"].__args__[0] + entity = CORE_AND_EXTENSIONS.claimant(CodecEntity, name) + assert entity is not None, name + assert issubclass(entity, expected) + + +@given(codec_chains()) +@_SLOW +def test_a_well_ordered_chain_always_produces_a_verdict(codecs: tuple[object, ...]) -> None: + """Untrusted input gets judged, never raises, however odd the values.""" + assert isinstance(validate_array_metadata_v3(document(codecs)), tuple) + + +# The rules a chain of arbitrary codec configurations reliably exercises, +# with the observed rate over 600 examples. The three chain rules absent +# here are out of this strategy's reach by construction, not by accident: +# pipeline ordering cannot fire because the chain is assembled in order, +# and variable-length data types and variable-size index codecs need a +# correlated data type and index chain that only a hand-written case +# supplies. Those have their own tests in `test_v3_array_rules.py`. +_WITNESSES: Mapping[str, str] = { + "rank against the incoming array": "incoming array has", # 63% + "positive chunk extents": "expected an integer >= 1", # 34% + "transpose order is a permutation": "expected a permutation", # 21% + "endianness for multi-byte types": "endian is required", # 20% +} + + +def test_the_chain_strategy_reaches_the_rules() -> None: + reached: set[str] = set() + + @given(codec_chains()) + @_SLOW + def sample(codecs: tuple[object, ...]) -> None: + messages = [problem.message for problem in validate_array_metadata_v3(document(codecs))] + reached.update( + label + for label, witness in _WITNESSES.items() + if any(witness in message for message in messages) + ) + + sample() + assert reached == set(_WITNESSES) + + +def test_the_shard_strategy_reaches_chunk_geometry() -> None: + # A shard whose rank matches is what exposes divisibility: drawn freely + # it almost never has the right rank, and the rank check returns first. + reached = [False] + + @given(rank_matched_shards()) + @_SLOW + def sample(codecs: tuple[object, ...]) -> None: + if any( + "does not evenly divide" in problem.message + for problem in validate_array_metadata_v3(document(codecs)) + ): + reached[0] = True + + sample() + assert reached[0] + + +@given(valid_documents()) +@_SLOW +def test_documents_valid_by_construction_are_accepted(doc: Mapping[str, object]) -> None: + """The accept side: extents drawn, then inner shapes from their divisors. + + Every fix to this layer has made it stricter, and almost nothing drawn + from the codec types is valid, so this is the only property guarding + the other direction over generated input. + """ + problems = validate_array_metadata_v3(doc) + assert problems == (), [problem.message for problem in problems] + + +@given(corrupted_chains()) +@_SLOW +def test_an_ill_typed_configuration_member_is_reported_not_raised( + codecs: tuple[object, ...], +) -> None: + """A member of the wrong type stands its rules down; it never crashes one. + + This is the property the `reads` gate exists for, and the only generated + input that exercises it: `st.from_type` honours the TypedDicts, so every + other strategy here produces well-typed members. Without this, four + separate one-token changes to the gate leave the suite green while + `validate_*` raises `TypeError` on ordinary malformed metadata. + """ + problems = validate_array_metadata_v3(document(codecs)) + assert isinstance(problems, tuple) + + +def test_the_corrupting_strategy_reaches_the_gate() -> None: + # It has to produce ill-typed members, or the property above is vacuous. + reached = [False] + + @given(corrupted_chains()) + @_SLOW + def sample(codecs: tuple[object, ...]) -> None: + if any( + problem.kind == "invalid_type" + for problem in validate_array_metadata_v3(document(codecs)) + ): + reached[0] = True + + sample() + assert reached[0] diff --git a/packages/zarr-metadata/tests/rules/test_chunk_grid.py b/packages/zarr-metadata/tests/rules/test_chunk_grid.py new file mode 100644 index 0000000000..a431d785eb --- /dev/null +++ b/packages/zarr-metadata/tests/rules/test_chunk_grid.py @@ -0,0 +1,274 @@ +"""What array a chunk grid governs, and what follows from knowing it.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +import pytest + +from tests.helpers import configuration_of +from zarr_metadata.rules import validate_array_metadata_v3 +from zarr_metadata.v3._parts import ChunkGrid, shard_index_grid +from zarr_metadata.v3._registry import CORE_AND_EXTENSIONS +from zarr_metadata.v3.entity import ChunkGridEntity, resolve + +if TYPE_CHECKING: + from collections.abc import Mapping + +BASE: Mapping[str, object] = { + "zarr_format": 3, + "node_type": "array", + "shape": (64, 64), + "data_type": "uint8", + "fill_value": 0, + "chunk_key_encoding": "default", +} +REGULAR: Mapping[str, object] = {"name": "regular", "configuration": {"chunk_shape": (32, 32)}} + + +def _rectilinear(chunk_shapes: object) -> Mapping[str, object]: + return { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": chunk_shapes}, + } + + +# The shard index is uint64, so its bytes codec needs an endianness; these +# tests are about geometry, not about that. +_INDEX_CODECS = ({"name": "bytes", "configuration": {"endian": "little"}},) + + +def _shard(inner: object, index: object = _INDEX_CODECS) -> Mapping[str, object]: + return { + "name": "sharding_indexed", + "configuration": {"chunk_shape": inner, "codecs": ("bytes",), "index_codecs": index}, + } + + +def _u(*lengths: int) -> tuple[frozenset[int], ...]: + return tuple(frozenset({length}) for length in lengths) + + +# (grid metadata, array shape, rank, the lengths each axis's chunks take). +# A `None` axis is one this package cannot read; a `None` extents is a grid +# that does not even pin the rank. +GRIDS: dict[str, tuple[object, object, object, object]] = { + "regular": (REGULAR, (64, 64), 2, _u(32, 32)), + # A zero extent is not a grid, so the entity refuses to exist and + # only the rank the array shape pins survives -- the same answer as + # for a grid this package cannot read at all. + "regular-zero-length-is-unreadable": ( + {"name": "regular", "configuration": {"chunk_shape": (0, 32)}}, + (64, 64), + 2, + (None, None), + ), + "rectilinear-uniform": (_rectilinear(((32, 32), (32, 32))), (64, 64), 2, _u(32, 32)), + "rectilinear-uniform-rle": (_rectilinear((((32, 2),), ((32, 2),))), (64, 64), 2, _u(32, 32)), + "rectilinear-bare-int-is-a-regular-step": (_rectilinear((32, 32)), (64, 64), 2, _u(32, 32)), + "rectilinear-varying-axis-keeps-its-lengths": ( + _rectilinear(((30, 34), (32, 32))), + (64, 64), + 2, + (frozenset({30, 34}), frozenset({32})), + ), + "rectilinear-rle-varying": ( + _rectilinear( + ((((30, 2)), ((34, 1))),), + ), + (64,), + 1, + (frozenset({30, 34}),), + ), + "unknown-grid-keeps-the-array-rank": ( + {"name": "mycorp.hilbert", "configuration": {"anything": 1}}, + (64, 64), + 2, + (None, None), + ), + "no-usable-array-shape": (None, "not a shape", None, None), +} + + +def _grid_of(grid: object, shape: object) -> ChunkGrid: + """The grid `grid` describes over an array of `shape`. + + A grid entity builds its own; one out of scope pins only the rank the + array shape gives it. + """ + entity, _ = resolve(grid, ChunkGridEntity, CORE_AND_EXTENSIONS) + if not isinstance(entity, ChunkGridEntity): + return ChunkGrid.unreadable(shape) + return entity.grid(shape) + + +@pytest.mark.parametrize(("grid", "shape", "rank", "extents"), GRIDS.values(), ids=list(GRIDS)) +def test_chunk_grid_of(grid: object, shape: object, rank: object, extents: object) -> None: + built = _grid_of(grid, shape) + assert built.rank == rank + assert built.extents == extents + + +def test_permuting_reorders_the_axes() -> None: + grid = _grid_of(_rectilinear(((30, 34), (32, 32))), (64, 64)) + assert grid.permuted((1, 0)).extents == (frozenset({32}), frozenset({30, 34})) + # An order that is not a permutation of the rank keeps the rank only. + assert grid.permuted((0, 1, 2)).extents is None + assert grid.permuted((0, 1, 2)).rank == 2 + + +def test_shard_index_grid_is_chunks_per_shard_plus_two() -> None: + # "a shape that matches the chunks per shard tuple with an appended + # dimension of size 2" -- 128/32 = 4 along both axes. + assert shard_index_grid(ChunkGrid.regular((128, 128)), (32, 32)).extents == _u(4, 4, 2) + # A shard whose own extents vary makes the chunk count vary with it. + assert shard_index_grid(ChunkGrid.derived((frozenset({30, 60}),)), (15,)).extents == ( + frozenset({2, 4}), + frozenset({2}), + ) + # The trailing 2 is fixed by the spec, so it survives knowing nothing. + assert shard_index_grid(ChunkGrid(None, None), (32, 32)).extents == (None, None, frozenset({2})) + + +@pytest.mark.parametrize( + "chunk_shapes", + [((32, 32), (32, 32)), (((32, 2),), ((32, 2),)), (32, 32)], + ids=["explicit", "run-length", "bare-int"], +) +def test_error_shard_must_divide_a_uniform_rectilinear_grid(chunk_shapes: object) -> None: + # A rectilinear grid has no single chunk shape, but a uniform one pins + # every extent, and 7 divides neither. + problems = validate_array_metadata_v3( + {**BASE, "chunk_grid": _rectilinear(chunk_shapes), "codecs": (_shard((7, 7)),)} + ) + assert len(problems) == 2 + assert all("does not evenly divide" in problem.message for problem in problems) + + +def test_error_a_shard_must_divide_every_chunk_a_varying_axis_has() -> None: + # The case the per-grid summary could not express: dim 0's chunks are + # 30 and 34 long, dim 1's are all 32, and a transpose swaps them before + # the shard sees them. The inner extent must divide *every* length the + # axis takes, so only a common divisor of 30 and 34 will do. + grid = _rectilinear(((30, 34), (32, 32))) + transpose = {"name": "transpose", "configuration": {"order": (1, 0)}} + + def verdict(inner: object) -> list[object]: + return [ + problem.message + for problem in validate_array_metadata_v3( + {**BASE, "chunk_grid": grid, "codecs": (transpose, _shard(inner))} + ) + ] + + assert verdict((16, 2)) == [] # 2 divides both 30 and 34 + assert verdict((16, 15)) == [ # 15 divides 30 but not 34 + "inner chunk extent 15 does not evenly divide the incoming extent 34" + ] + assert verdict((16, 30)) == [ # 30 divides itself but not 34 + "inner chunk extent 30 does not evenly divide the incoming extent 34" + ] + assert verdict((16, 34)) == [ # and the other way round + "inner chunk extent 34 does not evenly divide the incoming extent 30" + ] + + +def test_an_unreadable_axis_declines_while_its_neighbours_are_judged() -> None: + # A grid this package cannot read pins the rank and nothing else, so + # every axis declines; a rank mismatch is still caught. + grid = {"name": "mycorp.hilbert", "configuration": {"order": 3}} + assert ( + validate_array_metadata_v3({**BASE, "chunk_grid": grid, "codecs": (_shard((7, 7)),)}) == () + ) + problems = validate_array_metadata_v3( + {**BASE, "chunk_grid": grid, "codecs": (_shard((7, 7, 7)),)} + ) + assert [problem.message for problem in problems] == [ + "chunk_shape has 3 entries but the incoming array has 2 dimensions" + ] + + +def test_error_index_codecs_are_judged_against_the_index_rank() -> None: + # The index of a 2-D shard is rank 3, so a rank-1 transpose is wrong. + index = ( + {"name": "transpose", "configuration": {"order": (0,)}}, + {"name": "bytes", "configuration": {"endian": "little"}}, + ) + problems = validate_array_metadata_v3( + {**BASE, "chunk_grid": REGULAR, "codecs": (_shard((8, 8), index=index),)} + ) + assert [problem.message for problem in problems] == [ + "order has 1 entries but the incoming array has 3 dimensions" + ] + + +def test_error_a_bad_inner_extent_costs_the_shard() -> None: + # A shard with a zero inner extent is not a shard, so it does not + # exist and nothing inside it is interpreted. The JSON is still there + # on the `Opaque` that replaces it; what is gone is the reading, and + # the one report that matters is the one you must fix first. + inner = { + **_shard((0, 2)), + "configuration": { + **configuration_of(_shard((0, 2))), + "codecs": ({"name": "transpose", "configuration": {"order": (0, 1, 2)}}, "bytes"), + }, + } + problems = validate_array_metadata_v3( + {**BASE, "data_type": "uint16", "chunk_grid": REGULAR, "codecs": (inner,)} + ) + assert [problem.loc for problem in problems] == [ + ("codecs", 0, "configuration", "chunk_shape", 0) + ] + + +# An unmodelled codec is the ordinary case, not an exotic one: every +# `numcodecs.*` filter is one. What it costs must be only what it actually +# obscures. +_UNMODELLED = { + "name": "numcodecs.delta", + "configuration": {"dtype": " None: + # The inner grid is this codec's own `chunk_shape` and the index is a + # uint64 array, whatever reached the codec. Neither waits on upstream. + nested = { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": (16, 16), + "codecs": ({"name": "transpose", "configuration": {"order": (0, 1, 2)}}, "bytes"), + "index_codecs": ("bytes",), + }, + } + messages = [ + problem.message + for problem in validate_array_metadata_v3( + {**BASE, "data_type": "uint16", "chunk_grid": REGULAR, "codecs": (_UNMODELLED, nested)} + ) + ] + assert any("order has 3 entries" in message for message in messages) + assert any("uint64" in message for message in messages) + + +def test_an_unusable_data_type_does_not_hide_the_geometry() -> None: + # `data_type` costs itself and nothing else: the grid is still readable, + # so a reader sees every fault at once instead of one per round trip. + messages = [ + problem.message + for problem in validate_array_metadata_v3( + {**BASE, "data_type": 5, "chunk_grid": REGULAR, "codecs": (_shard((7, 7)),)} + ) + ] + assert any("expected a metadata field" in message for message in messages) + assert sum("does not evenly divide" in message for message in messages) == 2 + + +def test_permuting_by_a_non_permutation_declines_rather_than_raising() -> None: + # `order` is shape-validated only as a tuple of integers, so this must + # not be what decides whether a validator raises IndexError. + grid = ChunkGrid.regular((4, 4)) + for order in ((5, 0), (0, 0), (-1, 0)): + assert grid.permuted(order) == ChunkGrid(2, None) diff --git a/packages/zarr-metadata/tests/rules/test_documents.py b/packages/zarr-metadata/tests/rules/test_documents.py new file mode 100644 index 0000000000..1ef75df6dc --- /dev/null +++ b/packages/zarr-metadata/tests/rules/test_documents.py @@ -0,0 +1,133 @@ +"""Tests for the whole-document validators in `zarr_metadata.rules`.""" + +from __future__ import annotations + +import math +from typing import TYPE_CHECKING + +import pytest + +from zarr_metadata.model import MetadataValidationError +from zarr_metadata.model import ( + is_array_metadata_v3 as model_is_array_metadata_v3, +) +from zarr_metadata.rules import ( + parse_array_metadata_v2, + parse_array_metadata_v3, + validate_array_metadata_v2, + validate_array_metadata_v3, +) + +if TYPE_CHECKING: + from collections.abc import Callable, Mapping + + from zarr_metadata import ZarrV2ArrayMetadataJSON, ZarrV3ArrayMetadataJSON + from zarr_metadata.model import ValidationProblem + + # The validators are uniform in their inputs (any object) and differ + # only in the document type they hand back, which these tests never + # depend on. + Validator = Callable[[object], tuple[ValidationProblem, ...]] + Parser = Callable[[object], Mapping[str, object]] + +V3_ARRAY: ZarrV3ArrayMetadataJSON = { + "zarr_format": 3, + "node_type": "array", + "shape": (4, 4), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2, 2)}}, + "chunk_key_encoding": "default", + "codecs": ("bytes",), +} + +V2_ARRAY: ZarrV2ArrayMetadataJSON = { + "zarr_format": 2, + "shape": (4,), + "chunks": (2,), + "dtype": " None: + parsed = parse(doc) + assert validate(parsed) == () + shape = doc["shape"] + assert isinstance(shape, (list, tuple)) + assert parsed["shape"] == tuple(shape) + + +def test_error_v3_combined_report() -> None: + # One raise carrying problems from both passes: a structural problem + # (bad node_type) and a composition problem (fill_value vs data_type). + with pytest.raises(MetadataValidationError) as info: + parse_array_metadata_v3({**V3_ARRAY, "node_type": "grid", "fill_value": 300}) + kinds = {(p.loc, p.kind) for p in info.value.problems} + assert (("node_type",), "invalid_value") in kinds + assert (("fill_value",), "invalid_value") in kinds + + +def test_error_v3_dimension_names_reported_once() -> None: + # Regression: this fault used to be reported twice — once by the + # structural validator, once by the composition rule. The check now + # has one owner. + problems = validate_array_metadata_v3({**V3_ARRAY, "dimension_names": ("x",)}) + assert [(p.loc, p.kind) for p in problems] == [(("dimension_names",), "invalid_value")] + + +def test_error_v2_chunks_rank() -> None: + problems = validate_array_metadata_v2({**V2_ARRAY, "chunks": (2, 2)}) + assert [(p.loc, p.kind) for p in problems] == [(("chunks",), "invalid_value")] + + +def test_error_v2_parse_raises() -> None: + with pytest.raises(MetadataValidationError, match="same number of dimensions"): + parse_array_metadata_v2({**V2_ARRAY, "chunks": (2, 2)}) + + +def test_composition_invalid_document_still_satisfies_the_typeddict() -> None: + # The model layer's TypeIs narrows a composition-invalid document, + # which is why the rules layer offers no guard of its own: a + # fill_value out of range does not stop the value being an instance + # of ZarrV3ArrayMetadataJSON. + doc = {**V3_ARRAY, "fill_value": 300} + assert model_is_array_metadata_v3(doc) is True + assert validate_array_metadata_v3(doc) != () diff --git a/packages/zarr-metadata/tests/rules/test_rule_properties.py b/packages/zarr-metadata/tests/rules/test_rule_properties.py new file mode 100644 index 0000000000..e89dc80644 --- /dev/null +++ b/packages/zarr-metadata/tests/rules/test_rule_properties.py @@ -0,0 +1,221 @@ +"""Property tests for composition-rule boundaries and API agreement.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +import pytest +from hypothesis import given +from hypothesis import strategies as st + +from zarr_metadata.model import MetadataValidationError +from zarr_metadata.rules import ( + parse_array_metadata_v3, + validate_array_metadata_v2, + validate_array_metadata_v3, + validate_group_metadata_v2, + validate_group_metadata_v3, +) + +if TYPE_CHECKING: + from collections.abc import Callable, Mapping + + from zarr_metadata.model import ValidationProblem + + +JSON_SCALARS = st.none() | st.booleans() | st.integers() | st.floats(allow_nan=False) | st.text() +JSON_VALUES = st.recursive( + JSON_SCALARS, + lambda children: ( + st.lists(children, max_size=4) | st.dictionaries(st.text(max_size=12), children, max_size=4) + ), + max_leaves=20, +) + + +DOCUMENT_VALIDATORS = ( + validate_array_metadata_v2, + validate_array_metadata_v3, + validate_group_metadata_v2, + validate_group_metadata_v3, +) + + +@pytest.mark.parametrize("validator", DOCUMENT_VALIDATORS, ids=lambda validator: validator.__name__) +@given(JSON_VALUES) +def test_document_validators_are_total_for_arbitrary_json( + validator: Callable[[object], tuple[ValidationProblem, ...]], value: object +) -> None: + """Untrusted JSON always produces a verdict; it never crashes the validator.""" + assert isinstance(validator(value), tuple) + + +@given( + extent=st.integers(min_value=0, max_value=200), + chunk_shapes=st.lists(st.integers(min_value=1, max_value=50), min_size=1, max_size=8), +) +def test_rectilinear_explicit_chunks_may_overflow_extent( + extent: int, chunk_shapes: list[int] +) -> None: + """The final explicit chunk may extend past the array boundary.""" + if sum(chunk_shapes) < extent: + return + document = { + "zarr_format": 3, + "node_type": "array", + "shape": (extent,), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": (tuple(chunk_shapes),)}, + }, + "chunk_key_encoding": "default", + "codecs": ("bytes",), + } + assert validate_array_metadata_v3(document) == () + + +@given(depth=st.integers(min_value=1, max_value=5), leaf_fill=st.integers(0, 255)) +def test_nested_struct_fill_values_are_checked_recursively(depth: int, leaf_fill: int) -> None: + data_type: object = "uint8" + fill_value: object = leaf_fill + path: list[str] = [] + for level in range(depth): + name = f"level_{level}" + data_type = { + "name": "struct", + "configuration": {"fields": ({"name": name, "data_type": data_type},)}, + } + fill_value = {name: fill_value} + path.insert(0, name) + + document: dict[str, object] = { + "zarr_format": 3, + "node_type": "array", + "shape": (1,), + "data_type": data_type, + "fill_value": fill_value, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (1,)}}, + "chunk_key_encoding": "default", + "codecs": ("bytes",), + } + assert validate_array_metadata_v3(document) == () + + invalid: object = 256 + for name in reversed(path): + invalid = {name: invalid} + document["fill_value"] = invalid + problems = validate_array_metadata_v3(document) + assert any(problem.loc == ("fill_value", *path) for problem in problems) + + +@given( + exponents=st.lists(st.integers(min_value=0, max_value=6), min_size=1, max_size=4, unique=True) +) +def test_nested_sharding_pipelines_accept_divisible_inner_chunks(exponents: list[int]) -> None: + """Every generated nesting level is checked against the level enclosing it.""" + inner_shapes = [2**exponent for exponent in sorted(exponents, reverse=True)] + codecs: tuple[object, ...] = ("bytes",) + for inner_shape in reversed(inner_shapes): + codecs = ( + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": (inner_shape,), + "codecs": codecs, + "index_codecs": ( + {"name": "bytes", "configuration": {"endian": "little"}}, + "crc32c", + ), + }, + }, + ) + document = { + "zarr_format": 3, + "node_type": "array", + "shape": (64,), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (64,)}}, + "chunk_key_encoding": "default", + "codecs": codecs, + } + assert validate_array_metadata_v3(document) == () + + +@given( + data_type=st.sampled_from(("int8", "uint8", "int16", "uint16", "int32", "uint32")), + fill_value=st.integers(min_value=-(2**40), max_value=2**40), +) +def test_validator_and_parser_agree(data_type: str, fill_value: int) -> None: + document: Mapping[str, object] = { + "zarr_format": 3, + "node_type": "array", + "shape": (4,), + "data_type": data_type, + "fill_value": fill_value, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2,)}}, + "chunk_key_encoding": "default", + "codecs": ({"name": "bytes", "configuration": {"endian": "little"}},), + } + validator_accepts = validate_array_metadata_v3(document) == () + + try: + parse_array_metadata_v3(document) + except MetadataValidationError: + parser_accepts = False + else: + parser_accepts = True + + assert validator_accepts == parser_accepts + + +@pytest.mark.parametrize("position", ["data_type", "cast_value"]) +@pytest.mark.parametrize("name", ["numpy.datetime64", "numpy.timedelta64"]) +@given( + depth=st.integers(min_value=0, max_value=4), + scale=st.integers(min_value=-1, max_value=2**31), +) +def test_nested_time_data_types_obey_scale_factor_constraints( + position: str, name: str, depth: int, scale: int +) -> None: + """Embedding a known dtype must not bypass its composition rules.""" + data_type: object = { + "name": name, + "configuration": {"unit": "s", "scale_factor": scale}, + } + fill_value: object = 0 + nested_loc: tuple[str | int, ...] = ("configuration", "scale_factor") + for _ in range(depth): + data_type = { + "name": "struct", + "configuration": {"fields": ({"name": "value", "data_type": data_type},)}, + } + fill_value = {"value": fill_value} + nested_loc = ("configuration", "fields", 0, "data_type", *nested_loc) + document: dict[str, object] = { + "zarr_format": 3, + "node_type": "array", + "shape": (1,), + "data_type": data_type, + "fill_value": fill_value, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (1,)}}, + "chunk_key_encoding": "default", + "codecs": ({"name": "bytes", "configuration": {"endian": "little"}},), + } + if position == "cast_value": + document["data_type"] = "uint8" + document["fill_value"] = 0 + document["codecs"] = ( + {"name": "cast_value", "configuration": {"data_type": data_type}}, + {"name": "bytes", "configuration": {"endian": "little"}}, + ) + loc = ("codecs", 0, "configuration", "data_type", *nested_loc) + else: + loc = ("data_type", *nested_loc) + problems = validate_array_metadata_v3(document) + if 1 <= scale < 2**31: + assert problems == () + else: + assert any(problem.loc == loc for problem in problems) diff --git a/packages/zarr-metadata/tests/rules/test_v3_array_rules.py b/packages/zarr-metadata/tests/rules/test_v3_array_rules.py new file mode 100644 index 0000000000..1c302f2629 --- /dev/null +++ b/packages/zarr-metadata/tests/rules/test_v3_array_rules.py @@ -0,0 +1,905 @@ +"""Tests for the v3 array and group composition rules added with the +rules-layer promotion: chunk grid values/geometry, transpose orders, +sharding pipelines/geometry, and consolidated-entry recursion.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any, cast + +import pytest + +from zarr_metadata.model import MetadataValidationError +from zarr_metadata.rules import validate_array_metadata_v3, validate_group_metadata_v3 + +if TYPE_CHECKING: + from collections.abc import Mapping + + from zarr_metadata import ZarrV3ArrayMetadataJSON + +BASE: ZarrV3ArrayMetadataJSON = { + "zarr_format": 3, + "node_type": "array", + "shape": (4, 4), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2, 2)}}, + "chunk_key_encoding": "default", + "codecs": ("bytes",), +} + + +# The shard index is a uint64 array, so its bytes codec needs an endianness. +_INDEX_BYTES: Mapping[str, object] = {"name": "bytes", "configuration": {"endian": "little"}} + + +def _shard(**overrides: object) -> Mapping[str, object]: + """A sharding codec entry; overrides may be deliberately malformed.""" + configuration: dict[str, object] = { + "chunk_shape": (2, 2), + "codecs": ("bytes",), + "index_codecs": (_INDEX_BYTES, "crc32c"), + } + configuration.update(overrides) + return {"name": "sharding_indexed", "configuration": configuration} + + +# Documents that must be fully valid: the rules judge geometry and values +# without rejecting legitimate spellings of the same constructs. +VALID_CASES: dict[str, Mapping[str, object]] = { + "regular": BASE, + "rectilinear-explicit-sums": { + **BASE, + "chunk_grid": { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": ((2, 2), (1, 3))}, + }, + }, + "rectilinear-explicit-overflow": { + **BASE, + "shape": (6,), + "chunk_grid": { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": ((4, 4, 4),)}, + }, + }, + "rectilinear-rle-and-uniform": { + **BASE, + "chunk_grid": { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": (((2, 2),), 4)}, + }, + }, + "transpose": { + **BASE, + "codecs": ({"name": "transpose", "configuration": {"order": (1, 0)}}, "bytes"), + }, + "sharding": {**BASE, "codecs": (_shard(),)}, + "nested-sharding": { + **BASE, + "codecs": ( + _shard( + codecs=( + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": (1, 2), + "codecs": ("bytes",), + "index_codecs": (_INDEX_BYTES,), + }, + }, + ) + ), + ), + }, + "nested-struct": { + **BASE, + "data_type": { + "name": "struct", + "configuration": { + "fields": ( + {"name": "id", "data_type": "uint8"}, + { + "name": "point", + "data_type": { + "name": "struct", + "configuration": {"fields": ({"name": "x", "data_type": "int16"},)}, + }, + }, + ) + }, + }, + "fill_value": {"id": 1, "point": {"x": -2}}, + "codecs": ({"name": "bytes", "configuration": {"endian": "little"}},), + }, + "unknown-grid-passes": { + **BASE, + "chunk_grid": {"name": "hilbert", "configuration": {"level": 3}}, + }, + "unknown-codec-inconclusive": {**BASE, "codecs": ({"name": "zfpy"}, "bytes")}, + "scale-offset-scalars-of-the-array-type": { + **BASE, + "codecs": ({"name": "scale_offset", "configuration": {"offset": 1, "scale": 2}}, "bytes"), + }, + "scale-offset-scalars-of-the-type-cast-to": { + **BASE, + "data_type": "float32", + "fill_value": "NaN", + "codecs": ( + {"name": "cast_value", "configuration": {"data_type": "uint8"}}, + {"name": "scale_offset", "configuration": {"offset": 10}}, + "bytes", + ), + }, +} + + +@pytest.mark.parametrize("doc", VALID_CASES.values(), ids=list(VALID_CASES)) +def test_valid_documents(doc: Mapping[str, object]) -> None: + assert validate_array_metadata_v3(doc) == () + + +def _sole_problem(doc: Mapping[str, object]) -> tuple[tuple[str | int, ...], str]: + problems = validate_array_metadata_v3(doc) + assert len(problems) == 1, [p.message for p in problems] + return problems[0].loc, problems[0].message + + +def test_error_regular_chunk_extent_zero() -> None: + loc, message = _sole_problem( + {**BASE, "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (0, 2)}}} + ) + assert loc == ("chunk_grid", "configuration", "chunk_shape", 0) + assert "expected an integer >= 1" in message + + +def test_error_rectilinear_rank_mismatch() -> None: + loc, message = _sole_problem( + { + **BASE, + "chunk_grid": { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": ((2, 2),)}, + }, + } + ) + assert loc == ("chunk_grid", "configuration", "chunk_shapes") + assert "2 dimensions" in message + + +def test_error_rectilinear_sum_mismatch() -> None: + loc, message = _sole_problem( + { + **BASE, + "chunk_grid": { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": ((3,), (2, 2))}, + }, + } + ) + assert loc == ("chunk_grid", "configuration", "chunk_shapes", 0) + assert "sum to 3" in message + + +def test_error_rectilinear_nonpositive_rle() -> None: + problems = validate_array_metadata_v3( + { + **BASE, + "chunk_grid": { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": (((0, 4),), 4)}, + }, + } + ) + assert any("expected an integer >= 1" in p.message for p in problems) + + +def test_error_transpose_not_a_permutation() -> None: + loc, message = _sole_problem( + {**BASE, "codecs": ({"name": "transpose", "configuration": {"order": (5, 5)}}, "bytes")} + ) + assert loc == ("codecs", 0, "configuration", "order") + assert "permutation" in message + + +def test_error_transpose_rank_mismatch() -> None: + loc, message = _sole_problem( + {**BASE, "codecs": ({"name": "transpose", "configuration": {"order": (2, 0, 1)}}, "bytes")} + ) + assert loc == ("codecs", 0, "configuration", "order") + assert "incoming array has 2 dimensions" in message + + +def test_error_sharding_inner_pipeline_order() -> None: + loc, _ = _sole_problem({**BASE, "codecs": (_shard(codecs=("crc32c", "bytes")),)}) + assert loc == ("codecs", 0, "configuration", "codecs", 1) + + +def test_error_sharding_inner_no_array_bytes() -> None: + loc, message = _sole_problem({**BASE, "codecs": (_shard(codecs=("crc32c",)),)}) + assert loc == ("codecs", 0, "configuration", "codecs") + assert "no array->bytes codec" in message + + +def test_error_sharding_index_codecs_no_array_bytes() -> None: + loc, message = _sole_problem({**BASE, "codecs": (_shard(index_codecs=("crc32c",)),)}) + assert loc == ("codecs", 0, "configuration", "index_codecs") + assert "no array->bytes codec" in message + + +def test_error_sharding_index_codecs_are_variable_sized() -> None: + loc, message = _sole_problem( + { + **BASE, + "codecs": ( + _shard( + index_codecs=( + _INDEX_BYTES, + {"name": "gzip", "configuration": {"level": 1}}, + ) + ), + ), + } + ) + assert loc == ("codecs", 0, "configuration", "index_codecs", 1) + assert "fixed-size" in message + + +def test_error_sharding_rank_mismatch() -> None: + loc, message = _sole_problem({**BASE, "codecs": (_shard(chunk_shape=(2,)),)}) + assert loc == ("codecs", 0, "configuration", "chunk_shape") + assert "incoming array has 2 dimensions" in message + + +def test_error_sharding_not_divisible() -> None: + loc, message = _sole_problem( + { + **BASE, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (4, 4)}}, + "codecs": (_shard(chunk_shape=(3, 2)),), + } + ) + assert loc == ("codecs", 0, "configuration", "chunk_shape", 0) + assert "does not evenly divide" in message + + +def test_error_nested_sharding_not_divisible() -> None: + inner: Mapping[str, object] = { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": (2, 3), + "codecs": ("bytes",), + "index_codecs": (_INDEX_BYTES,), + }, + } + loc, message = _sole_problem( + { + **BASE, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (4, 4)}}, + "codecs": (_shard(codecs=(inner,)),), + } + ) + assert loc == ("codecs", 0, "configuration", "codecs", 0, "configuration", "chunk_shape", 1) + assert "does not evenly divide" in message + + +def test_error_sharding_inner_chunk_extent_zero() -> None: + problems = validate_array_metadata_v3({**BASE, "codecs": (_shard(chunk_shape=(0, 2)),)}) + assert any( + p.loc == ("codecs", 0, "configuration", "chunk_shape", 0) + and "expected an integer >= 1" in p.message + for p in problems + ) + + +def test_error_bytes_requires_endian_for_multibyte_data() -> None: + loc, message = _sole_problem({**BASE, "data_type": "int32", "codecs": ("bytes",)}) + assert loc == ("codecs", 0, "configuration", "endian") + assert "required" in message + + +def test_error_bytes_rejects_variable_length_data_type() -> None: + # The problem is about the codec, not about a member of its + # configuration — and this codec is spelled as a bare string, so it has + # no `configuration` node for a loc to point into. + loc, message = _sole_problem( + {**BASE, "data_type": "string", "fill_value": "", "codecs": ("bytes",)} + ) + assert loc == ("codecs", 0) + assert "not compatible" in message + + +def test_error_struct_fields_are_empty() -> None: + doc = { + **BASE, + "data_type": {"name": "struct", "configuration": {"fields": ()}}, + "fill_value": {}, + } + loc, message = _sole_problem(doc) + assert loc == ("data_type", "configuration", "fields") + assert "at least one" in message + + +def test_error_struct_field_is_variable_length() -> None: + doc = { + **BASE, + "data_type": { + "name": "struct", + "configuration": {"fields": ({"name": "label", "data_type": "string"},)}, + }, + "fill_value": {"label": ""}, + } + problems = validate_array_metadata_v3(doc) + assert any( + problem.loc == ("data_type", "configuration", "fields", 0, "data_type") + and "fixed-size" in problem.message + for problem in problems + ) + + +def test_error_struct_fill_is_missing_field() -> None: + doc = { + **BASE, + "data_type": { + "name": "struct", + "configuration": {"fields": ({"name": "x", "data_type": "uint8"},)}, + }, + "fill_value": {}, + } + loc, message = _sole_problem(doc) + assert loc == ("fill_value", "x") + assert "missing" in message + + +def test_error_struct_fill_field_is_invalid() -> None: + doc = { + **BASE, + "data_type": { + "name": "struct", + "configuration": {"fields": ({"name": "x", "data_type": "uint8"},)}, + }, + "fill_value": {"x": 300}, + } + loc, message = _sole_problem(doc) + assert loc == ("fill_value", "x") + assert "[0, 255]" in message + + +def test_error_gzip_level_is_out_of_range() -> None: + doc = { + **BASE, + "codecs": ("bytes", {"name": "gzip", "configuration": {"level": 99}}), + } + loc, message = _sole_problem(doc) + assert loc == ("codecs", 1, "configuration", "level") + assert "[0, 9]" in message + + +@pytest.mark.parametrize("data_type_name", ["numpy.datetime64", "numpy.timedelta64"]) +def test_error_numpy_time_scale_factor_is_out_of_range(data_type_name: str) -> None: + doc = { + **BASE, + "data_type": { + "name": data_type_name, + "configuration": {"unit": "ns", "scale_factor": 0}, + }, + "codecs": ({"name": "bytes", "configuration": {"endian": "little"}},), + } + loc, message = _sole_problem(doc) + assert loc == ("data_type", "configuration", "scale_factor") + assert "[1, 2147483647]" in message + + +@pytest.mark.parametrize("data_type_name", ["numpy.datetime64", "numpy.timedelta64"]) +def test_error_numpy_time_fill_is_out_of_range(data_type_name: str) -> None: + doc = { + **BASE, + "data_type": { + "name": data_type_name, + "configuration": {"unit": "ns", "scale_factor": 1}, + }, + "fill_value": 2**80, + "codecs": ({"name": "bytes", "configuration": {"endian": "little"}},), + } + loc, message = _sole_problem(doc) + assert loc == ("fill_value",) + assert "64-bit" in message + + +def test_error_consolidated_child_violates_array_rules() -> None: + doc: Mapping[str, object] = { + "zarr_format": 3, + "node_type": "group", + "consolidated_metadata": { + "kind": "inline", + "must_understand": False, + "metadata": {"a": {**BASE, "fill_value": 300}}, + }, + } + problems = validate_group_metadata_v3(doc) + assert [(p.loc, p.kind) for p in problems] == [ + (("consolidated_metadata", "metadata", "a", "fill_value"), "invalid_value") + ] + + +def test_error_consolidated_nested_group_recursion() -> None: + child_group: Mapping[str, object] = { + "zarr_format": 3, + "node_type": "group", + "consolidated_metadata": { + "kind": "inline", + "must_understand": False, + "metadata": {"b": {**BASE, "fill_value": 300}}, + }, + } + doc: Mapping[str, object] = { + "zarr_format": 3, + "node_type": "group", + "consolidated_metadata": { + "kind": "inline", + "must_understand": False, + "metadata": {"g": child_group}, + }, + } + problems = validate_group_metadata_v3(doc) + assert [p.loc for p in problems] == [ + ( + "consolidated_metadata", + "metadata", + "g", + "consolidated_metadata", + "metadata", + "b", + "fill_value", + ) + ] + + +def test_error_group_parse_raises() -> None: + from zarr_metadata.rules import parse_group_metadata_v3 + + with pytest.raises( + MetadataValidationError, match=r"consolidated_metadata\.metadata\.a\.fill_value" + ): + parse_group_metadata_v3( + { + "zarr_format": 3, + "node_type": "group", + "consolidated_metadata": { + "kind": "inline", + "must_understand": False, + "metadata": {"a": {**BASE, "fill_value": 300}}, + }, + } + ) + + +# -- unknown configuration members -------------------------------------------- +# +# The v3 spec does not say whether an extension's `configuration` is closed +# (zarr-developers/zarr-specs#270, open since 2023). This package takes the +# strict reading, matching most registered extension schemas and most other +# implementations — but reports it as its own `unknown_key` kind, and never +# lets it mask a real finding about the same entity. + + +def test_unknown_configuration_member_has_its_own_kind() -> None: + doc = { + **BASE, + "codecs": ({"name": "bytes", "configuration": {"endian": "little", "hint": 1}},), + } + problems = validate_array_metadata_v3(doc) + # The location names the offending key, so a consumer can route the + # problem without parsing the message. + assert [(p.loc, p.kind) for p in problems] == [ + (("codecs", 0, "configuration", "hint"), "unknown_key") + ] + + +def test_error_known_data_type_has_invalid_configuration() -> None: + doc = { + **BASE, + "data_type": { + "name": "numpy.datetime64", + "configuration": {"unit": "banana", "scale_factor": 1}, + }, + } + problems = validate_array_metadata_v3(doc) + assert [(p.loc, p.kind) for p in problems] == [ + (("data_type", "configuration", "unit"), "invalid_value") + ] + + +def test_error_known_chunk_key_encoding_has_invalid_configuration() -> None: + doc = { + **BASE, + "chunk_key_encoding": {"name": "default", "configuration": {"separator": "!"}}, + } + problems = validate_array_metadata_v3(doc) + assert [(p.loc, p.kind) for p in problems] == [ + (("chunk_key_encoding", "configuration", "separator"), "invalid_value") + ] + + +def test_unknown_member_does_not_mask_a_codec_rule() -> None: + # Regression: an unrecognized member used to make the whole entity + # uninterpretable, silently suppressing every other rule about it — so a + # cosmetic extra key hid a genuine permutation error. + doc = { + **BASE, + "codecs": ({"name": "transpose", "configuration": {"order": (5, 5), "hint": 1}}, "bytes"), + } + kinds = {(p.loc, p.kind) for p in validate_array_metadata_v3(doc)} + assert (("codecs", 0, "configuration", "hint"), "unknown_key") in kinds + assert (("codecs", 0, "configuration", "order"), "invalid_value") in kinds + + +def test_unknown_member_does_not_mask_a_chunk_grid_rule() -> None: + doc = { + **BASE, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2,), "hint": 1}}, + } + kinds = {(p.loc, p.kind) for p in validate_array_metadata_v3(doc)} + assert (("chunk_grid", "configuration", "hint"), "unknown_key") in kinds + assert (("chunk_grid", "configuration", "chunk_shape"), "invalid_value") in kinds + + +def test_unknown_member_survives_a_round_trip() -> None: + # Whatever the strict validator says, the package must never silently + # drop a member it does not model: a writer that knows more than we do + # must get its bytes back. (zarr-python's own chunk-grid path is lossy + # here; this asserts we are not.) + import json + + from zarr_metadata.model import ZarrV3ArrayMetadata + + raw = { + **BASE, + "shape": [4, 4], + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": [2, 2]}}, + "codecs": [ + { + "name": "blosc", + "configuration": { + "cname": "zstd", + "clevel": 5, + "shuffle": "shuffle", + "blocksize": 0, + "numThreads": 4, + }, + }, + "bytes", + ], + } + model = ZarrV3ArrayMetadata.from_json(json.loads(json.dumps(raw))) + emitted = model.to_json() + codec = cast("Mapping[str, Any]", emitted["codecs"][0]) + assert codec["configuration"]["numThreads"] == 4 + + +# -- gaps found by adversarial review ---------------------------------------- + + +def test_error_scale_offset_does_not_stand_down_later_rules() -> None: + # A no-op array->array codec used to stop spec propagation, silently + # switching off every rule after it. + loc, message = _sole_problem( + { + **BASE, + "data_type": "uint16", + "codecs": ( + {"name": "scale_offset", "configuration": {"offset": 0, "scale": 1}}, + "bytes", + ), + } + ) + assert loc == ("codecs", 1, "configuration", "endian") + assert "uint16" in message + + +def test_error_rank_is_judged_under_a_chunk_grid_with_unknown_extents() -> None: + # A rectilinear grid gives no single chunk shape, but every chunk still + # has the array's rank, so a rank-3 transpose over a 1-D array is a fault. + loc, message = _sole_problem( + { + **BASE, + "chunk_grid": { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": ((2, 2), (2, 2))}, + }, + "codecs": ({"name": "transpose", "configuration": {"order": (0, 1, 2)}}, "bytes"), + } + ) + assert loc == ("codecs", 0, "configuration", "order") + assert "2 dimensions" in message + + +def test_error_an_unusable_member_costs_the_entity() -> None: + # An entity exists only if its members are readable and its values + # allowed, so a bad `index_location` means there is no shard to ask + # about its pipelines. The JSON survives on the `Opaque` that stands + # in for it; what is gone is the interpretation, and the report that + # remains is the one that has to be fixed first. + problems = validate_array_metadata_v3( + { + **BASE, + "codecs": ( + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": (2, 2), + "codecs": ("bytes", "bytes"), + "index_codecs": ({"name": "bytes", "configuration": {"endian": "little"}},), + "index_location": "middle", + }, + }, + ), + } + ) + assert {problem.loc for problem in problems} == { + ("codecs", 0, "configuration", "index_location"), + } + + +def test_error_a_bare_entity_reports_at_the_entity_not_a_missing_node() -> None: + # "bytes" has no `configuration` node, so a problem about the codec as a + # whole must not point into one. + loc, _ = _sole_problem({**BASE, "data_type": "string", "fill_value": "", "codecs": ("bytes",)}) + assert loc == ("codecs", 0) + + +def test_error_endian_message_names_the_shard_index_type() -> None: + # Inside index_codecs the array is the shard index, whose uint64 type + # appears nowhere in the document; the message has to say so. + loc, message = _sole_problem( + { + **BASE, + "codecs": ( + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": (2, 2), + "codecs": ({"name": "bytes", "configuration": {"endian": "little"}},), + "index_codecs": ("bytes",), + }, + }, + ), + } + ) + assert loc == ("codecs", 0, "configuration", "index_codecs", 0, "configuration", "endian") + assert "uint64" in message + + +def test_error_a_malformed_must_understand_does_not_suppress_the_entity() -> None: + # `must_understand` is part of the envelope, not the configuration, so + # a bad one says nothing about whether the configuration is readable. + problems = validate_array_metadata_v3( + { + **BASE, + "codecs": ( + { + "name": "transpose", + "configuration": {"order": (2, 1, 0)}, + "must_understand": "yes", + }, + "bytes", + ), + } + ) + assert ("codecs", 0, "configuration", "order") in {problem.loc for problem in problems} + + +# -- blosc: the one member whose requiredness the spec makes conditional ------ + + +def _blosc_configuration(**overrides: object) -> dict[str, object]: + return { + "cname": "zstd", + "clevel": 5, + "shuffle": "shuffle", + "typesize": 2, + "blocksize": 0, + **overrides, + } + + +def _blosc(**overrides: object) -> Mapping[str, object]: + return {"name": "blosc", "configuration": _blosc_configuration(**overrides)} + + +def _with_blosc(**overrides: object) -> Mapping[str, object]: + return {**BASE, "codecs": ("bytes", _blosc(**overrides))} + + +def test_blosc_accepts_a_document_zarr_python_writes() -> None: + assert validate_array_metadata_v3(_with_blosc()) == () + + +def test_blosc_typesize_may_be_omitted_only_without_shuffling() -> None: + # "Required unless `shuffle` is `"noshuffle"`, in which case the value + # is ignored." A TypedDict cannot say that, so the rule does. + configuration = _blosc_configuration() + del configuration["typesize"] + for shuffle, required in (("noshuffle", False), ("shuffle", True), ("bitshuffle", True)): + codec = {"name": "blosc", "configuration": {**configuration, "shuffle": shuffle}} + problems = validate_array_metadata_v3({**BASE, "codecs": ("bytes", codec)}) + assert [problem.loc[-1] for problem in problems] == (["typesize"] if required else []) + + +def test_error_blosc_typesize_is_not_positive() -> None: + loc, message = _sole_problem(_with_blosc(typesize=0)) + assert loc == ("codecs", 1, "configuration", "typesize") + assert "positive" in message + + +def test_error_blosc_clevel_out_of_range() -> None: + loc, message = _sole_problem(_with_blosc(clevel=99)) + assert loc == ("codecs", 1, "configuration", "clevel") + assert "[0, 9]" in message + + +def test_error_blosc_blocksize_is_negative() -> None: + loc, message = _sole_problem(_with_blosc(blocksize=-1)) + assert loc == ("codecs", 1, "configuration", "blocksize") + assert "expected an integer >= 0" in message + + +@pytest.mark.parametrize( + ("level", "valid"), + [(0, True), (22, True), (-131072, True), (23, False), (-131073, False), (1000, False)], +) +def test_zstd_level_range(level: int, valid: bool) -> None: + # "An integer from -131072 to 22"; 0 selects the default level. + document = { + **BASE, + "codecs": ("bytes", {"name": "zstd", "configuration": {"level": level, "checksum": False}}), + } + problems = validate_array_metadata_v3(document) + if valid: + assert problems == () + else: + assert [problem.loc for problem in problems] == [("codecs", 1, "configuration", "level")] + + +# -- must_understand: false, wherever a document writes it -------------------- + +# (a document nesting an ignorable extension point, and where it sits) +IGNORABLE_NESTED: dict[str, tuple[Mapping[str, object], tuple[object, ...]]] = { + "inner-codec-of-a-shard": ( + { + **BASE, + "codecs": (_shard(codecs=({"name": "bytes", "must_understand": False},)),), + }, + ("codecs", 0, "configuration", "codecs", 0, "must_understand"), + ), + "index-codec-of-a-shard": ( + { + **BASE, + "codecs": (_shard(index_codecs=({**_INDEX_BYTES, "must_understand": False},)),), + }, + ("codecs", 0, "configuration", "index_codecs", 0, "must_understand"), + ), + "data-type-of-a-struct-field": ( + { + **BASE, + "data_type": { + "name": "struct", + "configuration": { + "fields": ( + { + "name": "a", + "data_type": {"name": "uint8", "must_understand": False}, + }, + ) + }, + }, + "fill_value": {"a": 0}, + }, + ("data_type", "configuration", "fields", 0, "data_type", "must_understand"), + ), +} + + +@pytest.mark.parametrize("case", IGNORABLE_NESTED) +def test_error_a_nested_extension_point_may_not_be_declared_ignorable(case: str) -> None: + # An extension point is something a reader must understand, and depth + # does not change that: a codec inside a shard is still a codec. The + # top-level check would otherwise be a check on where the flag was + # written rather than on what it says. + document, loc = IGNORABLE_NESTED[case] + problems = validate_array_metadata_v3(cast("Any", document)) + assert loc in {problem.loc for problem in problems} + + +def test_error_a_pipeline_that_is_not_an_array_is_not_judged_as_empty() -> None: + # Nothing was read, so there is no chain to find an array->bytes + # codec missing from: the one problem is the shape of the field. + problems = validate_array_metadata_v3(cast("Any", {**BASE, "codecs": "bytes"})) + assert [(problem.loc, problem.kind) for problem in problems] == [(("codecs",), "invalid_type")] + + +@pytest.mark.parametrize( + ("codecs", "loc"), + [ + ( + ({"name": "scale_offset", "configuration": {"scale": "0"}}, "bytes"), + ("codecs", 0, "configuration", "scale"), + ), + ( + ({"name": "scale_offset", "configuration": {"offset": -1}}, "bytes"), + ("codecs", 0, "configuration", "offset"), + ), + ( + ( + {"name": "cast_value", "configuration": {"data_type": "uint8"}}, + {"name": "scale_offset", "configuration": {"scale": 0.5}}, + "bytes", + ), + ("codecs", 1, "configuration", "scale"), + ), + ], + ids=["string-for-uint8", "out-of-range-for-uint8", "float-for-the-type-cast-to"], +) +def test_error_scale_offset_scalar_is_no_fill_value_of_the_type_it_receives( + codecs: tuple[object, ...], loc: tuple[str | int, ...] +) -> None: + # Each scalar is "encoded to JSON using the Zarr V3 fill value encoding + # for the input array's data type" -- the type that reaches the codec. + problems = validate_array_metadata_v3({**BASE, "codecs": codecs}) + assert [(p.loc, p.kind) for p in problems] == [(loc, "invalid_value")] + + +def test_error_scale_offset_data_type_has_no_arithmetic() -> None: + doc = {**BASE, "data_type": "bool", "fill_value": False, "codecs": ("scale_offset", "bytes")} + assert _sole_problem(doc) == ( + ("codecs", 0), + "scale_offset is defined for integer and floating-point data types, not 'bool'", + ) + + +# (a cast target, whether `out_of_range: "wrap"` is defined for it) +WRAP_TARGETS: dict[str, tuple[object, bool]] = { + "int32": ("int32", True), + "uint64": ("uint64", True), + "bool": ("bool", False), + "float32": ("float32", False), + "complex64": ("complex64", False), + "raw-bytes": ("r8", False), + "string": ("string", False), + "numpy-time": ( + {"name": "numpy.datetime64", "configuration": {"unit": "s", "scale_factor": 1}}, + False, + ), + # Out of scope, so it may be an integral extension type; declining + # beats guessing. + "unmodelled": ("mycorp.bigint", True), +} + + +@pytest.mark.parametrize(("target", "allowed"), WRAP_TARGETS.values(), ids=list(WRAP_TARGETS)) +def test_wrap_requires_a_twos_complement_integer_target(target: object, allowed: bool) -> None: + document = { + **BASE, + "codecs": ( + {"name": "cast_value", "configuration": {"data_type": target, "out_of_range": "wrap"}}, + {"name": "bytes", "configuration": {"endian": "little"}}, + ), + } + wrap = [ + problem + for problem in validate_array_metadata_v3(document) + if problem.loc == ("codecs", 0, "configuration", "out_of_range") + ] + assert (len(wrap) == 0) is allowed, wrap + + +def test_the_wrap_message_names_the_spelling_not_the_family() -> None: + # `r` is an invented lookup key, not a name any document writes. + document = { + **BASE, + "codecs": ( + {"name": "cast_value", "configuration": {"data_type": "r24", "out_of_range": "wrap"}}, + {"name": "bytes", "configuration": {"endian": "little"}}, + ), + } + messages = [problem.message for problem in validate_array_metadata_v3(document)] + assert any("got 'r24'" in message for message in messages), messages diff --git a/packages/zarr-metadata/tests/test_public_api.py b/packages/zarr-metadata/tests/test_public_api.py index 6613aa394b..a5b7b2d9cf 100644 --- a/packages/zarr-metadata/tests/test_public_api.py +++ b/packages/zarr-metadata/tests/test_public_api.py @@ -267,6 +267,18 @@ def test_all_is_grouped_and_unique() -> None: "FillValue", "Configuration", "Component", + # The configuration record an entity is built from, `BloscOptions`: + # the dataclass whose fields are the members and whose `problems` + # holds the rules. `Configuration` is taken by the TypedDict. + "Options", + # The bare role is the coerced entity itself: the dataclass that owns + # the extension's type checks, value checks and canonical spelling. + # Listed last so a longer role still wins the alternation. + "Codec", + "Entity", + "ChunkGrid", + "ChunkKeyEncoding", + "DataType", ) _EXTENSION_NAME = re.compile(r"^(?:[A-Z][a-z0-9]*)+?(?:" + "|".join(_EXTENSION_ROLES) + r")$") @@ -279,14 +291,29 @@ def test_all_is_grouped_and_unique() -> None: "Base64Bytes", "BloscCName", "BloscShuffle", + "Canonical", "CastOutOfRangeMode", "CastRoundingMode", + "StorageClass", + "Loc", + "Extents", + "Context", + "Coerced", + "Configuration", + "ChunkGrid", + "ArrayParts", + "ArrayDocumentV3", + "RefinedArrayV3", + "Pipeline", + "PipelineStage", "Endianness", + "Invalid", "HexFloat16", "HexFloat32", "HexFloat64", "JSONValue", "MetadataValidationError", + "Opaque", "NumpyDatetime64", "NumpyTimeUnit", "NumpyTimedelta64", @@ -409,7 +436,14 @@ def _literal_backed_constants() -> list[tuple[str, str, str]]: for const_name, value in vars(module).items(): if const_name.startswith("_") or not const_name.isupper(): continue - members = frozenset(value) if isinstance(value, tuple) else frozenset({value}) + if isinstance(value, tuple): + members = frozenset(value) + elif isinstance(value, str): + members = frozenset({value}) + else: + # Unhashable non-value constants (rule sets, factory + # registries) back no Literal type and carry no signal. + continue if not all(isinstance(m, str) for m in members): continue matches = [t for t, args in literals.items() if args == members] @@ -441,7 +475,14 @@ def _value_tied_constants() -> set[str]: for const_name, value in vars(module).items(): if const_name.startswith("_") or not const_name.isupper(): continue - members = frozenset(value) if isinstance(value, tuple) else frozenset({value}) + if isinstance(value, tuple): + members = frozenset(value) + elif isinstance(value, str): + members = frozenset({value}) + else: + # Unhashable non-value constants (rule sets, factory + # registries) back no Literal type and carry no signal. + continue if not all(isinstance(m, str) for m in members): continue if sum(1 for args in literals if args == members) > 1: diff --git a/packages/zarr-metadata/tests/v3/test_acme_affine.py b/packages/zarr-metadata/tests/v3/test_acme_affine.py new file mode 100644 index 0000000000..bd52395e0e --- /dev/null +++ b/packages/zarr-metadata/tests/v3/test_acme_affine.py @@ -0,0 +1,231 @@ +"""`acme.affine`, a third-party `array_array` codec, written against the door alone. + +Every element becomes `x * scale + offset`, and the result may be stored +as a different data type. Every import is from `zarr_metadata.v3.entity` +or another public module; this is the complete example the door's +docstring points at. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import TYPE_CHECKING, ClassVar, Literal, NotRequired, Self + +import pytest +from typing_extensions import TypedDict + +from zarr_metadata.rules import ( + Canonical, + canonicalize_array_metadata_v3, + validate_array_metadata_v3, +) +from zarr_metadata.v3.data_type.float32 import Float32DataType +from zarr_metadata.v3.entity import ( + CORE_AND_EXTENSIONS, + UNSET, + ArrayArrayCodec, + ArrayDocumentV3, + ArrayParts, + Configuration, + DataTypeEntity, + MetadataValidationError, + Opaque, + ValidationProblem, + ZarrV3MetadataFieldJSON, + problem, +) + +if TYPE_CHECKING: + from collections.abc import Iterator + + +class AcmeAffineConfiguration(TypedDict, closed=True): + scale: float + offset: NotRequired[float] + dtype: NotRequired[ZarrV3MetadataFieldJSON] + + +class AcmeAffineObject(TypedDict, closed=True): + name: Literal["acme.affine"] + configuration: AcmeAffineConfiguration + must_understand: NotRequired[bool] + + +@dataclass(frozen=True) +class AcmeAffineOptions(Configuration): + scale: float + offset: float | UNSET = UNSET + dtype: DataTypeEntity | Opaque | UNSET = UNSET + + def problems(self) -> Iterator[ValidationProblem]: + if self.scale == 0: + yield ValidationProblem( + ("scale",), "expected a non-zero number, got 0", "invalid_value" + ) + if ( + isinstance(self.dtype, DataTypeEntity) + and self.dtype.storage_class() == "variable_length" + ): + yield ValidationProblem( + ("dtype",), + f"expected a fixed-size data type, got {type(self.dtype).identifier!r}", + "invalid_value", + ) + + +@dataclass(frozen=True) +class AcmeAffineCodec(ArrayArrayCodec): + """`x * scale + offset`, stored as `dtype` if one is named.""" + + configuration: AcmeAffineOptions + + identifier: ClassVar[str] = "acme.affine" + variable_size: ClassVar[bool] = False + + def canonical(self) -> Self: + """An offset of 0 is the identity, and absent says the same; `dtype` in its own form.""" + return self.with_configuration( + offset=UNSET if self.configuration.offset == 0 else self.configuration.offset, + dtype=UNSET + if self.configuration.dtype is UNSET + else self.configuration.dtype.canonical(), + ) + + def incoming_problems(self, incoming: ArrayParts | None) -> tuple[ValidationProblem, ...]: + data_type = incoming.data_type if incoming is not None else None + if data_type is None or data_type.storage_class() != "variable_length": + return () + return problem( + (), + f"acme.affine cannot scale variable-length data_type {type(data_type).identifier!r}", + "invalid_value", + ) + + def transition(self, incoming: ArrayParts) -> ArrayParts | None: + if self.configuration.dtype is UNSET: + return incoming + return incoming.with_data_type( + self.configuration.dtype + if isinstance(self.configuration.dtype, DataTypeEntity) + else None + ) + + +SCOPE = CORE_AND_EXTENSIONS.extended_with(AcmeAffineCodec) +BYTES_LE = {"name": "bytes", "configuration": {"endian": "little"}} + + +def _affine(**configuration: object) -> dict[str, object]: + return {"name": "acme.affine", "configuration": configuration} + + +def _document(**overrides: object) -> dict[str, object]: + return { + "zarr_format": 3, + "node_type": "array", + "shape": [8], + "data_type": "float32", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": [8]}}, + "chunk_key_encoding": "default", + "codecs": [_affine(scale=2), BYTES_LE], + **overrides, + } + + +@pytest.mark.parametrize( + "configuration", + [ + {"scale": 2}, + {"scale": 2.5, "offset": -1}, + {"scale": -0.5, "offset": 0.25, "dtype": "float64"}, + {"scale": 3, "offset": 0, "dtype": {"name": "float32"}}, + ], +) +def test_a_valid_document_has_no_problems(configuration: dict[str, object]) -> None: + document = _document(codecs=[_affine(**configuration), BYTES_LE]) + assert validate_array_metadata_v3(document, context=SCOPE) == () + assert isinstance(ArrayDocumentV3.from_json(document, context=SCOPE).codecs[0], AcmeAffineCodec) + + +def test_error_problems_are_located_at_their_members() -> None: + document = _document( + codecs=[ + {"name": "transpose", "configuration": {"order": [0]}}, + _affine(scale=0, offset=None), + BYTES_LE, + ] + ) + problems = validate_array_metadata_v3(document, context=SCOPE) + # A null offset fails the type check, so the value rules do not run + # on the hole: one problem, at the member that has it. + assert [(found.loc, found.kind) for found in problems] == [ + (("codecs", 1, "configuration", "offset"), "invalid_type") + ] + problems = validate_array_metadata_v3( + _document(codecs=[_affine(scale=0), BYTES_LE]), context=SCOPE + ) + assert [(found.loc, found.kind) for found in problems] == [ + (("codecs", 0, "configuration", "scale"), "invalid_value") + ] + + +def test_error_a_variable_length_type_is_refused_in_and_out() -> None: + document = _document(data_type="string", fill_value="", codecs=[_affine(scale=2), "vlen-utf8"]) + problems = validate_array_metadata_v3(document, context=SCOPE) + assert [(found.loc, found.kind) for found in problems] == [(("codecs", 0), "invalid_value")] + document = _document(codecs=[_affine(scale=2, dtype="string"), BYTES_LE]) + problems = validate_array_metadata_v3(document, context=SCOPE) + assert [(found.loc, found.kind) for found in problems] == [ + (("codecs", 0, "configuration", "dtype"), "invalid_value") + ] + + +def test_dtype_changes_what_the_next_codec_sees() -> None: + # uint8 needs no endianness and float32 does, so a bare `bytes` codec + # is fine before the transition and wrong after it. + without = _document(data_type="uint8", codecs=[_affine(scale=2), "bytes"]) + assert validate_array_metadata_v3(without, context=SCOPE) == () + retyped = _document(data_type="uint8", codecs=[_affine(scale=2, dtype="float32"), "bytes"]) + problems = validate_array_metadata_v3(retyped, context=SCOPE) + assert [(found.loc, found.kind) for found in problems] == [ + (("codecs", 1, "configuration", "endian"), "missing_key") + ] + + +def test_an_out_of_scope_dtype_is_kept_and_not_judged() -> None: + entry = _affine(scale=2, dtype="mycorp.decimal") + document = _document(codecs=[entry, "bytes"]) + assert validate_array_metadata_v3(document, context=SCOPE) == () + codec = ArrayDocumentV3.from_json(document, context=SCOPE).codecs[0] + assert isinstance(codec, AcmeAffineCodec) + assert isinstance(codec.configuration.dtype, Opaque) + assert codec.configuration.dtype.reason == "out_of_scope" + assert codec.to_json() == entry + + +def test_round_trip_and_canonical() -> None: + entry = _affine(scale=2, offset=1, dtype="float64") + codec = ArrayDocumentV3.from_json(_document(codecs=[entry, BYTES_LE]), context=SCOPE).codecs[0] + assert isinstance(codec, AcmeAffineCodec) + assert codec.to_json() == entry + assert AcmeAffineCodec(AcmeAffineOptions(scale=2, offset=0)).canonical() == AcmeAffineCodec( + AcmeAffineOptions(scale=2) + ) + document = _document(codecs=[_affine(scale=2, offset=0.0), BYTES_LE]) + result = canonicalize_array_metadata_v3(document, context=SCOPE) + assert isinstance(result, Canonical) + assert result.document["codecs"][0] == _affine(scale=2) + + +def test_constructed_by_hand() -> None: + codec = AcmeAffineCodec(AcmeAffineOptions(scale=2.5, offset=-1, dtype=Float32DataType())) + assert codec.to_json() == { + "name": "acme.affine", + "configuration": {"scale": 2.5, "offset": -1, "dtype": "float32"}, + } + with pytest.raises(MetadataValidationError) as caught: + AcmeAffineCodec(AcmeAffineOptions(scale=0)) + assert [(found.loc, found.kind) for found in caught.value.problems] == [ + (("scale",), "invalid_value") + ] diff --git a/packages/zarr-metadata/tests/v3/test_acme_decimal.py b/packages/zarr-metadata/tests/v3/test_acme_decimal.py new file mode 100644 index 0000000000..09c5fc592b --- /dev/null +++ b/packages/zarr-metadata/tests/v3/test_acme_decimal.py @@ -0,0 +1,356 @@ +"""`acme.decimal`, a third-party data type, written against the door alone. + +A fixed-size decimal stored as a 16-byte integer. The configuration carries `precision` (1..38 significant digits) and +`scale` (0..precision digits after the point); a fill value is a JSON +string holding a decimal literal such as `"12.50"` whose digits fit both. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from typing import TYPE_CHECKING, ClassVar, Final, Literal, NotRequired + +import pytest +from typing_extensions import ReadOnly, TypedDict + +if TYPE_CHECKING: + from collections.abc import Iterable, Iterator + + +from zarr_metadata.v3.entity import ( + Configuration, + DataTypeEntity, + Loc, + MetadataValidationError, + StorageClass, + ValidationProblem, + problem, + resolve, +) + +ACME_DECIMAL_DATA_TYPE_NAME: Final = "acme.decimal" +"""The `name` field value of the `acme.decimal` data type.""" + +AcmeDecimalDataTypeName = Literal["acme.decimal"] +"""Literal type of the `name` field of the `acme.decimal` data type.""" + +ACME_DECIMAL_MAX_PRECISION: Final = 38 +"""The most significant digits a 16-byte integer holds in every case.""" + +DECIMAL_LITERAL: Final = re.compile(r"-?(?P[0-9]+)(?:\.(?P[0-9]+))?") +"""A plain decimal literal: an optional sign, digits, an optional fraction.""" + + +class AcmeDecimalConfiguration(TypedDict, closed=True): + """Configuration for the `acme.decimal` data type.""" + + precision: ReadOnly[int] + scale: ReadOnly[int] + + +class AcmeDecimal(TypedDict, closed=True): + """`acme.decimal` data type metadata.""" + + name: AcmeDecimalDataTypeName + configuration: AcmeDecimalConfiguration + must_understand: NotRequired[bool] + + +AcmeDecimalFillValue = str +"""Permitted JSON shape of the `fill_value` field for `acme.decimal`: a decimal literal.""" + + +__all__ = [ + "ACME_DECIMAL_DATA_TYPE_NAME", + "ACME_DECIMAL_MAX_PRECISION", + "AcmeDecimal", + "AcmeDecimalConfiguration", + "AcmeDecimalDataType", + "AcmeDecimalDataTypeName", + "AcmeDecimalFillValue", +] + + +@dataclass(frozen=True) +class AcmeDecimalOptions(Configuration): + precision: int + scale: int + + def problems(self) -> Iterator[ValidationProblem]: + if not 1 <= self.precision <= ACME_DECIMAL_MAX_PRECISION: + yield ValidationProblem( + ("precision",), + f"expected an integer in [1, {ACME_DECIMAL_MAX_PRECISION}], got {self.precision}", + "invalid_value", + ) + if self.scale < 0: + yield ValidationProblem( + ("scale",), f"expected an integer >= 0, got {self.scale}", "invalid_value" + ) + elif self.scale > self.precision: + yield ValidationProblem( + ("scale",), + f"expected an integer <= precision ({self.precision}), got {self.scale}", + "invalid_value", + ) + + +@dataclass(frozen=True) +class AcmeDecimalDataType(DataTypeEntity): + """The `acme.decimal` data type, coerced from its metadata.""" + + configuration: AcmeDecimalOptions + + identifier: ClassVar[str] = ACME_DECIMAL_DATA_TYPE_NAME + scalar_storage: ClassVar[StorageClass] = "multi_byte" + twos_complement: ClassVar[bool] = False + + def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]: + """A decimal literal whose digits fit `precision` and `scale`. + + Judged as written: `"12.50"` has two fractional digits whatever + its value, so it needs a scale of at least two. + """ + if not isinstance(value, str): + return problem( + loc, f"expected a decimal literal string, got {value!r}", "invalid_value" + ) + matched = DECIMAL_LITERAL.fullmatch(value) + if matched is None: + return problem( + loc, f"expected a decimal literal like '12.50', got {value!r}", "invalid_value" + ) + integer_digits = len(matched.group("integer").lstrip("0")) + fraction = matched.group("fraction") + fraction_digits = 0 if fraction is None else len(fraction) + allowed_integer_digits = self.configuration.precision - self.configuration.scale + found: list[ValidationProblem] = [] + if fraction_digits > self.configuration.scale: + found.extend( + problem( + loc, + f"{value!r} has {fraction_digits} fractional digits, but scale is {self.configuration.scale}", + "invalid_value", + ) + ) + if integer_digits > allowed_integer_digits: + found.extend( + problem( + loc, + f"{value!r} has {integer_digits} integer digits, but precision {self.configuration.precision} " + f"with scale {self.configuration.scale} allows {allowed_integer_digits}", + "invalid_value", + ) + ) + return tuple(found) + + +# --- the tests -------------------------------------------------------------- + +from zarr_metadata.rules import canonicalize_array_metadata_v3, validate_array_metadata_v3 +from zarr_metadata.v3.entity import CORE_AND_EXTENSIONS, ArrayDocumentV3, Context, Opaque + +SCOPE: Context = CORE_AND_EXTENSIONS.extended_with(AcmeDecimalDataType) + +LITTLE_ENDIAN_BYTES = {"name": "bytes", "configuration": {"endian": "little"}} + + +def _data_type(precision: int, scale: int) -> AcmeDecimal: + return { + "name": ACME_DECIMAL_DATA_TYPE_NAME, + "configuration": {"precision": precision, "scale": scale}, + } + + +def _document(**overrides: object) -> dict[str, object]: + return { + "zarr_format": 3, + "node_type": "array", + "shape": (8,), + "data_type": _data_type(4, 2), + "fill_value": "12.50", + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (8,)}}, + "chunk_key_encoding": "default", + "codecs": (LITTLE_ENDIAN_BYTES,), + **overrides, + } + + +def _locs(problems: Iterable[ValidationProblem]) -> list[Loc]: + return [problem.loc for problem in problems] + + +# --- documents, judged in a scope that knows the type --------------------- + + +def test_a_valid_document_has_no_problems() -> None: + assert validate_array_metadata_v3(_document(), context=SCOPE) == () + + +def test_a_bare_bytes_codec_is_refused_for_a_sixteen_byte_type() -> None: + # A 16-byte integer has a byte order, so the `bytes` codec needs to be + # told which one: the same rule the core multi-byte types are held to. + problems = validate_array_metadata_v3(_document(codecs=("bytes",)), context=SCOPE) + assert [(problem.loc, problem.kind) for problem in problems] == [ + (("codecs", 0, "configuration", "endian"), "missing_key") + ] + + +def test_error_scale_above_precision_is_located_at_the_member() -> None: + document = _document(data_type=_data_type(4, 5)) + problems = validate_array_metadata_v3(document, context=SCOPE) + assert _locs(problems) == [("data_type", "configuration", "scale")] + + +def test_error_a_fill_value_that_does_not_fit_is_located_at_fill_value() -> None: + problems = validate_array_metadata_v3(_document(fill_value="1234.5"), context=SCOPE) + assert _locs(problems) == [("fill_value",)] + + +def test_the_type_passes_through_a_sharding_codec() -> None: + shard = { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": (4,), + "codecs": (LITTLE_ENDIAN_BYTES,), + "index_codecs": (LITTLE_ENDIAN_BYTES, "crc32c"), + }, + } + document = _document(shape=(16,), codecs=(shard,)) + assert validate_array_metadata_v3(document, context=SCOPE) == () + array = ArrayDocumentV3.from_json(document, context=SCOPE) + assert isinstance(array.data_type, AcmeDecimalDataType) + + +def test_an_inner_bare_bytes_codec_is_refused_inside_a_shard_too() -> None: + shard = { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": (4,), + "codecs": ("bytes",), + "index_codecs": (LITTLE_ENDIAN_BYTES, "crc32c"), + }, + } + problems = validate_array_metadata_v3(_document(shape=(16,), codecs=(shard,)), context=SCOPE) + assert _locs(problems) == [ + ("codecs", 0, "configuration", "codecs", 0, "configuration", "endian") + ] + + +def test_an_unregistered_scope_reads_it_as_opaque_and_judges_nothing() -> None: + document = _document(fill_value="this is not judged") + assert validate_array_metadata_v3(document) == () + array = ArrayDocumentV3.from_json(document) + assert isinstance(array.data_type, Opaque) + assert array.data_type.reason == "out_of_scope" + + +# --- round trip ------------------------------------------------------------- + + +def test_to_json_writes_back_what_was_read() -> None: + document = _document() + array = ArrayDocumentV3.from_json(document, context=SCOPE) + assert isinstance(array.data_type, DataTypeEntity) + assert array.data_type.to_json() == document["data_type"] + assert array.data_type.canonical() == array.data_type + + +def test_canonical_form_keeps_the_configuration() -> None: + result = canonicalize_array_metadata_v3(_document(), context=SCOPE) + assert result.valid is True + assert result.document["data_type"] == _data_type(4, 2) + + +# --- hand construction ------------------------------------------------------ + + +def test_hand_construction() -> None: + entity = AcmeDecimalDataType(AcmeDecimalOptions(precision=4, scale=2)) + assert entity.storage_class() == "multi_byte" + assert entity.to_json() == _data_type(4, 2) + assert entity == AcmeDecimalDataType(AcmeDecimalOptions(4, 2)) + assert hash(entity) == hash(AcmeDecimalDataType(AcmeDecimalOptions(4, 2))) + + +@pytest.mark.parametrize( + ("precision", "scale", "value"), + [ + (4, 2, "12.50"), + (4, 2, "-99.99"), + (4, 2, "0"), + (4, 2, "0.5"), + (4, 2, "0012.5"), + (4, 0, "1234"), + (4, 4, "0.1234"), + (1, 1, "0.0"), + (38, 10, "1234567890123456789012345678.0123456789"), + ], +) +def test_fill_values_that_fit_are_accepted(precision: int, scale: int, value: str) -> None: + entity = AcmeDecimalDataType(AcmeDecimalOptions(precision=precision, scale=scale)) + assert entity.fill_value_problems(value, ("fill_value",)) == () + + +@pytest.mark.parametrize("precision", [0, 39]) +def test_error_precision_out_of_range(precision: int) -> None: + with pytest.raises(MetadataValidationError) as caught: + AcmeDecimalDataType(AcmeDecimalOptions(precision=precision, scale=0)) + assert _locs(caught.value.problems) == [("precision",)] + + +def test_error_scale_negative() -> None: + with pytest.raises(MetadataValidationError) as caught: + AcmeDecimalDataType(AcmeDecimalOptions(precision=4, scale=-1)) + assert _locs(caught.value.problems) == [("scale",)] + + +def test_error_scale_above_precision() -> None: + with pytest.raises(MetadataValidationError) as caught: + AcmeDecimalDataType(AcmeDecimalOptions(precision=4, scale=5)) + assert [(problem.loc, problem.message) for problem in caught.value.problems] == [ + (("scale",), "expected an integer <= precision (4), got 5") + ] + + +def test_error_the_constructor_stops_at_the_first_problem_and_coerce_reports_every_one() -> None: + with pytest.raises(MetadataValidationError) as caught: + AcmeDecimalDataType(AcmeDecimalOptions(precision=0, scale=-1)) + assert _locs(caught.value.problems) == [("precision",)] + _, problems = resolve( + {"name": "acme.decimal", "configuration": {"precision": 0, "scale": -1}}, + DataTypeEntity, + SCOPE, + ) + assert _locs(problems) == [("configuration", "precision"), ("configuration", "scale")] + + +def test_error_fill_value_must_be_a_string() -> None: + entity = AcmeDecimalDataType(AcmeDecimalOptions(precision=4, scale=2)) + assert _locs(entity.fill_value_problems(12.5, ("fill_value",))) == [("fill_value",)] + + +@pytest.mark.parametrize("value", ["", "1e2", " 12.5", "12.", ".5", "abc", "1,5", "NaN"]) +def test_error_fill_value_must_be_a_decimal_literal(value: str) -> None: + entity = AcmeDecimalDataType(AcmeDecimalOptions(precision=4, scale=2)) + assert _locs(entity.fill_value_problems(value, ("fill_value",))) == [("fill_value",)] + + +def test_error_fill_value_with_too_many_fraction_digits() -> None: + entity = AcmeDecimalDataType(AcmeDecimalOptions(precision=4, scale=2)) + problems = entity.fill_value_problems("1.234", ("fill_value",)) + assert [(problem.loc, problem.message) for problem in problems] == [ + (("fill_value",), "'1.234' has 3 fractional digits, but scale is 2") + ] + + +def test_error_fill_value_with_too_many_integer_digits() -> None: + entity = AcmeDecimalDataType(AcmeDecimalOptions(precision=4, scale=2)) + problems = entity.fill_value_problems("1234.5", ("fill_value",)) + assert [(problem.loc, problem.message) for problem in problems] == [ + ( + ("fill_value",), + "'1234.5' has 4 integer digits, but precision 4 with scale 2 allows 2", + ) + ] diff --git a/packages/zarr-metadata/tests/v3/test_entities.py b/packages/zarr-metadata/tests/v3/test_entities.py new file mode 100644 index 0000000000..1c6d0cd5fa --- /dev/null +++ b/packages/zarr-metadata/tests/v3/test_entities.py @@ -0,0 +1,1253 @@ +"""The correspondence between an entity's dataclass and its TypedDicts. + +A configuration TypedDict unpacked is exactly the dataclass constructor's +signature, and the object TypedDict is exactly what `to_json` returns. +Asserting the first is what lets the dataclass carry its own checks +instead of a hand-written per-member table elsewhere: the two cannot +drift, because one drifting makes this fail. +""" + +from __future__ import annotations + +import copy +import dataclasses +import math +import sys +from typing import ( + Any, + ClassVar, + Literal, + Self, + cast, + get_type_hints, +) + +import pytest +from hypothesis import given, settings +from typing_extensions import is_typeddict + +from tests.helpers import configuration_of, entry_at +from tests.rules.strategies import valid_documents +from zarr_metadata.model import UNSET, MetadataValidationError +from zarr_metadata.rules import validate_array_metadata_v3 +from zarr_metadata.v3._document import read_array_v3, refine_array_v3, well_formed_array_v3 +from zarr_metadata.v3._entity import BytesBytesCodec, Opaque, is_from_name +from zarr_metadata.v3._registry import CORE, CORE_AND_EXTENSIONS +from zarr_metadata.v3._typed_json import ( + describe, + field_hints, + is_class_var, + is_not_required, + is_optional, + no_writer_leaf, + shape_of, + writer_for, +) +from zarr_metadata.v3.chunk_grid.rectilinear import ( + RectilinearChunkGrid, +) +from zarr_metadata.v3.chunk_grid.regular import RegularChunkGrid +from zarr_metadata.v3.chunk_key_encoding.default import ( + DefaultChunkKeyEncoding, +) +from zarr_metadata.v3.chunk_key_encoding.v2 import ( + V2ChunkKeyEncoding, +) +from zarr_metadata.v3.codec.blosc import BloscCodec, BloscOptions +from zarr_metadata.v3.codec.bytes import BytesCodec +from zarr_metadata.v3.codec.cast_value import CastValueCodec +from zarr_metadata.v3.codec.crc32c import Crc32cCodec +from zarr_metadata.v3.codec.gzip import GzipCodec, GzipOptions +from zarr_metadata.v3.codec.scale_offset import ScaleOffsetCodec +from zarr_metadata.v3.codec.sharding_indexed import ( + ShardingIndexedCodec, + ShardingIndexedOptions, +) +from zarr_metadata.v3.codec.transpose import TransposeCodec +from zarr_metadata.v3.codec.zstd import ZstdCodec +from zarr_metadata.v3.data_type.bool import BoolDataType +from zarr_metadata.v3.data_type.bytes import BytesDataType +from zarr_metadata.v3.data_type.complex64 import Complex64DataType +from zarr_metadata.v3.data_type.complex128 import Complex128DataType +from zarr_metadata.v3.data_type.float16 import Float16DataType +from zarr_metadata.v3.data_type.float32 import Float32DataType +from zarr_metadata.v3.data_type.float64 import Float64DataType +from zarr_metadata.v3.data_type.int8 import Int8DataType +from zarr_metadata.v3.data_type.int16 import Int16DataType +from zarr_metadata.v3.data_type.int32 import Int32DataType +from zarr_metadata.v3.data_type.int64 import Int64DataType +from zarr_metadata.v3.data_type.numpy_datetime64 import ( + NumpyDatetime64DataType, +) +from zarr_metadata.v3.data_type.numpy_timedelta64 import ( + NumpyTimedelta64DataType, +) +from zarr_metadata.v3.data_type.raw import RawBytesDataType +from zarr_metadata.v3.data_type.string import StringDataType +from zarr_metadata.v3.data_type.struct import StructDataType, StructFieldComponent, StructOptions +from zarr_metadata.v3.data_type.uint8 import Uint8DataType +from zarr_metadata.v3.data_type.uint16 import Uint16DataType +from zarr_metadata.v3.data_type.uint32 import Uint32DataType +from zarr_metadata.v3.data_type.uint64 import Uint64DataType +from zarr_metadata.v3.entity import ( + ArrayDocumentV3, + ChunkGridEntity, + ChunkKeyEncodingEntity, + CodecEntity, + Configuration, + DataTypeEntity, + JSONValue, + MetadataEntity, + StorageTransformerEntity, + resolve, +) + +# Every registered entity, keyed by `:` -- an identifier +# is unique only within its extension point, and `bytes` is both a codec +# and a data type. +# The field of a v3 array document each kind is read from: the test ids. +POINT: dict[type[MetadataEntity], str] = { + DataTypeEntity: "data_type", + ChunkGridEntity: "chunk_grid", + ChunkKeyEncodingEntity: "chunk_key_encoding", + CodecEntity: "codecs", + StorageTransformerEntity: "storage_transformers", +} + +ENTITIES: dict[str, type[MetadataEntity]] = { + "codecs:blosc": BloscCodec, + "codecs:bytes": BytesCodec, + "codecs:cast_value": CastValueCodec, + "codecs:crc32c": Crc32cCodec, + "codecs:gzip": GzipCodec, + "codecs:scale_offset": ScaleOffsetCodec, + "codecs:sharding_indexed": ShardingIndexedCodec, + "codecs:transpose": TransposeCodec, + "codecs:zstd": ZstdCodec, + "chunk_grid:regular": RegularChunkGrid, + "chunk_grid:rectilinear": RectilinearChunkGrid, + "chunk_key_encoding:default": DefaultChunkKeyEncoding, + "chunk_key_encoding:v2": V2ChunkKeyEncoding, + "data_type:numpy.datetime64": NumpyDatetime64DataType, + "data_type:numpy.timedelta64": NumpyTimedelta64DataType, + "data_type:bool": BoolDataType, + "data_type:int8": Int8DataType, + "data_type:int16": Int16DataType, + "data_type:int32": Int32DataType, + "data_type:int64": Int64DataType, + "data_type:uint8": Uint8DataType, + "data_type:uint16": Uint16DataType, + "data_type:uint32": Uint32DataType, + "data_type:uint64": Uint64DataType, + "data_type:float16": Float16DataType, + "data_type:float32": Float32DataType, + "data_type:float64": Float64DataType, + "data_type:complex64": Complex64DataType, + "data_type:complex128": Complex128DataType, + "data_type:bytes": BytesDataType, + "data_type:struct": StructDataType, + "data_type:string": StringDataType, + "data_type:r": RawBytesDataType, +} + + +# One or more documents each entity reads, spelled to reach both shapes +# where the entity has both: the bare name when every member is absent, +# the object otherwise. `st.from_type` over the named types cannot serve +# here -- measured, it reaches a valid gzip or blosc in under 1% of draws. +EXAMPLES: dict[str, tuple[object, ...]] = { + "codecs:blosc": ( + { + "name": "blosc", + "configuration": { + "cname": "zstd", + "clevel": 5, + "shuffle": "shuffle", + "typesize": 4, + "blocksize": 0, + }, + }, + ), + "codecs:bytes": ("bytes", {"name": "bytes", "configuration": {"endian": "little"}}), + "codecs:cast_value": ( + {"name": "cast_value", "configuration": {"data_type": "int8"}}, + { + "name": "cast_value", + "configuration": {"data_type": "int8", "scalar_map": {"encode": (("NaN", 0),)}}, + }, + ), + "codecs:crc32c": ("crc32c", {"name": "crc32c"}), + "codecs:gzip": ({"name": "gzip", "configuration": {"level": 5}},), + "codecs:scale_offset": ( + "scale_offset", + {"name": "scale_offset", "configuration": {"offset": 2, "scale": 0.5}}, + ), + "codecs:sharding_indexed": ( + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": (4,), + "codecs": ("bytes",), + "index_codecs": ( + {"name": "bytes", "configuration": {"endian": "little"}}, + "crc32c", + ), + "index_location": "start", + }, + }, + ), + "codecs:transpose": ({"name": "transpose", "configuration": {"order": (2, 1, 0)}},), + "codecs:zstd": ({"name": "zstd", "configuration": {"level": 3, "checksum": False}},), + "chunk_grid:regular": ({"name": "regular", "configuration": {"chunk_shape": (4, 4)}},), + "chunk_grid:rectilinear": ( + { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": ((32, 32, 32),)}, + }, + { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": (((32, 3),),)}, + }, + ), + "chunk_key_encoding:default": ( + "default", + {"name": "default", "configuration": {"separator": "."}}, + ), + "chunk_key_encoding:v2": ("v2", {"name": "v2", "configuration": {"separator": "/"}}), + "data_type:numpy.datetime64": ( + {"name": "numpy.datetime64", "configuration": {"unit": "s", "scale_factor": 1}}, + ), + "data_type:numpy.timedelta64": ( + {"name": "numpy.timedelta64", "configuration": {"unit": "ms", "scale_factor": 10}}, + ), + "data_type:bool": ("bool",), + "data_type:int8": ("int8",), + "data_type:int16": ("int16",), + "data_type:int32": ("int32",), + "data_type:int64": ("int64",), + "data_type:uint8": ("uint8",), + "data_type:uint16": ("uint16",), + "data_type:uint32": ("uint32",), + "data_type:uint64": ("uint64",), + "data_type:float16": ("float16",), + "data_type:float32": ("float32",), + "data_type:float64": ("float64",), + "data_type:complex64": ("complex64",), + "data_type:complex128": ("complex128",), + "data_type:bytes": ("bytes",), + "data_type:struct": ( + { + "name": "struct", + "configuration": { + "fields": ( + {"name": "a", "data_type": "uint8"}, + { + "name": "b", + "data_type": { + "name": "numpy.datetime64", + "configuration": {"unit": "s", "scale_factor": 1}, + }, + }, + ), + }, + }, + ), + "data_type:string": ("string",), + "data_type:r": ("r16", "r008"), +} + + +def _round_trips(entity: type[MetadataEntity], document: JSONValue) -> MetadataEntity: + read, problems = entity.coerce(document, CORE_AND_EXTENSIONS) + assert problems == () + assert read is not None + again, problems = entity.coerce(read.to_json(), CORE_AND_EXTENSIONS) + assert problems == () + assert again == read + return read + + +@pytest.mark.parametrize( + ("entity", "document"), + [(ENTITIES[key], document) for key, documents in EXAMPLES.items() for document in documents], + ids=[ + f"{key}:{index}" for key, documents in EXAMPLES.items() for index in range(len(documents)) + ], +) +def test_to_json_reads_back_to_the_same_entity( + entity: type[MetadataEntity], document: JSONValue +) -> None: + # What `to_json` writes, `coerce` reads to the entity that wrote it: + # every member is written, in the spelling the reader expects. + _round_trips(entity, document) + + +@given(document=valid_documents()) +@settings(max_examples=50, deadline=None) +def test_to_json_reads_back_across_a_valid_document(document: dict[str, object]) -> None: + refined, structural = well_formed_array_v3(document) + assert refined is not None + assert structural == () + array, problems = read_array_v3(refined, CORE_AND_EXTENSIONS) + assert problems == () + for entity in (array.data_type, array.chunk_grid, array.chunk_key_encoding, *array.codecs): + assert isinstance(entity, MetadataEntity) + _round_trips(type(entity), entity.to_json()) + + +def test_every_registered_entity_is_checked_here() -> None: + registered = { + f"{POINT[kind]}:{identifier}" + for kind, entities in CORE_AND_EXTENSIONS.tables.items() + for identifier in entities + } + assert registered == set(ENTITIES) + assert set(EXAMPLES) == set(ENTITIES) + + +def test_core_is_a_subset_of_core_and_extensions() -> None: + both = CORE_AND_EXTENSIONS.tables + for kind, entities in CORE.tables.items(): + assert entities.items() <= both[kind].items() + + +def test_a_name_out_of_scope_resolves_to_nothing() -> None: + # Not an error: an unmodelled extension is left unjudged, not rejected. + assert CORE.claimant(CodecEntity, "mycorp.secret") is None + assert CORE.claimant(CodecEntity, "blosc") is BloscCodec + + +def test_an_entity_round_trips_through_its_json_form() -> None: + original = { + "name": "blosc", + "configuration": { + "cname": "zstd", + "clevel": 5, + "shuffle": "shuffle", + "blocksize": 0, + "typesize": 4, + }, + } + codec, problems = BloscCodec.coerce(original, CORE) + assert problems == () + assert codec is not None + assert codec.to_json() == original + + +def test_an_unknown_key_is_reported_without_losing_the_member() -> None: + # The document is invalid either way, but dropping the member would + # make the entity describe something the document does not say. + codec, problems = CastValueCodec.coerce( + { + "name": "cast_value", + "configuration": {"data_type": "int8", "scalar_map": {"encode": (), "enc": ()}}, + }, + CORE_AND_EXTENSIONS, + ) + assert [problem.kind for problem in problems] == ["unknown_key"] + assert codec is not None + assert codec.configuration.scalar_map == {"encode": (), "enc": ()} + + +# (a defect in a nested metadata field, the location it belongs at) +NESTED_ENVELOPES: dict[str, tuple[dict[str, object], tuple[str | int, ...]]] = { + "unexpected-member": ({"name": "bytes", "typo": 1}, ("typo",)), + "configuration-not-an-object": ({"name": "bytes", "configuration": 42}, ("configuration",)), + "must-understand-not-a-boolean": ( + {"name": "bytes", "must_understand": "no"}, + ("must_understand",), + ), +} + + +@pytest.mark.parametrize( + ("codec", "inner_loc"), NESTED_ENVELOPES.values(), ids=list(NESTED_ENVELOPES) +) +def test_a_nested_metadata_field_is_judged_like_a_top_level_one( + codec: dict[str, object], inner_loc: tuple[str | int, ...] +) -> None: + # A metadata field is a metadata field wherever it appears. These sit + # inside a shard's pipeline, which only the entity layer reads, so + # nothing else is going to notice them. + document = { + "zarr_format": 3, + "node_type": "array", + "shape": (8,), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (8,)}}, + "chunk_key_encoding": "default", + "codecs": ( + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": (4,), + "codecs": (codec,), + "index_codecs": ({"name": "bytes", "configuration": {"endian": "little"}},), + }, + }, + ), + } + problems = validate_array_metadata_v3(document) + assert [problem.loc for problem in problems] == [ + ("codecs", 0, "configuration", "codecs", 0, *inner_loc) + ] + + +def test_every_problem_location_indexes_into_the_document() -> None: + # Two defects in one shard, at different depths. A location that does + # not resolve is a location a consumer cannot use. + document = { + "zarr_format": 3, + "node_type": "array", + "shape": (8, 8), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (8, 8)}}, + "chunk_key_encoding": "default", + "codecs": ( + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": (4, 4), + "codecs": ( + {"name": "transpose", "configuration": {"order": (0, 0)}}, + {"name": "bytes", "configuration": {"endian": "little"}}, + ), + "index_codecs": ({"name": "bytes"},), + }, + }, + ), + } + problems = validate_array_metadata_v3(document) + assert len(problems) != 0 + for problem in problems: + node: object = document + for step in problem.loc: + # Two branches rather than one `or`: each narrows `step` to + # the key type its container takes. + if isinstance(node, dict) and isinstance(step, str) and step in node: + node = node[step] + continue + if isinstance(node, tuple) and isinstance(step, int) and step < len(node): + node = node[step] + continue + # A `missing_key` problem names where the key belongs, so it + # is allowed to run past the end of what is there. Any other + # kind must address a node that exists. + assert problem.kind == "missing_key", (problem.loc, step) + break + + +def test_an_unreadable_member_costs_the_entity() -> None: + # `clevel` is the wrong type, so this blosc cannot be built, and an + # entity that does not exist has no values to judge. The type problem + # is what you have to fix first, and the raw JSON is still on the + # `Opaque` standing in for the codec. + document = { + "zarr_format": 3, + "node_type": "array", + "shape": (4,), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (4,)}}, + "chunk_key_encoding": "default", + "codecs": ( + {"name": "bytes", "configuration": {"endian": "little"}}, + { + "name": "blosc", + "configuration": { + "cname": "lz4", + "clevel": "five", + "shuffle": "noshuffle", + "blocksize": -1, + }, + }, + ), + } + problems = validate_array_metadata_v3(document) + assert {problem.loc for problem in problems} == { + ("codecs", 1, "configuration", "clevel"), + } + + +def test_an_optional_member_of_the_wrong_type_is_not_judged_as_absent() -> None: + # `typesize` could not be read. Building the entity around the hole + # would have `__post_init__` see it as absent and add a second, + # contradictory problem at the same location. + document = { + "zarr_format": 3, + "node_type": "array", + "shape": (4,), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (4,)}}, + "chunk_key_encoding": "default", + "codecs": ( + "bytes", + { + "name": "blosc", + "configuration": { + "cname": "zstd", + "clevel": 5, + "shuffle": "shuffle", + "typesize": "four", + "blocksize": 0, + }, + }, + ), + } + problems = validate_array_metadata_v3(document) + assert [(problem.loc, problem.kind) for problem in problems] == [ + (("codecs", 1, "configuration", "typesize"), "invalid_type") + ] + + +def test_the_document_writes_back_only_the_fields_it_read() -> None: + # An absent field is read as an `Opaque` standing in for it; writing + # it back as `null` would invent a value the document never wrote. + array, problems = read_array_v3({"shape": (4,)}, CORE_AND_EXTENSIONS) + assert problems == () + assert array.to_json() == {"shape": (4,)} + + +@pytest.mark.parametrize( + "spelling", + [ + "crc32c", + {"name": "crc32c"}, + {"name": "crc32c", "configuration": {}}, + {"name": "crc32c", "must_understand": True}, + {"name": "crc32c", "configuration": {}, "must_understand": True}, + ], + ids=["bare", "object", "empty-configuration", "must-understand", "both"], +) +def test_the_document_writes_an_envelope_as_it_was_written(spelling: JSONValue) -> None: + # An entity writes its own spelling; the document knows the one it + # read and puts it back, around a codec in the pipeline and around + # one inside a shard alike. Only `canonical` simplifies it. + document = { + "codecs": ( + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": (2,), + "codecs": ("bytes",), + "index_codecs": ( + {"name": "bytes", "configuration": {"endian": "little"}}, + spelling, + ), + }, + }, + spelling, + ), + } + array, problems = read_array_v3(document, CORE_AND_EXTENSIONS) + assert problems == () + assert isinstance(array.codecs[0], ShardingIndexedCodec) + assert isinstance(array.codecs[1], Crc32cCodec) + assert array.to_json() == document + canonical = array.canonical().to_json()["codecs"] + assert isinstance(canonical, tuple) + assert canonical[1] == "crc32c" + + +def test_the_document_writes_a_changed_member_inside_the_envelope_it_read() -> None: + # What changed is what changes: the member is the entity's, the + # spelling around it the document's. An entity put in by hand has + # no spelling on record and writes itself. + document = { + "codecs": ( + "bytes", + {"name": "gzip", "configuration": {"level": 1}, "must_understand": True}, + ), + } + array, problems = read_array_v3(document, CORE_AND_EXTENSIONS) + assert problems == () + bytes_codec, gzip = array.codecs + assert isinstance(gzip, GzipCodec) + changed = dataclasses.replace(array, codecs=(bytes_codec, gzip.with_configuration(level=5))) + assert changed.to_json()["codecs"] == ( + "bytes", + {"name": "gzip", "configuration": {"level": 5}, "must_understand": True}, + ) + swapped = dataclasses.replace(array, codecs=(bytes_codec, Crc32cCodec())) + assert swapped.to_json()["codecs"] == ("bytes", "crc32c") + + +def test_an_unreadable_member_is_not_judged_by_its_default() -> None: + # `shuffle` could not be read, so it falls back to `noshuffle`, under + # which `typesize` means nothing. The absent `typesize` must not be + # reported as required -- that would be the default talking. + document = { + "zarr_format": 3, + "node_type": "array", + "shape": (4,), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (4,)}}, + "chunk_key_encoding": "default", + "codecs": ( + {"name": "bytes", "configuration": {"endian": "little"}}, + { + "name": "blosc", + "configuration": { + "cname": "lz4", + "clevel": 5, + "shuffle": 7, + "blocksize": 0, + }, + }, + ), + } + problems = validate_array_metadata_v3(document) + assert [problem.loc for problem in problems] == [("codecs", 1, "configuration", "shuffle")] + + +# Entities whose written form and canonical form differ, or could. +FAITHFUL: dict[str, tuple[type[MetadataEntity], object]] = { + "rectilinear-expanded": ( + ChunkGridEntity, + { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": ((32, 32, 32),)}, + }, + ), + "rectilinear-encoded": ( + ChunkGridEntity, + { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": (((32, 3),),)}, + }, + ), + "blosc-ignored-typesize": ( + CodecEntity, + { + "name": "blosc", + "configuration": { + "cname": "zstd", + "clevel": 5, + "shuffle": "noshuffle", + "blocksize": 0, + "typesize": 4, + }, + }, + ), + "raw-bytes-padded": (DataTypeEntity, "r008"), + "scale-offset-scalar": ( + CodecEntity, + {"name": "scale_offset", "configuration": {"offset": 2, "scale": 0.5}}, + ), + "struct-nested": ( + DataTypeEntity, + { + "name": "struct", + "configuration": { + "fields": ({"name": "a", "data_type": "uint8"},), + }, + }, + ), +} + + +@pytest.mark.parametrize(("field", "written"), FAITHFUL.values(), ids=list(FAITHFUL)) +def test_to_json_writes_back_what_was_read(field: type[MetadataEntity], written: object) -> None: + # Serialization is not canonicalization. A reader that reads a + # document and writes it back must not change bytes it was not asked + # to change -- `canonical()` is where you ask. + entity, problems = resolve(written, field, CORE_AND_EXTENSIONS) + assert problems == () + assert isinstance(entity, MetadataEntity) + assert entity.to_json() == written + + +def test_canonical_is_what_simplifies() -> None: + encoded, _ = resolve( + { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": ((32, 32, 32),)}, + }, + ChunkGridEntity, + CORE_AND_EXTENSIONS, + ) + assert isinstance(encoded, MetadataEntity) + assert encoded.canonical().to_json() == { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": (((32, 3),),)}, + } + blosc, _ = resolve( + { + "name": "blosc", + "configuration": { + "cname": "zstd", + "clevel": 5, + "shuffle": "noshuffle", + "blocksize": 0, + "typesize": 4, + }, + }, + CodecEntity, + CORE_AND_EXTENSIONS, + ) + assert isinstance(blosc, BloscCodec) + assert "typesize" not in configuration_of(blosc.canonical().to_json()) + + +def test_canonical_reaches_a_contained_entity() -> None: + shard, _ = resolve( + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": (4,), + "codecs": ( + {"name": "bytes", "configuration": {"endian": "little"}}, + { + "name": "blosc", + "configuration": { + "cname": "zstd", + "clevel": 5, + "shuffle": "noshuffle", + "blocksize": 0, + "typesize": 4, + }, + }, + ), + "index_codecs": ({"name": "bytes", "configuration": {"endian": "little"}},), + }, + }, + CodecEntity, + CORE_AND_EXTENSIONS, + ) + assert isinstance(shard, ShardingIndexedCodec) + inner = entry_at(shard.canonical().to_json(), "configuration", "codecs", 1) + assert "typesize" not in configuration_of(inner) + + +def test_error_an_explicit_null_scalar_is_refused() -> None: + # `null` is a value the document wrote, distinct from absence -- and + # no data type admits it as a scalar, so the codec cannot be built. + codec, problems = resolve( + {"name": "scale_offset", "configuration": {"offset": None}}, + CodecEntity, + CORE_AND_EXTENSIONS, + ) + assert codec is not None + assert not isinstance(codec, MetadataEntity) + assert [problem.loc for problem in problems] == [("configuration", "offset")] + + +# (an entity whose configuration holds a mutable JSON value) +MUTABLE_MEMBERS: dict[str, tuple[type[MetadataEntity], object]] = { + "scale-offset-object": ( + CodecEntity, + {"name": "scale_offset", "configuration": {"offset": {"a": 1}}}, + ), + "cast-value-scalar-map": ( + CodecEntity, + { + "name": "cast_value", + "configuration": {"data_type": "int8", "scalar_map": {"encode": (("NaN", 0),)}}, + }, + ), + "struct-fields": ( + DataTypeEntity, + {"name": "struct", "configuration": {"fields": ({"name": "a", "data_type": "uint8"},)}}, + ), +} + + +@pytest.mark.parametrize(("field", "written"), MUTABLE_MEMBERS.values(), ids=list(MUTABLE_MEMBERS)) +def test_to_json_shares_no_mutable_state_with_the_entity( + field: type[MetadataEntity], written: object +) -> None: + # The model layer has this test; the entity layer did not, and handed + # out its own dict -- so a caller mutating the document it was given + # mutated a frozen entity. + entity, problems = resolve(written, field, CORE_AND_EXTENSIONS) + assert problems == () + assert isinstance(entity, MetadataEntity) + baseline = copy.deepcopy(entity.to_json()) + handed_out = entity.to_json() + configuration = configuration_of(handed_out) + assert isinstance(configuration, dict) + for key in list(configuration): + value = configuration[key] + if isinstance(value, dict): + value["INJECTED"] = "boom" + else: + configuration[key] = "clobbered" + assert entity.to_json() == baseline + + +def test_a_member_the_entity_does_not_model_is_not_written_back() -> None: + # `unknown_key` is survivable so that one stray member cannot hide + # every other finding about its entity -- a concession about + # reporting, not a promise to carry the member. The entity holds what + # it models, so writing back drops it. + entry = { + "name": "blosc", + "configuration": { + "cname": "zstd", + "clevel": 5, + "shuffle": "shuffle", + "typesize": 2, + "blocksize": 0, + "typo_key": 1, + }, + } + codec, problems = resolve(entry, CodecEntity, CORE_AND_EXTENSIONS) + assert [(p.loc, p.kind) for p in problems] == [(("configuration", "typo_key"), "unknown_key")] + assert isinstance(codec, BloscCodec) + assert "typo_key" not in configuration_of(codec.to_json()) + + +def test_the_fail_fast_reader_refuses_a_member_it_would_drop() -> None: + # Which is why dropping it is survivable: the reader that does not + # hand back problems does not hand back the entity either. + document = { + "zarr_format": 3, + "node_type": "array", + "shape": (4,), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2,)}}, + "chunk_key_encoding": "default", + "codecs": ("bytes", {"name": "gzip", "configuration": {"level": 1, "typo_key": 1}}), + } + with pytest.raises(MetadataValidationError) as caught: + ArrayDocumentV3.from_json(cast("Any", document)) + assert (("codecs", 1, "configuration", "typo_key"), "unknown_key") in { + (problem.loc, problem.kind) for problem in caught.value.problems + } + + +# A storage transformer: the one extension point nothing in the package +# models, so the only way to reach it is to register one. +@dataclasses.dataclass(frozen=True) +class AcmeShardCacheOptions(Configuration): + verbose: bool | UNSET = UNSET + + +@dataclasses.dataclass(frozen=True) +class AcmeShardCache(StorageTransformerEntity): + """A third-party storage transformer with a member canonical form drops.""" + + configuration: AcmeShardCacheOptions + + identifier: ClassVar[str] = "acme.shard_cache" + + def canonical(self) -> Self: + return self.with_configuration(verbose=UNSET) + + +def test_the_document_writes_itself_back_and_canonical_reaches_every_point() -> None: + # `to_json` is faithful, entities included; `canonical` walks every + # field that holds an entity -- `storage_transformers` among them, + # which the hand-written walk it replaces never reached. + scope = CORE_AND_EXTENSIONS.extended_with(AcmeShardCache) + document = { + "zarr_format": 3, + "node_type": "array", + "shape": (4,), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2,)}}, + "chunk_key_encoding": "default", + "codecs": ( + "bytes", + { + "name": "blosc", + "configuration": { + "cname": "zstd", + "clevel": 5, + "shuffle": "noshuffle", + "typesize": 4, + "blocksize": 0, + }, + }, + ), + "storage_transformers": ({"name": "acme.shard_cache", "configuration": {"verbose": True}},), + "dimension_names": (None,), + } + array = ArrayDocumentV3.from_json(document, context=scope) + assert array.to_json() == document + canonical = array.canonical().to_json() + assert canonical["codecs"] == ( + "bytes", + { + "name": "blosc", + "configuration": {"cname": "zstd", "clevel": 5, "shuffle": "noshuffle", "blocksize": 0}, + }, + ) + assert canonical["storage_transformers"] == ("acme.shard_cache",) + assert "dimension_names" not in canonical + + +def test_the_fields_are_the_public_configuration_type() -> None: + # Each entity module's `*Configuration` TypedDict is the public JSON + # type of what the entity holds; the fields are what it reads and + # writes. Nothing else ties the two, so this does: same keys, same + # requiredness. An entity of no members has no such type. + for cls in CORE_AND_EXTENSIONS.entities(): + record = field_hints(cls).get("configuration") + hints = field_hints(record) if isinstance(record, type) else {} + members = {key for key, annotation in hints.items() if not is_from_name(annotation)} + required = {key for key in members if not is_optional(hints[key])} + declared = [ + value + for name, value in vars(sys.modules[cls.__module__]).items() + if name.endswith("Configuration") and is_typeddict(value) + ] + if len(declared) == 0: + assert members == set(), cls + continue + (configuration,) = declared + keys = get_type_hints(configuration, include_extras=True) + assert set(keys) == members, cls + assert { + key for key, annotation in keys.items() if not is_not_required(annotation) + } == required, cls + + +def test_a_bare_name_entity_holds_the_empty_configuration() -> None: + # One rule for every entity: a name and a record. The spec makes an + # absent configuration and an empty one the same, so a bare name + # holds the empty record, reads either spelling, and writes the name. + assert Crc32cCodec().configuration == Configuration() + for spelling in ("crc32c", {"name": "crc32c"}, {"name": "crc32c", "configuration": {}}): + assert resolve(spelling, CodecEntity, CORE_AND_EXTENSIONS) == (Crc32cCodec(), ()) + assert Crc32cCodec().to_json() == "crc32c" + + +def test_error_a_bare_name_entity_has_no_member_to_change() -> None: + with pytest.raises(TypeError, match="unexpected keyword argument 'level'"): + Crc32cCodec().with_configuration(level=1) + + +def test_error_an_entity_of_another_kind_is_invalid_not_out_of_scope() -> None: + # `transpose` is in scope, so it is not for another reader to + # resolve; it is an array->array codec where a bytes->bytes one goes. + value = {"name": "transpose", "configuration": {"order": (0,)}} + entity, problems = resolve(value, BytesBytesCodec, CORE_AND_EXTENSIONS, ("codecs", 2)) + assert entity == Opaque(value, "invalid") + assert [(problem.loc, problem.kind, problem.message) for problem in problems] == [ + ( + ("codecs", 2), + "invalid_value", + "expected a BytesBytesCodec, got 'transpose', an ArrayArrayCodec", + ) + ] + + +@pytest.mark.parametrize( + ("annotation", "shape"), + [ + (Literal[True], "bool"), + (Literal[1, 2], "int"), + (Literal["a", "b"], "str"), + (Literal[0, "auto"], None), + ], +) +def test_a_literal_has_the_shape_of_its_values(annotation: object, shape: str | None) -> None: + # `True` is an integer to Python and a boolean to JSON; a literal + # mixing shapes has none, so a union tries it for any value. + assert shape_of(annotation) == shape + + +def test_a_literal_of_mixed_shapes_is_described() -> None: + assert describe(Literal[0, "auto"]) == "one of ('auto', 0)" + + +@pytest.mark.parametrize( + ("annotation", "expected"), + [ + ("ClassVar[str]", True), + ("ClassVar", True), + ("typing.ClassVar[str]", True), + ("t.ClassVar[str]", True), + ("str", False), + ("ClassVarLike[str]", False), + ("Final[ClassVar[str]]", False), + ], +) +def test_a_class_var_is_read_from_any_spelling(annotation: str, expected: bool) -> None: + # Under PEP 649 the annotation is a string, spelled however the + # module imported the name. + assert is_class_var(annotation) is expected + + +def test_field_hints_is_shared_and_so_read_only() -> None: + hints = field_hints(GzipCodec) + assert field_hints(GzipCodec) is hints + with pytest.raises(TypeError, match="does not support item assignment"): + cast("dict[str, object]", hints)["configuration"] = int + + +def test_error_a_fixed_tuple_of_the_wrong_length_is_not_written() -> None: + write = writer_for(tuple[int, str], no_writer_leaf) + assert write is not None + with pytest.raises(TypeError, match="is not a JSON value"): + write((1,)) + + +def test_error_a_record_refuses_a_member_of_the_wrong_type() -> None: + # The runtime half of the record's type: pyright sees a value written + # by hand, the constructor sees the rest. + with pytest.raises(MetadataValidationError) as caught: + GzipOptions(level="high") # pyright: ignore[reportArgumentType] + assert [(p.loc, p.kind, p.message) for p in caught.value.problems] == [ + (("level",), "invalid_type", "expected an integer, got 'high'") + ] + + +def test_error_a_record_reports_every_mistyped_member() -> None: + with pytest.raises(MetadataValidationError) as caught: + BloscOptions(cname=1, clevel="x", shuffle="shuffle", blocksize=0, typesize=4) # pyright: ignore[reportArgumentType] + assert [p.loc for p in caught.value.problems] == [("cname",), ("clevel",)] + + +def test_error_a_required_member_may_not_be_unset() -> None: + with pytest.raises(MetadataValidationError) as caught: + GzipOptions(level=UNSET) # pyright: ignore[reportArgumentType] + assert [(p.loc, p.kind) for p in caught.value.problems] == [(("level",), "invalid_type")] + + +def test_error_with_configuration_refuses_a_member_of_the_wrong_type() -> None: + # `**changes` is typed `object`, so this is the path pyright cannot + # see; the record's constructor sees it, and `5.0` is not an integer. + codec = GzipCodec(GzipOptions(level=1)) + with pytest.raises(MetadataValidationError) as caught: + codec.with_configuration(level=5.0) + assert [(p.loc, p.kind) for p in caught.value.problems] == [(("level",), "invalid_type")] + + +def test_error_a_member_holding_an_entity_holds_one() -> None: + # A shard's pipelines are entities, not the JSON that names them. + with pytest.raises(MetadataValidationError) as caught: + ShardingIndexedOptions( + chunk_shape=(2,), + codecs=("bytes",), # pyright: ignore[reportArgumentType] + index_codecs=(Crc32cCodec(),), + ) + assert [(p.loc, p.kind, p.message) for p in caught.value.problems] == [ + ( + ("codecs", 0), + "invalid_type", + "expected an entity of CodecEntity or an Opaque, got 'bytes'", + ) + ] + + +def test_error_a_member_holding_a_record_holds_one_that_is_well_typed() -> None: + # A record inside a record is checked in turn, and located inside. + with pytest.raises(MetadataValidationError) as caught: + StructOptions(fields=(("a", Int8DataType()),)) # pyright: ignore[reportArgumentType] + assert [(p.loc, p.kind) for p in caught.value.problems] == [(("fields", 0), "invalid_type")] + with pytest.raises(MetadataValidationError) as caught: + StructOptions(fields=(StructFieldComponent(name=1, data_type=Int8DataType()),)) # pyright: ignore[reportArgumentType] + assert [(p.loc, p.kind) for p in caught.value.problems] == [ + (("fields", 0, "name"), "invalid_type") + ] + + +def test_error_an_entity_refuses_a_record_that_is_not_its_own() -> None: + # The runtime half of the entity's type: the record's members are + # well-typed, and they are blosc's, which gzip would write under its + # own name. + with pytest.raises(MetadataValidationError) as caught: + GzipCodec(BloscOptions(cname="zstd", clevel=5, shuffle="noshuffle", blocksize=0)) # pyright: ignore[reportArgumentType] + assert [(p.loc, p.kind, p.message) for p in caught.value.problems] == [ + ((), "invalid_type", "expected a GzipOptions configuration, got BloscOptions") + ] + + +def test_error_a_carried_name_is_a_string() -> None: + with pytest.raises(MetadataValidationError) as caught: + RawBytesDataType(16) # pyright: ignore[reportArgumentType] + assert [(p.loc, p.kind) for p in caught.value.problems] == [((), "invalid_type")] + + +def test_error_an_opaque_carries_a_reason_the_reader_gives() -> None: + with pytest.raises(MetadataValidationError) as caught: + Opaque({"name": "acme.x"}, "shrug") # pyright: ignore[reportArgumentType] + assert [(p.loc, p.kind) for p in caught.value.problems] == [(("reason",), "invalid_type")] + + +def test_error_a_document_holds_an_entity_of_each_field_s_kind() -> None: + # A document built by hand is held to what `read_array_v3` builds: + # each field an entity of its kind, or an `Opaque`. + with pytest.raises(MetadataValidationError) as caught: + ArrayDocumentV3( + document={}, + data_type=GzipCodec(GzipOptions(level=1)), # pyright: ignore[reportArgumentType] + chunk_grid=Opaque(None, "invalid"), + chunk_key_encoding=Opaque(None, "invalid"), + codecs=(Int8DataType(),), # pyright: ignore[reportArgumentType] + storage_transformers=(), + ) + assert [(p.loc, p.kind) for p in caught.value.problems] == [ + (("data_type",), "invalid_type"), + (("codecs", 0), "invalid_type"), + ] + + +def test_create_unchecked_is_the_one_way_around_the_constructors() -> None: + # For a caller that has just made the checks itself, as `coerce` + # has: the record's members are not type-checked, the entity's + # record and rules are not asked. + record = GzipOptions.create_unchecked(level="high") + assert record.level == "high" + codec = GzipCodec.create_unchecked(configuration=GzipOptions(level=99)) + assert codec.configuration.level == 99 + assert codec == GzipCodec.create_unchecked(configuration=GzipOptions(level=99)) + read, problems = GzipCodec.coerce({"name": "gzip", "configuration": {"level": 1}}, CORE) + assert problems == () + assert read == GzipCodec(GzipOptions(level=1)) + + +def test_the_first_layer_refines_json_and_judges_the_shape() -> None: + # Needs nothing but the value: arrays become tuples, the structural + # problems come with the document, and the next layer reads what it + # can regardless of them. + document, problems = well_formed_array_v3( + {"zarr_format": 3, "node_type": "array", "shape": [4]} + ) + assert document is not None + assert document["shape"] == (4,) + assert ("data_type",) in {problem.loc for problem in problems} + assert all(problem.kind == "missing_key" for problem in problems) + + +def test_error_a_value_that_is_not_json_gets_only_that_verdict() -> None: + document, problems = well_formed_array_v3({"shape": (4,), "attributes": {"x": object()}}) + assert document is None + assert [(p.loc, p.kind) for p in problems] == [(("attributes", "x"), "invalid_type")] + + +def test_error_a_document_that_is_not_an_object_has_nothing_to_read() -> None: + document, problems = well_formed_array_v3([1, 2]) + assert document is None + assert [(p.loc, p.kind) for p in problems] == [((), "invalid_type")] + + +def test_error_a_field_that_is_not_json_resolves_to_an_invalid_opaque() -> None: + entity, problems = resolve( + {"name": "gzip", "configuration": {"level": object()}}, CodecEntity, CORE + ) + assert entity == Opaque(None, "invalid") + assert [(p.loc, p.kind) for p in problems] == [(("configuration", "level"), "invalid_type")] + + +def test_the_third_layer_resolves_the_pipeline() -> None: + # What a codec pipeline is built from: at each position, the array + # that reaches the codec, and a shard's two pipelines refined inside + # it. Past the array->bytes boundary there is no array. + document = { + "zarr_format": 3, + "node_type": "array", + "shape": (64, 32), + "data_type": "float32", + "fill_value": 0.0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (32, 16)}}, + "chunk_key_encoding": "default", + "codecs": ( + {"name": "transpose", "configuration": {"order": (1, 0)}}, + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": (8, 8), + "codecs": ({"name": "bytes", "configuration": {"endian": "little"}},), + "index_codecs": ( + {"name": "bytes", "configuration": {"endian": "little"}}, + "crc32c", + ), + }, + }, + {"name": "gzip", "configuration": {"level": 5}}, + ), + } + array = ArrayDocumentV3.from_json(document) + refined, problems = refine_array_v3(array) + assert problems == () + transpose, shard, gzip = refined.pipeline.stages + assert isinstance(transpose.codec, TransposeCodec) + assert transpose.incoming is refined.parts + assert refined.parts.grid.axis(0) == frozenset({32}) + assert shard.incoming is not None + assert shard.incoming.grid.axis(0) == frozenset({16}) # transposed + assert isinstance(shard.incoming.data_type, Float32DataType) + assert gzip.incoming is None + (inner_bytes,) = shard.inner["codecs"].stages + assert inner_bytes.incoming is not None + assert inner_bytes.incoming.grid.axis(1) == frozenset({8}) + assert isinstance(inner_bytes.incoming.data_type, Float32DataType) + index_bytes, index_crc = shard.inner["index_codecs"].stages + assert index_bytes.incoming is not None + assert isinstance(index_bytes.incoming.data_type, Uint64DataType) + assert index_crc.incoming is None + + +def test_an_out_of_scope_codec_receives_the_array_and_passes_nothing_on() -> None: + document = { + "shape": (4,), + "data_type": "uint8", + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2,)}}, + "codecs": ("acme.unknown", "bytes"), + } + array, problems = read_array_v3(well_formed_array_v3(document)[0] or {}, CORE_AND_EXTENSIONS) + assert problems == () + refined, _ = refine_array_v3(array) + unknown, bytes_codec = refined.pipeline.stages + assert isinstance(unknown.codec, Opaque) + assert unknown.incoming is refined.parts + assert bytes_codec.incoming is None + + +def test_error_an_opaque_holds_json() -> None: + with pytest.raises(MetadataValidationError) as caught: + Opaque({"name": object()}, "invalid") # pyright: ignore[reportArgumentType] + assert [(p.loc, p.kind) for p in caught.value.problems] == [(("json",), "invalid_type")] + assert Opaque.create_unchecked({"name": "x"}, "out_of_scope").reason == "out_of_scope" + + +def test_the_document_reads_and_writes_attributes_as_user_data() -> None: + # zarr-python writes attributes with Python's `json` defaults, so an + # xarray `_FillValue` of NaN arrives as a bare `NaN`. Attributes are + # nobody's to interpret: every layer reads one, and the faithful + # writer puts it back. + document = { + "zarr_format": 3, + "node_type": "array", + "shape": (4,), + "data_type": "float32", + "fill_value": "NaN", + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2,)}}, + "chunk_key_encoding": "default", + "codecs": ({"name": "bytes", "configuration": {"endian": "little"}},), + "attributes": {"_FillValue": math.nan}, + } + written = ArrayDocumentV3.from_json(document).to_json()["attributes"] + fill = cast("dict[str, float]", written)["_FillValue"] + assert math.isnan(fill) + + +def test_the_document_names_the_fields_a_reader_must_understand() -> None: + # Recognition is the reader's own knowledge, as in the model: the + # document keeps an unknown field, writes it back, and says which + # ones a reader must refuse to open the array without understanding. + document = { + "zarr_format": 3, + "node_type": "array", + "shape": (4,), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2,)}}, + "chunk_key_encoding": "default", + "codecs": ("bytes",), + "provenance": {"tool": "x"}, + "notes": {"must_understand": False, "text": "y"}, + "flag": True, + } + array = ArrayDocumentV3.from_json(document) + assert array.must_understand_fields == {"provenance": {"tool": "x"}, "flag": True} + assert array.to_json() == document diff --git a/packages/zarr-metadata/tests/v3/test_extension_api.py b/packages/zarr-metadata/tests/v3/test_extension_api.py new file mode 100644 index 0000000000..5267bd5f34 --- /dev/null +++ b/packages/zarr-metadata/tests/v3/test_extension_api.py @@ -0,0 +1,927 @@ +"""A third party registering its own entity, through public API only. + +Every import here is from a module without a leading underscore. If this +file has to reach into a private one, the extension surface is not real. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Annotated, ClassVar, Literal, NotRequired, Self, get_args + +import pytest +from typing_extensions import TypedDict + +from zarr_metadata.model import UNSET, MetadataValidationError +from zarr_metadata.rules import ( + canonicalize_array_metadata_v3, + validate_array_metadata_v3, +) +from zarr_metadata.v3.codec.blosc import BloscCodec, BloscOptions +from zarr_metadata.v3.codec.gzip import GzipCodec +from zarr_metadata.v3.entity import ( + CORE_AND_EXTENSIONS, + FROM_NAME, + ArrayArrayCodec, + ArrayDocumentV3, + ArrayParts, + BytesBytesCodec, + ChunkGridEntity, + ChunkKeyEncodingEntity, + CodecEntity, + Configuration, + Context, + DataTypeEntity, + Extents, + IntegerDataType, + Loc, + MetadataEntity, + Opaque, + StorageClass, + ValidationProblem, + problem, + refine_array_v3, + resolve, +) + +if TYPE_CHECKING: + from collections.abc import Iterator + +ACME_MAX_ACCELERATION = 65537 + + +@dataclass(frozen=True) +class AcmeLz4Options(Configuration): + acceleration: int | UNSET = UNSET + + def problems(self) -> Iterator[ValidationProblem]: + if self.acceleration is not UNSET and not 1 <= self.acceleration <= ACME_MAX_ACCELERATION: + yield ValidationProblem( + ("acceleration",), + f"expected an integer in [1, {ACME_MAX_ACCELERATION}], got {self.acceleration}", + "invalid_value", + ) + + +@dataclass(frozen=True) +class AcmeLz4Codec(BytesBytesCodec): + """A third-party compressor.""" + + configuration: AcmeLz4Options + + identifier: ClassVar[str] = "acme.lz4" + variable_size: ClassVar[bool] = True + + +@dataclass(frozen=True) +class AcmeFloat8DataType(DataTypeEntity): + """A third-party one-byte float.""" + + configuration: Configuration = field(default_factory=Configuration) + + identifier: ClassVar[str] = "acme.float8" + scalar_storage: ClassVar[StorageClass] = "single_byte" + twos_complement: ClassVar[bool] = False + + def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]: + return () + + +def _scope() -> Context: + return CORE_AND_EXTENSIONS.extended_with(AcmeLz4Codec, AcmeFloat8DataType) + + +SCOPE = _scope() + + +def _document(**overrides: object) -> dict[str, object]: + return { + "zarr_format": 3, + "node_type": "array", + "shape": (8,), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (8,)}}, + "chunk_key_encoding": "default", + "codecs": ({"name": "bytes", "configuration": {"endian": "little"}},), + **overrides, + } + + +def test_an_unregistered_name_is_not_judged() -> None: + # Extension openness: out of scope means unjudged, not invalid. + document = _document( + codecs=( + {"name": "bytes", "configuration": {"endian": "little"}}, + {"name": "acme.lz4", "configuration": {"acceleration": 999999}}, + ) + ) + assert validate_array_metadata_v3(document) == () + + +def test_a_registered_entity_is_judged() -> None: + document = _document( + codecs=( + {"name": "bytes", "configuration": {"endian": "little"}}, + {"name": "acme.lz4", "configuration": {"acceleration": 999999}}, + ) + ) + problems = validate_array_metadata_v3(document, context=SCOPE) + assert [problem.loc for problem in problems] == [("codecs", 1, "configuration", "acceleration")] + + +def test_a_registered_entity_joins_the_pipeline_rules() -> None: + # Declared `bytes_bytes`, so it may not precede the array->bytes codec, + # and it is variable-size, so it may not encode a shard index. + document = _document( + codecs=("acme.lz4", {"name": "bytes", "configuration": {"endian": "little"}}) + ) + problems = validate_array_metadata_v3(document, context=SCOPE) + assert [problem.loc for problem in problems] == [("codecs", 1)] + + +def test_a_registered_data_type_drives_the_codecs_around_it() -> None: + # Single-byte, so the `bytes` codec needs no endianness for it. + document = _document(data_type="acme.float8", fill_value=0, codecs=("bytes",)) + assert validate_array_metadata_v3(document, context=SCOPE) == () + + +def test_a_registered_entity_canonicalizes_itself() -> None: + document = _document( + codecs=( + {"name": "bytes", "configuration": {"endian": "little"}}, + {"name": "acme.lz4", "configuration": {}}, + ) + ) + result = canonicalize_array_metadata_v3(document, context=SCOPE) + assert result.valid is True + assert result.document["codecs"][1] == "acme.lz4" + + +def test_error_an_entity_must_say_what_it_is() -> None: + @dataclass(frozen=True) + class Nameless(BytesBytesCodec): + """A codec that forgot to say what it is.""" + + variable_size: ClassVar[bool] = True + + with pytest.raises(TypeError, match="does not declare identifier"): + CORE_AND_EXTENSIONS.extended_with(Nameless) + + +def test_the_entity_layer_answers_what_a_reader_needs() -> None: + # The questions zarr-python asks before it can read a chunk. + data_type, problems = resolve("int32", DataTypeEntity, CORE_AND_EXTENSIONS) + assert problems == () + assert isinstance(data_type, DataTypeEntity) + assert data_type.storage_class() == "multi_byte" + + grid, problems = resolve( + {"name": "regular", "configuration": {"chunk_shape": (32, 32)}}, + ChunkGridEntity, + CORE_AND_EXTENSIONS, + ) + assert problems == () + assert isinstance(grid, ChunkGridEntity) + parts = ArrayParts(grid.grid((64, 64)), data_type) + assert parts.grid.rank == 2 + assert parts.grid.axis(0) == frozenset({32}) + + +@dataclass(frozen=True) +class DefaultedOptions(Configuration): + level: int | UNSET = 3 + + +def test_an_absent_optional_member_is_read_as_unset_whatever_its_default() -> None: + # A default is for hand construction; what a document left out is + # `UNSET` in the record, so no field's default decides what a + # document said. + @dataclass(frozen=True) + class Defaulted(BytesBytesCodec): + configuration: DefaultedOptions + + identifier: ClassVar[str] = "acme.defaulted" + + variable_size: ClassVar[bool] = False + + assert Defaulted(DefaultedOptions()).configuration.level == 3 + codec, problems = resolve( + "acme.defaulted", CodecEntity, CORE_AND_EXTENSIONS.extended_with(Defaulted) + ) + assert problems == () + assert isinstance(codec, Defaulted) + assert codec.configuration.level is UNSET + + +def test_a_reader_gets_entities_or_an_exception() -> None: + array = ArrayDocumentV3.from_json(_document()) + assert isinstance(array.data_type, DataTypeEntity) + assert array.data_type.storage_class() == "single_byte" + assert refine_array_v3(array)[0].parts.grid.rank == 1 + assert [type(codec).identifier for codec in array.codecs if isinstance(codec, CodecEntity)] == [ + "bytes" + ] + + +def test_error_a_reader_gets_every_reason_at_once() -> None: + document = _document(fill_value=-1, dimension_names=("x", "y")) + with pytest.raises(MetadataValidationError) as raised: + ArrayDocumentV3.from_json(document) + assert {problem.loc for problem in raised.value.problems} == { + ("fill_value",), + ("dimension_names",), + } + + +def test_an_unmodelled_extension_is_read_not_refused() -> None: + # Openness: a name this reader does not model is not a failure. It + # arrives as `Opaque`, saying which kind of not-an-entity it is, so + # the reader can resolve it elsewhere instead of guessing. + document = _document( + codecs=( + {"name": "numcodecs.bitround", "configuration": {"keepbits": 9}}, + {"name": "bytes", "configuration": {"endian": "little"}}, + ) + ) + array = ArrayDocumentV3.from_json(document) + first = array.codecs[0] + assert isinstance(first, Opaque) + assert first.reason == "out_of_scope" + assert first.json == {"name": "numcodecs.bitround", "configuration": {"keepbits": 9}} + assert isinstance(array.codecs[1], CodecEntity) + + +def test_every_extension_point_is_an_exhaustive_two_case_union() -> None: + # The property that makes the fields narrowable: an entity of the + # right kind, or an `Opaque`. Never a bare `object`. + array = ArrayDocumentV3.from_json(_document(data_type="mycorp.decimal", fill_value=0)) + assert isinstance(array.data_type, (DataTypeEntity, Opaque)) + assert isinstance(array.chunk_grid, (ChunkGridEntity, Opaque)) + assert isinstance(array.chunk_key_encoding, (ChunkKeyEncodingEntity, Opaque)) + assert all(isinstance(codec, (CodecEntity, Opaque)) for codec in array.codecs) + + +def test_a_reader_can_choose_its_own_scope() -> None: + document = _document( + codecs=( + {"name": "bytes", "configuration": {"endian": "little"}}, + {"name": "acme.lz4", "configuration": {"acceleration": 4}}, + ) + ) + assert isinstance(ArrayDocumentV3.from_json(document).codecs[1], Opaque) + in_scope = ArrayDocumentV3.from_json(document, context=SCOPE).codecs[1] + assert isinstance(in_scope, AcmeLz4Codec) + assert in_scope.configuration.acceleration == 4 + + +def test_error_a_family_member_must_declare_what_the_family_left_open() -> None: + # `bounds` is annotated on `IntegerDataType` and bound by none of it, + # so every concrete integer type owes one. Nothing lists it: the + # requirement is read off the annotation. + @dataclass(frozen=True) + class Int24DataType(IntegerDataType): + identifier: ClassVar[str] = "acme.int24" + + with pytest.raises(TypeError, match="does not declare bounds"): + CORE_AND_EXTENSIONS.extended_with(Int24DataType) + + +# A third-party *family*: one class covering a parameterized set of names, +# the way `r` covers every raw-byte width. +ACME_FIXED_PATTERN = re.compile(r"acme\.fixed(\d+)") + + +@dataclass(frozen=True) +class AcmeFixedDataType(DataTypeEntity): + """`acme.fixedN`, a fixed-width type for every N.""" + + configuration: Configuration = field(default_factory=Configuration, kw_only=True) + data_type_name: Annotated[str, FROM_NAME] + + identifier: ClassVar[str] = "acme.fixed" + scalar_storage: ClassVar[StorageClass] = "multi_byte" + twos_complement: ClassVar[bool] = False + + @classmethod + def name_problems(cls, name: str) -> Iterator[ValidationProblem]: + match = ACME_FIXED_PATTERN.fullmatch(name) + if match is not None and int(match.group(1)) % 8 != 0: + yield ValidationProblem((), "expected a width that is a multiple of 8", "invalid_value") + + @classmethod + def accepts(cls, name: str) -> bool: + return ACME_FIXED_PATTERN.fullmatch(name) is not None + + def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]: + return () + + +def test_a_third_party_can_register_a_family() -> None: + # One class for an unbounded set of names. Nothing in the package + # holds a table of spellings: the entity registers under an invented + # identifier and `resolve` asks it, so a family is registered exactly + # like a single name. + scope = CORE_AND_EXTENSIONS.extended_with(AcmeFixedDataType) + for name in ("acme.fixed8", "acme.fixed128"): + assert scope.claimant(DataTypeEntity, name) is AcmeFixedDataType + entity, problems = resolve(name, DataTypeEntity, scope) + assert problems == () + assert isinstance(entity, AcmeFixedDataType) + assert entity.to_json() == name + # The invented identifier is not a name a document may write, and a + # near-miss is still nobody's. + assert scope.claimant(DataTypeEntity, AcmeFixedDataType.identifier) is None + assert scope.claimant(DataTypeEntity, "acme.fixed") is None + # A rule about the name lands on the field: the document has no + # configuration to locate it under. + problems = validate_array_metadata_v3( + _document(data_type="acme.fixed12", fill_value=0), context=scope + ) + assert [(p.loc, p.kind) for p in problems] == [(("data_type",), "invalid_value")] + + +@dataclass(frozen=True) +class StructuredOptions(Configuration): + inner: object + + +def test_error_a_member_needs_a_check_from_somewhere() -> None: + # An annotation outside the shapes the parser reads implies no + # parser, so the entity owes one. Silently skipping the member would + # let anything through where the field promised a type. + @dataclass(frozen=True) + class Structured(BytesBytesCodec): + configuration: StructuredOptions + + identifier: ClassVar[str] = "acme.structured" + variable_size: ClassVar[bool] = False + + with pytest.raises(TypeError, match="inner is annotated .*, which is not a shape JSON takes"): + CORE_AND_EXTENSIONS.extended_with(Structured) + + +# A third-party codec that contains another codec. +@dataclass(frozen=True) +class AcmeWrapperOptions(Configuration): + inner: CodecEntity | Opaque + + +@dataclass(frozen=True) +class AcmeWrapperCodec(BytesBytesCodec): + """A codec that applies another codec after its own step.""" + + configuration: AcmeWrapperOptions + + identifier: ClassVar[str] = "acme.wrapper" + + variable_size: ClassVar[bool] = False + + def canonical(self) -> Self: + return self.with_configuration(inner=self.configuration.inner.canonical()) + + +def test_a_third_party_entity_containing_entities_reads_them_in_scope() -> None: + # `inner: CodecEntity | Opaque` is the whole declaration of the + # reading: the inner codec is resolved in the scope the wrapper is + # read in, and its problems are located inside. Writing and + # canonicalizing it are the wrapper's own two lines. + scope = CORE_AND_EXTENSIONS.extended_with(AcmeWrapperCodec) + entry = { + "name": "acme.wrapper", + "configuration": {"inner": {"name": "gzip", "configuration": {"level": 5}}}, + } + codec, problems = resolve(entry, CodecEntity, scope) + assert problems == () + assert isinstance(codec, AcmeWrapperCodec) + inner = codec.configuration.inner + assert isinstance(inner, GzipCodec) + assert inner.configuration.level == 5 + assert codec.to_json() == entry + + # An inner codec the scope does not model stays verbatim, as anywhere. + unknown = {"name": "acme.wrapper", "configuration": {"inner": "acme.unknown"}} + codec, problems = resolve(unknown, CodecEntity, scope) + assert problems == () + assert isinstance(codec, AcmeWrapperCodec) + assert isinstance(codec.configuration.inner, Opaque) + assert codec.to_json() == unknown + + # A problem inside is located inside. + bad = { + "name": "acme.wrapper", + "configuration": {"inner": {"name": "gzip", "configuration": {"level": 99}}}, + } + _, problems = resolve(bad, CodecEntity, scope) + assert [problem.loc for problem in problems] == [ + ("configuration", "inner", "configuration", "level") + ] + + # Canonical form reaches the contained entity. + verbose = { + "name": "acme.wrapper", + "configuration": { + "inner": { + "name": "blosc", + "configuration": { + "cname": "zstd", + "clevel": 5, + "shuffle": "noshuffle", + "typesize": 4, + "blocksize": 0, + }, + } + }, + } + codec, _ = resolve(verbose, CodecEntity, scope) + assert isinstance(codec, AcmeWrapperCodec) + inner = codec.canonical().configuration.inner + assert isinstance(inner, BloscCodec) + assert inner.configuration.typesize is UNSET + + +@dataclass(frozen=True) +class AcmeFramedOptions(Configuration): + inner: CodecEntity | Opaque + frame: int | UNSET = UNSET + + +def test_canonical_is_the_entity_s_own_and_reaches_what_it_contains() -> None: + # An entity that contains an entity and rewrites its own members does + # both in one `canonical` -- the contained blosc loses the `typesize` + # that `noshuffle` ignores, and the frame of 0 that means "unframed" + # is dropped -- with nothing to call `super()` for. + @dataclass(frozen=True) + class AcmeFramedCodec(BytesBytesCodec): + configuration: AcmeFramedOptions + + identifier: ClassVar[str] = "acme.framed" + + variable_size: ClassVar[bool] = False + + def canonical(self) -> Self: + return self.with_configuration( + inner=self.configuration.inner.canonical(), + frame=UNSET if self.configuration.frame == 0 else self.configuration.frame, + ) + + blosc = BloscCodec( + BloscOptions(cname="zstd", clevel=5, shuffle="noshuffle", typesize=4, blocksize=0) + ) + framed = AcmeFramedCodec(AcmeFramedOptions(inner=blosc, frame=0)) + assert framed.canonical() == AcmeFramedCodec( + AcmeFramedOptions(inner=blosc.with_configuration(typesize=UNSET)) + ) + assert framed.configuration.inner is blosc # a transformation, not a mutation + + +@dataclass(frozen=True) +class VagueOptions(Configuration): + inner: MetadataEntity | Opaque + + +def test_error_a_nested_field_names_a_kind() -> None: + # `MetadataEntity` is of no kind, so a field typed as one could not + # be resolved through any scope. + @dataclass(frozen=True) + class Vague(BytesBytesCodec): + configuration: VagueOptions + + identifier: ClassVar[str] = "acme.vague" + variable_size: ClassVar[bool] = False + + with pytest.raises(TypeError, match="inner holds an entity but is not written as its kind"): + CORE_AND_EXTENSIONS.extended_with(Vague) + + +# The JSON types third-party entities name, at module level so their +# annotations resolve. +class AcmeLvlConfiguration(TypedDict, closed=True): + lvl: int + + +class AcmeLvlObject(TypedDict, closed=True): + name: Literal["acme.lvl"] + configuration: AcmeLvlConfiguration + must_understand: NotRequired[bool] + + +class AcmeBlockConfiguration(TypedDict, closed=True): + block: int + + +class AcmeBlockObject(TypedDict, closed=True): + name: Literal["acme.block"] + configuration: AcmeBlockConfiguration + must_understand: NotRequired[bool] + + +# A third-party rule about a member: the record's own. + + +@dataclass(frozen=True) +class AcmeBlockOptions(Configuration): + block: int + + def problems(self) -> Iterator[ValidationProblem]: + if self.block < 1 or self.block & (self.block - 1) != 0: + yield ValidationProblem( + ("block",), f"expected a power of two, got {self.block}", "invalid_value" + ) + + +@dataclass(frozen=True) +class AcmeBlockCodec(BytesBytesCodec): + """A codec whose block size must be a power of two.""" + + configuration: AcmeBlockOptions + + identifier: ClassVar[str] = "acme.block" + + variable_size: ClassVar[bool] = False + + +def test_a_rule_about_a_member_is_the_record_s_own() -> None: + # The rule runs on the typed record and reports relative to the + # configuration; `coerce` runs it to the end and locates what it + # yields in the document; the constructor stops at the first. + scope = CORE_AND_EXTENSIONS.extended_with(AcmeBlockCodec) + codec, problems = resolve( + {"name": "acme.block", "configuration": {"block": 64}}, CodecEntity, scope + ) + assert problems == () + assert isinstance(codec, AcmeBlockCodec) + _, problems = resolve({"name": "acme.block", "configuration": {"block": 6}}, CodecEntity, scope) + assert [(p.loc, p.message) for p in problems] == [ + (("configuration", "block"), "expected a power of two, got 6") + ] + # And on the constructor, the same rule. + with pytest.raises(MetadataValidationError) as caught: + AcmeBlockCodec(AcmeBlockOptions(block=6)) + assert [p.loc for p in caught.value.problems] == [("block",)] + # A member that failed its type check never reaches the rule. + _, problems = resolve( + {"name": "acme.block", "configuration": {"block": "x"}}, CodecEntity, scope + ) + assert [p.kind for p in problems] == ["invalid_type"] + + +@dataclass(frozen=True) +class AcmeRangeOptions(Configuration): + low: int + high: int + + def problems(self) -> Iterator[ValidationProblem]: + if self.low < 0: + yield ValidationProblem( + ("low",), f"expected an integer >= 0, got {self.low}", "invalid_value" + ) + if self.high < self.low: + yield ValidationProblem( + ("high",), f"expected an integer >= low, got {self.high}", "invalid_value" + ) + + +@dataclass(frozen=True) +class AcmeRangeCodec(BytesBytesCodec): + """A codec with two rules, so that one can fail after another.""" + + configuration: AcmeRangeOptions + + identifier: ClassVar[str] = "acme.range" + + variable_size: ClassVar[bool] = False + + +def test_the_constructor_stops_at_the_first_problem_and_coerce_reports_every_one() -> None: + # One method, three consumers: the entity's constructor takes the + # first problem it yields, `coerce` runs it to the end, and a reader + # with a record asks it directly, and stops or collects. + with pytest.raises(MetadataValidationError) as caught: + AcmeRangeCodec(AcmeRangeOptions(low=-1, high=-2)) + assert [p.loc for p in caught.value.problems] == [("low",)] + scope = CORE_AND_EXTENSIONS.extended_with(AcmeRangeCodec) + _, problems = resolve( + {"name": "acme.range", "configuration": {"low": -1, "high": -2}}, CodecEntity, scope + ) + assert [p.loc for p in problems] == [("configuration", "low"), ("configuration", "high")] + assert list(AcmeRangeOptions(low=0, high=1).problems()) == [] + # A reader that wants every problem of a record asks the record. + assert [p.loc for p in AcmeRangeOptions(low=-1, high=-2).problems()] == [("low",), ("high",)] + + +def test_error_an_entity_may_not_define_post_init() -> None: + # `coerce` never runs it, so a rule written there would judge a + # hand-built entity and no document; the rules go on the record. + @dataclass(frozen=True) + class Checked(BytesBytesCodec): + identifier: ClassVar[str] = "acme.checked" + variable_size: ClassVar[bool] = False + + def __post_init__(self) -> None: + return None + + with pytest.raises(TypeError, match="defines __post_init__; write its rules as `problems`"): + CORE_AND_EXTENSIONS.extended_with(Checked) + + +@dataclass(frozen=True) +class LocalizedOptions(Configuration): + # `Local` is defined inside the test, so it is not here, where this + # class's annotations resolve: the case the message is for. + inner: Local # noqa: F821 # pyright: ignore[reportUndefinedVariable] + + +def test_error_a_field_annotation_names_what_is_not_defined() -> None: + # Annotations are resolved where the class is, at registration; a + # type defined inside a function is not there, however real it is. + class Local(TypedDict, closed=True): # pyright: ignore[reportUnusedClass] + depth: int + + @dataclass(frozen=True) + class Localized(BytesBytesCodec): + configuration: LocalizedOptions + + identifier: ClassVar[str] = "acme.localized" + variable_size: ClassVar[bool] = False + + with pytest.raises(TypeError, match="a field annotation names 'Local', which is not defined"): + CORE_AND_EXTENSIONS.extended_with(Localized) + + +def test_error_a_codec_says_whether_its_output_size_is_fixed() -> None: + # A default in either direction is a verdict: a compressor that + # said nothing would be accepted as a shard-index codec. + @dataclass(frozen=True) + class Sizeless(BytesBytesCodec): + identifier: ClassVar[str] = "acme.sizeless" + + with pytest.raises(TypeError, match="does not declare variable_size"): + CORE_AND_EXTENSIONS.extended_with(Sizeless) + + +def test_a_reader_gets_structural_and_semantic_reasons_together() -> None: + with pytest.raises(MetadataValidationError) as caught: + ArrayDocumentV3.from_json(_document(attributes=5, fill_value=-1)) + assert {problem.loc for problem in caught.value.problems} == {("attributes",), ("fill_value",)} + + +def test_a_malformed_envelope_is_one_problem() -> None: + # The envelope is judged once, by the scope; the entity is not asked + # to read what is not a metadata field. + _, problems = resolve(5, CodecEntity, CORE_AND_EXTENSIONS, ("codecs", 0)) + assert [(p.loc, p.kind) for p in problems] == [(("codecs", 0), "invalid_type")] + _, problems = resolve( + {"name": "gzip", "configuration": 42}, CodecEntity, CORE_AND_EXTENSIONS, ("codecs", 0) + ) + assert [(p.loc, p.kind) for p in problems] == [(("codecs", 0, "configuration"), "invalid_type")] + + +@dataclass(frozen=True) +class AcmeSlottedOptions(Configuration): + level: int + + +def test_a_slotted_entity_is_accepted() -> None: + # `@dataclass(slots=True)` builds the class twice; registration sees + # the second, whose members are slot descriptors. + @dataclass(frozen=True, slots=True) + class AcmeSlotted(BytesBytesCodec): + configuration: AcmeSlottedOptions + + identifier: ClassVar[str] = "acme.slotted" + + variable_size: ClassVar[bool] = False + + assert AcmeSlotted(AcmeSlottedOptions(level=1)).to_json() == { + "name": "acme.slotted", + "configuration": {"level": 1}, + } + + +def test_a_bare_class_var_is_a_class_variable() -> None: + @dataclass(frozen=True) + class AcmeNoted(BytesBytesCodec): + configuration: Configuration = field(default_factory=Configuration) + identifier: ClassVar[str] = "acme.noted" + variable_size: ClassVar[bool] = False + note: ClassVar = "not a member" + + assert AcmeNoted().to_json() == "acme.noted" + + +@dataclass(frozen=True) +class AcmeScaledOptions(Configuration): + scale: float + + +def test_a_number_member_is_a_float_field() -> None: + # JSON has one number type; `float` admits an int spelled without a + # point and refuses a bool, which is what a document's `2` and `true` + # deserve. + @dataclass(frozen=True) + class AcmeScaled(ArrayArrayCodec): + configuration: AcmeScaledOptions + + identifier: ClassVar[str] = "acme.scaled" + + variable_size: ClassVar[bool] = False + + def transition(self, incoming: ArrayParts) -> ArrayParts | None: + return incoming + + scope = CORE_AND_EXTENSIONS.extended_with(AcmeScaled) + for spelled in (2, 2.5): + codec, problems = resolve( + {"name": "acme.scaled", "configuration": {"scale": spelled}}, CodecEntity, scope + ) + assert problems == () + assert isinstance(codec, AcmeScaled) + _, problems = resolve( + {"name": "acme.scaled", "configuration": {"scale": True}}, CodecEntity, scope + ) + assert [(p.loc, p.message) for p in problems] == [ + (("configuration", "scale"), "expected a number, got True") + ] + + +@dataclass(frozen=True) +class UndecoratedOptions(Configuration): + level: int + + +def test_error_an_entity_must_be_a_dataclass() -> None: + # Class creation runs before `@dataclass` and cannot see it missing; + # registration can, and says so instead of the first `coerce` failing + # with the base class's `__init__`. + class Undecorated(BytesBytesCodec): + configuration: UndecoratedOptions + + identifier: ClassVar[str] = "acme.undecorated" + + with pytest.raises(TypeError, match="not a dataclass; decorate it with @dataclass"): + CORE_AND_EXTENSIONS.extended_with(Undecorated) + + +@dataclass(frozen=True) +class ClosedOptions(Configuration): + inner: CodecEntity + + +def test_error_a_nested_field_admits_opaque() -> None: + # What the field holds when the inner name is out of scope. + @dataclass(frozen=True) + class Closed(BytesBytesCodec): + configuration: ClosedOptions + + identifier: ClassVar[str] = "acme.closed" + variable_size: ClassVar[bool] = False + + with pytest.raises(TypeError, match="inner holds an entity but is not written as its kind"): + CORE_AND_EXTENSIONS.extended_with(Closed) + + +def test_error_an_array_array_codec_defines_transition() -> None: + # `transition` is abstract on the kind; a codec that leaves it so is + # refused where it is first used, with what to write. + @dataclass(frozen=True) + class Silent(ArrayArrayCodec): + identifier: ClassVar[str] = "acme.silent" + variable_size: ClassVar[bool] = False + + with pytest.raises( + TypeError, match="does not define transition, which its base leaves abstract" + ): + CORE_AND_EXTENSIONS.extended_with(Silent) + + +def test_error_a_codec_is_of_a_kind() -> None: + # The kind classes say what a codec does to the array; registration + # refuses one that skipped them. + @dataclass(frozen=True) + class Kindless(CodecEntity): + identifier: ClassVar[str] = "acme.kindless" + + with pytest.raises( + TypeError, match="subclasses CodecEntity directly; subclass ArrayArrayCodec" + ): + CORE_AND_EXTENSIONS.extended_with(Kindless) + + +def test_error_a_data_type_judges_its_fill_values() -> None: + # Abstract, so that accepting every fill value is said, not defaulted. + @dataclass(frozen=True) + class Lax(DataTypeEntity): + identifier: ClassVar[str] = "acme.lax" + scalar_storage: ClassVar[StorageClass] = "single_byte" + twos_complement: ClassVar[bool] = False + + with pytest.raises(TypeError, match="does not define fill_value_problems"): + CORE_AND_EXTENSIONS.extended_with(Lax) + + +def test_error_a_data_type_must_say_whether_it_wraps() -> None: + # `cast_value`'s `out_of_range: "wrap"` is defined only for two's + # complement integers. Defaulting either way would answer for a new + # data type silently, and one direction accepts an undefined cast. + @dataclass(frozen=True) + class Undecided(DataTypeEntity): + identifier: ClassVar[str] = "acme.undecided" + scalar_storage: ClassVar[StorageClass] = "single_byte" + + def fill_value_problems( + self, value: object, loc: Loc = () + ) -> tuple[ValidationProblem, ...]: + return () + + with pytest.raises(TypeError, match="does not declare twos_complement"): + CORE_AND_EXTENSIONS.extended_with(Undecided) + + +def test_error_a_list_of_problem_tuples_is_refused() -> None: + # `problem()` returns a one-element tuple; a list of those would pass + # the constructor and fail inside `coerce`, far from the mistake. + with pytest.raises(TypeError, match="collect with `extend`, not `append`"): + MetadataValidationError([problem(("a",), "bad a")]) # pyright: ignore[reportArgumentType] + + +@dataclass(frozen=True) +class NestedRecordOptions(Configuration): + depth: int + + +@dataclass(frozen=True) +class NestingOptions(Configuration): + inner: NestedRecordOptions + + +def test_error_a_member_may_not_be_a_configuration() -> None: + # `problems` is asked of the entity's configuration and of nothing + # inside it; a `Configuration` nested there would carry rules nothing + # runs. + @dataclass(frozen=True) + class Nesting(BytesBytesCodec): + configuration: NestingOptions + + identifier: ClassVar[str] = "acme.nesting" + variable_size: ClassVar[bool] = False + + with pytest.raises(TypeError, match="a Configuration, whose rules nothing would ask"): + CORE_AND_EXTENSIONS.extended_with(Nesting) + + +@dataclass(frozen=True) +class CheckedOptions(Configuration): + level: int + + def __post_init__(self) -> None: + return None + + +def test_error_a_record_may_not_define_post_init() -> None: + # It would stop at the first problem where `coerce` reports every one. + @dataclass(frozen=True) + class RecordChecked(BytesBytesCodec): + configuration: CheckedOptions + + identifier: ClassVar[str] = "acme.record_checked" + variable_size: ClassVar[bool] = False + + with pytest.raises( + TypeError, match="CheckedOptions defines __post_init__; write its rules as `problems`" + ): + CORE_AND_EXTENSIONS.extended_with(RecordChecked) + + +@dataclass(frozen=True) +class Window: + start: int + + def __post_init__(self) -> None: + return None + + +@dataclass(frozen=True) +class WindowedOptions(Configuration): + window: Window + + +def test_error_a_nested_record_may_not_define_post_init() -> None: + # A plain record is data, built whenever its keys read; a rule about + # it belongs with the other rules, in the configuration's `problems`. + @dataclass(frozen=True) + class Windowed(BytesBytesCodec): + configuration: WindowedOptions + + identifier: ClassVar[str] = "acme.windowed" + variable_size: ClassVar[bool] = False + + with pytest.raises(TypeError, match="Window defines __post_init__; a record is plain data"): + CORE_AND_EXTENSIONS.extended_with(Windowed) + + +def test_extents_is_a_type_an_extension_can_write() -> None: + # `Extents | None` in a third party's annotation needs a real alias, + # not a string one. + assert get_args(Extents | None) == (Extents, type(None)) diff --git a/packages/zarr-metadata/tests/v3/test_fill_values.py b/packages/zarr-metadata/tests/v3/test_fill_values.py new file mode 100644 index 0000000000..f0f97b5057 --- /dev/null +++ b/packages/zarr-metadata/tests/v3/test_fill_values.py @@ -0,0 +1,105 @@ +"""What each data type accepts as a fill value. + +The knowledge used to live in a table keyed by data type name; now each +type answers for itself, so these exercise the families directly rather +than through a document. +""" + +from __future__ import annotations + +import pytest + +from zarr_metadata.v3._registry import CORE_AND_EXTENSIONS +from zarr_metadata.v3.entity import DataTypeEntity, resolve + +# (data type metadata, a fill value it accepts) +ACCEPTED: dict[str, tuple[object, object]] = { + "bool": ("bool", True), + "int8-low": ("int8", -128), + "uint64-high": ("uint64", 2**64 - 1), + "float64-number": ("float64", 1.5), + "float64-integer": ("float64", 1), + "float32-special": ("float32", "NaN"), + "float16-hex": ("float16", "0x3e00"), + "complex64-pair": ("complex64", (1.0, "-Infinity")), + "string": ("string", "hello"), + "bytes-base64": ("bytes", "aGk="), + "bytes-array": ("bytes", (1, 2, 3)), + "raw-exact-width": ("r16", (0, 255)), + "time-integer": ( + {"name": "numpy.datetime64", "configuration": {"unit": "s", "scale_factor": 1}}, + -1, + ), + "time-nat": ( + {"name": "numpy.timedelta64", "configuration": {"unit": "s", "scale_factor": 1}}, + "NaT", + ), + "struct": ( + {"name": "struct", "configuration": {"fields": ({"name": "a", "data_type": "uint8"},)}}, + {"a": 7}, + ), +} + +# (data type metadata, a fill value it rejects, part of the reason) +REJECTED: dict[str, tuple[object, object, str]] = { + "bool-integer": ("bool", 1, "expected a boolean"), + "int8-above-range": ("int8", 128, "[-128, 127]"), + "uint8-negative": ("uint8", -1, "[0, 255]"), + "float64-not-a-number": ("float64", "nope", "hex string"), + "float64-boolean": ("float64", True, "expected a number or string"), + "complex64-single": ("complex64", 1.0, "[real, imag] pair"), + "complex64-bad-part": ("complex64", (1.0, "nope"), "invalid component"), + "string-integer": ("string", 3, "expected a string"), + "bytes-bad-base64": ("bytes", "!!", "standard-alphabet base64"), + "raw-wrong-width": ("r16", (1, 2, 3), "expected 2 byte values"), + "raw-byte-out-of-range": ("r8", (256,), "[0, 255]"), + "time-not-integer": ( + {"name": "numpy.datetime64", "configuration": {"unit": "s", "scale_factor": 1}}, + 1.5, + "signed 64-bit integer or 'NaT'", + ), + "struct-missing-field": ( + {"name": "struct", "configuration": {"fields": ({"name": "a", "data_type": "uint8"},)}}, + {}, + "missing fill value", + ), + "struct-unknown-field": ( + {"name": "struct", "configuration": {"fields": ({"name": "a", "data_type": "uint8"},)}}, + {"a": 1, "b": 2}, + "unknown struct fill field", + ), +} + + +def _data_type(metadata: object) -> DataTypeEntity: + entity, problems = resolve(metadata, DataTypeEntity, CORE_AND_EXTENSIONS) + assert problems == (), problems + assert isinstance(entity, DataTypeEntity), metadata + return entity + + +@pytest.mark.parametrize(("metadata", "fill"), ACCEPTED.values(), ids=list(ACCEPTED)) +def test_accepts(metadata: object, fill: object) -> None: + assert _data_type(metadata).fill_value_problems(fill) == () + + +@pytest.mark.parametrize(("metadata", "fill", "reason"), REJECTED.values(), ids=list(REJECTED)) +def test_error_rejects(metadata: object, fill: object, reason: str) -> None: + problems = _data_type(metadata).fill_value_problems(fill) + assert problems, f"expected {fill!r} to be rejected" + assert any(reason in problem.message for problem in problems), problems + + +def test_error_a_malformed_raw_name_has_no_entity_to_ask() -> None: + # `r12` is not a width, so the data type does not exist and there is + # nothing to put a fill value to. + entity, problems = resolve("r12", DataTypeEntity, CORE_AND_EXTENSIONS) + assert not isinstance(entity, DataTypeEntity) + assert [problem.message for problem in problems] == [ + "Expected 'r' where N is a positive multiple of 8, got 'r12'" + ] + + +def test_an_unmodelled_data_type_judges_nothing() -> None: + # Extension openness: a fill value we cannot interpret is not wrong. + assert CORE_AND_EXTENSIONS.claimant(DataTypeEntity, "mycorp.decimal") is None diff --git a/packages/zarr-metadata/tests/v3/test_resolve.py b/packages/zarr-metadata/tests/v3/test_resolve.py new file mode 100644 index 0000000000..83d9f533b2 --- /dev/null +++ b/packages/zarr-metadata/tests/v3/test_resolve.py @@ -0,0 +1,128 @@ +"""How a name reaches the entity that answers for it. + +`Context.resolve` tries the name as a key and, when nothing is keyed by +it, asks each entity at that point whether the name is one of its own -- +which is how a family such as `r` covers an unbounded set of names +from one registration. The examples pin the cases that matter; the +properties cover the family, which no example-based test can sample. +""" + +from __future__ import annotations + +import pytest +from hypothesis import given +from hypothesis import strategies as st + +from zarr_metadata.rules import validate_array_metadata_v3 +from zarr_metadata.v3.codec.blosc import BloscCodec +from zarr_metadata.v3.data_type.numpy_datetime64 import NumpyDatetime64DataType +from zarr_metadata.v3.data_type.raw import RAW_BYTES_FAMILY, RawBytesDataType +from zarr_metadata.v3.data_type.uint8 import Uint8DataType +from zarr_metadata.v3.entity import ( + CORE_AND_EXTENSIONS, + ChunkGridEntity, + CodecEntity, + DataTypeEntity, + MetadataEntity, +) + +# (field, name, the entity that answers for it — None when nothing does) +RESOLUTIONS: dict[str, tuple[type[MetadataEntity], str, type[MetadataEntity] | None]] = { + "plain-dtype": (DataTypeEntity, "uint8", Uint8DataType), + "dotted-dtype": (DataTypeEntity, "numpy.datetime64", NumpyDatetime64DataType), + "raw-8": (DataTypeEntity, "r8", RawBytesDataType), + "raw-24": (DataTypeEntity, "r24", RawBytesDataType), + # A malformed member reaches the family too: a misspelling of + # something we model must be reported as such, not pass as an unknown + # third-party extension. + "raw-not-multiple-of-8": (DataTypeEntity, "r12", RawBytesDataType), + "raw-zero": (DataTypeEntity, "r0", RawBytesDataType), + # Tables are per point, so the family cannot be reached from another. + "raw-shaped-codec-name": (CodecEntity, "r8", None), + "codec": (CodecEntity, "blosc", BloscCodec), + "unknown": (CodecEntity, "zfpy", None), + # The family's key is invented, so no document may write it. + "the-family-key-itself": (DataTypeEntity, RAW_BYTES_FAMILY, None), +} + + +@pytest.mark.parametrize(("field", "name", "expected"), RESOLUTIONS.values(), ids=list(RESOLUTIONS)) +def test_a_name_resolves_to_the_entity_that_answers_for_it( + field: type[MetadataEntity], name: str, expected: type[MetadataEntity] | None +) -> None: + assert CORE_AND_EXTENSIONS.claimant(field, name) is expected + + +@given(width=st.integers(min_value=0, max_value=2**32)) +def test_every_numeric_r_spelling_resolves_to_the_family(width: int) -> None: + # Including malformed widths (0, 12, anything not a multiple of 8): + # the family claims a name by grammar shape, not by validity, so a + # misspelled member of a family we model is reported as a misspelling + # rather than passing as an unknown third-party extension. + assert CORE_AND_EXTENSIONS.claimant(DataTypeEntity, f"r{width}") is RawBytesDataType + + +OTHER_KINDS: tuple[type[MetadataEntity], ...] = (CodecEntity, ChunkGridEntity) + + +@given(width=st.integers(min_value=0, max_value=2**32), field=st.sampled_from(OTHER_KINDS)) +def test_r_shaped_names_resolve_to_nothing_outside_data_types( + width: int, field: type[MetadataEntity] +) -> None: + # The family belongs to `data_type`; a codec that happens to be named + # `r8` must not reach it. + assert CORE_AND_EXTENSIONS.claimant(field, f"r{width}") is None + + +# The scan `resolve` falls back to asks every entity, so a name no entity +# claims has to come back as nothing however many are registered. +_UNCLAIMED = st.text(min_size=1).filter( + lambda name: ( + not (name.startswith("r") and name[1:].isdigit()) + and name not in CORE_AND_EXTENSIONS.tables[DataTypeEntity] + ) +) + + +@given(name=_UNCLAIMED) +def test_a_name_no_entity_claims_resolves_to_nothing(name: str) -> None: + assert CORE_AND_EXTENSIONS.claimant(DataTypeEntity, name) is None + + +def test_squatted_names_are_judged_against_the_definition_they_squat() -> None: + # Zarr identifiers are registry-allocated. A private codec named + # `bytes` has left the compatibility contract, and saying so is the + # correct answer rather than a limitation, so nothing here defends + # against collisions. + document = { + "zarr_format": 3, + "node_type": "array", + "shape": (4, 4), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2, 2)}}, + "chunk_key_encoding": "default", + "codecs": ({"name": "bytes", "configuration": {"width": 7}},), + } + problems = validate_array_metadata_v3(document) + assert [(p.loc, p.kind) for p in problems] == [ + (("codecs", 0, "configuration", "width"), "unknown_key") + ] + + +def test_forging_the_family_sentinel_cannot_change_a_verdict() -> None: + # A literal "r" data type mislabels nothing: the family claims its + # names through `accepts`, not through the table key, so no validation + # verdict depends on the sentinel being unforgeable. + document = { + "zarr_format": 3, + "node_type": "array", + "shape": (4, 4), + "data_type": RAW_BYTES_FAMILY, + "fill_value": (1,), + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2, 2)}}, + "chunk_key_encoding": "default", + "codecs": ("bytes",), + } + # Unjudged as an unknown data type, exactly as any unmodelled name is. + assert validate_array_metadata_v3(document) == ()