Conversation
`zarr.open` looks for an array first and opens a group when it doesn't find one. Both steps read `zarr.json` and `.zattrs`, so a format-detecting open of a group made seven requests where five would do, a cost paid on every call against a remote store. Split the array lookup into `_probe_array_metadata`, which reports "no array here" as a value instead of an exception and hands back the documents it read. `zarr.open` passes those to `AsyncGroup.open`, which then reads only the keys it is still missing. `get_array_metadata` keeps its signature and its exceptions, and becomes a thin wrapper over the probe. Assisted-by: ClaudeCode:claude-opus-5
github-actions
Bot
added
the
needs release notes
Assisted-by: ClaudeCode:claude-opus-5
github-actions
Bot
removed
the
needs release notes
Documentation build overview
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4366 +/- ##
==========================================
+ Coverage 94.22% 94.25% +0.02%
==========================================
Files 92 92
Lines 12942 12953 +11
==========================================
+ Hits 12195 12209 +14
+ Misses 747 744 -3
🚀 New features to boost your workflow:
|
`zarr.json` wins whenever it is present, so whether the probe's metadata came from it is a fact about `docs`, not a second field that has to agree with `docs`. Also name the fetcher for what it reads: the array document set, never `.zgroup`. Assisted-by: ClaudeCode:claude-fable-5-1
A dataclass field set to None could mean the key was read and held nothing, or was never read at all, so the group open could only reuse documents under a guard at the call site. With NotRequired keys, presence is the signal: the group open takes whatever was actually read, and zarr.open passes the probe's documents along unconditionally. Assisted-by: ClaudeCode:claude-fable-5-1
The probe learns one of three things: nothing is here, an array is, or a Zarr format 3 group is. Say that with `node_type: Literal["array", "group"] | None` instead of deriving `is_array` and `from_zarr_json` from the documents after the fact. That makes the classification explicit. A `.zarray` is an array. A `zarr.json` is whatever its `node_type` says, with a missing `node_type` read as a group, which is the leniency `GroupMetadata.from_dict` already grants and so what `zarr.open` already did. Any other value is now rejected by the probe with `NodeTypeValidationError`, the same error the group open raised for it after the fallback, only earlier and with a clearer message. Assisted-by: ClaudeCode:claude-fable-5-1
…in functions Drop the `_ArrayProbe` class. `_fetch_array_metadata_docs` returns the `_MetadataDocs` TypedDict and `_array_metadata_from_docs` interprets it, raising exactly what `get_array_metadata` raised before; that function is now their composition. `zarr.open` reads the docs once ahead of its existing try/except, which is otherwise unchanged, and passes them to the group open on the fallback. Pin the two odd-document behaviors `zarr.open` already had: a zarr.json without a node_type opens as a group, and one with an unknown node_type ends in GroupNotFoundError. Assisted-by: ClaudeCode:claude-fable-5-1
With zarr.json already in hand, the format-detecting branch of AsyncGroup.open knows the format before it reads anything else. A format 3 group needs nothing more, so it now costs no reads at all on top of the array lookup; a format 2 group reads .zgroup, plus the consolidated document only when use_consolidated is not False, which is the rule the explicit format 2 branch already followed. The same rule now applies to the un-pre-fetched path, which used to read .zmetadata and then discard it. zarr.open on a group: format 3 goes from 5 reads to 3, format 2 with use_consolidated=False from 5 to 4. The one thing given up is the warning for a store holding both zarr.json and .zgroup on the zarr.open path, since .zgroup is no longer read when zarr.json is present; open_group with zarr_format=None still emits it. Assisted-by: ClaudeCode:claude-fable-5-1
d-v-b
changed the title
zarr.open used to look for an array and, failing that, open a group, and the two steps read overlapping keys: seven requests for a group, five of them for a format 3 group whose single zarr.json already held everything. Now zarr.open builds the node itself from what it reads. _open_v3 reads zarr.json once; its node_type says array or group and the document holds everything needed to open either. Only if there is no zarr.json does _open_v2 read .zarray, .zgroup, .zattrs and, when it might be used, the consolidated document, in one concurrent round. The existing open_group fallback still handles create-or-raise, and the mode="w" quirk is kept: an existing array is returned, an existing group is overwritten. The pieces zarr.open needs come out of AsyncGroup.open without changing it: _resolve_use_consolidated settles use_consolidated against the store, the format 2 missing/discard check moves into _from_bytes_v2 so it mirrors _from_bytes_v3, and _from_dict_v3 builds a group from a parsed document. No signature that users see changes. Reads via zarr.open: a format 3 array or group costs one; a format 2 group five (four with use_consolidated=False), as before; a format 2 array five where it cost three, plus one round trip, the price of being the fallback. When both zarr.json and .zarray exist at a path, zarr.open takes the format 3 node without reading the other and no longer warns; open_array still does. zarr.open(mode="w", use_consolidated=...) now runs the consolidated metadata checks before overwriting instead of ignoring the request. Assisted-by: ClaudeCode:claude-fable-5-1
d-v-b
changed the title
…data reader Three read implementations existed: get_array_metadata for arrays, AsyncGroup.open for groups, and the _read_metadata_v2/v3 helpers behind AsyncGroup.getitem for either kind with an explicit format. Each fused reading with interpreting, the first two each detected the format on their own, and zarr.open, with no primitive to build on, was trial-and-error over the specific openers with exceptions carrying "found something else." Now read_node_metadata reads a node's documents once and returns parsed metadata whichever kind and format the node is, trying Zarr format 3 first when the format is not given; _open_node adds the use_consolidated policy and builds the node; and open, open_array, open_group, AsyncArray.open, AsyncGroup.open and get_node are mode policy and a kind filter over that. AsyncGroup._from_bytes_v2/v3 and _from_dict_v3 are gone, and _build_node takes the array config. open has a written contract, replacing the "mode check seems wrong" TODO: with shape it behaves as open_array; otherwise the reading modes open whatever node is there, 'a' creating a group when there is none, and the creating modes create a group, 'w' replacing whatever is there. One node_type policy: a format 3 document's node_type must be array or group, else NodeTypeValidationError; finding the wrong kind is ContainsArrayError or ContainsGroupError, never "not found." Behavior changes, each pinned by a test: open(mode="w") without shape replaces an existing array with a group instead of returning it; open(shape=...) on a group raises ContainsGroupError instead of opening it; open(mode="r") on nothing raises NodeNotFoundError; a missing or unknown node_type raises everywhere (a missing one used to open as a group); wrong kind raises Contains*Error in every format and mode; open_array(mode="w") on a group replaces it; open_array applies config to an existing array; a path holding both formats opens as format 3 without a warning; requesting missing format 2 consolidated metadata raises with the format 3 message. Reads via zarr.open: a format 3 node costs one; a format 2 group five (four with use_consolidated=False); a format 2 array five where it cost three, the price of being the fallback. Assisted-by: ClaudeCode:claude-fable-5-1
read_v3_metadata and read_v2_metadata are the reusable pieces: each reads one format's documents and returns metadata or None. read_node_metadata is their composition, trying format 3 first when the format is not given, rather than inlining both and hanging the per-format wrappers off itself. Assisted-by: ClaudeCode:claude-fable-5-1
Split the Zarr format 2 reader per kind: read_v2_array_metadata reads .zarray and .zattrs, read_v2_group_metadata reads .zgroup, .zattrs and the consolidated document, and read_v2_metadata, for a caller that does not know the kind, reads both kinds' documents in one round and shares their parsing. read_array_metadata and read_group_metadata compose these with the format 3 reader the way read_node_metadata does, and _open_array and _open_group put them under open_array, open_group, AsyncArray.open and AsyncGroup.open, which know what they want. open_array on a format 2 array now reads two documents instead of three; open_group three instead of four. Only zarr.open, which does not know the kind, still reads the union. Because a format 2 opener reads only its own kind's documents, a format 2 node of the other kind is not looked for and reads as missing in the read modes; the create modes still find it when they check before writing. A format 3 node's one document says what it is, so the wrong kind is still reported as ContainsArrayError or ContainsGroupError. Assisted-by: ClaudeCode:claude-fable-5-1
Documentation build overview
7 files changed ·
|
…ract tests/test_open_properties.py states the contract of open as an oracle over a scenario (what is at the path: nothing, an array or a group, in either format, consolidated or not; and the call: path, mode, zarr_format, whether shape is given, use_consolidated) and checks every invariant that follows from it: what comes back or is raised; that an opened node is the existing one with its attributes and format, was reached by reading only the documents it needs, each exactly once, with nothing written; that a created node has the requested format and is there to reopen; that a failed open leaves the store as it was; that opening again, and opening through open_array or open_group, gives the same node. test_open_option_space walks every discrete combination, 1260 in a second. test_open_properties lets Hypothesis vary the parts that are not discrete (path names, attribute contents, the consolidated-metadata key) and shrink. Assisted-by: ClaudeCode:claude-fable-5-1
A fresh-context review of the branch against its base found four things the code got wrong and two claims the description got wrong. Code: zarr.open(use_consolidated=True) on an array in a store that cannot hold consolidated metadata raised, because the store was asked before the node's kind was known; the question now waits until the node turns out to be a group. `config` was accepted or rejected depending on what was at the path; zarr.open now treats it as the array's, applied to an array and ignored for a group. The wrong-kind check ran after building the other kind's document, so a broken array document opened as a group failed with a KeyError; the kind is now read off node_type first. And WrapperStore did not forward supports_consolidated_metadata, answering True for any wrapped store; the property test found that one. Description: zarr.open(mode="w") on an existing array already made a new group on main, since make_store_path empties the path first, so that was never a change; and zarr.open(shape=...) on a group changes only for format 3, from TypeError to ContainsGroupError. The changelog and PR body say so now, and list the changes the review found unlisted. Tests cover each fix, and the property test gains a store that cannot hold consolidated metadata as a dimension: 2520 discrete scenarios. _build_metadata_v2 had no callers and is gone. Assisted-by: ClaudeCode:claude-fable-5-1
…g it is a bool Whether the format 2 readers fetch the consolidated-metadata document was carried as an Optional key, with None meaning "don't", which conflated which key with whether to read it. The readers now take `consolidated: bool` and read the constant .zmetadata; `_v2_consolidated_key` is gone. use_consolidated is bool | None everywhere; a str is rejected with a TypeError from _resolve_use_consolidated, for both formats. The open_group and AsyncGroup.open docstrings had said a str names a custom format 2 key, and AsyncGroup.open honored that in one line, but nothing in the tree tested or used it, and the key is .zmetadata. The docstrings no longer promise it. Assisted-by: ClaudeCode:claude-fable-5-1
Assisted-by: ClaudeCode:claude-fable-5-1
d-v-b
marked this pull request as ready for review
September 17, 2026 13:31
zarr.open raises NodeNotFoundError when there is no node at the path, and its docstring says so, but the class was missing from zarr.errors.__all__ and so from the rendered error documentation. It is the shared base of ArrayNotFoundError and GroupNotFoundError, which were both listed. Assisted-by: ClaudeCode:claude-opus-5
The fragment was `misc`, and towncrier's default config for that type discards the content: the Misc section of the release notes is a list of pull request links. Everything this change asks a user to do was therefore invisible. It is a `feature` now, wrapped like its neighbours, and the consolidated-key removal is split into a `removal` fragment so it lands in "Deprecations and Removals" where someone scanning for breakage will look. The prose leads with what to do rather than what moved: which errors changed type, which `except` clauses stop catching them, and what to catch instead. It no longer names the internal metadata readers, which are not public API. Assisted-by: ClaudeCode:claude-opus-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AI-authored cleanup of our basic open array / group / v3 / v2 / mode parameter space. Paradoxically, I wanted this to happen for a long time and also never wanted to do it. Thanks, Claude.
🤖 AI text below 🤖
What
zarr.openused to look for an array and, failing that, open a group, and the two steps read overlapping keys: seven requests for a group, five of them for a format 3 group whose singlezarr.jsonalready held everything. Fixing that exposed the shape of the code underneath, so this PR restructures how nodes are opened rather than patching the one path.How
Three read implementations existed:
get_array_metadata(arrays),AsyncGroup.open(groups), and the_read_metadata_v2/v3helpers behindAsyncGroup.getitem(either kind, explicit format only). Each fused reading with interpreting, and the first two each detected the format on their own.openhad no primitive to build on, so it was trial-and-error over the specific openers, with exceptions carrying "found something else."Now there is one reader and everything is a thin layer on it:
read_v3_metadata(store, path)is one read ofzarr.json, whosenode_typesays which kind. Format 2 has a document per kind, soread_v2_array_metadatareads.zarrayand.zattrs,read_v2_group_metadatareads.zgroup,.zattrsand, when asked to (consolidated: bool),.zmetadata, andread_v2_metadata, for a caller that doesn't know the kind, reads both kinds' documents in one round and shares their parsing. Whether to fetch the consolidated document is a plain bool; the key is the constant.zmetadata.read_node_metadata,read_array_metadataandread_group_metadatacompose these: withzarr_format=Nonethey try v3 first and read the format 2 documents only if there is nozarr.json._build_node(now takingconfig), and for groups_apply_use_consolidatedenforces the caller'suse_consolidatedon what was read.AsyncGroup._from_bytes_v2/v3and_from_dict_v3are gone;AsyncGroup.from_dictis a plain constructor._open_node,_open_array,_open_group= read + consolidated policy + build, orNone. No mode policy, no exceptions for "nothing here." The kind-specific two read only their kind's documents.openandget_nodeare mode policy over_open_node;open_array,AsyncArray.openover_open_array;open_group,AsyncGroup.openover_open_group.get_array_metadatais kept for callers that want the document.openhas a written contract (in its docstring, replacing theTODO: the mode check below seems wrong!): withshapeit behaves asopen_array; otherwise the reading modes (r,r+,a) open whatever node is there,r/r+raisingNodeNotFoundErrorandacreating a group when there is none, and the creating modes (w,w-) create a group,wreplacing whatever is there.node_typepolicy: a format 3 document'snode_typemust bearrayorgroup, elseNodeTypeValidationError. A format 3 node of the wrong kind isContainsArrayError/ContainsGroupError, since its one document says what it is. A format 2 opener reads only its own kind's documents, so a format 2 node of the other kind is not looked for and reads as missing in the read modes; the create modes still find it when they check before writing.Reads
mainopen, v3 array or groupopen, v2 groupuse_consolidated=False)open, v2 arrayopen_array, v2 arrayopen_group, v2 groupzarr.opendoesn't know which kind it's opening, so on format 2 it fetches both kinds' documents in one round; that's the price of not saying what you want. An opener that knows reads only its kind's documents. Theopenv2 array row is the price of format 2 being the fallback; format 3 is the default, so one read for any format 3 node is the number that matters.Behavior changes
All deliberate, each pinned by a test. A fresh-context reviewer audited the diff against the base commit with a 645-cell A/B probe; its corrections to earlier drafts of this list are folded in.
zarr.open(shape=...)on a format 3 group raisesContainsGroupError; it used to raiseTypeErrorfrom forwardingshapetoopen_group. Format 2 is unchanged:ArrayNotFoundErrorin the read modes,ContainsGroupErrorfrom the create check ina.zarr.open(mode="w-")withoutshapeon an empty path creates a group; it used to raiseTypeErrorfrom trying to create an array without a shape.zarr.open(mode="r" | "r+")on nothing raisesNodeNotFoundErrorrather thanGroupNotFoundError(both areFileNotFoundError;GroupNotFoundErroris the narrower class, so code catching only that no longer catches this).AsyncGroup.open/Group.openon nothing raiseGroupNotFoundErrorrather than a bareFileNotFoundError(a subclass; compatible).node_typeraisesNodeTypeValidationErroreverywhere, includingGroup.__getitem__andmembers(). A missingnode_typeused to open as a group throughopen/open_group; an unknown one raisedGroupNotFoundError.ContainsArrayError; opening a group as an array raisesContainsGroupError(it used to sayNodeTypeValidationError). The kind is read offnode_typebefore the document is built, so a broken document of the other kind still says what it is.open_groupon a format 3 array withuse_consolidated=Trueor a key now reportsContainsArrayError(it used to fail on the consolidated argument first). Format 2 openers read only their own kind's documents, so the other kind reads asGroupNotFoundError/ArrayNotFoundErrorin the read modes, as onmain, and asContainsArrayError/ContainsGroupErrorin the create modes, from the check before writing.open_array(config=...)on an existing array now appliesconfig; it was silently ignored.zarr.opentreatsconfigas the array's: applied to an array, ignored for a group whether found or created (it used to be rejected withTypeErrorwhen a group was found or created).zarr.open(use_consolidated=True)on an array in a store that cannot hold consolidated metadata returns the array; the question is only asked of a group. (An earlier commit on this branch raised; the reviewer caught it.)ValueError(".zmetadata"); a consolidated document with nometadataobject raisesMetadataValidationErrorinstead of being swallowed intoGroupNotFoundError.get_node(..., zarr_format=None)detects the format (it used to raise). A format 2.zarraywithoutshapefails to parse as an array instead of silently reading as a group.get_array_metadata(no in-tree callers) returns the parsed metadata serialized again, normalized, not the stored document verbatim.WrapperStorenow forwardssupports_consolidated_metadatato the store it wraps; it used to answerTruefor any wrapped store. Found by the property test below.use_consolidatedisbool | None; a string is rejected withTypeError. Theopen_group/AsyncGroup.opendocstrings used to say a string names a custom format 2 consolidated-metadata key, andAsyncGroup.openhonored that in one line, but nothing in the tree tested or used it, and the key is.zmetadata. This is a deliberate removal of a documented-but-unused option; veto if it matters.Not changed, contrary to earlier drafts of this description:
zarr.open(mode="w")on an existing array already produced a new group onmain, becausemake_store_pathempties the path in modewbefore anything is read; likewiseopen_array(mode="w")on a group already replaced it.Known cost: on the create path (
zarr.open(mode="a")on an empty path)openlooks, then hands creation toopen_group, which validates the arguments and looks again: 7 reads on format 2 wheremainmade 5; format 3 is 3 on both. Avoiding it means duplicatingopen_group's argument handling inopen, so it is left as is.Public API
Audited by introspecting every public callable's signature against the base commit and A/B-ing 162 user-visible calls; 36 differ.
zarr.__all__is unchanged and no public function gained or lost a parameter.use_consolidatednarrows frombool | str | Nonetobool | None(see Deprecations); the error-type changes in the list above, of which the one most likely to be caught in the wild iszarr.openraisingNodeNotFoundError, the base ofGroupNotFoundError, soexcept GroupNotFoundErrorstops catching it.get_nodeandget_node_asyncacceptzarr_format=None;Group.open/AsyncGroup.openon nothing raiseGroupNotFoundErrorin place of a bareFileNotFoundError, a subclass.zarr.api.asynchronousandzarr.core.array(get_array_metadata,parse_node_type_array,buffer_to_json_object, and two error classes) were incidental imports, in no__all__and in no documentation;get_array_metadatastill lives atzarr.core.array.read_node_metadata,read_array_metadata,read_group_metadata,read_v3_metadata,read_v2_metadata,read_v2_array_metadataandread_v2_group_metadataare public-named but not public API, matchingget_node,parse_attributes,parse_node_type,parse_zarr_format,create_nodesandcreate_rooted_hierarchy, which already sit inzarr.core.groupunder the same convention — the module has no__all__and is not a documented page, and onlycreate_hierarchyis promoted tozarr.__all__. They are named without underscores for the same reason those are: they are composed by each other across the module. The release note does not mention them.zarr.errors.NodeNotFoundErrorwas missing fromzarr.errors.__all__, so it was absent from the rendered error docs even thoughzarr.opennow raises it and says so in its docstring. Both its subclasses were listed.Tests
tests/test_open_properties.pystates the contract ofopenas an oracle over a scenario — what is at the path (nothing, an array or a group, in either format, consolidated or not) and the call (path, mode,zarr_format, whethershapeis given,use_consolidated) — and checks every invariant that follows:open_array/open_group, gives the same node.test_open_option_spacewalks every discrete combination (1260, in about a second).test_open_propertieslets Hypothesis vary what isn't discrete — path names, attribute contents, the consolidated-metadata key — and shrink. The oracle also documents one pre-existing behavior the walk confirms: creating a node checks the path only in the format being created, so with a mismatchedzarr_formata new node is made alongside the other format's.test_open_mode_contract(30 cells) andtest_open_reads_only_what_the_node_needs(84 cases) remain as the readable, example-based versions of the same contract, and small tests pin each error type above.This supersedes d-v-b#192.
🤖 Generated with Claude Code