feat(zarr-indexing): LazyArray — generic lazy indexing over array-API arrays - #4222
Merged
Conversation
… arrays A generic wrapper for any array-API-like source (numpy, zarr, cupy, ...) adding TensorStore-style lazy indexing with a positional NumPy dialect: eager __getitem__, .lazy/.oindex/.vindex composing transforms without data access, result()/__array__ materializing. Resolution is partition-based: parts() iterates the base array's partitions projected through the view as resolvable sub-LazyArrays (Partition carries global box coordinates, out placement, and completeness); with_parts() re-partitions the same base explicitly; a single lowering engine serves both partitioned and whole-array sources. Partitioning is discovered from the source (read_chunk_sizes, .chunks) or declared, and never surfaced as a chunks vocabulary. Box selections (no index arrays; affine, interval-representable) are first-class: is_box, bounding_box() (exact hull up to stride), and strides() complete the slab-read story; the design-notes page records the box-vs-query taxonomy and the relationship to TensorStore. Dunders: __dask_tokenize__ (deterministic, canonical-ndsel-body-based), __len__, __iter__, 0-d conversions, pickling. Degenerate all-singleton index-array maps now collapse to constant maps in the transform algebra, and NumPy advanced-index placement rules are implemented faithfully. Assisted-by: ClaudeCode:claude-fable-5
`arr[::-1]` reverses. One desugaring rule covers both signs, as TensorStore 0.1.84 does and as ndsel PR #2 now specifies: omitted bounds resolve on the side the traversal starts and stops (`hi-1` and `lo-1` going down), the source interval is [start, stop) going up and [stop+1, start+1) going down, an empty interval is legal at any coordinate, an interval running the wrong way is an error rather than a silent empty, and the origin is trunc(start/step) for either sign. The corpus is re-vendored from ndsel 92d6a32 in this same commit, because it is the definition of correct here: `slice.json` gains ten negative-step fixtures, `errors.json` retires `negative_step_unsupported` for three `bounds_out_of_order` fixtures, and the message layer is changed to satisfy them. The retired reason code is documented as such rather than removed. A latent bug that only negative steps could reach: `_reindex_array` built `slice(pos, pos + size*step, step)`, and a downward walk reaching the front of the array computes a negative stop, which NumPy reads as counting from the end — `slice(6, -1, -1)` selects nothing where `slice(6, None, -1)` selects seven elements reversed. Both reindex helpers now go through `_positional_slice`. The stride<0 branches of `_intersect_dimension_map` and `iter_chunk_transforms` were written defensively and had never been reachable. They are now, and they were right: the parts-coverage test gains two reversing views, and the seeded sweep generates downward slices (4,352 of 7,200 chains carry one) across every partitioning. At the wrapper boundary the dialect stays NumPy's, which differs in one place: a reversed *positional* interval like `lazy[2:5:-1]` is empty, not an error, because that is what `x[2:5:-1]` means. Only literal coordinates call it a direction error. Tests: the study's recorded TensorStore corpus lands in `test_tensorstore_parity.py` — fifteen desugarings with their domains, offsets and strides, the three rows that discriminate trunc from floor and ceil, both error families, empty-outside-the-domain, five recorded compositions, and the recorded index-array reversal (a negative step over a gathered axis reverses the array rather than attaching a stride). Assisted-by: ClaudeCode:claude-fable-5
…ly Partition, strides docs - slice step zero now raises ValueError, matching NumPy in the wrapper's positional dialect (was IndexError) - Partition is keyword-only: box was inserted mid-field-list, so positional construction would silently misbind - strides() documents the empty-box case (bounding_box None, strides still defined) Assisted-by: ClaudeCode:claude-fable-5
github-actions
Bot
added
the
needs release notes
Rewrite the package's documentation surfaces — docs/index.md, docs/design-notes.md, docs/ndsel.md, docs/api/index.md, the LazyArray, boundary, and transform docstrings, and the 267 changelog fragment — in plain declarative English. Metaphor, personification, rhetorical framing, and emphasis used for effect are replaced with statements of the same technical content. No technical claim, API name, example, or example output changes. Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
A new ruff release (the CI job floats via uvx) flags BLE001/S110 at the chunk-discovery tolerance and tokenize-fallback sites. Both catches are intentional contracts: discovery must degrade to no-information on any foreign-object failure, and a token call must never raise. Configured as per-file-ignores rather than noqa comments because the pinned pre-commit ruff strips the comments as unused (RUF100) while the floating CI ruff requires them. Assisted-by: ClaudeCode:claude-fable-5
Mirrors the main-branch pin (#271) so this PR's workflow runs the same ruff version; bump together with the pyproject pin. Assisted-by: ClaudeCode:claude-fable-5
Two runnable examples in the house style: wrapping a NumPy array in LazyArray (attribute forwarding, composing selections, box vs query selections, partitions), and using a LazyArray with Dask (from_array, one task per partition, deterministic tokens). Assisted-by: ClaudeCode:claude-fable-5
Adds a timed comparison of chained selections through dask.array against the same selections composed into one transform, and a section on when each is the right tool: dask's graph earns its cost when there is computation across chunks, and is overhead when it only defers indexing. Assisted-by: ClaudeCode:claude-fable-5
The example runner rewrote only the `zarr` dependency to the local checkout, so an example depending on an in-repo package resolved it from git main instead — and the lazy-indexing examples failed in CI, since LazyArray is not on main yet. Rewrite every package this repository ships, leaving dependencies an example does not declare alone. Assisted-by: ClaudeCode:claude-fable-5
`LazyArray` assumed every wrapped array could do basic slicing and did all fancy work itself, reading a block and post-indexing it with NumPy. That over-reads when the source could gather natively, and it walks a source axis by axis with `take` where one request would do. Each wrapper now carries an `IndexingSupport` level — BASIC, OUTER, OUTER_1VECTOR, VECTORIZED, the taxonomy and member names of xarray's `IndexingSupport` — and every read is split into the largest part of the selection that level can express, asked of the source in one call through `oindex`/`vindex` when it has them, and a residual transform applied to the block that comes back. The split is applied per partition as well as per whole-array read, so a part costs one request. The level is resolved at construction: an explicit `with_indexing_support` wins, then the source's own `__zarr_indexing_support__` (read defensively), then conservative inference — a NumPy array or zarr's `oindex`/`vindex` pair reads as VECTORIZED, everything else as BASIC, the only assumption that is always correct. A multi-array outer request only ever goes to an `oindex` accessor, because a bare `__getitem__` key with two arrays means an outer product to HDF5 and a correlated gather to NumPy. The level decides how much data crosses the boundary, never what `result()` returns. Tests hold that invariant directly: every selection case, at all four levels, against NumPy and zarr sources, partitioned and not; plus test doubles that raise when handed a key their declared level forbids, so exceeding a declaration fails loudly rather than working by accident. Assisted-by: ClaudeCode:claude-fable-5
The token included the partitioning while deliberately excluding the indexing-support level, though both are read strategies that leave the values unchanged. Excluding both means two wrappers that describe the same data token alike, so a consumer caching on tokens reuses one result across partitionings and support levels. Assisted-by: ClaudeCode:claude-fable-5
A slice with a negative step whose start lies before the front of the axis selects nothing, but the positional slice was written as `slice(start, stop, step)` with a negative stop, which NumPy reads as counting from the end: an empty selection of a fancy axis returned the whole axis reversed, and the correlated form raised a broadcast error. Write an empty selection out explicitly. The randomized basic-selection generator drew negative-step starts from `[0, size)` only, so a start before the front of the axis was unreachable and the suite could not see this. It now draws from below `-size` as well, and three cases pin the behavior directly. Assisted-by: ClaudeCode:claude-fable-5
An `oindex`/`vindex` step whose entries are all slices carries no coordinates: it narrows the view's own axes and must compose like basic indexing. `_reindex_array_oindex` instead applied each entry positionally to the corresponding axis of the existing index array, without asking whether that axis is one the array varies over or a singleton it merely broadcasts along — the distinction its basic-indexing sibling `_reindex_array` has always made. A slice starting past 0 therefore indexed a size-1 broadcast axis out of range and truncated the whole index array to size 0. A view with no coordinates left resolves to no parts, and `result()` handed back its unwritten `np.empty` buffer: live, on the default path, for any source that advertises chunks. `_reindex_array_oindex` now takes the `ArrayMap` and applies an entry only along its dependency axes (plus the `input_dimension` that breaks the tie for a degenerate length-1 orthogonal selection), preserving a broadcast singleton whatever the slice says. Coordinates never reach a broadcast axis — `_guard_fancy_after_fancy` still rejects genuine fancy-after-fancy with `NotImplementedError`, now under test. Four defects from the same review ride along: - `_array_map_dependency_axes` counted a length-**0** axis as an axis the array varies over, so an empty orthogonal selection classified as correlated and `array_map_dependent_axis` rejected it — a raise on the unpartitioned path where every partitioned path returned the right empty answer. An axis of size 0 carries no dependency any more than a singleton does. - `parts()` raised on a view emptied by a slice over an axis of extent 1. A correlated selection of one point normalizes to an all-singleton index array, so emptying the domain leaves the array at size 1 and the resolver went looking for a chunk. An empty input domain now yields no parts and meets no output domain, matching `result()`. - `sub_transform_to_selections` built `slice(stop + 1, start + 1, stride)` for a negative stride — endpoints swapped, step still negative, so it selected nothing where the reversed axis was meant. Both branches now lower through `_positional_slice`, the same walk an `ArrayMap` axis is reindexed by, which knows a downward walk reaching the front must stop at `None`. - `compose()` evaluated an inner index array over `range(size)` rather than over the outer domain's own range, and addressed it from 0 rather than from the inner domain's origin. Every coordinate resolved to the wrong cell whenever a domain did not start at 0 — which a step-1 slice and a negative-step slice both produce routinely here. - `transform_from_canonical` now rejects a non-integer `index_array` with an `NdselError` carrying `invalid_json`, instead of silently truncating `[0.9, 1.9]` to cells 0 and 1, coercing booleans, or leaking NumPy's own conversion error for strings. The fuzzer missed the first two because `_random_chain` drew at most one fancy step and had no way to spell a step that goes through a fancy accessor while carrying only slices. It now draws such a step separately, and the seeded sweep gained a `parts()` counterpart: `result()` can absorb a defect that the iteration contract cannot, since an empty view assembles correctly from no parts at all. Both sweeps fail on the pre-fix source, as does the new exhaustive stride/extent sweep over the chunk-selection bridge. Assisted-by: ClaudeCode:claude-fable-5
A `vindex` coordinate array with a singleton broadcast axis contributes an axis it does not vary over. A later basic index that consumes the axis it *does* vary over collapses the map to a `ConstantMap` and leaves the broadcast axis in the domain, referenced by nothing. Three places assumed that could not happen: - `sub_transform_to_selections` built `out_selection` with one entry per output map, so a view with such an axis got an index tuple of lower rank than the buffer. `out[out_selection] = value` then placed the part against the leading axes and broadcast the rest — silently wrong data on a partitioned read, and a `parts()` walk that left cells unwritten. - `_restore_domain_axis_order` put an unreferenced axis back as a singleton whatever the domain said. At extent 0 that fabricated a row for a selection whose own `shape` reported it empty. - `_lower_correlated` built its flat gather index from the domain's broadcast shape but added coordinates straight off the stored index array, which is singleton on the axes it does not vary over. The two disagree exactly when a correlated map is constant along a shared broadcast axis. `out_selection` is now built per domain dimension throughout, an unreferenced axis is restored at its own extent, and a correlated map's coordinates are broadcast to the block before being combined. The randomized chain sweep never generated the shape at fault: `_random_vindex` only produced `(length,)` and `(length, 1)` coordinate arrays, neither of which leaves a singleton axis for a later step to strand. It now draws a broadcast rank and places each array's varying axis within it, which reproduces all three failures on the unfixed code. Assisted-by: ClaudeCode:claude-fable-5
Five fixes to the wrapper's edges, none of which changes what a selection means. `result()` and `__array__` no longer alias the wrapped array. An unpartitioned read of a basic selection lowers to plain slicing, so it came back as a *view* of the source; NumPy 2 hands whatever `__array__` returns straight to the caller, so `numpy.array(view, copy=True)` aliased it and a write reached through. Under any partitioning the same read allocates, so this also made the answer depend on how the read was divided. The result is now detached whenever it may share memory with the wrapped array, and the `copy=False` refusal no longer justifies itself with a claim the other branch violated. `result()` verifies that the partition walk covered the output before returning it. The buffer is deliberately uninitialized, so any defect in the walk was reported as plausible-looking numbers rather than as an error. The cells each part addresses are counted from the selectors' own shapes — nothing is read — and a walk that does not add up to the view's size raises. Measured on a 16 MiB read: 51 us of accounting against 6.5 ms of read for 64 parts, within noise end to end, and +1.7% at 512 parts. The `BASIC` floor is a promise about the *source*, not about the blocks it returns. The residual is finished with `take`, `reshape` and `transpose`, which were applied to the block unconverted — so a source meeting exactly the documented floor crashed on `oindex[[4, 0, 0], :, :]`. A block that is neither a NumPy array nor an array-API namespace of its own is now coerced, which leaves a device array where it is. `numpy.matrix` is refused at construction: it never reduces rank, so a view's shape and its result disagree on every rank-reducing selection. A `numpy.ma` source keeps its mask through a partitioned read, which allocates a masked buffer. A declaration holding a *foreign* enum member that names one of these four levels — xarray's `IndexingSupport`, whose members these are borrowed from — is honored rather than discarded, since discarding it fell through to inference and answered with a *more* permissive level than the source asked for. `is_complete` is true for a reversing view, which reads every cell of its box back to front; the stride-1 test it failed was about direction, not coverage. `with_parts` accepts `(0,)` and `(0, 0)` for a zero-length axis, which said the same thing as the `()` and uniform spellings it already took, and the positivity error names the working form. Above the token digest limit and without dask, `__dask_tokenize__` returns a value that matches nothing rather than a shape-and-dtype description that two different 4 MiB arrays shared. A cache keyed on it misses instead of lying. Assisted-by: ClaudeCode:claude-fable-5
Every statement below was executed before being rewritten, and the replacement was executed too. - "`view + 1`" / "arithmetic materializes through `__array__`" is false. `LazyArray` defines no arithmetic dunders, so `view + 1` raises `TypeError`. What does work is a NumPy *function* — `numpy.add(view, 1)`, `numpy.sum(view)`, `numpy.stack([view, view])` — and an ndarray on the left of the operator. Corrected in the module docstring, `docs/index.md` and the changelog fragment. - "An empty selection returns `None` from both" is false: an empty *box* reports `strides()` and only `bounding_box()` is `None`. The `strides()` docstring already said so; the design notes now agree with it. - "A box touches a contiguous run of parts" is false for a strided box — `[::4]` over 2-wide parts visits every other part. The true property, and the one a partition-walk optimizer would want, is a regularly-spaced run in increasing order, each part at most once. - "TensorStore permits a lower-rank index array" is backwards. Checked against tensorstore 0.1.84: its JSON parser rejects a rank-1 array over a rank-2 domain and accepts full rank with singletons, which is what we emit. *Our* loader is the permissive one. The passage now says both models want full rank, keeps the real rationale (the singletons are what makes the orthogonal/vectorized distinction derivable), and describes our lower-rank acceptance as the compatibility affordance it is. - "Two limits remain" omitted fancy-after-fancy, which is a live `NotImplementedError` reachable from the documented surface, while `index.md` invited chaining fancy steps "anywhere in the chain". Current scope now lists five limits, including the diagonal-view and mixed correlated/orthogonal ones, and both prose pages point at it. - "A single whole-array part stays in the wrapped array's namespace" is only true with *no* partitioning: `result()` branches on whether a partitioning is in force, not on how many boxes it has, so `with_parts((4, 6))` on a 4x6 array returns a plain ndarray. - The changelog stated the support-detection precedence backwards (declaration wins, not inference); `index.md` had a sentence missing its noun; the module docstring's one-line `bounding_box()` summary dropped the stride caveat the three other locations keep; and the package README, the PyPI long description, never mentioned `LazyArray`. Assisted-by: ClaudeCode:claude-fable-5
The index-array rank was checked only from above, so a lower-rank array could exist inside the engine and be read for dependency axes it did not have. Nothing produced one: the tolerance was there for a test asserting compatibility with a body TensorStore itself rejects (verified against 0.1.84 — a rank-1 array over a rank-3 domain is an error in its JSON parser). Require the full input rank in the type, widen a lower-rank array at the JSON boundary where external input arrives, and give the test the shape TensorStore accepts. Assisted-by: ClaudeCode:claude-fable-5
An index array axis must be the domain's extent or a singleton it broadcasts over. Any other size leaves input coordinates with no entry, which read as a smaller selection rather than as the error it is: the truncated array behind one of this review's silent-corruption bugs was a (3, 0) array over a (3, 2) domain, which this rejects at construction. Two fixtures carried the inconsistency they were meant to exercise — an empty array over a domain with room for two coordinates, and a widening case whose array covered three of four positions — and now describe domains their arrays span. Assisted-by: ClaudeCode:claude-fable-5
…nk-0 part it found Adds `zarr_indexing.testing`, behind a `testing` extra: a Hypothesis state machine that composes indexing steps onto a LazyArray and checks each step's shape, `result()`, and `parts()` assembly against NumPy, plus the selection strategies on their own. A project can point it at its own array by overriding one method. The machine asserts the documented assembly literally — a part's values must arrive at the shape its out_selection addresses — which is how it found the defect it also fixes: intersecting a correlated transform with a part's bounds collapsed the surviving broadcast block into one axis even when the block was already rank 0, so a view narrowed to a single point produced parts of rank 1. A rank-0 block now stays rank 0, and `result()` drops the reshape that was absorbing the mismatch. Merged from the branch that produced it, which predates the coverage guard in `result()`; the guard stays and the reshape it compensated with goes. Assisted-by: ClaudeCode:claude-fable-5
…undaries Six reviewers went at the package, each proving findings by execution. The algebra held: ~85k chained selections against NumPy, ~24k part assemblies with poisoned buffers, 3.8k transform round-trips evaluated as coordinate maps, all clean. Everything below was at a boundary. `result()` and `__array__(copy=True)` could hand back a live view of the source. `_detach` asked whether the *source* was an ndarray, but a duck array that merely stores its data in NumPy returns NumPy views, and those went straight to the caller — while three docstrings promised the opposite unconditionally. It now asks whether the *result* owns its buffer, so memory is released only when sharing is disproved rather than when it cannot be established. The wire format could not reload its own output. `tolist()` renders every empty array as `[]` once the leading axis is the zero-length one, so an ordinary empty selection lost the axis it varied over, and the loader put it back on a different one by prepending singletons. Nested lists cannot express the shape either, so the body carries it. `index_domain_from_json` was a second undefended way into the same objects: a bare `int()` that truncated 3.9, coerced "3" and True, and let a non-string label into a tuple[str, ...]. It goes through the message layer now, as a transform body always did. With it: a rank ceiling, i64 checks on desugared bounds so normalization stays idempotent, ordered index_array_bounds, and typed errors where raw ones leaked. `sub_transform_to_selections` transposed its blocks two ways. A ConstantMap was emitted as a bare integer, which NumPy counts among the *advanced* indices whenever an index array is present, moving the broadcast axis to the front when a slice separates them — while `out_selection` is built positionally. And the correlated branch documented a points-major block but assembled the chunk selection in output order, so a residual slice before the coordinates arrived slice-major. Constants are length-one slices now, named in drop_axes, and the correlated scatter is permuted to the block NumPy actually returns. Nothing caught either one because `LazyArray` resolves a part through its own lowering and never reads `chunk_selection`; one of the two was even asserted as correct in a passing test's comment. Three further failures were one stale field: `_apply_vindex` carried `input_dimension` onto a map the vindex had just made correlated. The value outlived the shape that justified it and was believed later by a scatter that filed positions under the wrong axis — which is why one view's answer depended on how it was partitioned. The dependency is read back off the array now, and `__post_init__` checks the field it was wrong about: ArrayMap was the one map whose `input_dimension` nothing validated. Also: `oindex` over a correlated view applied its index tuple positionally, NumPy's vectorized rule, collapsing two arrays into one axis; `compose()` indexed an inner array's broadcast singletons by the raw coordinate and sized its one-dimensional shortcut by the input rank while gating on the output rank; and neither checked that the outer transform's output lands inside the inner domain, where a negative would have wrapped. Assisted-by: ClaudeCode:claude-fable-5
A mutation audit ran 31 mutations and 10 survived. They were not scattered: the invariants check thoroughly what a view *returns* and never what it *claims about itself*, and the strategies could not draw whole classes of selection. The generators now draw them. Orthogonal slices carry a step and may stop early, so a strided or reversed slice reaches `oindex` at all. Coordinate lists and masks can be empty, so a fancy selection that selects nothing exists — the shape that lost its axis on the way through JSON. Vectorized coordinate arrays can be multi-dimensional, so a rank-raising `vindex` is generated. Measured over 4000 draws each, every one of those counts was previously 0. The first run of the widened generators found a live defect: an empty Python list carries no element type and NumPy defaults it to float64, so `oindex[[]]` was refused as a non-integer index array. `json.py` already had that case; the boundary did not. NumPy takes `a[np.ix_([])]`, and so does this now. `Partition.is_complete` gets an invariant. It is what a consumer reads to decide it may take a whole-box read, so a wrongly-`True` one is silent corruption — and three separate mutations to `_covers_whole_part` survived the entire suite, including ones reporting `is_complete` for a part carrying two of its three cells. Asserted one way only: the flag is documented as conservative and only the claim to cover everything has to be earned. Two more state machines. The sorted one-dimensional fancy path needs both ranks to be 1, and the output rank is the *source's*, so the rank-3 default source walled it off entirely — 0 hits from the machine against 105 from the unit tests, in the path where reordering and duplicate coordinates are partitioned. A rank-1 source now reaches it 322 times per run. A source with an extent-1 axis covers the other side of a distinction the code draws from the domain rather than the array. Finally the two remaining mutants, both checked by mutating and confirming the failure: the index-array bound checks are probed one past the boundary rather than comfortably outside it, and `_out_selection_cell_count`'s guard is tested directly, since a partition walk only ever produces the forward in-bounds intervals that never reach it. Assisted-by: ClaudeCode:claude-fable-5
…ter 1.0 `with_parts` decided what it had been given by inspecting the type of it: a sequence of integers meant uniform boxes, a sequence of sequences meant per-axis sizes, and `None` meant no partitioning at all — which also sent `result()` down an entirely different code path. Three semantics behind one parameter, and no way to ask for one of them and be told when you had spelled it wrong. They are now `with_parts`, `with_parts_per_axis` and `unpartitioned`. A harness that draws from a list of mixed partitionings still needs the dispatch, so it exists once, as `zarr_indexing.testing.repartition`. The sizes those methods take are relative to the array being read, not to the view reading it, and a narrowed view partitions the base extents — which could only be discovered from an error message. `base_shape` says it. `ArrayMap`, `IndexTransform` and `Partition` are `frozen=True`, which reads as a promise that a value can be compared and hashed. Both raised: `==` on the index arrays returned an array and then `ValueError: the truth value of an array is ambiguous`, and `hash()` refused an ndarray outright. So no transform could enter a set or key a cache, and the package had already grown an internal `_is_identity_transform` because of it. An `ArrayMap` was frozen but the array inside it was not, so reaching through a view's transform to `index_array[0] = 9` silently changed what the view returned. It is held through a read-only view now — a view rather than a flag on the caller's array, since constructing a map should not take away the right to write to an array you still own. `IndexDomain.narrow` clamped a slice bound to the domain, in a package whose stated invariant is no clamping and no negative wrapping. `narrow(slice(-3, None))` on `[0, 10)` therefore returned the whole axis — reading as the NumPy spelling of "the last three" and answering with something else — and a stop past the end returned a domain its own parent did not contain. Both raise `BoundsCheckError` now, and a stride raises `ValueError` like every other unimplemented request rather than `IndexError`. `Partition.array` was a view of the array while `LazyArray.array` was the raw source: adjacent types, one name, inverted meanings. The part's is `view`. Smaller: `BoundsCheckError` and `VindexInvalidSelectionError` are exported at the top level, being what exported functions raise; `errors.py` no longer claims `zarr.errors` re-exports them by identity, which is false and would have been believed; `parts()` says it is single-use; and `sub_transform_to_selections` says it is provisional rather than implying the rest of the API's stability. Assisted-by: ClaudeCode:claude-fable-5
…wo dialect gaps `boundary.py` justified applying scalars before the advanced indices by asserting NumPy does the same, citing `a[0, [1, 2], :]`. NumPy groups a scalar *with* the advanced indices for placement, so the two disagree the moment a slice separates them: `a[0, ..., [1, 2]]` is `(2, 3)` where `a[0][..., [1, 2]]` is `(3, 2)`. Scalar-first is this package's documented dialect and stays; the reasoning was wrong, and a wrong reason invites someone to "fix" the correct end later. Two places where the dialect really did diverge, both now matching NumPy. A zero-dimensional integer array is a scalar — `a[np.array(2), :]` drops its axis — but only Python and NumPy integers counted, so a 0-d array was widened into a length-1 index array and kept an axis; that was a third answer, agreeing with neither NumPy nor eager zarr. And a multi-dimensional array in an orthogonal selection is refused where the rule lives, instead of surfacing two layers down as a rank complaint about an `index_array` the caller never wrote. `iter_chunk_transforms` documented two shapes for `out_indices` and returns three: the `dict[int, ndarray]` an orthogonal selection with several index arrays produces was missing, from the function downstream integrators use most. Packaging: the README is the PyPI long description, so the monorepo-only development section moved to CONTRIBUTING.md and the dead relative link to the justfile went with it. The examples pinned `zarr-indexing` to a moving `main` rather than to a release they document. Three of the six docs pins claimed to match the repo root and did not, and the justfile's ruff comment contradicted the package's own pin. Assisted-by: ClaudeCode:claude-fable-5
…g the format
The previous commit fixed the empty-selection round trip by carrying an
`index_array_shape` field, on the reasoning that JSON nested lists cannot
express the shape of an array with a leading zero axis — `[]` is the only
spelling of every empty shape, and `[[]]` is (1, 0) with nothing for (0, 1).
That much is true, but the conclusion was wrong: it invented a field ndsel does
not define, so the documents this package wrote stopped being ndsel documents.
The reference implementation does not have the problem, because it never emits
an empty index array. `t[ts.d[0][[]]]` in TensorStore is `out[0] = 0`, emitted
as `{}` — a constant map. An empty index array names no cell, and it can only
be empty because an input dimension is, since the full-rank invariant makes
every axis either 1 or the domain's extent. Nothing is ever read through it, the
emptiness is carried by the domain, and the map is degenerate in exactly the way
a size-1 array is — which this format already collapses.
So it collapses the same way, and the extension is gone. The loader still
recovers the axis from the domain for an empty array arriving from a producer
that does emit one, since ndsel does not forbid it; that path just no longer has
a first-party caller.
Checked against tensorstore 0.1.84: every canonical body from 4000 randomized
transforms loads into `ts.IndexTransform`, and every one re-emits itself
unchanged. No spec change needed.
Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: Codex:gpt-5
…s are in force An empty view is now answered without reading the source, and the shortcut reached for the array namespace's own `empty`, which knows nothing about masks. An unpartitioned empty view over a masked source therefore came back a plain array while the same view partitioned came back masked — no cells either way, so no value changed, but the caller's type depended on how the read had been divided, which `result()` promises it never does. A masked source goes through `_output_buffer`, which is the branch that knows. Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: Codex:gpt-5.6
Assisted-by: Codex:gpt-5.6
…othing 'No data is touched' in apply/apply_many and 'without I/O' on the oindex/vindex accessors stated what the functions do not do without clarifying what they do: nothing about a coordinate lookup, or about an accessor documented to return a new transform, suggests data movement. Removed. The statements that earn their negation stay: the module thesis, the class contract, and __getitem__ — where subscription syntax genuinely suggests an eager read to anyone arriving from zarr. Assisted-by: ClaudeCode:claude-fable-5
'Use lazy indexing' held one page: a disclosure triangle and a competing label with no organization gained. 'Practical reference' classified nothing and grouped two pages serving different audiences — the exact split the guide's opening and the landing page's annotated links already route explicitly. Both dissolve into top-level entries. Examples and API Reference keep their sections, being the only real collections at this site's size; the ndsel wire format and design notes gain explicit labels and sit in the for-builders tail before the API. Assisted-by: ClaudeCode:claude-fable-5
Visual guide, indexing patterns, and integration boundaries sat at top level with the same rank as the fifteen-page API Reference — three pages wearing category clothes. They are one collection, and the docs/guide/ directory said so all along: learn it, look it up, apply it at the boundary. Pydantic's sidebar confirms the pattern (its narrative layer is one Concepts section) while cautioning against copying its tab bar and zero-loose-pages norm, which pay off at fifty pages and cost discovery at eight. navigation.indexes makes the Guide entry itself land on the visual guide, so the common click is free; standalone artifacts (landing, ndsel spec, design notes, changelog) keep their top-level standing. Assisted-by: ClaudeCode:claude-fable-5
The indexing-pattern reference framed every row through LazyArray, as if the wrapper defined the semantics. It does not: every dialect compiles to an IndexTransform, and the wrapper is a regular array-like API whose only distinction is that operations on .lazy return views. The matrix now speaks the algebra — t[1:5, ::2], t.oindex[rows, columns] on IndexTransform.from_shape((6, 8)) — and the executable snippet compiles each selection through the transform accessors, reads the box/query category structurally off the output maps, and resolves values through the public reader. The wrapper is demoted to a closing note, and the test suite keeps it honest by running the same matrix through LazyArray: the snippet proves the algebra, the test proves the wrapper agrees. Assisted-by: ClaudeCode:claude-fable-5
…sform Showing t[1:5, ::2] demonstrated only that IndexTransform supports __getitem__ — syntax, not the object. Each idiom is now modeled by hand: domain plus output maps, with the anatomy stated per case (the dropped axis surviving as a ConstantMap, reversal as nothing but a negative stride, emptiness living in the domain, a mask being its nonzero coordinates, and fancy flavor spelled entirely by index-array shape — distinct axes for the outer product, a shared axis for pointwise). The table shows each idiom's maps; the executable matrix checks every model's shape, category, and NumPy values, then proves the selection compiler derives the same transform — after translate_domain_to, since compiled basic selections keep literal domains, a wrinkle the page now names. Assisted-by: ClaudeCode:claude-fable-5
… gaps An adversarial pedagogy review (five reader personas, three skeptics per finding) confirmed thirteen distinct gaps; all are closed. Rendered excerpts no longer reference invisible code: the map-kind examples' resolve helper is now shown and introduced — doubling as the guide's first statement of how a bare transform meets data — and the axis-manipulation excerpts carry their own setup. The sharpest finding, a genuine contradiction, is corrected: the guide claimed an integer index is not a ConstantMap while the pattern matrix models image[2, :] with exactly one; the responsibility now sits where the model puts it — the domain decides axes, the map only fixes a coordinate, and a broadcast is a domain axis no map consumes. The wrapper's partitioning vocabulary is introduced before use: with_parts leaves the composition snippet (it taught nothing there), 'parts' is defined in the chunk-plan section, Partition/.view/.projection/ .out_selection get an introducing sentence before the frame warning, and the integrations page states the chunks-attribute discovery its read counts silently relied on. plan_chunks' argument is described truthfully (one per-dimension grid, four-method contract, dimension_grids_from_chunks as the built-in), oindex is defined at first use, 'cover' and 'bounding hull' are defined where the dense-box policy leans on them, the reads-through-the-chunk-local-side claim now matches the code, the pattern table uses the executable's real variable names, and the guide's footer no longer skips the rest of its own collection. Assisted-by: ClaudeCode:claude-fable-5
Each row of the idiom-to-model matrix now carries the domain beside the output maps — the complete IndexTransform model, not half of it. The redundant result-shape column folds into the domain spelling, with one sentence stating why that is no loss: the domain is the result's coordinates, so its shape is the result shape. Assisted-by: ClaudeCode:claude-fable-5
…re form
A table cell reading from_shape((4, 4)) was a receiverless method call —
syntax debris, not a model. Each idiom now appears as its complete
transform in the ndsel canonical body: explicit domain bounds, one output
map per source dimension, nothing to squint at. The wire form also
teaches for free — the dropped axis of image[2, :] is visibly the bare
{"offset": 2} constant form, emptiness is visibly a zero-width bound, and
outer-product versus pointwise is visibly nesting versus flat arrays.
The nine bodies cannot rot: a new test pins each JSON block to
transform_to_canonical of the corresponding executable model, in order.
Assisted-by: ClaudeCode:claude-fable-5
Each idiom now shows both spellings of its model in linked content tabs: the Python construction and the ndsel canonical body. content.tabs.link keeps every pair switched together, so a reader can walk the whole matrix in either language. The pinning test grows teeth on both sides: the JSON tab must equal the model's canonical form, and the Python tab must evaluate — in the executable matrix's own namespace — to the model itself, so neither tab can drift from the code. Assisted-by: ClaudeCode:claude-fable-5
…lared The wrapper defines no __setitem__, but nothing said so: the class docstring, the module docstring, the guide, and the landing page were all silent, leaving a real contract to be discovered by TypeError. The class docstring and the guide's materialization warning now state it, together with where writing does belong — a consumer plans the selection with plan_chunks and owns the read-modify-write, because chunk atomicity and concurrent-writer policy are the backend's to decide. The flip side is stated where a source is described: a wrapped array needs shape, dtype, and __getitem__ and nothing more, so a read-only source (an HTTP endpoint, a snapshot, a decoded-chunk cache) wraps as well as a writable one. Naming was considered and rejected. LazyArrayView would mislead, since a NumPy view shares memory and writes through — the opposite of the truth — and LazyReadOnlyArray names a non-capability and implies a writable sibling. Read-only-ness follows from the object being a deferred description of a read; a documented contract closes the gap that a loud, immediate TypeError already made non-silent. Assisted-by: ClaudeCode:claude-fable-5
The canonical converters were free functions in json.py, and the reason given for that — keeping the algebra core ignorant of the wire format — did not survive inspection: importing the package already loads json.py and messages.py through __init__, so nothing was decoupled for anyone. And the usual reason to keep a type ignorant of its serialization is to avoid privileging one of several; this repo admits exactly one, spec- defined and TensorStore-compatible, whose own IndexTransform carries to_json(). So the conversions are methods now: IndexTransform.to_json/from_json, IndexDomain.to_json/from_json, and to_json on each output map kind. output_index_map_from_json stays a function, moved to output_map.py, because the wire form is a tagged union — loading it dispatches rather than belonging to any one kind. The surface shrinks rather than grows. Each conversion had two public spellings (the canonical name plus a historical *_to_json alias); both are gone, leaving one. json.py keeps the wire vocabulary — the TypedDicts and JSON type aliases — and the lowering rules the types share move to a package-private _wire.py, which is what they always were: pyright caught underscore-private helpers being read from three other modules. Assisted-by: ClaudeCode:claude-fable-5
…s go private Applying the lesson from the serialization move to the rest of the package, and checked against TensorStore, whose split is unusually clear: its public index_space headers are the types, while every transform operation — compose_transforms, inverse_transform, transpose, translate, the slice ops — lives in internal/ and surfaces as a method. Its Python IndexTransform is all methods and no free functions. composition.py was the same defect json.py had: 242 lines, zero types, one public function over a type defined elsewhere, with its own API page as if it were a subsystem. The algorithm is now private in _composition.py and the public spelling is outer.compose(inner). selection_to_transform becomes transform.select(selection, mode), index_array_structure becomes a property, and array_map_dependent_axis becomes ArrayMap.dependent_axis — it described an ArrayMap while living in transform.py. Two smaller repairs fall out. affine.py, never exported and never documented, takes the underscore it had earned. And the dependency-axes helper that three modules read across a private boundary becomes ArrayMap.dependency_axes, which is what it always was: something a map knows about itself. Assisted-by: ClaudeCode:claude-fable-5
… shipped API Three fragments carried fork PR numbers (267, 272, 273), which towncrier renders as links to zarr-developers/zarr-python issues of those numbers — unrelated upstream issues. They are this PR's work and now say so. Their contents had also aged past the code. The LazyArray note described array_map_dependent_axis, compose() and transform_from_canonical, none of which survive; the composition note named index_array_structure as a function. All now name the shipped spelling. The merged ndsel fragment already on main said index_transform_to_json and its siblings 'now produce and consume the canonical body' — true when written, but this PR removes them, and both fragments render in the same unreleased changelog, so it would have introduced and withdrawn one API in a single release. One recategorization: the note covering with_parts becoming three named methods, Partition.array becoming Partition.view, and the new equality and hashing sat under misc, whose towncrier default shows the link and discards the text — user-visible breaking changes, invisible in the changelog. Moved to removals, which also drops a duplicated zarr-developers#4222 link from the misc line. Assisted-by: ClaudeCode:claude-fable-5
One fragment had grown to 1844 words — two thirds of all release-note text, rendering as a single bullet. It was the PR's commit log: the feature, then nine paragraphs of defects found and fixed along the way. Those defects belong to the pull request, not the changelog. This package has never released — CHANGELOG.md holds nothing but the towncrier marker — so every one of them was a bug in code no reader could have run, and a first release that recounts them describes a journey nobody took. What remains is what the package offers, split by capability rather than by the order the work happened: LazyArray and its partitioning, source-independent chunk planning, the reader boundary, the testing subpackage, and negative-step slices. Each is scannable, and the detail they used to carry is in the guide and design notes, which they now link. Assisted-by: ClaudeCode:claude-fable-5
…evelopment With no release before this PR merges, the Bugfixes and Deprecations sections were describing work no reader could have experienced: defects in code that never shipped, and the withdrawal of functions nobody could have imported. Both sections are gone. Their surviving content is stated as what the package offers. The algebra's types carry their own operations — composition, selection, application, classification and serialization on IndexTransform, dependency axes on ArrayMap, value semantics on all three — which is a capability, not a migration. base_shape joins the partitioning entry. The remaining entries drop the last of their before-and-after framing: a first release has no 'now', and nothing in it was 'fixed'. Assisted-by: ClaudeCode:claude-fable-5
… does not
CI's ruff runs a broader rule set than the version pinned in
.pre-commit-config.yaml, so three violations passed locally and failed
there. All three are worth fixing on their own terms rather than
suppressing.
The BadIndex fixtures return 2.5 from __index__ deliberately — the point is
an object that violates the protocol, so the package can be shown rejecting
it. PLE0305 reads that as a mistake. `cast("int", 2.5)` keeps the runtime
lie the test needs while saying the lie is intentional, and drops the
`# type: ignore` it needed for mypy.
The stateful test imported `zarr_indexing.testing.stateful as stateful`;
`from zarr_indexing.testing import stateful` is the form the rest of the
suite uses, and the import block re-sorts around it.
Assisted-by: ClaudeCode:claude-fable-5
d-v-b
force-pushed
the
feat/lazy-array-wrapper
branch
from
August 11, 2026 19:21
afc52dc to
c950ef3
Compare
It was taught to substitute a local `zarr-indexing` checkout into a PEP 723 header, but no example at the repository root declares that dependency — the three there are zarr-only — and `set_dep` leaves an undeclared one alone, so the entry never fired. The runner also globs `examples/` only, so it could never have reached the package's own examples in `packages/zarr-indexing/examples/`; the package's suite asserted as much, checking they were absent from the root. Those examples do now run, as subprocesses in the package's own `test_doc_examples.py`, which is where a package's examples belong. This restores `tests/test_examples.py` to what main has: root examples, the one local package they actually use, and no knowledge of `packages/`. Assisted-by: ClaudeCode:claude-fable-5
Contributor
Author
|
I'm going to self-merge this tomorrow! The changes here are entirely scoped to |
zarr-indexing 0.1.0 is on PyPI, tagged at a994a4f. Its CHANGELOG.md holds nothing but the towncrier marker — the release was cut without ever building its notes — and I read that emptiness as "never released", then deleted the Bugfixes and Deprecations sections on the grounds that nobody could have experienced them. They could. Comparing the installed 0.1.0 against this branch, ten public names disappear: compose, selection_to_transform, transform_to_canonical, transform_from_canonical, index_transform_to_json, index_transform_from_json, index_domain_to_json, index_domain_from_json, iter_chunk_transforms and sub_transform_to_selections. Every one is an import that breaks on upgrade, and the notes said nothing about any of them. The four removal fragments and the bugfix fragment are restored, and the last two removals — the provisional tuple resolver and selector bridge, which only a feature fragment had mentioned in passing — are named. The fixes that were buried in the LazyArray feature entry move to Bugfixes where they belong, as a list rather than the nine paragraphs they were. The duplicate feature entry I had written to carry the surviving API names goes, its content being what the removal entries already say from the migrating reader's side. Assisted-by: ClaudeCode:claude-fable-5
…ying around hatchling had no sdist configuration, so a source distribution carried everything in the package directory. Building from a working tree with scratch files in it put eight of them in the tarball. A tagged release builds from a fresh CI checkout and so was never actually at risk, but nothing made that a property of the package rather than of the runner. The allowlist is chosen to keep an sdist able to test itself: tests/ carries the vendored ndsel conformance corpus, and test_doc_examples.py executes docs/snippets/*.py and examples/*/*.py, so those directories are part of the suite rather than documentation shipped for its own sake. Verified by unpacking the built sdist into a bare venv and running its tests there — 1058 pass, the rest skipping on optional dependencies. Assisted-by: ClaudeCode:claude-fable-5
The docs job already called `just docs-check`, and zarr-metadata's workflow runs entirely on `just`, but zarr-indexing's test, ruff and pyright jobs spelled their commands out again. Every recipe existed twice, so adding the src/ doctest collection meant editing both places, and forgetting either would have let them drift silently. They drift already. The justfile pinned ruff 0.15.22 while the repo moved to 0.16.0 in zarr-developers#4213 — the divergence behind this branch's lint failure — and its own comment says the two are meant to be bumped together. The pin is now 0.16.0, and because CI reads it from the justfile there is one place to bump next time. The tensorstore parity run gains the recipe it never had, so `just check` finally means what its comment claims: everything CI runs. The test job moves into the package directory to give the recipes their expected working directory, and syncs `--project ../..` so it still resolves against the repo-root environment that provides `zarr`. Verified by running every recipe CI now calls: test (1290 passed), test-tensorstore (88), lint, typecheck, docs-check. Assisted-by: ClaudeCode:claude-fable-5
Contributor
Author
|
I am ready to merge this. It does not affect here's an AI-written summary of the state of this PR: 🤖 AI text below 🤖 SummaryAdds Around that, the PR adds source-independent chunk planning, an explicit reader boundary, negative-step slice support per merged ndsel 1.0-draft.2, and a 164 commits, 93 files, +16.7k/−1.9k, essentially all under What's new
Breaking changes (from published 0.1.0)Ten public names are removed. Each has a direct replacement — operations and serialization now live on the types that own them:
Also: CorrectnessAn adversarial review pass fixed defects reachable from 0.1.0 — index-array aliasing after an integer index, Docs and infrastructureA visual guide, indexing-pattern reference, and integration guide, all with executable snippets; every public object carries a doctest that runs in CI. The ndsel conformance corpus is vendored and checked, and TensorStore parity suites now run in CI. Package CI is driven by the package's |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a generic
LazyArrayadapter class tozarr-indexing. This is a class that wraps an eagerly indexed array with a lazy indexing accessor field.lazy. The idea is to support grafting lazy indexing semantics on top of existing eager arrays. Maybe this is useful for zarr-python, we will see.written by claude, origin PR at d-v-b#267
edit: this PR evolved considerably since its creation. see this comment for an updates summary.
Author attestation
TODO
docs/user-guide/*.mdchanges/