fix chunk/shard iteration

[d-v-b]

In main there are some routines for iterating over the chunks of an array, but these routines do not distinguish between chunks and shards (i.e., stored objects) for arrays with sharding.

This PR adds a separate set of shard-specific iteration routines to complement our chunk-specific iteration routines. Various bugs related to iterating over chunks, when shards were the intended iteration target, have been fixed by these changes, notably bugs causing memory races when creating arrays via create_array (xref ##3169)

I think this supersedes #3217, @bojidar-bg I credited you as a co-author on one of these commits because your idea to change the iteration from chunks to shards was correct.

[d-v-b]

I added a new property called chunk_grid_shape because cdata_shape is ambiguous. cdata_shape is still around, but it now uses chunk_grid_shape

[d-v-b]

this complements chunk_grid_shape.

[bojidar-bg]

Ooh, that looks like a much more complete implementation of what I did in that other PR! Kudos; I wouldn't have been able to push things this far ✨✨

[d-v-b]

Ooh, that looks like a much more complete implementation of what I did in that other PR! Kudos; I wouldn't have been able to push things this far ✨✨

It turned out to be more than I expected 😅 .

[d-v-b]

this is a new function for iterating over contiguous regions. When we support irregular chunking, we can overload the type of region_shape accordingly.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.69%. Comparing base (9498336) to head (dfb4ee5).
⚠️ Report is 79 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3299      +/-   ##
==========================================
+ Coverage   94.62%   94.69%   +0.07%     
==========================================
  Files          79       79              
  Lines        9468     9520      +52     
==========================================
+ Hits         8959     9015      +56     
+ Misses        509      505       -4     

Files with missing lines	Coverage Δ
src/zarr/core/array.py	97.44% <100.00%> (+0.33%)	⬆️
src/zarr/core/indexing.py	96.40% <100.00%> (+0.31%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dstansby]

Just to make sure we're on the same page here, this is where we're heading in my mind, as something that can go in the user guide:

In zarr-python, a chunk of data represents a single file in storage. There is one exception: when a single sharding codec is part of the codecs. In this case a shard represents a single file in storage, and a chunk represents a unit of data the same size or smaller that can be read independently of other chunks in the shard. This distinction is made to allow zarr-python to optimize the creation and processing of sharded arrays.

[dcherian]

LGTM. I have some concerns about better testing these functions, but they can be addressed later.

[lumberbot-app]

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

git checkout 3.1.x
git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 c9eefe663b34b019e9673bcaa37a88aae6158280

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #3299: fix chunk/shard iteration'

4. Push to a named branch:

git push YOURFORK 3.1.x:auto-backport-of-pr-3299-on-3.1.x

5. Create a PR against branch 3.1.x, I would have named this PR:

"Backport PR #3299 on branch 3.1.x (fix chunk/shard iteration)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.