fix: accept universal-pathlib UPath as a StoreLike value - #280
Open
d-v-b wants to merge 22 commits into
Open
Conversation
…#176) Bumps the actions group with 8 updates in the / directory: | Package | From | To | | --- | --- | --- | | [prefix-dev/setup-pixi](https://github.com/prefix-dev/setup-pixi) | `0.9.5` | `0.9.6` | | [codecov/codecov-action](https://github.com/codecov/codecov-action) | `6.0.0` | `6.0.1` | | [github/issue-metrics](https://github.com/github/issue-metrics) | `4.2.2` | `4.2.7` | | [j178/prek-action](https://github.com/j178/prek-action) | `2.0.3` | `2.0.4` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `7.0.0` | `7.0.1` | | [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` | | [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.0` | | [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.3` | `0.5.6` | Updates `prefix-dev/setup-pixi` from 0.9.5 to 0.9.6 - [Release notes](https://github.com/prefix-dev/setup-pixi/releases) - [Commits](prefix-dev/setup-pixi@1b2de7f...5185adf) Updates `codecov/codecov-action` from 6.0.0 to 6.0.1 - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@57e3a13...e79a696) Updates `github/issue-metrics` from 4.2.2 to 4.2.7 - [Release notes](https://github.com/github/issue-metrics/releases) - [Commits](github-community-projects/issue-metrics@c9e9838...1e38d5e) Updates `j178/prek-action` from 2.0.3 to 2.0.4 - [Release notes](https://github.com/j178/prek-action/releases) - [Commits](j178/prek-action@6ad8027...bdca6f1) Updates `actions/upload-artifact` from 7.0.0 to 7.0.1 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v7...043fb46) Updates `actions/download-artifact` from 7.0.0 to 8.0.1 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](actions/download-artifact@v7...3e5f45b) Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0 - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](pypa/gh-action-pypi-publish@v1.13.0...cef2210) Updates `zizmorcore/zizmor-action` from 0.5.3 to 0.5.6 - [Release notes](https://github.com/zizmorcore/zizmor-action/releases) - [Commits](zizmorcore/zizmor-action@b1d7e1f...5f14fd0) --- updated-dependencies: - dependency-name: prefix-dev/setup-pixi dependency-version: 0.9.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: codecov/codecov-action dependency-version: 6.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: github/issue-metrics dependency-version: 4.2.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: j178/prek-action dependency-version: 2.0.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: actions/download-artifact dependency-version: 8.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: pypa/gh-action-pypi-publish dependency-version: 1.14.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: zizmorcore/zizmor-action dependency-version: 0.5.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
A remote UPath now creates an FsspecStore from the filesystem and storage
options the UPath already carries, and a local UPath creates a LocalStore so
that UPath("/data") and Path("/data") agree.
This previously worked only by accident. In universal-pathlib < 0.3 every
UPath subclassed pathlib.Path and implemented __fspath__, so a remote path was
either converted to a URI string by the caller (xarray does this) or wrapped
in a LocalStore that happened to dispatch through fsspec. Since
universal-pathlib 0.3 remote paths do neither, and passing one raised
TypeError: Unsupported type for store_like.
The UPath branch is checked before the Path branch so that routing is
identical on both universal-pathlib 0.2 and 0.3.
FsspecStore.from_upath now converts the UPath's filesystem to async mode
instead of raising TypeError for synchronous filesystems and warning for
sync-mode instances of async ones.
The memory:// routing test wraps a synchronous MemoryFileSystem, which needs
fsspec's AsyncFileSystemWrapper, so it is skipped below fsspec 2024.12.0 as
test_wrap_sync_filesystem already does.
Closes zarr-developers#4244
Assisted-by: ClaudeCode:claude-fable-5
d-v-b
force-pushed
the
claude/zarr-python-issue-4244-3863cc
branch
from
August 10, 2026 20:04
63e3930 to
e023851
Compare
* deps: bump cast-value.rs to >= 0.4.2 * deps: update uv.lock for cast-value-rs >= 0.4.2 The pyproject floor alone left uv.lock pinning 0.4.0, so any lock-honoring install (uv sync --locked/--frozen) kept the version that silently transposes non-row-major input. Assisted-by: ClaudeCode:claude-fable-5 * fix(cast_value): enforce the cast-value-rs floor at runtime The pyproject floor only binds installs that go through the zarr[cast-value-rs] extra. An environment that already has an older cast-value-rs installed kept silently corrupting non-row-major input after upgrading zarr, which is the failure this floor exists to stop. Check the installed version at import and raise from _do_cast, so the error surfaces when the codec is used rather than breaking `import zarr` for everyone else. A backend without distribution metadata (a `maturin develop` build) has no version to compare and is left alone. Assisted-by: ClaudeCode:claude-fable-5 * test(cast_value): cover cast_value next to the transpose codec Regression test for zarr-developers#4237: a cast_value codec on either side of a transpose codec must round-trip, because transpose hands the next codec a non-row-major view. Imported unchanged from zarr-developers#4238, where this test was written. The np.ascontiguousarray workaround that accompanied it there is deliberately left out: cast-value-rs 0.4.2 normalizes layout itself, and the workaround promotes 0-d arrays to shape (1,), breaking 0-d arrays. Co-authored-by: Raphael Jolivet <raphael.jolivet@minesparis.psl.eu> Assisted-by: ClaudeCode:claude-fable-5 --------- Co-authored-by: Raphael Jolivet <contact@raphael-jolivet.name> Co-authored-by: Raphael Jolivet <raphael.jolivet@minesparis.psl.eu>
…pers#4264) * fix(zarr-metadata): make JSONValue's array arm covariant `list["JSONValue"] | tuple["JSONValue", ...]` is invariant in the element type, so a value typed with any narrower element -- a `list[str]` field on a TypedDict, a `Sequence[float]` -- was not assignable to `JSONValue`, and a TypedDict carrying such fields was not assignable to `Mapping[str, JSONValue]`. pyright's diagnostic for the failure suggests the fix verbatim: "Consider switching from list to Sequence which is covariant." The array arm is now `Sequence["JSONValue"]`. The docstring records the deliberate type-level cost (`Sequence` admits `str`/`bytes`; runtime narrowing must exclude them regardless of the alias's spelling). Found while aliasing zarr-cm's JsonValue to this type: the two aliases are structurally identical except for this arm, and with it changed, pyright unifies them across the package boundary. Assisted-by: ClaudeCode:claude-opus-5 * Rename 295.bugfix.md to 4264.bugfix.md
7 tasks
The changelog fragment was named for 4245, which towncrier renders as a link
to an unrelated issue ("Link Checker Report"). Rename it to the PR number.
test_make_store_upath only asserted the store type, so a mangled local path
would still produce a LocalStore and pass. Assert the root as well.
Silence PLC0414 on the UPath re-export: the alias looks redundant but is the
explicit re-export mypy requires under strict mode, and ruff 0.16.0 enables
the rule by default.
Assisted-by: ClaudeCode:claude-opus-4.8
* chore(zarr-metadata): note the sdist allowlist, and let misc entries speak zarr-developers#4248 gave this package's sdist an explicit allowlist and merged without a news fragment, so a user-visible packaging change was about to miss the 0.5.0 notes. Add one. Filing it as `misc` exposed that towncrier's built-in `misc` type sets `showcontent = false`: the entry would render as a bare PR link, which tells a reader nothing. Restate all five types the `changes/README.md` menu offers — declaring any type replaces the built-in set — as the defaults verbatim except for `misc`, which now shows its content. A change worth a release note is worth a sentence, whatever its category. Assisted-by: ClaudeCode:claude-opus-5 * chore(zarr-metadata): build 0.5.0 changelog Consume the pending news fragments — zarr-developers#4232's constant-naming-grammar removal note, zarr-developers#4264's note widening `JSONValue`'s array arm to the covariant `Sequence`, and zarr-developers#4248's sdist allowlist — into CHANGELOG.md via towncrier for the zarr_metadata-v0.5.0 release. Minor, not patch: the `JSONValue` widening changes a published type's meaning for every consumer that annotates against it, and zarr-developers#4232 removes the old version-last constant spellings outright. Assisted-by: ClaudeCode:claude-opus-5
The sentence ended in a colon introducing an example of passing storage_options alongside a UPath, but the snippet shows the working case and never passes storage_options. Worse, the `anon=True` on the UPath sits right after the sentence and reads as the thing being called out, when it is the recommended way to supply the option. State the rule as prose and let the example just show normal usage. Assisted-by: ClaudeCode:claude-opus-4.8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Closes zarr-developers#4244.
What was going on
Passing a remote
UPathto the top-level API raisedTypeError: Unsupported type for store_like: 'S3Path'. Nothing changed on zarr's side — universal-pathlib 0.3.0 (2025-09-29) rebasedUPathontopathlib_abc, and remote paths likeS3Pathstopped beingpathlib.Pathsubclasses and stopped implementing__fspath__.That broke two things that had been working by accident:
os.PathLikestore to a string before handing it to zarr, so zarr used to receive"s3://..."and route it toFsspecStore.from_url. Zarr never actually saw aUPath. With no__fspath__, the raw object now comes through.zarr.open(S3Path). I confirmed against zarr 3.1.3 + upath 0.2.6 that this used to hit theisinstance(store_like, Path)branch and build aLocalStorerooted at anS3Path— which worked only because upath 0.2 overrode every pathlib method to dispatch through fsspec.So the report is filed as a feature request, and that framing is accurate: remote
UPathsupport was never deliberate. ButFsspecStore.from_upathalready exists and universal-pathlib is already an optional dependency, so wiring it up is small.The change
make_storegains aUPathbranch. Remote UPaths go toFsspecStore.from_upath; local ones (protocol""or"file") go to aLocalStore, soUPath('/data')andPath('/data')agree, mirroring how the equivalent strings are already routed.The branch is checked before the
Pathbranch. On universal-pathlib 0.2 everyUPath— includingS3Path— subclassespathlib.Path, so otherwise remote paths would keep landing in aLocalStorethere. I verified routing is now identical on 0.2.0 and 0.3.10:UPath('s3://bucket/foo.zarr')FsspecStoreFsspecStoreUPath('memory://bucket/foo.zarr')FsspecStoreFsspecStoreUPath('file:///tmp/x.zarr')LocalStoreLocalStoreUPath('/tmp/x.zarr')LocalStoreLocalStoreDrive-by fix:
FsspecStore.from_upathpassedupath.fsstraight to the constructor without going through_make_async(unlikefrom_mapper). AUPathhands back a filesystem in whatever mode it was built with, which is synchronous unless the caller passedasynchronous=True. Sofrom_upathraisedTypeError: Filesystem needs to support async operations.for genuinely sync filesystems, and emitted the "not created withasynchronous=True" warning for sync-mode instances of async ones. Both reproduce on upath 0.3.10.storage_optionspassed alongside aUPathstill raises, since theUPathcarries its own — the existing check already covers this, and there's now a test pinning it.Notes for review
UPathwithstorage_optionsis an error rather than a merge. Ambiguous either way; erroring seemed better than silently picking one. Happy to change. Note xarray always forwardsstorage_optionsto zarr, but asNonewhen unset, so the reported case is unaffected.LocalStorerather than through fsspec'sLocalFileSystem. This is a judgement call for consistency with the string handling (and it's faster); the alternative is to send everything toFsspecStoreand keep the branch simpler.UPathre-export in_utils.pyneeded an explicitas UPathalias to satisfyno_implicit_reexport.Testing
storage_options.open_group(UPath(...))round trip reproducing the issue, and one assertingfrom_upathon a sync-modeUPathyields an async filesystem with storage options preserved and no warning.filterwarningsfor aDeprecationWarningthat universal-pathlib 0.2.0 raises from its own subclass registry when any localUPathis constructed. It fires on a bareUPath('/tmp/x')with zarr uninvolved, and would otherwise break themin_depsCI env, which pinsuniversal_pathlib==0.2.0.ruffandmypyclean. Store tests also verified against universal-pathlib 0.2.0.🤖 Generated with Claude Code