Skip to content

fix: accept universal-pathlib UPath as a StoreLike value - #280

Open
d-v-b wants to merge 22 commits into
mainfrom
claude/zarr-python-issue-4244-3863cc
Open

fix: accept universal-pathlib UPath as a StoreLike value#280
d-v-b wants to merge 22 commits into
mainfrom
claude/zarr-python-issue-4244-3863cc

Conversation

@d-v-b

@d-v-b d-v-b commented Aug 10, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Closes zarr-developers#4244.

What was going on

Passing a remote UPath to the top-level API raised TypeError: Unsupported type for store_like: 'S3Path'. Nothing changed on zarr's side — universal-pathlib 0.3.0 (2025-09-29) rebased UPath onto pathlib_abc, and remote paths like S3Path stopped being pathlib.Path subclasses and stopped implementing __fspath__.

That broke two things that had been working by accident:

  1. Via xarray (how the issue was reported). xarray converts any os.PathLike store to a string before handing it to zarr, so zarr used to receive "s3://..." and route it to FsspecStore.from_url. Zarr never actually saw a UPath. With no __fspath__, the raw object now comes through.
  2. Direct zarr.open(S3Path). I confirmed against zarr 3.1.3 + upath 0.2.6 that this used to hit the isinstance(store_like, Path) branch and build a LocalStore rooted at an S3Path — which worked only because upath 0.2 overrode every pathlib method to dispatch through fsspec.

So the report is filed as a feature request, and that framing is accurate: remote UPath support was never deliberate. But FsspecStore.from_upath already exists and universal-pathlib is already an optional dependency, so wiring it up is small.

The change

  • make_store gains a UPath branch. Remote UPaths go to FsspecStore.from_upath; local ones (protocol "" or "file") go to a LocalStore, so UPath('/data') and Path('/data') agree, mirroring how the equivalent strings are already routed.

  • The branch is checked before the Path branch. On universal-pathlib 0.2 every UPath — including S3Path — subclasses pathlib.Path, so otherwise remote paths would keep landing in a LocalStore there. I verified routing is now identical on 0.2.0 and 0.3.10:

    value 0.2.0 0.3.10
    UPath('s3://bucket/foo.zarr') FsspecStore FsspecStore
    UPath('memory://bucket/foo.zarr') FsspecStore FsspecStore
    UPath('file:///tmp/x.zarr') LocalStore LocalStore
    UPath('/tmp/x.zarr') LocalStore LocalStore
  • Drive-by fix: FsspecStore.from_upath passed upath.fs straight to the constructor without going through _make_async (unlike from_mapper). A UPath hands back a filesystem in whatever mode it was built with, which is synchronous unless the caller passed asynchronous=True. So from_upath raised TypeError: Filesystem needs to support async operations. for genuinely sync filesystems, and emitted the "not created with asynchronous=True" warning for sync-mode instances of async ones. Both reproduce on upath 0.3.10.

  • storage_options passed alongside a UPath still raises, since the UPath carries its own — the existing check already covers this, and there's now a test pinning it.

Notes for review

  • Passing a UPath with storage_options is an error rather than a merge. Ambiguous either way; erroring seemed better than silently picking one. Happy to change. Note xarray always forwards storage_options to zarr, but as None when unset, so the reported case is unaffected.
  • Local UPaths are routed to LocalStore rather than through fsspec's LocalFileSystem. This is a judgement call for consistency with the string handling (and it's faster); the alternative is to send everything to FsspecStore and keep the branch simpler.
  • The UPath re-export in _utils.py needed an explicit as UPath alias to satisfy no_implicit_reexport.

Testing

  • New parametrized routing test over the four cases in the table, plus an error-case test for storage_options.
  • New moto-backed S3 tests: an end-to-end open_group(UPath(...)) round trip reproducing the issue, and one asserting from_upath on a sync-mode UPath yields an async filesystem with storage options preserved and no warning.
  • I checked the new tests fail without the source change (5 of 6 do; the other two cases are regression guards that already passed).
  • One new test needed a filterwarnings for a DeprecationWarning that universal-pathlib 0.2.0 raises from its own subclass registry when any local UPath is constructed. It fires on a bare UPath('/tmp/x') with zarr uninvolved, and would otherwise break the min_deps CI env, which pins universal_pathlib==0.2.0.
  • Full suite: 7069 passed, 380 skipped, 4 xfailed. ruff and mypy clean. Store tests also verified against universal-pathlib 0.2.0.

🤖 Generated with Claude Code

dependabot Bot and others added 15 commits May 31, 2026 19:28
…#176)

Bumps the actions group with 8 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [prefix-dev/setup-pixi](https://github.com/prefix-dev/setup-pixi) | `0.9.5` | `0.9.6` |
| [codecov/codecov-action](https://github.com/codecov/codecov-action) | `6.0.0` | `6.0.1` |
| [github/issue-metrics](https://github.com/github/issue-metrics) | `4.2.2` | `4.2.7` |
| [j178/prek-action](https://github.com/j178/prek-action) | `2.0.3` | `2.0.4` |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `7.0.0` | `7.0.1` |
| [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` |
| [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.0` |
| [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.3` | `0.5.6` |



Updates `prefix-dev/setup-pixi` from 0.9.5 to 0.9.6
- [Release notes](https://github.com/prefix-dev/setup-pixi/releases)
- [Commits](prefix-dev/setup-pixi@1b2de7f...5185adf)

Updates `codecov/codecov-action` from 6.0.0 to 6.0.1
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](codecov/codecov-action@57e3a13...e79a696)

Updates `github/issue-metrics` from 4.2.2 to 4.2.7
- [Release notes](https://github.com/github/issue-metrics/releases)
- [Commits](github-community-projects/issue-metrics@c9e9838...1e38d5e)

Updates `j178/prek-action` from 2.0.3 to 2.0.4
- [Release notes](https://github.com/j178/prek-action/releases)
- [Commits](j178/prek-action@6ad8027...bdca6f1)

Updates `actions/upload-artifact` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v7...043fb46)

Updates `actions/download-artifact` from 7.0.0 to 8.0.1
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](actions/download-artifact@v7...3e5f45b)

Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0
- [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases)
- [Commits](pypa/gh-action-pypi-publish@v1.13.0...cef2210)

Updates `zizmorcore/zizmor-action` from 0.5.3 to 0.5.6
- [Release notes](https://github.com/zizmorcore/zizmor-action/releases)
- [Commits](zizmorcore/zizmor-action@b1d7e1f...5f14fd0)

---
updated-dependencies:
- dependency-name: prefix-dev/setup-pixi
  dependency-version: 0.9.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: codecov/codecov-action
  dependency-version: 6.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: github/issue-metrics
  dependency-version: 4.2.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: j178/prek-action
  dependency-version: 2.0.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/upload-artifact
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/download-artifact
  dependency-version: 8.0.1
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: pypa/gh-action-pypi-publish
  dependency-version: 1.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: zizmorcore/zizmor-action
  dependency-version: 0.5.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
A remote UPath now creates an FsspecStore from the filesystem and storage
options the UPath already carries, and a local UPath creates a LocalStore so
that UPath("/data") and Path("/data") agree.

This previously worked only by accident. In universal-pathlib < 0.3 every
UPath subclassed pathlib.Path and implemented __fspath__, so a remote path was
either converted to a URI string by the caller (xarray does this) or wrapped
in a LocalStore that happened to dispatch through fsspec. Since
universal-pathlib 0.3 remote paths do neither, and passing one raised
TypeError: Unsupported type for store_like.

The UPath branch is checked before the Path branch so that routing is
identical on both universal-pathlib 0.2 and 0.3.

FsspecStore.from_upath now converts the UPath's filesystem to async mode
instead of raising TypeError for synchronous filesystems and warning for
sync-mode instances of async ones.

The memory:// routing test wraps a synchronous MemoryFileSystem, which needs
fsspec's AsyncFileSystemWrapper, so it is skipped below fsspec 2024.12.0 as
test_wrap_sync_filesystem already does.

Closes zarr-developers#4244

Assisted-by: ClaudeCode:claude-fable-5
@d-v-b
d-v-b force-pushed the claude/zarr-python-issue-4244-3863cc branch from 63e3930 to e023851 Compare August 10, 2026 20:04
d-v-b and others added 2 commits August 13, 2026 21:50
* deps: bump cast-value.rs to >= 0.4.2

* deps: update uv.lock for cast-value-rs >= 0.4.2

The pyproject floor alone left uv.lock pinning 0.4.0, so any
lock-honoring install (uv sync --locked/--frozen) kept the version
that silently transposes non-row-major input.

Assisted-by: ClaudeCode:claude-fable-5

* fix(cast_value): enforce the cast-value-rs floor at runtime

The pyproject floor only binds installs that go through the
zarr[cast-value-rs] extra. An environment that already has an older
cast-value-rs installed kept silently corrupting non-row-major input
after upgrading zarr, which is the failure this floor exists to stop.

Check the installed version at import and raise from _do_cast, so the
error surfaces when the codec is used rather than breaking `import zarr`
for everyone else. A backend without distribution metadata (a `maturin
develop` build) has no version to compare and is left alone.

Assisted-by: ClaudeCode:claude-fable-5

* test(cast_value): cover cast_value next to the transpose codec

Regression test for zarr-developers#4237: a cast_value codec on either side of a
transpose codec must round-trip, because transpose hands the next codec
a non-row-major view.

Imported unchanged from zarr-developers#4238, where this test was written. The
np.ascontiguousarray workaround that accompanied it there is
deliberately left out: cast-value-rs 0.4.2 normalizes layout itself, and
the workaround promotes 0-d arrays to shape (1,), breaking 0-d arrays.

Co-authored-by: Raphael Jolivet <raphael.jolivet@minesparis.psl.eu>
Assisted-by: ClaudeCode:claude-fable-5

---------

Co-authored-by: Raphael Jolivet <contact@raphael-jolivet.name>
Co-authored-by: Raphael Jolivet <raphael.jolivet@minesparis.psl.eu>
…pers#4264)

* fix(zarr-metadata): make JSONValue's array arm covariant

`list["JSONValue"] | tuple["JSONValue", ...]` is invariant in the element
type, so a value typed with any narrower element -- a `list[str]` field on
a TypedDict, a `Sequence[float]` -- was not assignable to `JSONValue`, and
a TypedDict carrying such fields was not assignable to
`Mapping[str, JSONValue]`. pyright's diagnostic for the failure suggests
the fix verbatim: "Consider switching from list to Sequence which is
covariant."

The array arm is now `Sequence["JSONValue"]`. The docstring records the
deliberate type-level cost (`Sequence` admits `str`/`bytes`; runtime
narrowing must exclude them regardless of the alias's spelling).

Found while aliasing zarr-cm's JsonValue to this type: the two aliases
are structurally identical except for this arm, and with it changed,
pyright unifies them across the package boundary.

Assisted-by: ClaudeCode:claude-opus-5

* Rename 295.bugfix.md to 4264.bugfix.md
d-v-b added 5 commits August 14, 2026 13:55
The changelog fragment was named for 4245, which towncrier renders as a link
to an unrelated issue ("Link Checker Report"). Rename it to the PR number.

test_make_store_upath only asserted the store type, so a mangled local path
would still produce a LocalStore and pass. Assert the root as well.

Silence PLC0414 on the UPath re-export: the alias looks redundant but is the
explicit re-export mypy requires under strict mode, and ruff 0.16.0 enables
the rule by default.

Assisted-by: ClaudeCode:claude-opus-4.8
* chore(zarr-metadata): note the sdist allowlist, and let misc entries speak

zarr-developers#4248 gave this package's sdist an explicit allowlist and merged without a
news fragment, so a user-visible packaging change was about to miss the
0.5.0 notes. Add one.

Filing it as `misc` exposed that towncrier's built-in `misc` type sets
`showcontent = false`: the entry would render as a bare PR link, which tells
a reader nothing. Restate all five types the `changes/README.md` menu offers
— declaring any type replaces the built-in set — as the defaults verbatim
except for `misc`, which now shows its content. A change worth a release
note is worth a sentence, whatever its category.

Assisted-by: ClaudeCode:claude-opus-5

* chore(zarr-metadata): build 0.5.0 changelog

Consume the pending news fragments — zarr-developers#4232's constant-naming-grammar removal
note, zarr-developers#4264's note widening `JSONValue`'s array arm to the covariant
`Sequence`, and zarr-developers#4248's sdist allowlist — into CHANGELOG.md via towncrier
for the zarr_metadata-v0.5.0 release.

Minor, not patch: the `JSONValue` widening changes a published type's
meaning for every consumer that annotates against it, and zarr-developers#4232 removes the
old version-last constant spellings outright.

Assisted-by: ClaudeCode:claude-opus-5
The sentence ended in a colon introducing an example of passing
storage_options alongside a UPath, but the snippet shows the working case and
never passes storage_options. Worse, the `anon=True` on the UPath sits right
after the sentence and reads as the thing being called out, when it is the
recommended way to supply the option.

State the rule as prose and let the example just show normal usage.

Assisted-by: ClaudeCode:claude-opus-4.8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Universal-pathlib support for zarr.open

1 participant