Skip to content

fix(store): retry the LocalStore rename when the destination is busy - #4358

Merged
d-v-b merged 1 commit into
zarr-developers:mainfrom
Tomatokeftes:fix/retry-busy-destination-on-local-store
Sep 15, 2026
Merged

d-v-b merged 1 commit into
zarr-developers:mainfrom
Tomatokeftes:fix/retry-busy-destination-on-local-store

Conversation

@Tomatokeftes

@Tomatokeftes Tomatokeftes commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Summary

Zarr v2 retried DirectoryStore's rename because Windows intermittently refuses
to replace a destination (#597, fixed by #698). Atomic writes arrived in v3's
LocalStore in #3412 without that retry, so the failure is back — _atomic_write's
tmp_path.replace(path) raises

PermissionError: [WinError 5] Access is denied:
  '...zarr.<hex>.partial' -> '...zarr.json'

and aborts the write. That's #3522, and @d-v-b already suggested on #597 that #698's
retry should be ported over. This does that.

_move_with_retry wraps the final move and retries only the two Windows codes that
mean the destination couldn't be superseded right now. It needs no platform check:
off Windows an OSError carries no winerror, so the first attempt either succeeds
or raises, and the tests still run everywhere.

The exclusive path goes through it too but is unaffected by construction — the
FileExistsError it relies on to report an existing node is ERROR_ALREADY_EXISTS
(183), which isn't in the retried set, so it still propagates on the first attempt.

On Windows 11, 4,000 group-attr rewrites (each a replace onto an existing
zarr.json): 155–171 raised before, 0 after, for 3.68 s → 4.08 s of wall clock on a
workload that is nothing but replace-onto-existing.

Two things that may be worth knowing, since #3522 is framed as a multiprocessing
problem and #698 is titled as an antivirus one:

  • it isn't concurrency — it reproduces single-threaded, in one process, with zarr not
    imported, at roughly 4% of replaces onto an existing destination;
  • it doesn't look like antivirus either — replacing a fresh name never failed here
    (0 in 4,000) while reusing the same name fails ~5%, same write pattern and same
    scanner. That points at NTFS file tunnelling, and there's a
    Microsoft thread
    where it reproduces with Defender disabled and is gone on Server 2025.

Either way the retry is the same fix.

For reviewers

The bit I'd most value a second look at is routing the exclusive branch through
_move_with_retry at all. I believe 183 excludes it safely and
test_move_with_retry_does_not_retry_other_errors pins that, but if you'd rather it
bypassed the retry entirely that's a one-line change.

I'd also welcome a view on _RETRY_DELAYS. It sums to under a second so a genuine
failure still surfaces promptly, and in a 20,000-replace loop 475 of 498 recoveries
needed only the second attempt with the worst needing the fourth — but the tail is
judgement, not measurement.

Behaviour is unchanged off Windows.

Author attestation

  • I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

TODO

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.md — n/a, internal behaviour
  • Changes documented as a new file in changes/
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

Zarr v2 retried DirectoryStore's rename because Windows intermittently
refuses to replace a destination (zarr-developers#597, fixed by zarr-developers#698). Atomic writes
arrived in v3's LocalStore in zarr-developers#3412 without that retry, so the failure is
back: _atomic_write's tmp_path.replace(path) raises

    PermissionError: [WinError 5] Access is denied:
      '...zarr.<hex>.partial' -> '...zarr.json'

and aborts the write. Reported in zarr-developers#3522.

_move_with_retry wraps the final move, retrying only the two Windows codes
that mean the destination could not be superseded right now. It needs no
platform test: off Windows an OSError carries no winerror, so the first
attempt either succeeds or raises.

The exclusive path is routed through it too but is unaffected by
construction -- the FileExistsError it relies on to report an existing node
is ERROR_ALREADY_EXISTS (183), which is not in the retried set, so it still
propagates on the first attempt.

Measured on Windows 11, 4,000 group-attr rewrites (each a replace onto an
existing zarr.json): 155-171 raised before, 0 after, for 3.68 s -> 4.08 s of
wall clock on a workload that is nothing but replace-onto-existing. In a
narrower stdlib-only loop of 20,000 replaces, 475 of 498 recoveries needed
only the second attempt and the worst needed the fourth.
@codecov

codecov Bot commented Sep 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.34%. Comparing base (0de6077) to head (4bb856d).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4358   +/-   ##
=======================================
  Coverage   94.34%   94.34%           
=======================================
  Files          92       92           
  Lines       12935    12950   +15     
=======================================
+ Hits        12203    12218   +15     
  Misses        732      732           
Files with missing lines Coverage Δ
src/zarr/storage/_local.py 97.69% <100.00%> (+0.17%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@d-v-b

d-v-b commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

this looks good, I'm going to merge as-is then apply some polishing changes in a follow-up PR

@d-v-b
d-v-b merged commit 6947dd8 into zarr-developers:main Sep 15, 2026
39 checks passed
d-v-b added a commit that referenced this pull request Sep 15, 2026
Follow-up cleanup to the retry landed in #4358, no behaviour change:

- drop the leading 0.0 sentinel, the bare `last_error` annotation, and
  the possibly-unbound re-raise from `_move_with_retry`; the final
  attempt now runs after the loop and propagates its own traceback
- correct the comment on ERROR_ACCESS_DENIED: Windows also reports it
  for destinations that will never clear (directory, read-only, ACL),
  so those surface the same error after the bounded delay
- align the changelog with the code: the exclusive path is routed
  through the retry, only FileExistsError is excluded; rename the
  fragment to the PR number and describe #3522 as mitigated, since a
  second process holding the destination open past the budget still
  fails
- tests record `time.sleep` instead of sleeping (~1.2 s per run before,
  now instant) and assert the actual delay schedule; the closure test
  double becomes a small callable class, removing four type-ignores
- add a regression test for `_atomic_write` onto an existing directory

Assisted-by: ClaudeCode:claude-fable-5-1

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Sep 16, 2026
Resolves the import conflict in tests/test_store/test_local.py: main's
LocalStore rename-retry tests (zarr-developers#4358, zarr-developers#4359) import `time`, this branch's
event-loop detector imports `dataclass`/`field` and typing names. Both are
kept.

The retry helpers main added (`_move_with_retry`, `_atomic_write`) are only
reached from `_put`, which this branch already runs under `asyncio.to_thread`,
so their `time.sleep` blocks a worker thread rather than the event loop and the
detector stays quiet. Main's new tests are module-level, so the class-scoped
autouse detector does not apply to them.

Assisted-by: ClaudeCode:claude-opus-5
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants