From 8d045a67526261066a246848d550e302cafa17f1 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 17 Aug 2026 20:26:04 +0000 Subject: [PATCH] lifecycle: record hpc-sync-first-push-race as complete (shipped 2026-08-17) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The hpc/sync first-push race shipped today across every assistant carrying the script, so the prompt advances draft/ -> complete/2026/08/. Written via `lifecycle.py record --prompt` (the prompt was staged through active/ so the script could fold and remove it; it never had a GitHub issue — the work ran straight from the draft). complete/index.md regenerated with `lifecycle.py index --apply`, dashboard.md with `intake --apply dashboard`; `lifecycle.py check` passes. Merged: autolens_assistant#112 (4b4d8a7), autofit_assistant#29 (8acfbe1), autogalaxy_assistant#15 (c06f190), PyAutoBrain#228 (b909804). --- complete/2026/08/hpc-sync-first-push-race.md | 107 ++++++++++++++++++ complete/index.md | 3 +- dashboard.md | 8 +- .../hpc_sync_first_push_race.md | 14 --- 4 files changed, 112 insertions(+), 20 deletions(-) create mode 100644 complete/2026/08/hpc-sync-first-push-race.md delete mode 100644 draft/bug/autolens_assistant/hpc_sync_first_push_race.md diff --git a/complete/2026/08/hpc-sync-first-push-race.md b/complete/2026/08/hpc-sync-first-push-race.md new file mode 100644 index 00000000..c0798284 --- /dev/null +++ b/complete/2026/08/hpc-sync-first-push-race.md @@ -0,0 +1,107 @@ +- completed: 2026-08-17 +- issue: none — worked straight from the draft prompt +- repos: + - autolens_assistant + - autofit_assistant + - autogalaxy_assistant + - PyAutoBrain +- prs: + - https://github.com/PyAutoLabs/autolens_assistant/pull/112 + - https://github.com/PyAutoLabs/autofit_assistant/pull/29 + - https://github.com/PyAutoLabs/autogalaxy_assistant/pull/15 + - https://github.com/PyAutoLabs/PyAutoBrain/pull/228 +- merge-commits: autolens_assistant `4b4d8a7` · autofit_assistant `8acfbe1` · + autogalaxy_assistant `c06f190` · PyAutoBrain `b909804` (all 2026-08-17) +- summary: `hpc/sync push()` fired the `CODE_DIRS` rsyncs in parallel before + anything created `${HPC_BASE}/${PROJECT_NAME}`. rsync only creates the last + path level, so on a first push all of them died with `mkdir failed: No such + file or directory` — then the `[root files]` rsync created the base dir, so + `dataset/` synced and the command exited 0. Fixed by `ssh mkdir -p` before the + parallel rsyncs, plus per-PID `wait` so a failed background rsync is no longer + swallowed. Landed in all three assistants carrying the script. +- validation: fake-HPC harness (real rsync, stubbed ssh) — reproduced the exit-0 + failure per repo, then confirmed the full tree transfers; injected failure now + exits 1 and stops `push-submit` before `sbatch`; re-push, `--no-data` and + `status` unchanged. +- release: n/a — assistant repos, no package release. + +## The silence was the bug, not the mkdir + +The missing `mkdir` is a one-line fix. What made it cost a GPU job was that +`push()` backgrounded each rsync with `&` and then called bare `wait`. + +**`wait` with no arguments reports its own status, not the jobs'.** So under +`set -euo pipefail` five failed rsyncs still left `$?` at 0. The `[root files]` +rsync that ran next happened to create the base directory, so `dataset/` — the +slow, visible part — synced perfectly. Every signal a human reads said success. + +The failure surfaced only later, as `sbatch` unable to find `hpc/batch_gpu` +(job 330464, slope_hierarchy first push, 2026-07-16). + +So the fix is two changes, and the second is the durable one: collect the PIDs, +`wait` on each, name the directory that failed, and return 1. A first push that +half-lands now stops `push-submit` before it queues work against an incomplete +tree, instead of deferring the error to SLURM. + +## It was three repos, not one + +The prompt named `autolens_assistant`. `hpc/sync` is cloned across the +assistants, so the same defect sat in two more: + +- `autofit_assistant` — blob `1560e64`, **byte-identical** to the file the bug + was reported against. +- `autogalaxy_assistant` — differs only in `CODE_DIRS` (no `slam_pipeline`); + `push()` identical, upstream patch applied with no conflicts. +- `autocti_assistant` — has `hpc/` but no `sync` script. Unaffected. + +Each was reproduced independently before fixing rather than assumed from the +shared ancestry. Worth remembering for the next `hpc/` change: the script has +three live copies and no shared source. + +## Trap: the fix is invisible to the repo's own CI + +Nothing in these repos tests `hpc/sync` — it needs a real SSH endpoint. The +whole verification was a local harness that stubs `ssh` (runs the command +locally) and wraps **real** rsync with a `HOST:path` → `path` rewrite, so +rsync's actual one-level-mkdir behaviour is exercised rather than mocked. That +distinction is the point: a mocked rsync would have "passed" against the buggy +script. + +`hpc/sync` is also mode `100644` in `autolens_assistant` but `100755` in the +other two — noticed while patching, left alone as out of scope. + +## Detour: a red `boundary` check that was not ours + +`autolens_assistant#112` opened onto a failing `clone-boundary` job — +`CHOOSING_YOUR_AI_TOOL.md` unclassified. It predated the branch (it arrived in +`64018f8`, "Release 2026.8.17.1", the PR's own base commit) and so was failing +**every** PR against the repo, and blocking every future assistant birth. + +The classification was already decided: `modes/maintainer.md`, the prose that +owns the boundary, files the file under `**Mixed**` alongside `llms-chat.txt`. +Only PyAutoBrain's `_clone.py` lacked the pattern — the two sources had drifted. +Fixed in `_SHARED_MIXED` (PyAutoBrain#228), which is a one-repo change; no +`maintainer.md` edit was needed. + +Useful mechanism found while doing it: `clone-boundary.yml` honours a +`Brain-ref: ` line in the PR body, running the boundary against that +PyAutoBrain ref instead of `main`, so paired PRs can both be green before an +ordered merge. Note that **editing a PR body does not re-trigger +`pull_request`** — the declaration only takes effect on a fresh event. + +## Original prompt + +# hpc/sync first-push race — parallel rsyncs before remote base dir exists + +Type: bug +Target: autolens_assistant +Repos: +- autolens_assistant +Difficulty: small +Autonomy: safe +Priority: normal +Status: formalised + +Bug in autolens_assistant hpc/sync: on the FIRST push to a new remote project, push() launches the CODE_DIRS rsyncs in parallel before anything has created the remote base directory, so all of them fail with 'mkdir failed: No such file or directory' (rsync only creates one path level). The '[root files]' rsync then creates the base dir, so dataset/ syncs and the overall command exits 0 — the failure is silent until sbatch can't find hpc/batch_gpu. Fix: ssh mkdir -p the remote project dir before the parallel rsyncs (or add --mkpath). Found during slope_hierarchy first push (job 330464 postmortem, 2026-07-16). + + diff --git a/complete/index.md b/complete/index.md index 34ec8c15..055a0feb 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -1000 records across 7 buckets. +1001 records across 7 buckets. ## Highlights @@ -60,6 +60,7 @@ _(curate hard-won records here — survives regeneration.)_ - [heart-red-guarded-sample-escape](2026/08/heart-red-guarded-sample-escape.md) — Closed the escape path that kept release validation RED after - [howto-smoke-all-tutorials](2026/08/howto-smoke-all-tutorials.md) - [howtofit-stub-fork-cleanup](2026/08/howtofit-stub-fork-cleanup.md) +- [hpc-sync-first-push-race](2026/08/hpc-sync-first-push-race.md) — `hpc/sync push()` fired the `CODE_DIRS` rsyncs in parallel before - [hst-dq-bits-dial](2026/08/hst-dq-bits-dial.md) - [hygiene-coverage-drift](2026/08/hygiene-coverage-drift.md) - [hygiene-detail-flag](2026/08/hygiene-detail-flag.md) diff --git a/dashboard.md b/dashboard.md index 75eefc14..726448f5 100644 --- a/dashboard.md +++ b/dashboard.md @@ -11,7 +11,7 @@ Tasks only — the organism's health lives with the Heart (`/health`), not here. | [In flight](#in-flight) (`active/`) | 7 | | [Parked](#parked) (`parked.md`) | 6 | | [Planned](#planned) (`planned.md`) | 7 | -| [Backlog](#backlog) (`draft/`) | 141 | +| [Backlog](#backlog) (`draft/`) | 140 | Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3Aissue+is%3Aopen&type=issues) · [open pull requests](https://github.com/search?q=org%3APyAutoLabs+is%3Apr+is%3Aopen&type=prs) @@ -34,7 +34,6 @@ Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3A **Quick wins** (small enough, and safe enough to run unattended) -- [hpc/sync first-push race — parallel rsyncs before remote base dir](draft/bug/autolens_assistant/hpc_sync_first_push_race.md) — autolens_assistant · small · safe · normal - [Audit HowTo tutorials for missing setup_notebook() line](draft/bug/howto/missing_setup_notebook_audit.md) — howto · small · safe · normal - [PyAutoFit CLI-noise batch: unclosed search.log handler + four small warning](draft/maintenance/pyautofit/cli_noise_pyautofit_batch.md) — pyautofit · small · safe · normal - [Tenant firewall: release_run.py carries an unlisted 'PyAutoLabs' instance fact](draft/bug/pyautoheart/tenant_firewall_release_run_instance_fact.md) — pyautoheart · small · safe · normal @@ -93,10 +92,10 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**141** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). +**140** filed prompts, not started. Each section is sorted most-pickable first (priority, then size).
-bug — 42 +bug — 41 - [pre_build stages untracked files, publishing uncommitted human work](draft/bug/pyautohands/pre_build_stages_untracked_wip.md) — pyautohands · small · supervised · high - [UniformPrior bounds are not enforced in the objective on the](draft/bug/autofit/uniform_prior_bounds_unenforced_on_numpy_path.md) — autofit · medium · human-required · high @@ -119,7 +118,6 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. - [`autofit.plot` functions accept `**kwargs` and silently discard them](draft/bug/autofit/plot_functions_discard_kwargs.md) — autofit · small · supervised · normal - [TEST_MODE bypass crashes on ordered-parameter assertion ties](draft/bug/autofit/test_mode_bypass_ordered_assertion_ties.md) — autofit · small · supervised · normal - [point.py JAX-vmap parity assert is non-deterministic under the smoke env](draft/bug/autolens/point_jax_vmap_parity_nondeterministic.md) — autolens · small · supervised · normal -- [hpc/sync first-push race — parallel rsyncs before remote base dir](draft/bug/autolens_assistant/hpc_sync_first_push_race.md) — autolens_assistant · small · safe · normal - [Scripts derive geometry from a hardcoded pixel_scale while the dataset](draft/bug/autolens_workspace/script_local_pixel_scale_vs_dataset_pixel_scales.md) — autolens_workspace · small · supervised · normal - [Audit HowTo tutorials for missing setup_notebook() line](draft/bug/howto/missing_setup_notebook_audit.md) — howto · small · safe · normal - [HowToGalaxy small API drifts: ellipse kwargs + plot_grid_lines (parked NEEDS_FIX)](draft/bug/howtogalaxy/small_api_drift_ellipse_and_plot_grid_lines.md) — howtogalaxy · small · supervised · normal diff --git a/draft/bug/autolens_assistant/hpc_sync_first_push_race.md b/draft/bug/autolens_assistant/hpc_sync_first_push_race.md deleted file mode 100644 index 4c7024dc..00000000 --- a/draft/bug/autolens_assistant/hpc_sync_first_push_race.md +++ /dev/null @@ -1,14 +0,0 @@ -# hpc/sync first-push race — parallel rsyncs before remote base dir exists - -Type: bug -Target: autolens_assistant -Repos: -- autolens_assistant -Difficulty: small -Autonomy: safe -Priority: normal -Status: formalised - -Bug in autolens_assistant hpc/sync: on the FIRST push to a new remote project, push() launches the CODE_DIRS rsyncs in parallel before anything has created the remote base directory, so all of them fail with 'mkdir failed: No such file or directory' (rsync only creates one path level). The '[root files]' rsync then creates the base dir, so dataset/ syncs and the overall command exits 0 — the failure is silent until sbatch can't find hpc/batch_gpu. Fix: ssh mkdir -p the remote project dir before the parallel rsyncs (or add --mkpath). Found during slope_hierarchy first push (job 330464 postmortem, 2026-07-16). - -