Fix conda-create cascade-restore gap (Active Backlog Item 23)

.github/workflows/batch-check.yml

@@ -768,6 +768,32 @@ jobs:
         run: |
           & tests\selfapps_cascade.ps1
 
+      # CLAUDE.md Active Backlog Item 23: a genuine conda-create failure during a REQ-009 cascade
+      # re-entry previously fell through :die instead of gracefully keeping the previous working
+      # uv build. Must run AFTER the step directly above (selfapps_cascade.ps1) so Miniconda is
+      # already installed/cached from that step -- otherwise :cascade_acquire_conda would need a
+      # real Miniconda download of its own, same CI-ordering reasoning as
+      # selfapps_conda_bothfail.ps1's own placement note, just the opposite direction.
+      - name: "Self-test: cascade-reentry conda-create failure keeps previous build (uv lane, non-gating)"
+        if: ${{ matrix.mode == 'uv' }}
+        continue-on-error: true
+        shell: pwsh
+        run: |
+          & tests\selfapps_cascade_conda_create_fail.ps1
+
+      # PR #413 CodeRabbit review finding: the scenario above only exercises the fix's
+      # :conda_create_failed call site (create itself fails); this exercises the OTHER call site
+      # inside :conda_create_done (create genuinely succeeds but python.exe is missing
+      # afterward). Same placement requirement as the step above (Miniconda already cached).
+      - name: "Self-test: cascade-reentry conda-create missing python.exe keeps previous build (uv lane, non-gating)"
+        if: ${{ matrix.mode == 'uv' }}
+        continue-on-error: true
+        shell: pwsh
+        env:
+          CASCADE_CCF_SCENARIO: missing_python
+        run: |
+          & tests\selfapps_cascade_conda_create_fail.ps1
+
       # docs/agent-closed-backlog.md's Item 22: a real, non-simulated end-to-end test proving the
       # uv-to-conda cascade, warnfix repair (both success and failure in the same round), and
       # --hidden-import auto-recovery all fire for real in ONE run -- no HP_TEST_FORCE_*/

CLAUDE.md

@@ -523,57 +523,6 @@ Once an item is fully resolved it is removed from here entirely and archived (ke
 original number) in `docs/agent-closed-backlog.md`, which is why the numbering below does not
 start at 1 and has gaps.
 
-- **Item 23: a genuine (non-test) conda-create failure during a REQ-009 cascade re-entry does
-  not restore the previous working build via `HP_CASCADE_SAVED_PY`, unlike every other
-  cascade-target failure.** Found via a CodeRabbit review finding on PR #412, verified by
-  reading the source directly (not taken on faith; re-verified again while fixing this entry's
-  own citations, which corrected the mechanism description below). `:try_conda_create`'s own
-  failure label (`:conda_create_failed`) does NOT hard-fail immediately -- it first calls
-  `:handle_conda_failure`, the same linear embed/venv/system fallback chain the ORIGINAL
-  (non-cascade) conda-create failure path already relies on; if any of those tiers succeeds,
-  `HP_ENV_READY` is set and control correctly `goto :after_env_mode_selection`. Only when
-  `:handle_conda_failure` ALSO exhausts every tier does `:conda_create_failed` fall through to
-  `call :die`, and only then does the actual bug surface: `:die` returns via `exit /b`
-  (subroutine return, not a process halt; see `docs/agent-lessons-learned.md`'s `:die` entry),
-  so execution falls straight through past it into `:conda_create_done`, which sets
-  `HP_PY=%CONDA_PREFIX%\python.exe` and checks `if not exist "%HP_PY%"` -- true in this case, so
-  it retries the identical `:handle_conda_failure` chain a second time (redundant, since nothing
-  changed) before a second `call :die`, after which execution again falls through, now carrying
-  a genuinely broken `HP_PY` into whatever code follows. Neither fall-through ever routes through
-  `:after_cascade_decision` (the label every OTHER cascade-target failure --
-  `:cascade_conda_unavailable`, `:cascade_embed_unavailable`, `:cascade_venv_unavailable`,
-  `:cascade_system_unavailable` -- correctly uses, logging `[WARN] ... unavailable; keeping
-  current build.` and restoring `HP_CASCADE_SAVED_PY` into `HP_PY` so the bootstrap gracefully
-  continues on the PREVIOUS successful build, e.g. uv, if cascading uv-to-conda). `:try_conda_
-  create`'s failure handling has no cascade-context-awareness at all -- it behaves identically
-  whether this is the very first creation attempt (where there is no earlier build to restore,
-  so eventually hard-failing is correct) or a `:cascade_from_uv` re-entry (where `HP_CASCADE_
-  SAVED_PY` holds a known-working uv build that never gets restored).
-  **Practical impact, tempered but real:** `:die` already sets `HP_BOOTSTRAP_STATE=error`
-  unconditionally as of an earlier fix (see `docs/agent-lessons-learned.md`), so the FINAL
-  `~bootstrap.status.json` should still correctly read `state=error` rather than falsely
-  claiming success -- this is not the same severity as the empty-interpreter-command bug the
-  prior commit fixed. But a genuine, plausibly-transient conda-create failure during a cascade
-  (e.g. a real network blip while acquiring Miniconda on demand for the cascade, distinct from
-  "Miniconda not installed at all" which `:cascade_conda_unavailable` already handles gracefully)
-  now hard-fails the WHOLE bootstrap instead of gracefully keeping the already-working uv build,
-  and burns through the doubled `:handle_conda_failure` retry plus whatever broken-`HP_PY` log
-  noise follows before a terminal point is reached.
-  **Deliberately not fixed in the same commit as the Item 22 companion fix** -- properly fixing
-  this needs `:try_conda_create`'s failure branches to become cascade-context-aware (e.g. check
-  `if defined HP_CASCADE_APPROVED` or an equivalent re-entry signal and route to
-  `:after_cascade_decision`'s "keeping current build" pattern instead of `:die`, but ONLY when
-  there is a genuine earlier build to fall back to -- the first-attempt case must keep failing
-  hard), plus a new regression test that forces a genuine (not `HP_TEST_FORCE_CONDA_FAIL`-style
-  simulated) conda-create failure specifically during a cascade re-entry, which does not
-  currently exist. This is real design work, not a quick fix -- scoping it into the same commit
-  as the cStringIO warnfix-filter fix would have risked a rushed, undertested change to
-  already-sensitive cascade logic. `:hp_test_conda_fail` (the existing `HP_TEST_FORCE_CONDA_FAIL`
-  test hook) has the identical fallthrough shape and reaches the same `:after_env_mode_selection`
-  clear, but is scoped to the FIRST-attempt path only (its only call site is the top of
-  `:try_conda_create`, before any cascade re-entry could reach it) -- it is NOT itself evidence
-  this gap is already covered by existing tests.
-
 - **Item 24: PyInstaller does not bundle `pygrib`'s native `eccodes.dll` dependency under the
   conda provider, so the frozen EXE fails at runtime even though the build itself succeeds.**
   Found via `self.layered_e2e.chain`'s real CI evidence (run `30875520181`, cache-lane job

docs/agent-closed-backlog.md

@@ -1160,6 +1160,75 @@ this belongs to).
    and non-gating (`cache` lane, `continue-on-error`) in the interim, so no CI lane that gates PR
    merges is affected.
 
+### Item 23 (closed 2026-08-04)
+
+- **A genuine (non-test) conda-create failure during a REQ-009 cascade re-entry did not restore
+  the previous working build via `HP_CASCADE_SAVED_PY`, unlike every other cascade-target
+  failure.** Found via a CodeRabbit review finding on PR #412, verified by reading the source
+  directly. `:try_conda_create`'s own failure label (`:conda_create_failed`) does NOT hard-fail
+  immediately -- it first calls `:handle_conda_failure`, the same linear embed/venv/system
+  fallback chain the ORIGINAL (non-cascade) conda-create failure path already relies on; if any
+  of those tiers succeeds, `HP_ENV_READY` is set and control correctly `goto
+  :after_env_mode_selection`. Only when `:handle_conda_failure` ALSO exhausts every tier does
+  `:conda_create_failed` fall through to `call :die`, and only then did the actual bug surface:
+  `:die` returns via `exit /b` (subroutine return, not a process halt; see
+  `docs/agent-lessons-learned.md`'s `:die` entry), so execution fell straight through past it
+  into `:conda_create_done`, which sets `HP_PY=%CONDA_PREFIX%\python.exe` and checks `if not
+  exist "%HP_PY%"` -- true in this case, so it retried the identical `:handle_conda_failure`
+  chain a second time (redundant, since nothing changed) before a second `call :die`, after which
+  execution again fell through, now carrying a genuinely broken `HP_PY` into whatever code
+  followed. Neither fall-through ever routed through `:after_cascade_decision` (the label every
+  OTHER cascade-target failure -- `:cascade_conda_unavailable`, `:cascade_embed_unavailable`,
+  `:cascade_venv_unavailable`, `:cascade_system_unavailable` -- correctly uses, logging `[WARN]
+  ... unavailable; keeping current build.` and restoring `HP_CASCADE_SAVED_PY` into `HP_PY` so
+  the bootstrap gracefully continues on the PREVIOUS successful build, e.g. uv, if cascading
+  uv-to-conda). `:try_conda_create`'s failure handling had no cascade-context-awareness at all --
+  it behaved identically whether this was the very first creation attempt (where there is no
+  earlier build to restore, so eventually hard-failing is correct) or a `:cascade_from_uv`
+  re-entry (where `HP_CASCADE_SAVED_PY` holds a known-working uv build that never got restored).
+  **Fixed** by inserting `if defined HP_CASCADE_SAVED_PY goto :cascade_conda_create_failed`
+  immediately before both `call :die` fall-through sites (`:conda_create_failed`'s own line and
+  the companion `python.exe`-missing check inside `:conda_create_done`), and adding a new
+  `:cascade_conda_create_failed` label mirroring the existing sibling template exactly -- logs
+  `[WARN] REQ-009: cascade target conda create failed; keeping current build.` and `goto
+  :after_cascade_decision`, deliberately never calling `:die` (that label's own restore logic
+  only works correctly when `HP_BOOTSTRAP_STATE` is left as whatever it already was -- the prior
+  successful build's `ok` -- not overwritten to `error`). `HP_CASCADE_SAVED_PY`-definedness is a
+  safe signal to distinguish "first attempt" (never defined) from "cascade re-entry" (always
+  defined by `:provider_cascade` before dispatching to `:cascade_from_uv`, the only cascade
+  source that can reach `:try_conda_create`).
+  **New regression test, forcing a GENUINE (not `HP_TEST_FORCE_CONDA_FAIL`-style simulated)
+  failure through the real create/retry code path** -- `tests/selfapps_cascade_conda_create_fail.
+  ps1` (uv lane, non-gating, `self.cascade.conda_create_fail`; see `docs/agent-ndjson.md` for the
+  full assertion list). New hook `HP_TEST_FORCE_CONDA_CREATE_BOTH_FAIL=1` (distinct from the
+  existing `HP_TEST_FORCE_CONDA_CREATE_NETWORK_FAIL`, which only fails the first attempt then
+  clears itself so the retry can genuinely succeed) persists through both the initial attempt and
+  the retry, so `:conda_create_failed` is reached deterministically without depending on real
+  network conditions. Combined with `HP_TEST_FORCE_EMBED_FAIL=1`/`HP_TEST_FORCE_VENV_FAIL=1`/
+  `HP_TEST_SYSCON_ANSWER=N` to exhaust `:handle_conda_failure`'s own fallback chain so the fix's
+  new check is actually reached. Placement is load-bearing: wired into `batch-check.yml`
+  immediately AFTER `selfapps_cascade.ps1`'s own step so Miniconda is already cached from that
+  step (mirrors `self.conda.bothfail`'s own placement note, opposite direction). `:hp_test_conda_
+  fail` (the existing `HP_TEST_FORCE_CONDA_FAIL` test hook) has the identical fallthrough shape
+  and reaches the same `:after_env_mode_selection` clear, but is scoped to the FIRST-attempt path
+  only -- it was never evidence this gap was already covered by existing tests.
+  **Follow-up, same PR (#413), from a CodeRabbit review finding**: the original regression test
+  above only exercised `:conda_create_failed`'s own call site (create itself genuinely fails). The
+  fix's OTHER call site -- the `if not exist "%HP_PY%"` check inside `:conda_create_done`, reached
+  when conda create genuinely SUCCEEDS but the resulting environment is somehow missing
+  `python.exe` -- had zero coverage. Closed by adding a `missing_python` scenario
+  (`CASCADE_CCF_SCENARIO` env var) to the SAME test file, via a new hook
+  `HP_TEST_FORCE_CONDA_MISSING_PYTHON=1` (`run_setup.bat`, `:conda_create_done`) that lets the real
+  create succeed and then deletes the `python.exe` it just produced -- genuine success followed by
+  a genuinely missing interpreter, not a simulated create failure. Wired as a second CI step
+  (`CASCADE_CCF_SCENARIO: missing_python`) immediately after the original, same placement
+  constraint (Miniconda already cached from `selfapps_cascade.ps1`). See `docs/agent-ndjson.md`'s
+  updated `self.cascade.conda_create_fail` entry for the full two-scenario assertion list,
+  including the message-occurrence-COUNT technique used to distinguish "handle_conda_failure
+  logged the failure once, as expected" from "the old :die fall-through regression is back"
+  (both scenarios' own failure messages are logged once by `:handle_conda_failure` regardless of
+  outcome; a second occurrence would come only from `:die`'s own separate echo).
+
 ### Item 13 (closed 2026-08-01)
 
 - **`self.warn.longpath`'s own real CI run showed an INCONCLUSIVE result (`ranBootstrap:false`),

docs/agent-cold-storage.md

@@ -114,3 +114,26 @@ its own named trigger genuinely fires -- do not speculatively build any of these
   was correctly ruled out elsewhere (its wheel bundles an ~40 MB SQLite package database, a
   non-starter for a single-file bootstrapper).
 
+- **Conda native-DLL bundling repair loop (CLAUDE.md Active Backlog Item 24 -- pygrib/eccodes and
+  the general case).** Full PRD at `docs/prd-conda-native-dll-bundling.md`: a frozen EXE built
+  under the conda provider can fail at runtime with `ImportError: DLL load failed` when
+  PyInstaller doesn't discover/bundle a native DLL a conda-forge package depends on (confirmed via
+  real CI evidence for `pygrib`'s `eccodes.dll` dependency, `self.layered_e2e.chain`). The PRD
+  lays out a reactive, bounded repair loop mirroring `:hidden_import_recover`'s proven shape (see
+  `docs/agent-lessons-learned.md`'s "--hidden-import auto-recovery must stay STRICT" entry) --
+  scan for the DLL-load-failure signature (or PyInstaller's own earlier build-time warning),
+  locate the missing DLL under `%CONDA_PREFIX%\Library\bin`, bundle via `--add-binary`, rebuild,
+  and iterate to catch transitive native deps one at a time. Not pursued now: the single highest-
+  leverage next step (verify whether `pyinstaller-hooks-contrib`'s existing `hook-gribapi.py`
+  already solves this for free via a plain `--hidden-import=gribapi` addition -- Finding 1 in the
+  PRD) is unverified and was not resolvable in the sandbox this PRD was drafted in (no access to
+  fetch `pyinstaller-hooks-contrib`'s source). Building any new mechanism before that verification
+  risks duplicating work an existing upstream hook may already cover.
+  **Trigger to thaw**: the owner explicitly brings this forward as the next active work item (per
+  their own stated intent when requesting this PRD: "I think I will bring it forward soon if level
+  check passes and the current todo is fully closed out") -- at that point, start with the PRD's
+  own Requirement 1 (the zero-new-code `--hidden-import=gribapi` verification), not with building
+  the repair loop directly. The PRD's own Open Questions section (narrow pygrib-only fix vs. a
+  general conda-native-DLL pattern-matcher) also needs a maintainer call before implementation
+  proceeds past that first verification step -- see `docs/open-questions.md`.
+