Release-prep: reorg demo doc, fix stale content, small product fixes

.github/workflows/batch-check.yml

@@ -95,48 +95,24 @@ jobs:
         env:
           HP_CACHE_EXACT_HIT: ${{ steps.conda_cache_restore.outputs.cache-hit }}
         run: |
-          $condaMain = 'C:\Users\Public\Documents\Miniconda3\condabin\conda.bat'
-          $condaAlt  = 'C:\Users\Public\Documents\Miniconda3\Scripts\conda.bat'
-          $condaBat = if (Test-Path $condaMain) { $condaMain } elseif (Test-Path $condaAlt) { $condaAlt } else { $null }
-          if ($null -eq $condaBat) {
-            Write-Host "No conda binary found; fresh install will proceed normally."
-          } else {
-            $output = & cmd /c "`"$condaBat`" info" 2>&1
-            if ($LASTEXITCODE -ne 0) {
-              if ($env:HP_CACHE_EXACT_HIT -eq 'true') {
-                # derived requirement: an EXACT primary-key hit that's corrupted can never be
-                # replaced in place (a GitHub Actions cache entry's blob is immutable once saved
-                # under a key) -- see docs/agent-closed-backlog.md's Item 19 (cache-lane
-                # self-perpetuating-corruption) for the full trace. Keep the original
-                # skip-this-run behavior for this narrow case; fully closing it needs an explicit
-                # cache-deletion API call, a smaller follow-on not implemented here.
-                Write-Host "::warning::Conda binary health check failed (exit=$LASTEXITCODE) on an EXACT cache-key hit; cache corrupted, skipping fast-path tests this run."
-                'HP_CACHE_CORRUPTED=1' | Out-File -FilePath $env:GITHUB_ENV -Encoding ascii -Append
-              } else {
-                # derived requirement: a restore-keys PREFIX match (the common case -- the cache
-                # key hashes run_setup.bat, which changes on nearly every PR) is not a guarantee
-                # the restored blob is still valid. Treating this like HP_CACHE_CORRUPTED=1 was
-                # the actual bug: the ONLY step that can do a fresh install is gated on that same
-                # flag, so a poisoned blob could never be replaced -- a permanent, self-
-                # perpetuating loop. Fix: treat it like a genuine cache miss instead -- delete the
-                # stale directory and let the run fall through to a real fresh install and a real
-                # fresh save under the current key, breaking the loop.
-                Write-Host "::warning::Conda binary health check failed (exit=$LASTEXITCODE) on a restore-keys prefix match; deleting stale cache directory and proceeding as a fresh install."
-                Remove-Item -LiteralPath 'C:\Users\Public\Documents\Miniconda3' -Recurse -Force -ErrorAction SilentlyContinue
-                if (Test-Path 'C:\Users\Public\Documents\Miniconda3') {
-                  # derived requirement: same AV/indexer file-lock hazard class already documented
-                  # for :try_embed_fallback's own directory swap in run_setup.bat -- if deletion
-                  # didn't fully succeed, do not proceed into an uncertain half-deleted state;
-                  # fall back to the original, safe skip-this-run behavior instead.
-                  Write-Host "::warning::Stale cache directory could not be fully removed (possible file lock); falling back to HP_CACHE_CORRUPTED=1 for this run."
-                  'HP_CACHE_CORRUPTED=1' | Out-File -FilePath $env:GITHUB_ENV -Encoding ascii -Append
-                } else {
-                  Write-Host "Stale cache directory removed; fresh install will proceed normally."
-                }
-              }
-            } else {
-              Write-Host "Conda health OK: $output"
-            }
+          # derived requirement: the actual health-check-and-heal logic now lives in
+          # tools/ci_cache_selfheal.ps1 so tests/test_ci_cache_selfheal.ps1 can exercise it
+          # deterministically on every CI run (a GATING lane), independent of whether GitHub's
+          # own cache happens to be organically corrupted this run -- see that file and
+          # docs/agent-closed-backlog.md's Item 19 entry for why the ambient `cache` lane alone
+          # (informational, job-level continue-on-error) was not enough signal on its own.
+          $exactArgs = @()
+          if ($env:HP_CACHE_EXACT_HIT -eq 'true') { $exactArgs = @('-ExactHit') }
+          & .\tools\ci_cache_selfheal.ps1 -CondaDir 'C:\Users\Public\Documents\Miniconda3' @exactArgs
+          $rc = $LASTEXITCODE
+          if ($rc -eq 1 -or $rc -eq 3) {
+            'HP_CACHE_CORRUPTED=1' | Out-File -FilePath $env:GITHUB_ENV -Encoding ascii -Append
+          }
+          if ($rc -eq 3) {
+            'HP_CACHE_SELFHEAL_FAILED=1' | Out-File -FilePath $env:GITHUB_ENV -Encoding ascii -Append
+          }
+          if ($rc -eq 2 -or $rc -eq 3) {
+            'HP_CACHE_SELFHEAL_ATTEMPTED=1' | Out-File -FilePath $env:GITHUB_ENV -Encoding ascii -Append
           }
           exit 0  # health check is informational; never fail this step
 
@@ -152,6 +128,49 @@ jobs:
           $row | Add-Content 'tests\~test-results.ndjson' -Encoding Ascii
           $row | Add-Content 'ci_test_results.ndjson' -Encoding Ascii
 
+      - name: "Record cache self-heal outcome (visibility row; never fails)"
+        # derived requirement: this is the gap the owner flagged directly -- when the self-heal
+        # SUCCEEDS (the ordinary case), HP_CACHE_CORRUPTED is never set, so no NDJSON row was
+        # ever emitted saying "this run actually had to self-heal a corrupted cache" -- the run
+        # just looked like an ordinary fresh install, with the only trace being a ::warning::
+        # buried in raw job logs. Emitting this unconditionally whenever the self-heal branch is
+        # entered (success OR failure) makes both outcomes queryable on the diagnostics site over
+        # time, instead of requiring a manual raw-log dig to notice either one.
+        if: ${{ !cancelled() && env.HP_CACHE_SELFHEAL_ATTEMPTED == '1' }}
+        shell: pwsh
+        run: |
+          if (-not (Test-Path 'tests')) { New-Item -ItemType Directory 'tests' | Out-Null }
+          $healed = ($env:HP_CACHE_SELFHEAL_FAILED -ne '1')
+          $row = [ordered]@{
+            id      = 'self.cache.selfheal.fired'
+            pass    = $true
+            lane    = 'cache'
+            desc    = 'Restored cache failed its health check on a restore-keys prefix match; self-heal (delete stale dir, fall through to fresh install) was attempted'
+            details = [ordered]@{ healed = $healed }
+          } | ConvertTo-Json -Compress
+          $row | Add-Content 'tests\~test-results.ndjson' -Encoding Ascii
+          $row | Add-Content 'ci_test_results.ndjson' -Encoding Ascii
+
+      - name: "Enforce cache self-heal success (Item 19 follow-on)"
+        # derived requirement: an ordinary corrupted-and-healed cache is routine infra noise and
+        # stays informational -- run_setup.bat hashes into the cache key on nearly every PR, so
+        # this fires often and is expected. But the self-heal ITSELF failing to clear the stale
+        # directory is not routine noise: it means this repo's own Item 19 fix has regressed back
+        # into the exact pre-fix "always corrupted, never self-heals" trap that item existed to
+        # close. This step's own failure is still absorbed by the `cache` lane's job-level
+        # continue-on-error (see CLAUDE.md's CI lane gating maturity notes -- this lane stays
+        # intentionally non-gating for ordinary organic flakiness), so on its own this does not
+        # block a PR; it exists so the failure is a loud, explicit ::error:: annotation and a red
+        # step marker instead of a buried ::warning::. tests/test_ci_cache_selfheal.ps1 (a GATING
+        # regression test wired into the `real` lane) is what actually catches a regression in
+        # this logic on every single CI run, independent of whether GitHub's own cache happens to
+        # be organically corrupted this run.
+        if: ${{ !cancelled() && env.HP_CACHE_SELFHEAL_FAILED == '1' }}
+        shell: pwsh
+        run: |
+          Write-Host "::error::Cache self-heal failed to clear the stale Miniconda3 directory -- see 'Validate restored conda binary' step output above."
+          exit 1
+
       # probe fires in real, conda-full, uv, and contract-uv* lanes; cache lane skips it intentionally
       - name: Enable Miniconda probe (real/conda-full/uv/contract-uv mode)
         if: ${{ matrix.mode == 'real' || matrix.mode == 'conda-full' || matrix.mode == 'uv' || matrix.mode == 'contract-uv' || matrix.mode == 'contract-uv-fail' }}
@@ -253,6 +272,17 @@ jobs:
             }
           }
 
+      # derived requirement: wired into `real` specifically (a GATING lane, not in the
+      # job-level continue-on-error list at the top of this file) so a regression in
+      # tools/ci_cache_selfheal.ps1 actually fails CI -- see that file and
+      # docs/agent-closed-backlog.md's Item 19 entry for why the ambient `cache` lane alone
+      # (informational, only fires on organic corruption) was not sufficient signal on its own.
+      - name: "Self-test: cache-lane self-heal logic (real lane only, GATING)"
+        if: ${{ matrix.mode == 'real' }}
+        shell: pwsh
+        run: |
+          & tests\test_ci_cache_selfheal.ps1
+
       - name: "Pre-bootstrap: ensure Python file exists"
         shell: pwsh
         run: |

docs/agent-closed-backlog.md

@@ -13,6 +13,13 @@ CLAUDE.md's ~4200 lines (>60%) despite being pure historical record with no forw
 action attached to any entry. Nothing here needs re-reading by default; it exists so a specific
 past decision or fix can be looked up when its details actually matter.
 
+**A `docs/demo-bootstrapper-output.md` "Part N, Scenario N" citation below reflects that doc's
+structure AT THE TIME the entry was written, not necessarily its current numbering** -- that file
+went through a flow-only reorg pass (2026-08-02) that renumbered every Part and Scenario, and will
+likely be reorganized again as it grows. Treat a Part/Scenario citation here as "roughly where to
+look," not a precise current coordinate; this file is an append-only historical record, so its own
+entries are not retroactively renumbered to track that doc's current structure.
+
 **Two sections below.** "Closed Active Backlog Items" holds items that were promoted out of
 `CLAUDE.md`'s own "Active Backlog" section once fully resolved (each keeps its original item
 number for cross-reference stability -- other docs cite these by number). "Closed Backlog" is
@@ -755,6 +762,52 @@ this belongs to).
    opportunistically across this and future PRs' own `cache`-lane runs, not via a dedicated
    verification loop.
 
+   **Follow-on shipped 2026-08-02, prompted by the owner directly asking "is the fix holding" and
+   then "make it go red so it actually helps something."** The "observed opportunistically" plan
+   above turned out to have two real gaps, both found while answering the first question
+   honestly rather than assuming: (1) verification meant manually pulling raw job logs each time
+   -- nothing queryable recorded whether a run's self-heal had actually fired (a corrupted
+   restore-keys-prefix-match cache with a successful heal never sets `HP_CACHE_CORRUPTED`, so no
+   NDJSON row was ever emitted for it -- the only trace was a `::warning::` buried in the log);
+   (2) even a hypothetical step-level failure inside the `cache` lane could never have surfaced as
+   a real CI failure, because that whole lane is job-level `continue-on-error` (see the top of
+   `batch-check.yml`) -- confirmed directly against the first real post-fix run (`30684739923`,
+   job `91328449387`, commit `40e6187`/PR #409): the corrupted-prefix-match branch fired for real
+   (`##[warning]Conda binary health check failed (exit=1) on a restore-keys prefix match; deleting
+   stale cache directory and proceeding as a fresh install.` at 04:55:51Z, `Stale cache directory
+   removed; fresh install will proceed normally.` at 04:56:21Z, `Cache saved with key:
+   win-Windows-py311b-conda-...` at 05:16:43Z) -- genuine end-to-end proof the fix works, found
+   only by pulling the raw log by hand, not by anything CI itself surfaced.
+   - Extracted the inline health-check-and-heal PowerShell out of `batch-check.yml`'s "Validate
+     restored conda binary" step into `tools/ci_cache_selfheal.ps1`, a small parameterized script
+     (`-CondaDir`, `-ExactHit`) with 4 distinct exit codes (0=healthy/no-op, 1=exact-hit-corrupted
+     [unchanged accepted gap], 2=prefix-corrupted-and-healed, 3=prefix-corrupted-heal-FAILED --
+     the new case: the stale directory could not be fully cleared, meaning this fix has regressed
+     back toward its own pre-fix trap).
+   - Added `tests/test_ci_cache_selfheal.ps1`: a deterministic regression test exercising all 4
+     exit codes against a scratch temp directory with fake `conda.bat` stand-ins (including a
+     genuine locked-file reproduction of the heal-FAILED case via a held `FileStream` handle) --
+     no dependency on GitHub's cache ever being organically corrupted. Wired into the `real`
+     lane specifically (a GATING lane, not in the job-level `continue-on-error` list), so a future
+     regression in this logic fails CI for real, unlike the ambient `cache` lane which cannot.
+     Windows-only (shells out to `conda.bat` via `cmd.exe`; the lock scenario needs Windows
+     file-locking semantics), matching this repo's usual `$IsWindows`-gated-skip convention.
+   - Added `self.cache.selfheal.fired`: an always-`pass:true` visibility row emitted by the
+     ambient `cache` lane itself whenever the self-heal branch is entered (success or failure),
+     recording the real outcome in `details.healed` -- closes gap (1) above; an organic
+     occurrence is now queryable on the diagnostics site instead of requiring a raw-log dig.
+   - Added a loud `::error::` tripwire step (`Enforce cache self-heal success`) for the
+     heal-FAILED sub-case specifically, mirroring `diag.conda.available.gate`'s established
+     "default to a loud failure, not a silent one" pattern -- explicitly documented as *not* a
+     substitute for the gating test above (still absorbed by the `cache` lane's own job-level
+     `continue-on-error`), only a clearer annotation than the `::warning::` it replaces for that
+     one case.
+   - Deliberately did NOT flip the `cache` lane itself to gating, and did NOT touch the still-open
+     exact-key-hit-corrupted gap (still needs the `gh cache delete`/cache-deletion-API follow-on
+     noted above) -- both out of scope for this pass; the ask was specifically "make a regression
+     in the self-heal mechanism visible and blocking," not "make ordinary organic cache flakiness
+     block PRs."
+
 ### Item 15 (closed 2026-08-01)
 
 - **`:exe_smokerun_hints`'s diagnostic re-run of a freshly-failed EXE had no timeout, unlike every

docs/agent-ndjson.md

@@ -27,6 +27,7 @@ self.failfast.probe.fastfail, self.failfast.probe.alive, self.failfast.probe,
 self.checkpoint.accept, self.checkpoint.decline,
 self.entry.entry1, self.entry.entryA, self.entry.entryB, self.entry.entryC, self.entry.entryD,
 self.entry.helper.invoke.absent, self.entry.results, self.entry.spaced-path, self.entry.picker,
+self.entry.picker.overflow,
 self.entry.req011.crossdir, self.entry.req011.sameDir, self.isolation.req010.pythonpath,
 entry.single.direct, entry.expected, helper.invoke,
 self.envname.hyphen, self.size.tripwire,
@@ -163,7 +164,7 @@ visa.detect, emit.helpers, env.state.write, dep.check.parse_lock,
 dp.compat, prep.multi.constraint, batch.paren.balance, env.foldername,
 conda.path, conda.url, env.mode,
 self.warnfix.platform_filter, self.exe.smokerun, helper.find_entry.syntax, entry.helper.ok,
-self.cache.corrupted, self.cache.bootstrap.failed,
+self.cache.corrupted, self.cache.bootstrap.failed, self.cache.selfheal.fired,
 meta.env.mode, workflow.lint,
 version.metadata,
 host.env.os, host.env.ps, host.env.python,
@@ -557,6 +558,34 @@ substitute for a human's own interactive session).
 self.interactive.stdin.roundtrip
 ```
 
+## selfapps-cache-selfheal NDJSON rows (test_ci_cache_selfheal.ps1, `real` lane only, GATING)
+
+Item 19 follow-on (docs/agent-closed-backlog.md): the cache-lane self-heal logic
+(`tools/ci_cache_selfheal.ps1`) previously had no deterministic CI coverage -- the ambient
+`cache` lane only reaches it when GitHub's own cache happens to be organically corrupted, and
+that lane is entirely informational (job-level `continue-on-error`) besides, so a regression in
+the self-heal logic itself would ship silently. This test exercises all 4 branches of that
+script directly against a scratch temp directory with fake `condabin\conda.bat` stand-ins (no
+real conda/network dependency), wired into `real` -- a GATING lane -- so a regression actually
+fails CI. Windows-only (the script under test shells out to `conda.bat` via `cmd.exe`, and the
+locked-directory scenario needs Windows file-locking semantics); skips with `skip=true` on
+non-Windows via the bare `self.ci.cache_selfheal` row.
+
+```
+self.ci.cache_selfheal,
+self.ci.cache_selfheal.healthy, self.ci.cache_selfheal.prefix_healed,
+self.ci.cache_selfheal.exact_hit_corrupted, self.ci.cache_selfheal.prefix_heal_failed,
+self.ci.cache_selfheal.no_binary
+```
+
+`self.cache.selfheal.fired` (inline `batch-check.yml`, `cache` lane, `HP_CACHE_SELFHEAL_ATTEMPTED`-gated)
+is the companion VISIBILITY row -- unlike the deterministic test above, this
+fires only when the AMBIENT `cache` lane's own restored cache is organically corrupted on a
+restore-keys prefix match, and records whether that real self-heal attempt actually succeeded
+(`details.healed`). Always `pass:true` (informational, matching `self.cache.corrupted`'s own
+convention) -- its purpose is to make an organic occurrence queryable on the diagnostics site
+over time instead of requiring a raw-log dig to notice it happened at all, not to gate.
+
 ---
 
 ## Key facts for debugging missing rows