Item 52: distinguish pyproj_deps.py's genuine errors from its exit-1 not-found case

CLAUDE.md

@@ -1025,27 +1025,6 @@ way (no live Windows execution available here), that is noted explicitly rather
   plain-language message naming Constrained Language Mode specifically if it fails, rather than
   letting the failure surface piecemeal as five-plus separate "Could not write ~x" messages later.
 
-- **Item 52: `tools/pyproj_deps.py`'s exit code 1 is overloaded between its intentional
-  "no `[project].dependencies` found" contract and a catch-all for any genuinely unexpected
-  exception, making a real bug in that script indistinguishable from the normal case.** CONFIRMED
-  directly against `tools/pyproj_deps.py` source and `run_setup.bat`'s own consumption of it, in
-  the pyproject.toml dependency-extraction block (`if exist "pyproject.toml" (...)`, the block that
-  calls `:emit_from_base64 "~pyproj_deps.py" HP_PYPROJ_DEPS`). The documented contract (exit
-  0/1/2 = ok/not-found/malformed-TOML) is
-  correct and intentional -- `run_setup.bat`'s silent no-op on exit 1 is CORRECT for the
-  "not-found" case, not a bug (an earlier external review of this same code mischaracterized this
-  as "swallowing a standard exception," which is not accurate -- exit 1 for "not found" is by
-  design). The real, narrower gap: `pyproj_deps.py`'s own top-level `except Exception:
-  sys.exit(1)` catch-all means a genuinely unexpected exception ALSO exits 1, so
-  `run_setup.bat`'s `if errorlevel 1 ( if errorlevel 2 (...) )` structure -- which only logs a
-  WARN for errorlevel >= 2 -- silently treats a real crash exactly like the benign "nothing to do
-  here" case. Low severity (the script is small and stable; this would only bite if a future
-  Python version or TOML edge case triggers an unhandled exception somewhere not already caught by
-  the script's own narrower `except` blocks) and low priority given that. Fix, if picked up:
-  either a distinct exit code for the top-level catch-all (e.g. 3), or an unconditional low-tier
-  log line (not a WARN) on any errorlevel 1 so the fact is at least visible in `~setup.log` for a
-  future debugging session, without changing user-facing behavior.
-
 - **Item 59: CodeRabbit's automated review did not run on PR #435, and should be manually
   triggered on every future PR -- owner-requested standing process fix, not a code change.**
   This repo (fewer than 10 stars, Organization UI config) requires a manual trigger for
@@ -1106,6 +1085,79 @@ way (no live Windows execution available here), that is noted explicitly rather
   OLD rules before running the bootstrapper, asserting the file ends up with `-text` and no
   leftover `eol=crlf` line for `*.bat`/`*.cmd`.
 
+- **Item 61: `check_delimiters.py` does not catch a cross-line `(`/`)` pair inside `rem` comment
+  text, even though cmd.exe's own parser is just as vulnerable to it as it is for `echo` text --
+  a real gap that caused a genuine CI-breaking regression across all 8 lanes (PR #445, Item 52's
+  own fix).** `docs/agent-lessons-learned.md`'s "A literal `(`/`)` inside `echo` text..." entry
+  documents both the original 2026-07 echo-text incident AND this newly-confirmed `rem`-text
+  sibling -- read that entry for the full mechanism and incident trace before starting this item.
+
+  **Root cause, already diagnosed**: `check_delimiters.py`'s `.bat`/`.cmd` handling treats a `rem`
+  line as fully opaque (`if upper.startswith("REM ") or ...: continue`, skipping it from
+  paren-scanning entirely) instead of scanning its characters the way the `echo`-line path does
+  (`is_bat_echo_line`, tracked via `is_echo_open` on the bracket stack, flagging a `(` that opens
+  on an echo line already nested inside a real block and closes on a LATER line). Real cmd.exe
+  does not distinguish `rem` from `echo` for this purpose -- its block-closing search is a raw
+  character scan across the whole block's text regardless of which command a given line belongs
+  to -- so `rem` comments are exactly as exposed to this hazard as `echo` text, but the checker
+  only defends the `echo` case today.
+
+  **High-level fix**: extend the existing `is_echo_open`/bracket-stack machinery to also apply to
+  `rem` lines -- drop the current `continue`-and-skip shortcut for REM lines, route them through
+  the same character scan `echo` lines already get, and reuse the identical "already nested inside
+  an open bracket, closes on a different line" flagging logic (scoped the same way, so a harmless
+  top-level `rem` header block with no enclosing `if`/`for` doesn't false-positive -- this repo's
+  own file header, `run_setup.bat` lines 1-40ish, has several legitimately-balanced-per-line or
+  intentionally `^`-escaped parens that must stay clean).
+
+  **Scope note, NOT yet done as part of the Item 52 fix that surfaced this**: a full audit of
+  every PRE-EXISTING cross-line `rem`-comment paren pair already in `run_setup.bat` (there are
+  many, scattered throughout the file's ~5300 lines) was explicitly NOT performed -- Item 52's own
+  fix only reworded the ONE `rem` block it had just introduced and broken. Whether any of the
+  pre-existing ones are ALSO genuinely hazardous (nested inside a real open block, not just a
+  top-level header) is unknown and unverified; the checker fix above would surface them
+  automatically once implemented -- do not assume the file is currently clean of other latent
+  instances of this same bug just because CI has been green so far (an existing hazard only
+  manifests when the SPECIFIC surrounding code happens to also be reached/reparsed in a way that
+  exposes it, exactly as this one sat undetected until Item 52 added new code near it).
+
+  **Coverage gap to close in the same slice**: `tests/test_check_delimiters_import.py`'s existing
+  `test_paren_*` cases cover the `echo`-line hazard; add an analogous `rem`-line case (a `rem`
+  block whose `(` opens on one line and matching `)` closes on a later line, nested inside a real
+  `if`/`for` block) proving the extended checker catches it, plus a negative case (a top-level
+  `rem` header block with no enclosing bracket) proving it doesn't false-positive.
+
+  **Scope WIDENED, same PR (#445), via a second real CI incident on the SAME code block: a
+  SAME-LINE, self-contained, balanced `(`/`)` pair -- not just cross-line pairs -- can ALSO corrupt
+  parsing when nested deep enough inside real `if (...)` blocks, contradicting this checker's own
+  (and this repo's own documented) assumption that same-line pairs are unconditionally safe.**
+  After the rem-comment fix above was pushed, CI still failed identically; root-caused via a
+  downloaded diagnostics artifact's real `~envsmoke_bootstrap.log` showing the exact same
+  corruption signature as the original PR #408 incident (`falling was unexpected at this time.`),
+  traced to a NEW `>> "%LOG%" echo ... (exit 3); falling back ...` line whose `(exit 3)` pair opens
+  and closes on the SAME line -- yet still corrupted parsing, nested FOUR levels deep. This is a
+  DIFFERENT shape from `:print_fastpath_ambiguous_note`'s own precedent (a plain top-level `echo`
+  with no enclosing block, confirmed safe) -- whether the redirection prefix (`>>` before `echo`)
+  or the nesting depth (4, one deeper than any previously-confirmed case) is the actual
+  distinguishing condition was NOT isolated; the fix (remove the parens) resolved it regardless.
+  See `docs/agent-lessons-learned.md`'s corresponding entry and `docs/agent-closed-backlog.md`'s
+  Item 52 entry for the full trace, and `tests/test_check_delimiters_import.py`'s
+  `test_paren_pair_on_redirected_echo_line_deeply_nested_is_a_known_false_negative` for a
+  regression fixture documenting the checker's current false-negative on this exact shape.
+
+  **Revised item scope**: the high-level fix above (extend `is_echo_open`-style tracking to `rem`
+  lines) is necessary but NOT sufficient on its own -- it still only catches CROSS-line pairs.
+  Whoever picks up this item should ALSO investigate whether extending the same-line-pair
+  "always safe" assumption is correct at all once genuinely nested (vs. top-level), and if not,
+  design a check for that case too (e.g. flag ANY `(`/`)` pair -- same-line or cross-line -- found
+  inside `echo`/`rem` text that is already nested inside a real open bracket, not just cross-line
+  ones) -- balanced against the real risk of false-positiving on the MANY existing, presumably-safe
+  same-line nested echo statements already in `run_setup.bat` (not audited; needs its own careful
+  pass, likely requiring live-cmd.exe verification per this repo's own established practice for
+  this hazard class, not static reasoning alone -- static reasoning about this exact hazard class
+  has now been wrong multiple times in this repo's history, per `docs/agent-lessons-learned.md`'s
+  "`:log` echoes UNQUOTED" entry's own general warning).
+
 ## Cold Storage (promising ideas, deliberately shelved -- revisit only if a named trigger fires)
 
 Moved to `docs/agent-cold-storage.md` (2026-07-31, to reduce this file's per-session context

docs/agent-closed-backlog.md

@@ -2460,6 +2460,108 @@ run of the same regex logic before landing, not just reasoned about).
   the expanded (wrong) path instead of the real subfolder and the hint would never fire.
   `docs/agent-ndjson.md` and `batch-check.yml`'s "Upload test logs" step updated to match.
 
+### Item 52 (closed 2026-08-18)
+
+- **`tools/pyproj_deps.py`'s exit code 1 was overloaded between its intentional "no
+  `[project].dependencies` found" contract and a catch-all for any genuinely unexpected
+  exception, making a real bug in that script indistinguishable from the normal case.** The
+  documented contract (exit 0/1/2 = ok/not-found/malformed-TOML) was correct and intentional --
+  `run_setup.bat`'s silent no-op on exit 1 was already correct for the "not-found" case. The real
+  gap: the script's own top-level `except Exception: sys.exit(1)` catch-all meant a genuinely
+  unexpected exception (a bug, an unusual I/O failure) also exited 1, so `run_setup.bat`'s `if
+  errorlevel 1 ( if errorlevel 2 (...) )` structure -- which only logs a WARN for errorlevel >= 2
+  -- silently treated a real crash exactly like the benign "nothing to do here" case.
+
+  **Fix shipped**: the catch-all now exits 3 instead of 1, and `run_setup.bat`'s consuming block
+  (the pyproject.toml dependency-extraction block right after `:emit_from_base64 "~pyproj_deps.py"
+  HP_PYPROJ_DEPS`) gained a new `if errorlevel 3` branch, checked BEFORE `if errorlevel 2` (since
+  `if errorlevel N` is a `>=N` test, checking the higher threshold first is required or errorlevel
+  3 would also satisfy the errorlevel-2 check and get mislabeled as a TOML parse error) -- logs an
+  unconditional, log-file-only line (`>> "%LOG%" echo ...`, not `call :log`, so it does not also
+  echo to console -- per the item's own "without changing user-facing behavior" scope) so the fact
+  is at least visible in `~setup.log` for a future debugging session.
+
+  **A real bug found and fixed while implementing this, caught by the existing test suite, not
+  found in review.** The first version of the fix broke `test_no_pyproject_toml_exits_1` --
+  a MISSING `pyproject.toml` (the common, documented "not found" case, and also the case any
+  bare invocation of this script outside `run_setup.bat`'s own `if exist "pyproject.toml"` guard
+  would hit) raises `FileNotFoundError` at the same `read_text()` call site as the genuinely
+  unexpected cases (a directory named `pyproject.toml`, a permission failure) -- both fell into
+  the SAME outer `except Exception:` block, so simply changing that block's exit code broke the
+  documented exit-1 contract for the primary, most common case.
+
+  **A second real bug found via CodeRabbit's review on PR #445, in the FIRST fix for the bug
+  above.** That fix checked `pathlib.Path('pyproject.toml').exists()` explicitly before
+  `read_text()`, exiting 1 immediately if absent. CodeRabbit (backed by a real web search against
+  CPython's own pathlib changelog, not just static reasoning) found that Python 3.14+ changed
+  `Path.exists()` (and `is_dir`/`is_file`/etc.) to swallow ANY `OSError` -- including
+  `PermissionError` -- and return `False` instead of raising, specifically to make these query
+  methods consistent with `os.path.exists()`. Since this bootstrapper's own embedded-helper
+  Python baseline explicitly targets always-latest conda-forge Python (3.14 is already the
+  current stable, already in the REQ-009 Tier 5 embed version table's own `LATEST_MINOR`), a
+  genuinely permission-denied `pyproject.toml` on 3.14+ would make `exists()` return `False` --
+  silently misclassified as "not found" (1) instead of the intended "real error" (3), the exact
+  ambiguity this whole fix exists to close. Fixed by removing the `exists()` check entirely and
+  instead catching `FileNotFoundError` specifically around the `read_text()` call itself -- any
+  OTHER exception (a directory, a permission failure, or anything else) is not caught by this
+  narrower handler and falls through to the outer `except Exception:` (exit 3), while a genuinely
+  missing file still exits 1 via the specific `FileNotFoundError` catch.
+
+  **Regression coverage**: `tests/test_pyproj_deps.py::TomllibPath::test_unexpected_exception_exits_3_not_1`
+  creates `pyproject.toml` as a DIRECTORY (not a file) -- cross-platform (raises
+  `IsADirectoryError` on POSIX, `PermissionError` on Windows; either way uncaught by the narrower
+  `FileNotFoundError` handler) -- and asserts exit code 3, not 1. The pre-existing
+  `test_no_pyproject_toml_exits_1` (genuinely missing file) continues to assert exit code 1, now
+  exercising the new `FileNotFoundError`-specific catch instead of incidentally sharing the same
+  catch-all as the directory case. `HP_PYPROJ_DEPS` payload re-synced via `tools/sync_payload.py`
+  after each fix.
+
+  **A third real bug found the same day, this one a genuine CI-breaking regression across ALL 8
+  lanes (`real`, `uv`, `contract-uv`, `contract-uv-fail`, `uv-dl-fallback`, `justme-test`, `cache`
+  all failed identically) -- confirmed via real CI logs, not caught locally by any tool in this
+  repo's own sanity sweep.** The `run_setup.bat` consuming block's own `rem` comment (explaining
+  "check the highest threshold first" for the new `if errorlevel 3`/`if errorlevel 2` dispatch)
+  split a parenthetical remark's `(`/`)` across THREE separate `rem` lines, nested three levels
+  deep inside real `if (...)` blocks. This is the identical hazard class already documented and
+  fixed once before for `echo` text (`docs/agent-lessons-learned.md`'s "A literal `(`/`)` inside
+  `echo` text..." entry, PR #408) -- cmd.exe's block-closing parser counts `(`/`)` characters in
+  `rem` comment text exactly the same way it does in `echo` text, with no concept of "this is just
+  a comment." `check_delimiters.py` did NOT catch it: unlike its `echo`-line handling (which scans
+  characters and tracks cross-line-opened parens), its `.bat`/`.cmd` path treats `rem` lines as
+  fully opaque and skips them from paren-scanning entirely -- a genuine, previously-undiscovered
+  blind spot in the checker itself, not just a one-off authoring mistake. Every CI lane doing a
+  real bootstrap failed with zero console output a few seconds in, since cmd.exe must determine
+  where the corrupted block's raw text ends BEFORE it can decide whether to execute or skip it --
+  the corruption happens at parse time, independent of whether `pyproject.toml` itself exists on
+  disk for any given test fixture. Fixed by rewording the comment to avoid the literal `(`/`)`
+  characters entirely (` -- ` in place of the parenthetical), per the identical rule the `echo`
+  case already established. **The general checker gap (extending `check_delimiters.py`'s existing
+  `echo`-line cross-line-paren tracking to also cover `rem` lines) is filed separately as CLAUDE.md
+  Active Backlog Item 61** -- not implemented as part of this fix, which only needed to unblock the
+  one broken instance, not audit or re-armor every pre-existing `rem` block already in the file.
+
+  **A FOURTH real bug, in the SAME commit as the third: the rem-comment fix alone did not resolve
+  the regression -- CI still failed identically after it was pushed, confirmed via a second round
+  of live CI evidence.** The new `>> "%LOG%" echo ... (exit 3); falling back to requirements.txt or
+  pipreqs.` line (added alongside the rem comment, in the SAME original commit) has its own literal
+  `(`/`)` pair -- `(exit 3)` -- that opens AND closes on the SAME line, inside echo text. Per the
+  established rule and `check_delimiters.py`'s own code, a same-line self-contained pair is
+  supposed to be harmless (the checker deliberately never flags one) -- but a downloaded diagnostics
+  artifact's real `~envsmoke_bootstrap.log` showed the IDENTICAL corruption signature as the
+  original PR #408 incident: `falling was unexpected at this time.`, with "falling" being the exact
+  next word after `(exit 3)` in this line. This same-line pair sits nested FOUR levels deep inside
+  real `if (...)` blocks -- one level deeper than any previously-confirmed safe case -- and is a
+  `>> file echo ...` REDIRECTED form, not a plain top-level `echo` statement; which factor (depth,
+  or the redirection prefix) actually matters was not isolated, since the live evidence made the
+  fix (remove the parens) unambiguous without needing to. Fixed by rewording `(exit 3)` to `, exit
+  3` -- no literal parens at all. **Revises the established rule**: "same-line, self-contained,
+  balanced" is not a blanket safe-harbor for text nested inside a real open block -- it is only
+  confirmed-safe for a genuinely top-level statement with no enclosing bracket (the
+  `:print_fastpath_ambiguous_note` precedent). See `docs/agent-lessons-learned.md`'s corresponding
+  entry for the full trace. Both this bug and the rem-comment bug shipped in the same original
+  commit and had to be found and fixed in two SEPARATE rounds, each confirmed only by live Windows
+  CI evidence pulled from a downloaded diagnostics artifact -- local tooling caught neither one.
+
 ## Known Findings (diagnosed, no action warranted)
 
 - **Backlog item numbering: renumber-on-collision convention dropped, 2026-07-31 owner decision.**