From a26e61700cbf6472973f0f6cf8ce9adfd905d182 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 16:05:08 +0000 Subject: [PATCH 1/2] Item 34 loop 3: distill dependency-strategy essay, table-ify payload list Closes CLAUDE.md Active Backlog Item 34's final loop (follow-up to Loops 1-2, PR #429/#430). - Distilled three "essay" sections (pipreqs pin rationale, pipreqs invocation rationale, the warnfix SKIP-set walkthrough) down to a 2-4 sentence load-bearing rule per topic, moving the full multi-paragraph justification verbatim to docs/agent-closed-backlog.md's new "Dependency Strategy Rationale" reference section -- zero information loss, same pointer pattern Loops 1-2 already established. - Trimmed the 6 Bootstrap Architecture Principles in place (wording-only; there was no separate narrative to move out, just redundant clauses to cut while keeping all 6 principles and their concrete examples intact). - Table-ified "run_setup.bat Rules"'s 7-payload paragraph list into a 4-column table (payload / decodes-to / purpose / canonical source). - Item 34 itself is now fully resolved -- moved from CLAUDE.md's Active Backlog to docs/agent-closed-backlog.md's Closed Active Backlog Items (keeping its number, summarizing all 3 loops), and CLAUDE.md's Active Backlog reset to the explicit "Nothing here." empty marker. Net: CLAUDE.md drops from 775 to 687 lines (6496 to 5573 words, ~14.2% further reduction) -- combined with Loops 1-2, CLAUDE.md is now 687 lines versus the pre-Item-34 baseline of 916 (~25% total reduction across the three loops), with zero information loss (every fact moved, not deleted). Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01A4iE1BRSkUETwz237XeuTW --- CLAUDE.md | 208 ++++++++++------------------------- docs/agent-closed-backlog.md | 134 +++++++++++++++++++++- 2 files changed, 191 insertions(+), 151 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index fa6a0b56..9a549e3a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -180,37 +180,21 @@ This is the deliverable. Treat changes carefully. hand-roll the encode/splice/write sequence (see docs/agent-lessons-learned.md's "Embedded Helper Update Workflow" and the near-miss that motivated automating it). - Current embedded payloads (in addition to legacy HP_FAST_CHECK): - - `HP_DEP_CHECK` -- decodes to `~dep_check.py`; compares pipreqs output against - `~environment.lock.txt` and exits 0 (skip) or 1 (install needed). - - `HP_ENV_STATE` -- decodes to `~env_state.py`; reads/writes `~env.state.json` to - cache conda env validity across runs. - - `HP_PYPROJ_DEPS` -- decodes to `~pyproj_deps.py`; parses pyproject.toml - `[project].dependencies` and writes one dep per line; exit 0 on success, 1 on - not-found/error, 2 on malformed TOML. Used by the layered dep resolution block. - - `HP_PREP_REQUIREMENTS` -- decodes to `~prep_requirements.py`; applies heuristic - dep-augmentation rules (REQ-005.8); strips pip extras (`[excel]`) before name lookup. - - `HP_COLLECT_SUBMODULES` -- decodes to `~collect_submodules.py`; emits pre-build - `--collect-submodules=PKG` flags for curated packages (sklearn, matplotlib, scipy, - plotly) that load submodules dynamically (the warn file is silent about them). - Double-gated: a flag is emitted only when the package is BOTH imported by the user's - project source AND importable in the build interpreter, so a fat global env never - bloats a lean app's EXE. Canonical source `tools/collect_submodules.py`; PayloadSync - in `tests/test_collect_submodules.py` asserts byte-equality of the embedded base64. - - `HP_HIDDEN_IMPORT_SCAN` -- decodes to `~hidden_import_scan.py`; for the Slice 2 - `--hidden-import` auto-recovery loop (`:hidden_import_recover` in run_setup.bat). Reads a - frozen EXE's stderr and emits the next hidden-import target ONLY when stderr shows - `ModuleNotFoundError: No module named 'X'` AND X is installed in the build interpreter -- - so a user typo or `ImportError: cannot import name` causes ZERO rebuilds. Bounded to 3 - rebuilds (helper tried-list + iter cap). Canonical source `tools/hidden_import_scan.py`; - PayloadSync in `tests/test_hidden_import_scan.py`. - - `HP_DLL_PCT_SANITIZE` -- decodes to `~dll_pct_sanitize.ps1`; strips `%`/`^` from one or more - env var values (`(envVarName, outFile)` argv pairs), for `:log`'s UNQUOTED-echo safety in the - native-DLL bundling loop. Emitted as a real `.ps1` file specifically so cmd.exe's own - tokenizer never parses its body -- the predecessor inline `-Command` version of this logic - went through three separate real-CI-confirmed bugs from cmd.exe's `%`-pairing behavior before - landing here (see `docs/agent-lessons-learned.md`'s ":log echoes UNQUOTED" entry). Canonical - source `tools/dll_pct_sanitize.ps1`; PayloadSync in `tests/test_dll_pct_sanitize.py`. + Current embedded payloads (in addition to legacy `HP_FAST_CHECK`): + + | Payload | Decodes to | Purpose | Canonical source | + |---------|-----------|---------|-------------------| + | `HP_DEP_CHECK` | `~dep_check.py` | Compares pipreqs output against `~environment.lock.txt`; exits 0 (skip) or 1 (install needed) | `tools/dep_check.py` | + | `HP_ENV_STATE` | `~env_state.py` | Reads/writes `~env.state.json` to cache conda env validity across runs | `tools/env_state.py` | + | `HP_PYPROJ_DEPS` | `~pyproj_deps.py` | Parses `pyproject.toml` `[project].dependencies`, one dep/line; exit 0/1/2 = ok/not-found/malformed TOML; feeds the layered dep resolution block | `tools/pyproj_deps.py` | + | `HP_PREP_REQUIREMENTS` | `~prep_requirements.py` | Applies REQ-005.8 heuristic dep-augmentation rules; strips pip extras (`[excel]`) before name lookup | `tools/prep_requirements.py` | + | `HP_COLLECT_SUBMODULES` | `~collect_submodules.py` | Pre-build `--collect-submodules=PKG` flags for curated dynamic-import packages (sklearn/matplotlib/scipy/plotly), double-gated on used-by-source AND installed-in-interpreter so a fat global env never bloats a lean app's EXE | `tools/collect_submodules.py` | + | `HP_HIDDEN_IMPORT_SCAN` | `~hidden_import_scan.py` | Slice 2 `--hidden-import` auto-recovery target selection for `:hidden_import_recover`; strict `ModuleNotFoundError` + installed-in-build-interpreter gate (a typo or `ImportError: cannot import name` costs zero rebuilds); capped at 3 rebuilds | `tools/hidden_import_scan.py` | + | `HP_DLL_PCT_SANITIZE` | `~dll_pct_sanitize.ps1` | Strips `%`/`^` from env var values for `:log`'s UNQUOTED-echo safety in the native-DLL bundling loop; emitted as a real `.ps1` (invoked via `-File`) so cmd.exe's own tokenizer never parses its body -- see `docs/agent-lessons-learned.md`'s ":log echoes UNQUOTED" entry for why the earlier inline `-Command` version needed three separate fixes | `tools/dll_pct_sanitize.ps1` | + + Each payload's canonical `tools/` source has a `PayloadSync` unit test asserting + byte-equality between the embedded base64 and the source file (see the Testing section + below for which test file covers which payload). 2. **Delimiter-check after every edit**: ```bash @@ -379,47 +363,26 @@ and cv2/opencv mapping limitation. ## Dependency Discovery: pipreqs pin rationale (as of 2026-06-18) -**pipreqs is pinned to 0.4.13, NOT 0.5.0.** This is deliberate and load-bearing: - -- pipreqs 0.5.0 (the latest release) added Jupyter notebook scanning, which hard-pins `ipython==8.12.3` - (the last ipython supporting Python 3.8). ipython 8.12.3 does not support Python 3.13+, so 0.5.0's - metadata declares `Requires-Python >=3.8.1,<3.13`. -- The bootstrapper always targets the latest conda-forge Python (currently 3.14+). On that Python, pip - refuses to install 0.5.0 (version cap), so pipreqs would be lost entirely and every run would fall back - to warnfix. -- pipreqs 0.4.13 has `Requires-Python >=3.7` (no upper cap), deps only `docopt`+`yarg`, supports the same - `--mode compat` / `--force` / `--savepath` flags, uses only stable stdlib (ast-based scan), and runs on - Python 3.14. It restores pipreqs as the primary discovery tool. -- **Do NOT "upgrade" the pin back to 0.5.0** -- it reintroduces the `<3.13` cap and silently disables - pipreqs on modern Python. The only feature lost by 0.4.13 is `.ipynb` scanning, which was already - non-functional on latest Python (0.5.0 cannot run there). - -The `pipreqs.flags` CI gate validates the invocation flags, not the version, so the pin is free to change. -The setup log line `[INFO] pipreqs installed successfully` confirms pipreqs is active on a given run. +**pipreqs is pinned to 0.4.13, NOT 0.5.0 -- do not "upgrade" this pin.** 0.5.0 hard-caps +`Requires-Python <3.13` (via its Jupyter-scanning `ipython==8.12.3` dependency), which would +silently disable pipreqs entirely on this bootstrapper's always-latest-conda-forge-Python target +and fall every run back to warnfix; 0.4.13 has no upper Python cap and runs fine on 3.14. The +only feature lost by staying on 0.4.13 is `.ipynb` scanning, already non-functional on latest +Python since 0.5.0 can't run there anyway. Full version-comparison rationale in +`docs/agent-closed-backlog.md`'s "Dependency Strategy Rationale" reference section. The +`pipreqs.flags` CI gate validates invocation flags, not the version, so the pin is free to change; +the setup log line `[INFO] pipreqs installed successfully` confirms pipreqs is active on a +given run. ## Dependency Discovery: pipreqs invocation (bootstrap determinism) -**pipreqs is invoked via `python -m pipreqs.pipreqs`, NOT the console script (`pipreqs` command).** -This is an intentional bootstrap execution strategy, not a workaround for pipreqs limitations. - -**Constraints driving this choice:** -- Windows batch bootstrap never depends on shell state (PATH, activation, environment variables) -- Bootstrap runs immediately after environment creation in the same shell session -- Console scripts require PATH correctness and activation to persist--neither is guaranteed -- Bootstrap reliability > API purity in this system class - -**Why internal module invocation is safe here:** -- pipreqs is pinned to 0.4.13 permanently (no automatic upgrades) -- Version freeze makes internal module structure (`pipreqs/pipreqs.py`) stable by contract -- Internal coupling is a low-risk controlled assumption due to the pinned dependency version - -**Comparison of approaches:** -| Approach | Reliability in Bootstrap | Architecture | Scope | -|----------|--------------------------|--------------|-------| -| `pipreqs` (console script) | WARN Fragile (PATH dependent) | OK Official API | General use | -| `python -m pipreqs.pipreqs` | OK Deterministic (no PATH) | WARN Internal mechanism | Bootstrap only | - -See `run_setup.bat` lines ~813-820 for the invocation comment and rationale. This is a **deterministic execution pattern required for bootstrap reliability**, not a sign of fragility or a temporary workaround. +**pipreqs is invoked via `python -m pipreqs.pipreqs`, NOT the console script.** Deterministic +(no PATH/activation dependency) and safe only because pipreqs is permanently version-pinned +(0.4.13), making its internal module structure stable by contract -- the exact tradeoff this +repo's Bootstrap Architecture Principles (below) codify generally. See `run_setup.bat`'s own +invocation comment (search `python -m pipreqs.pipreqs`) for the in-code pointer, and +`docs/agent-closed-backlog.md`'s "Dependency Strategy Rationale" reference section for the full +constraints/comparison writeup. ## Dependency Discovery Fallback: warnfix (secondary safety net) @@ -428,38 +391,17 @@ cannot build), the bootstrapper still falls back to `warnfix`: 1. PyInstaller builds the EXE (static analysis finds many imports) 2. Read the `warn` file (list of modules PyInstaller couldn't find) 3. Parse warn file via `parse_warn.py`: extract top-level, delayed, and conditional imports -4. Filter out modules warnfix must never try to install as an application dependency (either - because the name isn't a real installable package at all, or because the interpreter/its own - import machinery already provides it and re-installing would be pointless noise). - `parse_warn.py` uses two distinct mechanisms, not one -- keep them separate when describing or - extending this filter: - - A generic, SKIP-independent rule (`if mod.startswith("_"): continue`) drops any - leading-underscore internal module by name pattern alone -- this is what actually filters - `_scproxy`, `_posixsubprocess`, and `_frozen_importlib_external`; none of the three is a - `SKIP` entry. - - The `SKIP` frozenset covers everything else warnfix must not treat as an application - dependency, across a few distinct groups: packaging/import-machinery internals - (`pkg_resources`, `distutils`, `setuptools`, `importlib` and its submodules) -- these ARE - real, installable PyPI packages, but PyInstaller's own bundling of the interpreter's import - machinery can surface them as "missing" even though the app never actually needs a separate - install; `collections` - (also covers `collections.abc`, since dotted names truncate to their top-level package - before the `SKIP` lookup) -- this one is the stdlib's own submodules surfacing as "missing," - not a platform gap; Unix-only platform modules absent on Windows (`grp`, `pwd`, `posix`, - `resource`, `fcntl`, `readline`, `termios`, `tty`, `pty`, `crypt`, `spwd`, `nis`, `syslog`, - `ossaudiodev`); and Python-2-only stdlib shims removed entirely in Python 3 (`cStringIO`, - `StringIO`) that still surface via real packages' own Python 2/3 compatibility code -- e.g. - `xlrd`'s `xlrd/timemachine.py` does `try: from cStringIO import StringIO except ImportError: - from io import StringIO`, a dead code path under Python 3 that PyInstaller's static - analysis still flags. Confirmed via a real, unflagged CI run of the layered-dependency - E2E test (`self.layered_e2e.chain`) -- warnfix was genuinely attempting and failing to - `conda install cStringIO`, which can never succeed, forcing an unnecessary extra - provider cascade. Every entry in `SKIP` is covered by - `tests/test_parse_warn.py::ParseWarnFileEdgeCasesTest:: - test_every_skip_entry_filtered_in_realistic_warn_line`, which iterates the current set - and proves each one filters in the `(conditional)`/`(delayed)`/`(top-level)` PyInstaller - 6.x forms -- any future `SKIP` addition is covered automatically, no separate test - needed per entry. +4. Filter modules warnfix must never install as an application dependency: a generic + `if mod.startswith("_"): continue` rule drops leading-underscore internals by name pattern + alone; the `SKIP` frozenset additionally covers packaging/import-machinery internals + (`pkg_resources`, `distutils`, `setuptools`, `importlib`+submodules), `collections` (+`.abc`), + Unix-only platform modules absent on Windows, and Python-2-only stdlib shims (`cStringIO`, + `StringIO`) that still surface via real packages' own Python 2/3 compat code. Every `SKIP` + entry is covered automatically by + `tests/test_parse_warn.py::ParseWarnFileEdgeCasesTest::test_every_skip_entry_filtered_in_realistic_warn_line` + -- no separate test needed per addition. Full per-group rationale and the real CI evidence + (`self.layered_e2e.chain`) that motivated the Unix-platform and Python-2-shim groups in + `docs/agent-closed-backlog.md`'s "Dependency Strategy Rationale" reference section. 5. Install detected missing packages via conda or pip 6. Rebuild EXE 7. Retry interpreter smoke test @@ -482,21 +424,22 @@ See README.md section Dependency strategy for full details. ## Bootstrap Architecture Principles -This system prioritizes **deterministic execution during bootstrap** over packaging purity. These principles guide decisions about tool invocation, dependency handling, and error handling in `run_setup.bat`: - -1. **Bootstrap reliability > API correctness.** If a feature depends on "maybe PATH is set" or "activation might work," it is invalid for bootstrap paths. Determinism is non-negotiable. - -2. **Never depend on console scripts during bootstrap.** Console scripts (`pipreqs`, `pytest`, etc.) are forbidden in bootstrap logic because they require: Scripts/ on PATH, activation state correctness, OS-level shim resolution. Instead: use explicit interpreter paths or direct Python APIs. - -3. **All execution must be interpreter-anchored.** Every tool invocation roots in an explicit Python executable path (`%HP_PY%` or `%CONDA_PREFIX%\python.exe`), never relying on PATH or activation to supply the correct interpreter. - -4. **Pinned dependencies are assumed stable.** For version-frozen tools (pipreqs 0.4.13), internal behavior and module structure may be relied upon as stable by contract. Internal coupling is acceptable when version is locked. - -5. **Bootstrap must fail fast and explicitly.** If bootstrap cannot guarantee interpreter, environment, or dependency availability, it fails loudly and early. No silent fallbacks unless explicitly logged. - -6. **Non-obvious decisions must be self-documenting.** If bootstrap does something like `python -m pipreqs.pipreqs` instead of `pipreqs`, it must include a comment explaining why PATH/CLI/activation was not used. Future maintainers must not be tempted to "fix" it incorrectly. - -**Application:** These principles validate the pipreqs invocation strategy, justify the dep-check cache optimization, and guide all future bootstrap-critical decisions. See pipreqs invocation section above for a concrete example. +This system prioritizes **deterministic execution during bootstrap** over packaging purity -- +these six principles govern tool invocation, dependency handling, and error handling in +`run_setup.bat` (the pipreqs invocation strategy above is the canonical worked example): + +1. **Bootstrap reliability > API correctness.** A feature depending on "maybe PATH is set" or + "activation might work" is invalid for bootstrap paths -- determinism is non-negotiable. +2. **Never depend on console scripts during bootstrap** (`pipreqs`, `pytest`, etc. all require + `Scripts/` on PATH and activation state neither is guaranteed) -- use explicit interpreter + paths or direct Python APIs instead. +3. **All execution must be interpreter-anchored**: every tool invocation roots in an explicit + Python executable path (`%HP_PY%` or `%CONDA_PREFIX%\python.exe`), never PATH/activation. +4. **Pinned dependencies are assumed stable** -- for a version-frozen tool (pipreqs 0.4.13), + relying on its internal module structure as stable by contract is an acceptable coupling. +5. **Bootstrap must fail fast and explicitly** -- no silent fallbacks unless explicitly logged. +6. **Non-obvious decisions must be self-documenting** (e.g. `python -m pipreqs.pipreqs` instead + of `pipreqs`) so a future maintainer isn't tempted to "fix" it incorrectly. --- @@ -534,38 +477,7 @@ text (see below) rather than leaving the section silently blank** -- an empty se marker is easy to mistake for a rendering glitch or an accidental deletion; an explicit statement is not. -- **Item 34: restructure the three auto-loaded knowledge docs (`docs/agent-interconnect.md`, - `docs/agent-lessons-learned.md`, `docs/agent-ndjson.md`) around a "distill to the load-bearing - rule" house rule, moving narrative/provenance out to `docs/agent-closed-backlog.md`.** Follow-up - to the Item 30 compression pass (PR #428), which found that further sentence-level trimming was - a saturated lever on `agent-interconnect.md` specifically (~0.2% word reduction) -- the file's - bulk is chronological incident-log narrative ("bug found via review, first fix was wrong, second - fix was wrong, third fix confirmed via CI run N") wearing reference-doc clothes, not restated - prose. Scoped as three separate loops (see `docs/agent-closed-backlog.md`'s Item 23/30 entries - for the prior compression history this builds on): - - **Loop 1 (DONE, PR #429):** added the house-rule line to the top of all three auto-loaded - docs codifying "distill to the load-bearing rule; move the narrative/provenance to - `docs/agent-closed-backlog.md`" as the ongoing editing principle, and deduped the two confirmed - cross-file duplications: the cmd.exe `%`-pairing sanitizer saga told at length in both - `agent-interconnect.md`'s DLL-bundling section and `agent-lessons-learned.md`'s ":log echoes - UNQUOTED" entry (canonical home: lessons-learned.md); and the `self.dll_bundle.recover` NDJSON - row's mechanism explained at length in both `agent-interconnect.md` and `agent-ndjson.md` - (canonical home: agent-ndjson.md). Zero information loss -- each fact still lives at its one - canonical home, with a one-line pointer left everywhere else. - - **Loop 2 (DONE, PR #430):** moved CLAUDE.md's own "Known Findings" section to - `docs/agent-closed-backlog.md` -- it is explicitly "diagnosed, no action warranted," which is - verbatim that file's own stated scope ("read on demand... when investigating something that - feels like it was already done"), so it doesn't belong in the always-loaded file at all. - - **Loop 3 (open):** distill CLAUDE.md's dependency-strategy essay (pipreqs pin rationale, - pipreqs invocation rationale, the warnfix `SKIP`-set walkthrough, the 6 Bootstrap Architecture - Principles) down to the load-bearing rule per topic, moving the multi-paragraph justification - to a reference doc. Table-ify the per-payload paragraph descriptions in "run_setup.bat Rules" - (payload name / decodes to / purpose / canonical source, one line each) instead of a paragraph - per payload. - **Deliberately keeping "Periodic Maintenance Checks" in CLAUDE.md, NOT moved out** -- owner - decision: the quarterly Claude Code Remote trigger's own reliability isn't confirmed yet, so this - section staying always-loaded is an intentional fallback until that trigger has enough of a track - record to be trusted as the sole delivery mechanism. +**Nothing here.** All items above have been resolved and moved to `docs/agent-closed-backlog.md`. ## Cold Storage (promising ideas, deliberately shelved -- revisit only if a named trigger fires) diff --git a/docs/agent-closed-backlog.md b/docs/agent-closed-backlog.md index a2b35dd1..a72fb363 100644 --- a/docs/agent-closed-backlog.md +++ b/docs/agent-closed-backlog.md @@ -20,15 +20,18 @@ likely be reorganized again as it grows. Treat a Part/Scenario citation here as look," not a precise current coordinate; this file is an append-only historical record, so its own entries are not retroactively renumbered to track that doc's current structure. -**Three sections below.** "Closed Active Backlog Items" holds items that were promoted out of +**Four sections below.** "Closed Active Backlog Items" holds items that were promoted out of `CLAUDE.md`'s own "Active Backlog" section once fully resolved (each keeps its original item number for cross-reference stability -- other docs cite these by number). "Known Findings" (moved here verbatim from `CLAUDE.md`, 2026-08-09, Active Backlog Item 34 Loop 2) holds diagnosed-no-action-warranted investigations -- a question was raised, researched, and answered with "no code change needed" or "considered and rejected, with reasoning" -- distinct from a Closed Active Backlog item (resolved with a real change) or an open Active Backlog item (still -pending). "Closed Backlog" is `CLAUDE.md`'s own pre-existing changelog-style record of completed -feature/fix work, moved here verbatim. +pending). "Dependency Strategy Rationale" (moved here 2026-08-09, Active Backlog Item 34 Loop 3) +holds the full multi-paragraph justification behind standing CLAUDE.md rules (the pipreqs version +pin, its invocation strategy, the warnfix `SKIP`-set) -- CLAUDE.md keeps only the load-bearing +rule plus a pointer here. "Closed Backlog" is `CLAUDE.md`'s own pre-existing changelog-style +record of completed feature/fix work, moved here verbatim. --- @@ -1896,6 +1899,53 @@ this belongs to). genuinely differ in actionability (`missing_output` suggests an AV/indexer conflict a user could investigate; `build_error`/`test_forced_fail` do not) and the token costs nothing to add. +### Item 34 (closed 2026-08-09) + +- **Restructure the three auto-loaded knowledge docs (`docs/agent-interconnect.md`, + `docs/agent-lessons-learned.md`, `docs/agent-ndjson.md`) around a "distill to the load-bearing + rule" house rule, moving narrative/provenance out to this file.** Follow-up to the Item 30 + compression pass, which found that further sentence-level trimming was a saturated lever on + `agent-interconnect.md` specifically (~0.2% word reduction) -- the file's bulk was chronological + incident-log narrative ("bug found via review, first fix was wrong, second fix was wrong, third + fix confirmed via CI run N") wearing reference-doc clothes, not restated prose. Shipped as three + separate loops: + - **Loop 1 (PR #429):** added the house-rule line to the top of all three auto-loaded docs + codifying "distill to the load-bearing rule; move the narrative/provenance to this file" as + the ongoing editing principle, and deduped the two confirmed cross-file duplications: the + cmd.exe `%`-pairing sanitizer saga told at length in both `agent-interconnect.md`'s + DLL-bundling section and `agent-lessons-learned.md`'s ":log echoes UNQUOTED" entry (canonical + home: lessons-learned.md); and the `self.dll_bundle.recover` NDJSON row's mechanism explained + at length in both `agent-interconnect.md` and `agent-ndjson.md` (canonical home: + agent-ndjson.md). Zero information loss -- each fact still lives at its one canonical home, + with a one-line pointer left everywhere else. Net: `agent-interconnect.md` 1981 -> 1918 lines + (20991 -> 20224 words, ~3.7% reduction). + - **Loop 2 (PR #430):** moved CLAUDE.md's own "Known Findings" section here verbatim (this + file's "Known Findings" section, below) -- it was explicitly "diagnosed, no action warranted," + verbatim this file's own stated scope, so it didn't belong in the always-loaded file. Also + fixed every stale "CLAUDE.md's Known Findings" cross-reference across the repo + (`docs/agent-interconnect.md` x2, `docs/plan-pvw-quickstart.md`, `docs/prd-av-safe-build-path.md` + x2), plus two genuinely pre-existing stale pointers caught by the same sweep (CLAUDE.md's own + "Next-pin probe" note wrongly cited "Known Findings" for REQ-AV Tier B's rationale, which + actually lives in `docs/agent-cold-storage.md`; `tests/selfapps_ux_hardening.ps1`'s `:die` + comment wrongly cited "Known Findings" for a fact that has always lived in + `docs/agent-lessons-learned.md`'s ":die" entry). Net: CLAUDE.md 916 -> 775 lines (8186 -> 6496 + words, ~20.6% reduction) -- the largest single cut in the series, since this section was pure + historical record with zero forward-looking action attached to any entry. + - **Loop 3 (this pass):** distilled CLAUDE.md's dependency-strategy essay (pipreqs pin + rationale, pipreqs invocation rationale, the warnfix `SKIP`-set walkthrough) down to the + load-bearing rule per topic, moving the full multi-paragraph justification to this file's + "Dependency Strategy Rationale" section (below) -- zero information loss, same pointer + pattern as Loops 1-2. Trimmed the 6 Bootstrap Architecture Principles in place (no separate + narrative to move; the compression there was wording-only, cutting redundant clauses while + keeping all 6 principles and their concrete examples). Table-ified the "run_setup.bat Rules" + section's 7-payload paragraph list into a 4-column table (payload / decodes-to / purpose / + canonical source). + **Deliberately kept "Periodic Maintenance Checks" in CLAUDE.md, NOT moved out** -- owner + decision: the quarterly Claude Code Remote trigger's own reliability wasn't confirmed at the + time, so that section staying always-loaded was an intentional fallback until the trigger has + enough of a track record to be trusted as the sole delivery mechanism. Revisit whether it can + move out once that track record exists. + ## Known Findings (diagnosed, no action warranted) - **Backlog item numbering: renumber-on-collision convention dropped, 2026-07-31 owner decision.** @@ -2047,6 +2097,84 @@ this belongs to). off-CI (expected ~30-45 min per the maintainer's prior experience). Until then, treat the "environmental" classification as strongly-supported-but-provisional. +## Dependency Strategy Rationale (reference) + +Full multi-paragraph justification behind the load-bearing rules CLAUDE.md states in its +"Dependency Discovery" sections -- moved here 2026-08-09 (Active Backlog Item 34 Loop 3) since +this is background evidence an agent needs only when questioning or extending one of those +rules, not on every session. + +### pipreqs pin rationale (as of 2026-06-18): why 0.4.13, not 0.5.0 + +- pipreqs 0.5.0 (the latest release) added Jupyter notebook scanning, which hard-pins + `ipython==8.12.3` (the last ipython supporting Python 3.8). ipython 8.12.3 does not support + Python 3.13+, so 0.5.0's metadata declares `Requires-Python >=3.8.1,<3.13`. +- The bootstrapper always targets the latest conda-forge Python (currently 3.14+). On that + Python, pip refuses to install 0.5.0 (version cap), so pipreqs would be lost entirely and + every run would fall back to warnfix. +- pipreqs 0.4.13 has `Requires-Python >=3.7` (no upper cap), deps only `docopt`+`yarg`, supports + the same `--mode compat` / `--force` / `--savepath` flags, uses only stable stdlib (ast-based + scan), and runs on Python 3.14. It restores pipreqs as the primary discovery tool. +- **Do NOT "upgrade" the pin back to 0.5.0** -- it reintroduces the `<3.13` cap and silently + disables pipreqs on modern Python. The only feature lost by 0.4.13 is `.ipynb` scanning, which + was already non-functional on latest Python (0.5.0 cannot run there). + +### pipreqs invocation rationale: why `python -m pipreqs.pipreqs`, not the console script + +This is an intentional bootstrap execution strategy, not a workaround for pipreqs limitations. + +**Constraints driving this choice:** +- Windows batch bootstrap never depends on shell state (PATH, activation, environment variables) +- Bootstrap runs immediately after environment creation in the same shell session +- Console scripts require PATH correctness and activation to persist -- neither is guaranteed +- Bootstrap reliability > API purity in this system class + +**Why internal module invocation is safe here:** +- pipreqs is pinned to 0.4.13 permanently (no automatic upgrades) +- Version freeze makes internal module structure (`pipreqs/pipreqs.py`) stable by contract +- Internal coupling is a low-risk controlled assumption due to the pinned dependency version + +**Comparison of approaches:** + +| Approach | Reliability in Bootstrap | Architecture | Scope | +|----------|--------------------------|--------------|-------| +| `pipreqs` (console script) | Fragile (PATH dependent) | Official API | General use | +| `python -m pipreqs.pipreqs` | Deterministic (no PATH) | Internal mechanism | Bootstrap only | + +See `run_setup.bat`'s own invocation comment (search `python -m pipreqs.pipreqs`) for the +in-code pointer. This is a **deterministic execution pattern required for bootstrap +reliability**, not a sign of fragility or a temporary workaround. + +### warnfix `SKIP`-set walkthrough: why each group exists + +`parse_warn.py` uses two distinct mechanisms, not one -- keep them separate when describing or +extending this filter: +- A generic, `SKIP`-independent rule (`if mod.startswith("_"): continue`) drops any + leading-underscore internal module by name pattern alone -- this is what actually filters + `_scproxy`, `_posixsubprocess`, and `_frozen_importlib_external`; none of the three is a + `SKIP` entry. +- The `SKIP` frozenset covers everything else warnfix must not treat as an application + dependency, across a few distinct groups: packaging/import-machinery internals + (`pkg_resources`, `distutils`, `setuptools`, `importlib` and its submodules) -- these ARE + real, installable PyPI packages, but PyInstaller's own bundling of the interpreter's import + machinery can surface them as "missing" even though the app never actually needs a separate + install; `collections` (also covers `collections.abc`, since dotted names truncate to their + top-level package before the `SKIP` lookup) -- this one is the stdlib's own submodules + surfacing as "missing," not a platform gap; Unix-only platform modules absent on Windows + (`grp`, `pwd`, `posix`, `resource`, `fcntl`, `readline`, `termios`, `tty`, `pty`, `crypt`, + `spwd`, `nis`, `syslog`, `ossaudiodev`); and Python-2-only stdlib shims removed entirely in + Python 3 (`cStringIO`, `StringIO`) that still surface via real packages' own Python 2/3 + compatibility code -- e.g. `xlrd`'s `xlrd/timemachine.py` does `try: from cStringIO import + StringIO except ImportError: from io import StringIO`, a dead code path under Python 3 that + PyInstaller's static analysis still flags. Confirmed via a real, unflagged CI run of the + layered-dependency E2E test (`self.layered_e2e.chain`) -- warnfix was genuinely attempting and + failing to `conda install cStringIO`, which can never succeed, forcing an unnecessary extra + provider cascade. Every entry in `SKIP` is covered by + `tests/test_parse_warn.py::ParseWarnFileEdgeCasesTest::test_every_skip_entry_filtered_in_realistic_warn_line`, + which iterates the current set and proves each one filters in the + `(conditional)`/`(delayed)`/`(top-level)` PyInstaller 6.x forms -- any future `SKIP` addition + is covered automatically, no separate test needed per entry. + ## Closed Backlog - **Cascade-vs-postexec fix (Active Backlog item 9), 2026-07-25, owner-directed follow-up to a From d9546d00cedaadda5c686dcffe8ce9f336ae1771 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 16:11:04 +0000 Subject: [PATCH 2/2] Fix PR #431 CodeRabbit finding: pipreqs CI gate note read as contradicting the pin directive The compression pass moved "the pipreqs.flags CI gate validates invocation flags, not the version, so the pin is free to change" into the same sentence group as "do not upgrade this pin", making a pre-existing narrow technical note (the CI gate doesn't assert an exact version string) read as directly contradicting the strong prohibition right above it. Reworded to state plainly that the gate's flag-only scope does not authorize changing 0.4.13. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01A4iE1BRSkUETwz237XeuTW --- CLAUDE.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 9a549e3a..b8744b3d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -370,9 +370,9 @@ and fall every run back to warnfix; 0.4.13 has no upper Python cap and runs fine only feature lost by staying on 0.4.13 is `.ipynb` scanning, already non-functional on latest Python since 0.5.0 can't run there anyway. Full version-comparison rationale in `docs/agent-closed-backlog.md`'s "Dependency Strategy Rationale" reference section. The -`pipreqs.flags` CI gate validates invocation flags, not the version, so the pin is free to change; -the setup log line `[INFO] pipreqs installed successfully` confirms pipreqs is active on a -given run. +`pipreqs.flags` CI gate validates invocation flags only, not the pinned version -- it does not +authorize changing 0.4.13; the setup log line `[INFO] pipreqs installed successfully` +confirms pipreqs is active on a given run. ## Dependency Discovery: pipreqs invocation (bootstrap determinism)