feat(mcp): report truthful index freshness from checkout evidence

[tmonestudio]

What does this PR do?

Makes verbose index_status distinguish the live checkout from the generation that was actually indexed.

• persists indexed_checkout_sha only at the staged-generation publication boundary
• reports bounded tracked/untracked checkout evidence
• returns machine-readable current, stale, or fail-closed unknown
• preserves the lean default response and legacy databases
• handles Windows porcelain-v2 rename/copy records and bounded git subprocess failures

This is intentionally separate from #1181 and #1065.

Local verification

• production Windows/MinGW build passed with the repository warnings-as-errors flags
• focused test runner: git_context mcp store_nodes exited 0
• existing database smoke: 27,369 nodes / 78,605 edges; legacy generation returned unknown/indexed_checkout_unavailable
• database SHA-256 was identical before and after the smoke

Checklist

• Every commit is signed off (git commit -s)
• Full test suite is delegated to CI
• Focused tests pass locally
• New behavior is covered by reproduce-first tests

No runtime binary, cache, ACL, or corpus was modified.

[github-actions]

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

• It will not be closed for inactivity. No stale bot touches pull requests here.
• It may still sit a while before a human reads it. That is on us, not on you.
• Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

• Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
• Get CI green, or say which failures you believe are pre-existing.
• Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
• Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

[DeusData]

Thank you @tmonestudio. The underlying issue is confirmed on current main: verbose index_status reports live checkout context, while the indexed generation stores no corresponding checkout identity. Persisting identity at the publication boundary is the right contract to review.

This is a broad correctness change despite the focused product claim: 11 files and about 1,300 added lines across git subprocess handling, MCP output, pipeline publication, store migration, and tests. It therefore needs a full storage/schema and fail-closed review rather than a quick UI pass. The current head is mergeable, but lint / lint and consequently ci-ok are red and the main test matrix was skipped. Please clear the lint failure before review.

Thank you for the reproduce-first coverage and legacy-database checks. The queue is full, so detailed feedback may take some time, but this is labeled and queued.

[DeusData]

Checking in — this has been quiet since 12 August and it is still open and still wanted, so here is where it stands with the friction removed.

The lint blocker is six clang-format violations in two files:

src/store/store.c:3171:94
src/store/store.c:3172:34
src/mcp/mcp.c:4504:80
src/mcp/mcp.c:4505:57
src/mcp/mcp.c:4505:78
src/mcp/mcp.c:4506:60

make -f Makefile.cbm lint-format reproduces it, and running clang-format over just those two files fixes it. One caveat that has bitten people here: it must be the Homebrew LLVM build — a standalone clang-format-20 produces spurious whole-file drift.

And an honest note on the rest, so the delay does not read as all yours. Even with lint green, this is not a fast merge: 11 files and ~1,300 lines spanning git subprocess handling, MCP output, pipeline publication, store migration and tests. It needs a full storage/schema and fail-closed review, and that review is ours to do and has not happened yet. Clearing lint is what lets the main test matrix actually run — it was skipped entirely on the last CI pass, so nobody has seen this change tested.

The underlying problem you identified is confirmed on current main and the contract you proposed — persisting checkout identity at the publication boundary — is still the right shape. Your reproduce-first coverage and the legacy-database checks were noted at the time and still stand.

If you have moved on, say so and we will take it from here rather than leaving it to age. If not, clear the lint and the matrix will finally have something to say.

[DeusData]

An update, and a decision that went in your favour.

#1727 independently added a top-level freshness to index_status as well — a string (fresh / stale / unknown) rather than your object, derived from clean Git snapshots taken before and after indexing. Neither of you could have seen the other: the PRs are independent, and git would have merged them without a conflict, leaving index_status with one key emitted twice at different types depending on merge order.

Your object stands. The reasoning is that a caller which learns why the graph is not current, and what to do about it, is better served than one that gets a single word — your reason, reasons[] and recommended_action are the difference. This PR is also standalone and already reviewed, so it does not wait on a six-PR stack. #1727 has been asked to drop its field and keep its last_index_attempt record, which is genuinely additive to yours.

One idea from that PR worth folding into yours if you touch the docs: "a dirty tree is never treated as evidence in either direction". Your implementation already behaves that way — a dirty tree cannot reach current because tracked changes force stale, and status_unavailable prevents a confident answer — but that sentence states the principle more clearly than anything currently written down here.

Where this stands otherwise: the storage/schema and fail-closed review you were owed is done and it passes — the three-DB-state migration (fresh, legacy-writable, legacy read-only) and the verdict ladder that never reaches current without indexed SHA, live HEAD, status availability and zero tracked/untracked. The only thing between this and merge is a clearance marker on the three cbm_popen calls in your test files, which is a maintainer step, not something you need to change.

Thanks for your patience — this has been open a long time, and it is the design that is being kept.

[DeusData]

Clearance is done and I've now verified this end to end on today's main rather than on the August base: trial-merged onto fe85a6b2, built with ASan/UBSan, and ran git_context (10/10, seven of them yours), store_nodes 71/0, index_resilience 6/0, daemon_frontend 13/0, mcp 318/0, lint clean; then with the real binary in an isolated cache: index → current with the SHA matching HEAD, tracked edit → stale / tracked_changes_present, commit → stale / indexed_checkout_mismatch, untracked-only → unknown, re-index → current with the new SHA. The revert-check binds exactly where it should (never reading the persisted SHA reddens the store round-trip and four MCP verdict tests; dropping the tracked≠stale rule reddens the tracked tests and rename_counts_once). The fit is right too: one top-level freshness object, nothing else on the surface beyond the nullable indexed_checkout_sha column with its idempotent migration.

Two things are needed before it merges, both mechanical, and both caused by main moving under you:

1. One conflict, src/mcp/mcp.c:44-53 (the enum block): your branch adds MCP_STATUS_SAMPLE_MAX / MCP_REASONS_MAX, main added MCP_QUERY_MAX_VISIBLE_ROWS / MCP_OUTPUT_BYTES_PER_TOKEN_ESTIMATE in the same spot. Keep both.
2. Nine tests need "format":"json" in their index_status call: feat: make CLI and MCP output lean by default #1597 (merged this week) made the tree format the default for tool output, so JSON-substring assertions no longer match the default response. The calls are at tests/test_mcp.c:5711, 5736, 5843, 5881, 5937, 5988, 6046, 6076, 6131; main's own tool_index_status_includes_git_metadata shows the pattern. With that, mcp is 318/0. Line 5736 then runs past 100 columns, so run clang-format (the Homebrew LLVM one) over the file.

Non-blocking, take or leave: the verbose description at src/mcp/mcp.c:680-683 still says only "add worktree/shadow Git paths" — worth a clause about freshness; and recommended_action maps every unknown to reindex_to_record_indexed_checkout, which misleads for untracked_not_indexed / status_unavailable where the identity is recorded — a follow-up is fine.

Keep the sign-offs on the rebased commits and this merges on green. Thanks for the fast turnaround on the lint fix in August and for sticking with it.

[tmonestudio]

Thanks for the CI report. I traced both failures to the same underlying issue: ci-ok was only aggregating the failed Unix shard, whose sole failing suite was complexity.

The failure was scheduler-sensitive, not a freshness regression: the shared-package scaling test observed nodes 244 -> 842 (ratio 3.45) while edges stayed 659 -> 1319 (ratio 2.00), exceeding the 2.75 node-ratio guard under the parallel wave. Re-running the suite locally produced the expected nodes 244 -> 484 (ratio 1.98) consistently.

Fix pushed in commit 6875483f: classify complexity as a serial/quiet-tail suite in scripts/run-tests-parallel.sh, alongside extraction, because both measure replicated-corpus scaling and are sensitive to concurrent allocator/CPU contention. No product logic or freshness behavior was changed.

Local verification on the updated source:

• git_context: 7 passed, 3 platform skips
• store_nodes: 71 passed
• index_resilience: 6 passed, 1 platform skip
• mcp: 315 passed, 11 platform skips
• complexity: 5 passed repeatedly

Please rerun the Unix shard/CI checks for 6875483f.

[DeusData]

Thank you for tracking down the failing shard and recording both the parallel and isolated counts. That evidence is useful.

Please separate the serial-suite workaround in 6875483 from this freshness PR. The reported change from 484 to 842 nodes is a difference in graph contents, not just elapsed time; moving the suite to the quiet tail may avoid the symptom, but does not yet establish whether the cause is shared test state, extraction nondeterminism, or another defect. Please preserve the failing run and fixture details so we can investigate that discrepancy separately rather than treating a serial green run as a correctness fix.

This does not withdraw our agreement on the freshness design. We also owe you the final integration review after our earlier merge commitment. Thank you for the repeated updates and your patience.