fix: redact current_stage from student_public bundles

[shaypal5]

Summary

Fixes the current_stage leakage that was surviving into student_public task splits at the 90-day horizon (where the column held terminal stages closed_won/closed_lost, perfectly encoding the label). Closes the "Known issue" entry under Phase 5 of .agent-plan.md.

Why this approach (option 2 — exposure-layer redaction)

I considered both options listed in .agent-plan.md:

1. Windowed snapshot (snapshot at day < label horizon, like v6/v7 datasets at day 14/20). Fundamentally more correct — features anchored before label resolution — but blast radius is huge: every aggregated feature value shifts, all downstream documentation and conversion-rate copy needs rewriting, and every byte of every published Parquet changes for non-leakage reasons. The v6/v7 datasets that do use windowed snapshots are simplified one-off teaching CSVs (lead_scoring_intro_v*.csv), not the full relational release/ bundle.
2. Exposure-layer column redaction (chosen). Strip leakage_risk and not is_leakage_trap columns in student_public mode. Smaller blast radius (only the leaky columns disappear), generalisable rule, naturally reuses the existing BundleFilter machinery, and preserves the deliberately included pedagogical trap total_touches_all. The simulation, the relational tables, and every non-leakage feature are unchanged.

A new is_leakage_trap field on FeatureSpec distinguishes the trap (total_touches_all, intentionally included for teaching) from a true leak (current_stage, the bug). The redaction set is derived from features at import time — any future leakage_risk=True column without is_leakage_trap=True is automatically stripped from student_public exports.

is_mql / is_sql were reviewed at the same time per the task brief: they are point-in-time achievement flags that correlate with the label but don't deterministically encode it (is_sql=True for many non-converted leads who reached SQL but didn't close). Their existing leakage_risk=False is correct; no change needed.

Changes

Schema layer

• schema/features.py: new is_leakage_trap: bool = False on FeatureSpec; set True on total_touches_all; export STUDENT_PUBLIC_REDACTED_COLUMNS.
• schema/dictionaries.py: emit is_leakage_trap column in feature_dictionary.csv so consumers can tell traps from leaks.

Exposure layer

• exposure/filters.py: BundleFilter.redacted_columns: frozenset[str]. Populated for student_public from the feature spec; empty for research_instructor.

Bundle assembly

• api/bundle.py: drop redacted columns from the snapshot DataFrame before writing task splits, and pass the visible feature tuple to the dictionary writer and dataset card renderer. Manifest hashes therefore reflect the published column set.
• narrative/dataset_card.py: accept a features arg so the categories / leakage section reflects what's actually published.

Validation

• validation/bundle_checks.py: new _check_exposure_redaction() reads exposure_mode from the manifest and asserts that the redacted columns are not present in any task split or in the feature dictionary. Wires CLAUDE.md invariant docs: add mandatory branch/PR workflow rules #1 into validate_bundle() instead of relying on hope.
• validation/invariants.py: relax check_exposure_monotonicity() from byte identity on feature_dictionary.csv and task splits to subset-with-shared-content semantics (student rows ⊆ instructor rows; shared columns hold identical values).

Build pipeline

• scripts/build_public_release.py: drop _FLAT_CSV_DROP_COLS — the bundle writer now redacts at source, so the flat CSV inherits the redaction transitively.

Documentation

• release/README.md and release/HF_DATASET_CARD.md: update column counts (33 + trap + target in student_public; 34 + trap + target in instructor) and rewrite the leakage section to explain stripped vs. deliberately retained.
• .agent-plan.md: flip the "Known issue: current_stage leakage" entry to resolved.

Tests

tests/exposure/test_redaction.py (19 new tests):

• Static checks: BundleFilter.redacted_columns matches the derived feature-spec set; research_instructor redacts nothing; the redaction set contains current_stage and excludes total_touches_all.
• End-to-end (student_public): every task split lacks current_stage, every task split keeps total_touches_all, no redacted column survives in any split, the target column is intact, the feature dictionary excludes current_stage, includes the trap, and has the expected row count, and validate_bundle() produces no redaction errors.
• End-to-end (research_instructor): every task split keeps current_stage and total_touches_all; the dictionary still has all 35 rows.
• Cross-mode: student columns ⊆ instructor columns; the difference is exactly the redacted set; for shared columns the values match row-by-row (post-shuffle, by determinism).
• Validation enforcement: an instructor bundle whose manifest has been hand-edited to claim student_public is correctly flagged by validate_bundle().

tests/schema/test_features.py extended with five new assertions covering the trap flag, the redaction set, and current_stage's role.

Verification

• pytest: 899 passing (was 880; +19 new redaction tests).
• ruff check / ruff format / mypy: clean.
• python scripts/build_public_release.py release: all 4 bundles generated and validated. Confirmed via pyarrow.parquet.read_schema:
  • release/{intro,intermediate,advanced}/.../train.parquet → 34 columns, no current_stage, total_touches_all present.
  • release/intermediate_instructor/.../train.parquet → 35 columns including current_stage.
• python scripts/verify_hash_determinism.py: 73/73 files hash-identical across two consecutive builds. Determinism preserved.

Out of scope

Per the task brief: uploading to Kaggle / HuggingFace and announcing the release (manual user steps, last in Phase 5).

Test plan

• pytest passes (899/899)
• ruff check and ruff format --check clean
• mypy leadforge/ clean
• scripts/build_public_release.py regenerates all 4 bundles, all pass validate_bundle()
• scripts/verify_hash_determinism.py reports 73/73 identical
• Manual inspection confirms current_stage is absent from student_public task splits and present in research_instructor
• feature_dictionary.csv row counts: 34 in student_public (no current_stage), 35 in research_instructor

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR fixes a label leakage issue in student_public bundles by introducing an exposure-layer redaction rule that removes leakage-risk columns (except explicitly marked pedagogical traps) from published task splits and their associated metadata (feature dictionary + dataset card).

Changes:

• Add is_leakage_trap to FeatureSpec and derive STUDENT_PUBLIC_REDACTED_COLUMNS from the canonical feature spec.
• Apply redaction during bundle writing (before task splits / hashes) and thread the “visible” feature list into the feature dictionary + dataset card.
• Strengthen validation/invariants and add end-to-end tests to ensure redacted columns never appear in student_public outputs.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
leadforge/schema/features.py	Adds is_leakage_trap and derives the student-public redaction set from feature specs.
leadforge/schema/dictionaries.py	Emits is_leakage_trap in feature_dictionary.csv.
leadforge/exposure/filters.py	Extends BundleFilter with redacted_columns and wires student-public defaults.
leadforge/api/bundle.py	Drops redacted columns pre-write and passes visible features to dictionary + dataset card rendering.
leadforge/narrative/dataset_card.py	Accepts a features argument so the card reflects the published column set.
leadforge/validation/bundle_checks.py	Adds _check_exposure_redaction() and wires it into validate_bundle().
leadforge/validation/invariants.py	Updates exposure monotonicity checks to allow subset semantics for redacted outputs.
scripts/build_public_release.py	Removes flat-CSV-specific dropping in favor of source redaction.
tests/exposure/test_redaction.py	Adds end-to-end tests covering student/instructor behavior, cross-mode consistency, and validation enforcement.
tests/schema/test_features.py	Adds schema-level assertions for traps, redaction set, and dictionary columns.
release/README.md	Updates leakage section and column-count documentation to match redaction behavior.
release/HF_DATASET_CARD.md	Updates dataset card leakage explanation and feature/column counts.
.agent-plan.md	Marks the current_stage leakage issue as resolved with the chosen approach.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[shaypal5]

Self-review pass: 10 issues raised, 10 addressed in commit 66b96ee

I took an outsider's view on the original two commits and found ten substantive problems. Concrete fixes per critique:

#	Issue	Action
1	Symptomatic vs structural fix (event aggregates also leak)	Documented as a clearly-scoped follow-up in .agent-plan.md covering windowed-snapshot, is_sql, and is_mql. Not opening a tracked issue without explicit confirmation.
2	is_leakage_trap contradicts leakage_risk	Collapsed: leakage_risk is now purely descriptive; new redact_in_modes: frozenset[ExposureMode] is the single prescriptive field.
3	_check_exposure_redaction was self-referential	Validator now derives expected redactions from LEAD_SNAPSHOT_FEATURES directly via redacted_columns_for, independent of BundleFilter. New regression test mutates a real parquet to reinsert current_stage and asserts validation fails.
4	is_mql / is_sql review hand-waved	Measured on regenerated bundle. P(conv | is_sql=False) = 0.038 / 0.015 / 0.006 (intro/intermediate/advanced) — near-deterministic for the negative class. P(conv | is_mql=True) = base rate (because is_mql is constant True — separate zero-variance bug). Captured in the follow-up note.
5	Bundle wasn't self-describing	manifest.json now carries redacted_columns: [...]; validator cross-checks. New test verifies the manifest matches the spec-derived expected set.
6	README duplicated auto-generated counts	Removed the per-category table; readers are pointed at dataset_card.md and feature_dictionary.csv.
7	Added is_leakage_trap column to feature_dictionary.csv without asking	Reverted. CSV is back to its pre-PR 6-column schema. The redaction policy stays package-internal; the published schema is observable from parquet + manifest.redacted_columns.
8	check_exposure_monotonicity was too lenient	Now asserts instructor_cols - student_cols == redacted_columns_for(student_public) exactly, not just superset.
9	Module-level STUDENT_PUBLIC_REDACTED_COLUMNS not parameterizable	Replaced by redacted_columns_for(mode, features=LEAD_SNAPSHOT_FEATURES) -> frozenset[str].
10	Tautological leakage_trap_implies_leakage_risk test	Dropped with #2. New schema tests assert positive properties (target never redacted; current_stage is in the student_public set; trap is not).

Verification after refactor:

• pytest: 902 passing (was 899; +3 new validator/manifest tests, no regressions)
• ruff / mypy: clean
• scripts/build_public_release.py regenerates all four bundles, all pass validate_bundle()
• scripts/verify_hash_determinism.py: 73/73 files identical across two consecutive builds
• feature_dictionary.csv reverted to pre-PR schema; manifest.json now has redacted_columns

[github-actions]

pr-agent-context report:

This run includes unresolved review comments on PR #56 in repository https://github.com/leadforge-dev/leadforge

For each unresolved review comment, recommend one of: resolve as irrelevant, accept and implement
the recommended solution, open a separate issue and resolve as out-of-scope for this PR, accept and
implement a different solution, or resolve as already treated by the code.

After I reply with my decision per item, implement the accepted actions, resolve the corresponding
PR comments, and push all of these changes in a single commit.

# Copilot Comments

## COPILOT-1
Location: release/HF_DATASET_CARD.md
URL: https://github.com/leadforge-dev/leadforge/pull/56#discussion_r3180311658
Status: outdated
Root author: copilot-pull-request-reviewer

Comment:
    The summary table’s “Features” count appears inconsistent with the bundle’s actual published schema described elsewhere (e.g., `student_public` task splits are 34 columns total = 33 non-target columns + 1 target, where the 33 includes the `total_touches_all` trap). “33 (+ 1 trap, + 1 target)” reads like 35 columns. Please clarify whether the 33 includes the trap or update the numbers/labels to match the actual Parquet column counts.

## COPILOT-2
Location: release/README.md
URL: https://github.com/leadforge-dev/leadforge/pull/56#discussion_r3180311717
Status: outdated
Root author: copilot-pull-request-reviewer

Comment:
    The “Columns (instructor)” cell is confusing/inconsistent: “35 (34 features + 1 target, +`current_stage`)” suggests `current_stage` is in addition to the 34 features, but it should be included in the 34 non-target columns if the total is 35. Please rephrase to avoid double-counting (e.g., “35 (34 features incl. `current_stage` + 1 target)”).

## COPILOT-3
Location: leadforge/validation/invariants.py:174
URL: https://github.com/leadforge-dev/leadforge/pull/56#discussion_r3180311750
Root author: copilot-pull-request-reviewer

Comment:
    In `check_exposure_monotonicity()`, `extra_in_student = set(s_df.index) - set(i_df.index)` represents feature names that are present in the student dictionary but *not* in the instructor dictionary. The error text currently says “student has rows missing from instructor”, which reads backwards/ambiguous; it should say that student has rows *not present* in instructor (or similar) so the violation is clear.

## COPILOT-4
Location: leadforge/validation/invariants.py:265
URL: https://github.com/leadforge-dev/leadforge/pull/56#discussion_r3180311775
Root author: copilot-pull-request-reviewer

Comment:
    `extra_in_student = set(s_df.columns) - set(i_df.columns)` means the student task split has columns that are *not present* in the instructor split. The emitted message says “student has columns missing from instructor”, which is misleading (it sounds like the inverse). Please reword to something like “student has extra columns not in instructor” to match the condition being checked.

## COPILOT-5
Location: leadforge/validation/bundle_checks.py
URL: https://github.com/leadforge-dev/leadforge/pull/56#discussion_r3180311801
Status: outdated
Root author: copilot-pull-request-reviewer

Comment:
    The docstring for `_check_exposure_redaction()` says it enforces “no post-snapshot features in student_public exports”, but the code explicitly allows leakage-risk features marked as pedagogical traps (e.g. `total_touches_all`). Consider updating the wording to reflect the actual contract (e.g., “no leakage-risk columns except explicit traps”), otherwise the comment/invariant reference is misleading.

Run metadata:

Tool ref: v4
Tool version: 4.0.21
Trigger: commit pushed
Workflow run: 25313065556 attempt 1
Comment timestamp: 2026-05-04T10:06:24.394223+00:00
PR head commit: 66b96ee1d567ad4fb1080c8326b25960401cda3f