PR 5.2: HuggingFace release packager + load_dataset smoke test

[shaypal5]

Summary

Phase 5 PR 5.2 — second and final PR of the platform-packaging phase. Adds scripts/package_hf_release.py, the HuggingFace counterpart to PR 5.1's Kaggle packager. Generates release/huggingface/README.md with G12.1-compliant YAML frontmatter and the rewritten dataset-card body, and supports a --variant=instructor flag for the companion repo (G12.4).

What ships

• scripts/package_hf_release.py — generates release/huggingface/README.md with YAML frontmatter satisfying G12.1: pretty_name, license: mit, language: [en], task_categories: [tabular-classification], size_categories: [1K<n<10K] per tier, sorted tags: [b2b, crm, datasets, lead-scoring, pandas, synthetic-data, tabular], three configs blocks pointing at the parquet task splits in the assembled upload tree. G12.2 wired in: intermediate is the only config with default: true, locked by validate_card. Body inlines the rewritten release/README.md (PR 4.1) with HF-specific link rewrites (source-repo tree → upload-tree, ](../foo) → GitHub blob URL, ](validation/...) → GitHub blob URL). Hand-rolled YAML renderer (PyYAML's flow style collapses configs[] onto one line, which the HF viewer rejects). CLI: --release-dir, --huggingface-dir, --variant, --default-config, --owner, --dataset-slug, --cover-image, --dry-run. Exit codes: 0 pass / 1 validation failure / 2 pre-flight error.
• scripts/_release_common.py — extracted shared release-packaging primitives so the Kaggle + HF packagers share one source of truth: GITHUB_BLOB_BASE, the ](../foo) regex, the validation-report blob-URL substitution, validate_cover_image (HF reuses the 560×280 floor), and validate_upload_dir_safe (refuses to assemble into cwd / release_dir / its parent / the filesystem anchor). FieldDescriptor / ResourceSchema / dtype maps are intentionally NOT extracted — HF infers schema from parquet via load_dataset and does not need a Frictionless declaration. scripts/package_kaggle_release.py refactored to import these primitives; release/kaggle/dataset-metadata.json is byte-stable across the refactor.
• --variant=instructor — packages the leadforge-lead-scoring-v1-instructor companion repo (G12.4) from release/intermediate_instructor/. The source dir flattens to intermediate/ in the upload tree so load_dataset(..., name=\"intermediate\") works without naming friction. Single-config card; separate release/huggingface-instructor/ output dir.
• Upload-tree assembly — real-file copies (mirrors PR 5.1; datasets walks the upload tree and silently skips broken symlinks in some versions). release/dataset-cover-image.png reused as the HF thumbnail. release/huggingface{,-instructor}/* gitignored except for README.md; only the dataset card is committed.
• release/HF_DATASET_CARD.md legacy stub deleted — superseded by the generated card.

Tests (21 new)

• Every G12.1 YAML field, including the failure modes for empty / mistyped values.
• G12.2 — exactly one config has default: true (zero defaults and two defaults both rejected).
• YAML round-trip through yaml.safe_load; tags-sorted-at-render-time.
• README rewriting (tree + ../ + validation-report links) and the HF_TREE_BLOCK_SOURCE silent-failure guard (mirrors PR 5.1's Kaggle-side).
• Unsafe-dir rejection (release dir / parent / cwd).
• Assembled-tree resource resolution (every declared data_files[*].path materialises as a real file, not a symlink).
• Idempotent re-assembly.
• Byte-determinism (audit-artifact-sync) and committed-README-matches-fresh-regeneration sync check.
• Instructor variant — single config, flattened source dir.
• G12.3 + G12.4 — parametrised load_dataset(local_path, name=<config>) round-trip per public config + instructor companion, with row-count assertions (3500 / 750 / 750 train / valid / test). Gated on the optional datasets SDK (pytest.importorskip); skipped in this CI install.

Local verification

• 1216/1216 tests pass (+ 5 gated skips: 4 datasets-SDK round-trip + 1 kaggle-SDK from PR 5.1).
• ruff check + ruff format --check + mypy clean across the new modules and the refactored Kaggle packager.
• python scripts/package_hf_release.py --dry-run and python scripts/package_hf_release.py --variant=instructor --dry-run both regenerate their committed README byte-identically.
• python scripts/probe_relational_leakage.py release/{intro,intermediate,advanced} --max-accuracy 0.65 exits 0 on every public tier.
• python scripts/verify_hash_determinism.py PASS 67/67.
• python scripts/validate_release_candidate.py --no-rebuild exits 0.
• BUNDLE_SCHEMA_VERSION unchanged at 5.

Test plan

• CI passes on this branch.
• Reviewer confirms YAML frontmatter shape matches the HF dataset-card schema.
• Reviewer spot-checks the rewritten body — no ](../, no intermediate_instructor/ in the tree diagram, no broken validation/... links.
• (Optional) Reviewer with pip install datasets in their env runs the gated round-trip tests and confirms 3500 / 750 / 750 row counts.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds Hugging Face release packaging support (dataset-card generation + upload-tree assembly) alongside shared helpers refactored out of the Kaggle packager, and locks the contract with a comprehensive new test suite (including optional datasets.load_dataset smoke tests).

Changes:

• Introduces scripts/package_hf_release.py to generate HF dataset-card README.md (YAML frontmatter + rewritten body) and optionally assemble an HF-loadable upload directory, including an instructor variant.
• Extracts shared release-packaging utilities into scripts/_release_common.py and refactors the Kaggle packager to consume them.
• Adds extensive tests for HF packaging behavior and adjusts Kaggle tests for the refactored shared cover-image validator messaging; commits generated HF README artifacts and updates .gitignore accordingly.

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/scripts/test_package_kaggle_release.py	Updates cover-image validation assertion to match shared validator messaging.
tests/scripts/test_package_hf_release.py	New test suite locking HF packaging contract (YAML, rewriting, safety guards, determinism, optional load_dataset).
scripts/package_kaggle_release.py	Refactors to import shared primitives from _release_common.
scripts/package_hf_release.py	New HF packager: builds/validates dataset card, rewrites README, assembles upload tree, supports instructor variant.
scripts/_release_common.py	New shared module for link rewriting, cover-image validation, and upload-dir safety validation.
release/huggingface/README.md	Committed generated HF dataset card for public tiers.
release/huggingface-instructor/README.md	Committed generated HF dataset card for instructor companion variant.
release/HF_DATASET_CARD.md	Removes legacy stub dataset card superseded by generated READMEs.
.gitignore	Ignores generated HF upload trees while keeping committed README artifacts.
.agent-plan.md	Marks Phase 5.2 tasks complete and documents shipped artifacts/tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[shaypal5]

Self-review pass (commit f2fc4a2)

Brutal senior-dev review of the first revision caught several real issues; folded the fixes back in before requesting human review.

Bugs fixed

• docs: add mandatory branch/PR workflow rules #1 — run_packager (both Kaggle + HF) was writing the README/metadata even when validation failed. Now early-returns with errors populated and zero artifacts on disk; new no-write tests on both sides.
• docs: add label taxonomy and milestone map to PR workflow #2 — Instructor README was inlining the public 3-tier README for a 1-tier dataset (the published HF card would have said "three-tier" 6 times for a single-config dataset). Replaced with a dedicated INSTRUCTOR_BODY constant that opens by linking to the public dataset and describes only the instructor-specific additions.
• feat: Milestone 0 — project foundation and package skeleton (v0.1.0) #3 — validate_upload_dir_safe was too narrow; --huggingface-dir release/intro would have rmtree'd the intro bundle. Now blocks strict descendants of release_dir with an allow-list for the canonical release/{kaggle,huggingface,huggingface-instructor} direct-children.

Architecture cleanups

• feat: Milestone 2 — narrative layer (NarrativeSpec, WorldSpec, dataset card) #5 — Finished the shared-primitives extraction. SOURCE_TREE_BLOCK, validate_readme_substitution, replace_file, replace_dir, and load_manifest all live in scripts/_release_common.py now. Both packagers reduced to imports.
• feat: carry primary_task and label_window_days into WorldSpec for dataset card rendering #6 — Replaced the 60-line hand-rolled YAML renderer (with its brittle quoting heuristic) with yaml.safe_dump + a 4-line _IndentedDumper subclass that forces indent-2 on top-level sequences.
• feat: Milestone 3 — schema layer (entities, IDs, FK validation, feature dictionary, task manifest) #7 — Deleted dead --owner / --dataset-slug CLI flags. PR 7.2 will add them when actually needed.
• feat: Milestone 4 — world structure layer (node types, graph, motifs, rewiring, sampler) #8 — assemble_upload_dir now takes rendered_readme and writes it itself; the public name no longer lies about producing a complete tree.
• Wire sample_hidden_graph RNG through RNGRoot for cross-component reproducibility #9 — build_config_for_tier is now pure (no I/O); a cheap _assert_tier_dir_exists does the manifest-stat preflight.
• feat: Milestone 5 — population generation and latent state initialisation #10 — --default-config with --variant=instructor now errors loudly instead of silently ignoring.

CI gap acknowledged + addressed

• feat: Milestone 1 — canonical config, recipe, and model objects #4 — Added [publish] extra in pyproject.toml (datasets>=2.14, kaggle>=1.6) so the gated G12.3/G12.4/G11.3 tests install in one line. Wiring a [publish] job into CI is a follow-up; this PR makes it trivial. Honest update to the PR description: G12.3/G12.4 are exercised only when datasets is installed; CI does not currently install it.

Smaller stuff

• feat: Milestone 8 — render/bundle layer and end-to-end Generator.generate() #13–feat: Milestone 10 — CLI generate/inspect/validate commands #16 — Visual cleanups (duplicate divider, ruff-split imports, COVER_IMAGE_FILENAME-vs-Path.name redundancy, dropped speculative comment about HF split rename).
• perf: use Parquet metadata for row counts in validate command #17 — Test cruft removed (unused tmp_path parameter, dead tag_lines computation); em-dash YAML round-trip parametrised for the instructor pretty_name (regression guard).
• Tighter messages in the YAML round-trip test; new audit-artifact-sync test for the instructor README; new --default-config-with-instructor rejection test; new descendant-of-release-dir rejection test; new no-write-on-validation-failure tests for BOTH packagers.

Verification

• 1223/1223 tests pass + 5 gated skips (4 datasets-SDK round-trip + 1 kaggle-SDK from PR 5.1).
• ruff check + ruff format --check + mypy clean.
• Hash determinism PASS 67/67.
• Leakage probes 0/3 reconstruct on every tier.
• validate_release_candidate --no-rebuild exits 0.
• BUNDLE_SCHEMA_VERSION unchanged at 5.

Files changed in this commit

• scripts/_release_common.py — absorbed the shared helpers
• scripts/package_kaggle_release.py — simplified imports, write-on-error fix
• scripts/package_hf_release.py — INSTRUCTOR_BODY, PyYAML rendering, dropped dead flags, validate→write gate
• tests/scripts/test_package_*.py — new tests for the fixes above
• pyproject.toml — [publish] extra
• release/huggingface{,-instructor}/README.md — regenerated (audit-artifact-sync)
• .agent-plan.md — captured the self-review fixes

🤖 Generated with Claude Code

[github-actions]

pr-agent-context report:

This run includes unresolved review comments on PR #72 in repository https://github.com/leadforge-dev/leadforge

For each unresolved review comment, recommend one of: resolve as irrelevant, accept and implement
the recommended solution, open a separate issue and resolve as out-of-scope for this PR, accept and
implement a different solution, or resolve as already treated by the code.

After I reply with my decision per item, implement the accepted actions, resolve the corresponding
PR comments, and push all of these changes in a single commit.

# Copilot Comments

## COPILOT-1
Location: scripts/package_hf_release.py:814
URL: https://github.com/leadforge-dev/leadforge/pull/72#discussion_r3196919377
Root author: copilot-pull-request-reviewer

Comment:
    `run_packager()` writes `README.md` to `huggingface_dir` before any unsafe-dir validation runs. If a user passes an unsafe `--huggingface-dir` (e.g. `.`) the script can overwrite/create `README.md` in that location even though `assemble_upload_dir()` later raises. Consider calling `validate_upload_dir_safe(huggingface_dir, release_dir, kind=...)` before *any* writes (and optionally also in `--dry-run` mode) so the safety net truly prevents clobbering.

Replies:
- shaypal5
  Addressed in commit 42260a8. `validate_upload_dir_safe` is now hoisted into `run_packager` (both
Kaggle + HF packagers), called immediately after the release-dir pre-flight, BEFORE any mkdir or
write — including in dry-run mode. The inner call inside `assemble_upload_dir` stays as
defence-in-depth for direct callers that bypass `run_packager`. New tests on both sides: a dry-run
with `--{huggingface,kaggle}-dir release` (i.e. release_dir itself) raises `ValueError` without
writing; the same path through `main()` returns rc=2 with "unsafe" in stderr.

## COPILOT-2
Location: scripts/package_hf_release.py
URL: https://github.com/leadforge-dev/leadforge/pull/72#discussion_r3196919426
Status: outdated
Root author: copilot-pull-request-reviewer

Comment:
    Cover image source resolution is inconsistent with validation: `run_packager()` validates `cover_image` exactly as passed, but `assemble_upload_dir()` may instead copy `release_dir / cover_image.name` if it exists (or skip copying if neither exists). This can produce false validation failures (e.g. passing just `dataset-cover-image.png`) and can also copy the wrong file if two paths share the same basename. Resolve the effective `cover_src` once (prefer the explicit path; optionally treat relative paths as relative to `release_dir`) and use that same path for both validation and copying.

## COPILOT-3
Location: scripts/package_hf_release.py
URL: https://github.com/leadforge-dev/leadforge/pull/72#discussion_r3196919453
Status: outdated
Root author: copilot-pull-request-reviewer

Comment:
    `assemble_upload_dir()` docstring says it assembles the rewritten README, but the function currently only copies the cover image, LICENSE, and bundle directories; `README.md` is written earlier by `run_packager()`. Either have `assemble_upload_dir()` also write/copy the README (so it’s self-contained as a helper) or update the docstring to reflect the actual responsibility split.

Run metadata:

Tool ref: v4
Tool version: 4.0.21
Trigger: commit pushed
Workflow run: 25461015698 attempt 1
Comment timestamp: 2026-05-06T21:04:02.523880+00:00
PR head commit: 42260a80e9dbea4d12ae725e7912741a89355c6a