feat(clean_slate): remove auto-simulated datasets by simulator provenance

[Jammy2211]

What

clean_slate.sh restored modified datasets but never reclaimed the simulated ones, so every workspace accumulated regenerable dataset/ payload indefinitely. This adds a step 1b that removes them.

Why not just "delete everything untracked under dataset/"

Because untracked does not mean regenerable. Every workspace ignores dataset/** wholesale and force-adds the real datasets back via !dataset/<name>/**, so some real data is deliberately never committed:

Untracked but real	Why
autolens_workspace/dataset/cluster/smacs0723/ (52 MB)	Downloaded by scripts/cluster/lenstool/data.py. The Mahler et al. Lenstool files are GPL-licensed and "downloaded at runtime with attribution rather than redistributed here" — committing them would be a redistribution problem.
autolens_workspace/dataset/cluster/a2744/data.fits	hips2fits HST cutout, explicitly re-ignored one line after the a2744 allowlist entry
autolens_workspace/dataset/cluster/csv_api_example/	Hand-authored CSV inputs
dataset/multi/rxj1131, dataset/weak/a2744_pyrrg	Real lensed-quasar / weak-lensing data

I also tested two intrinsic markers hoping for a cheap discriminator — neither works. README.md is absent from real and simulated dirs alike, and tracer.json is present in real-input dirs (autolens_profiling/dataset/imaging/hst, .../interferometer/alma) while missing from genuinely simulated ones (simulated_lens, deblending).

The rule

A dataset is removed only when a simulator script in the same repo demonstrably writes it: both its dataset type and its name appear as string literals in one scripts/**/simulator*.py or scripts/**/simulators/*.py. Anything without that provenance is kept, so the rule errs toward keeping — start_here.py-generated datasets survive uncleaned, which is the safe direction.

Requiring both components was free: name-only and name+type matching gave identical results across all 7 repos, so the stricter form costs no coverage.

Two guards worth calling out:

• Only directories are candidates. An earlier revision descended to individual files and matched dataset/cluster/a2744/data.fits — its generic name matches almost any simulator. Loose untracked files inside a committed dataset are never candidates.
• No parent test. A first attempt also skipped candidates whose parent held tracked files; that protected everything, because a type directory like dataset/imaging/ legitimately holds committed datasets (cosmos_web_ring) alongside simulated ones. It cut the delete list 89 → 62 and is deliberately not present.

Scope

4 workspaces + 3 HowTo repos. autolens_profiling and autolens_jax_joss are excluded outright — their dataset/ dirs hold real instrument data (alma/sma/hst inputs, JWST cosmos_web_ring).

HowToLens has no simulator scripts, so nothing there is cleaned.

Size tripwire

Warns when a committed dataset file exceeds 5 MB, guarding against large real data being added to a workspace. Currently fires on autolens_workspace/dataset/cluster/simple/{data,noise_map}.fits (7 MB each, pre-existing).

Verification

• 88 paths removed across 7 repos, ~13 MB
• zero tracked files in the delete list
• all 9 known-real datasets byte-size identical afterwards
• all 7 repos report 0 dirty files after the run, confirming only ignored paths were touched
• second run is a no-op (idempotent)
• DRY_RUN=1 previews unchanged; --packaging unchanged

🤖 Generated with Claude Code

https://claude.ai/code/session_012EZTtyLUAyKWuATyk4mkcm

[Jammy2211]

Correction: HowToLens does have simulator scripts

The original description claimed HowToLens has none, so nothing there was cleaned. That was a bug in my glob, not a property of the repo.

The pattern matched files named simulator*.py or anything under a directory named simulators/ (plural). HowToLens keeps its five simulators in scripts/simulator/ (singular):

scripts/simulator/lens_sersic.py
scripts/simulator/lens_x2.py
scripts/simulator/no_lens_light.py
scripts/simulator/no_lens_light__mass_sis.py
scripts/simulator/source_complex.py

That matches neither form, so the repo resolved to zero simulator scripts and the sweep silently no-opped — no error, just an empty result.

Fixed in 10d0708 by listing both directory spellings. I surveyed all seven in-scope repos; HowToLens is the only one affected (old pattern 0 scripts, new 6 — every other repo unchanged at 43/25/4/25/4/4).

HowToLens now removes 5 of its 6 dataset dirs:

Removed	Kept
source_complex, simple__no_lens_light__mass_sis, lens_sersic, simple__no_lens_light, x2_lens_galaxies	howtolens — written by the tutorial chapters, not a simulator

Re-verified after the fix: zero tracked files in the delete list, all 9 known-real datasets intact, HowToLens 0 dirty after the run.

This is exactly the failure mode the rule is meant to have — a missing spelling disables an entire repo silently — so the glob now names both forms and carries a comment explaining why.

[Jammy2211]

Added: empty-simulator-set warning (74e8749)

Closing the silent-failure hole that let the HowToLens miss go unnoticed.

The problem was that "no simulator scripts found" and "this repo has nothing to clean" produced identical output — an empty result and a silent no-op. The sweep reported nothing and looked correct.

An in-scope repo with zero simulator scripts is almost certainly an unrecognised layout, so it now says so:

=== euclid_strong_lens_modeling_pipeline ===
  WARNING: no simulator scripts found — dataset sweep disabled for this repo

Also extracted the glob into simulator_scripts() so the sweep and the check share one definition and cannot drift apart — a fix to one that missed the other would reintroduce exactly this class of bug.

Verified both directions:

• fires — added a simulator-less repo to DATASET_REPOS in a scratch copy; warning appears as above
• silent — no warning for any of the seven real in-scope repos

Safety checks still pass unchanged: zero tracked files in the delete list, all 9 known-real datasets protected, excluded repos absent.

Branch state

Commit
8486331	feat: remove auto-simulated datasets by simulator provenance
10d0708	fix: match scripts/simulator/ (singular) as well as simulators/
74e8749	feat: warn when an in-scope repo yields no simulator scripts