Skip to content

feat(clean_slate): write-site dataset provenance, orphan reporting, gc phase, tests #170

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Jammy2211 merged 1 commit into from
Jul 27, 2026
Merged

feat(clean_slate): write-site dataset provenance, orphan reporting, gc phase, tests #170

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
193 changes: 127 additions & 66 deletions bin/clean_slate.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,15 @@
# 1. restores tracked files under any dataset/ dir that a run modified in place
# — the shipped datasets (cosmos_web_ring, simple, …) go back to their
# committed state; they are never deleted.
# 1b. removes AUTO-SIMULATED datasets from the workspace/tutorial repos only
# (see DATASET_REPOS), and warns about oversized committed datasets.
# 1b. removes REGENERABLE datasets from the workspace/tutorial repos only
# (see DATASET_REPOS), reports orphans, prunes emptied dataset dirs, and
# warns about oversized committed datasets.
# 2. clears every output/ and scratch/ directory (model fits, scratch space).
# 3. removes generated test_report.md files.
# 3. removes generated test_report.md files, then untracked .ipynb_checkpoints/
# directories repo-wide.
# 4. removes ignored, fully-untracked top-level *.egg-info/ and build/
# packaging directories from the managed library repos.
# 5. runs `git gc --auto` per repo to compact the object store.
#
# It is git-aware and conservative:
# - it never deletes a tracked file (except reverting in-place dataset edits);
Expand All @@ -31,18 +34,32 @@
# problem. No intrinsic marker separates them from simulated data either — a
# README or a tracer.json sits in both kinds.
#
# So a dataset is removed only when a simulator script in the SAME repo
# demonstrably writes it: both its dataset type and its name appear as string
# literals in one scripts/**/simulator*.py, scripts/**/simulator/*.py or
# scripts/**/simulators/*.py. Anything with no such provenance is kept. The rule
# therefore errs toward keeping — datasets written by start_here.py-style scripts
# survive, which is the safe direction.
# __The write-site rule__
#
# A dataset is removed only when a script in the SAME repo demonstrably WRITES
# it — a name mention is never enough. bin/dataset_provenance.py parses every
# scripts/**/*.py with `ast` and, in source order, tracks which dataset path each
# variable currently holds and whether that variable reaches an output call
# (output_to_fits, fits_imaging, json.dump, open(..., "w"), a helper whose body
# writes the parameter, …). That order sensitivity is the point:
# scripts/interferometer/start_here.py binds `dataset_path` to the real sdp81
# data (which it only reads), then rebinds it to simulated_lens (which it
# writes) — a grep for the dataset name cannot tell those apart, and the old
# name-literal rule missed every dataset written by a start_here.py (#167).
#
# Three verdicts come back per candidate:
# REGENERABLE positive write evidence, no network — deleted.
# DOWNLOADED its binding script also fetches over the network — kept
# silently; this is real data cached rather than redistributed.
# ORPHAN no writer found — kept and REPORTED, so a human can look.
# Deletion requires positive evidence; every uncertainty lands on ORPHAN.
#
# Workspace root: PYAUTO_ROOT (default ~/Code/PyAutoLabs).
# Preview without changing anything: DRY_RUN=1 clean_slate.sh
# Packaging products only: clean_slate.sh --packaging

set -u
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
ROOT="${PYAUTO_ROOT:-$HOME/Code/PyAutoLabs}"
cd "$ROOT" || { echo "workspace root not found: $ROOT" >&2; exit 1; }
DRY_RUN="${DRY_RUN:-0}"
Expand Down Expand Up @@ -78,23 +95,13 @@ is_dataset_repo() {
return 1
}

# Print each untracked path under dataset/ that a simulator script in this repo
# writes. Expands git-clean's collapsed entries down to dataset/<type>/<name>
# granularity so a wholly-untracked dataset/ tree is still judged per dataset.
# Both directory spellings are in use: HowToLens uses scripts/simulator/
# (singular), everything else scripts/**/simulators/ (plural) or a simulator*.py
# filename. A spelling this misses yields zero scripts, which is indistinguishable
# from "nothing to clean" — hence the empty-set warning at the call site.
simulator_scripts() {
find "$1/scripts" -type f -name '*.py' \
\( -name 'simulator*' -o -path '*/simulator/*' -o -path '*/simulators/*' \) 2>/dev/null
}

simulated_datasets() {
local repo="$1" path depth child type name matched
local -a sims queue found
mapfile -t sims < <(simulator_scripts "$repo")
[ "${#sims[@]}" -eq 0 ] && return 0
# Print every untracked dataset/<type>/<name> DIRECTORY worth judging. Expands
# git-clean's collapsed entries down to dataset/<type>/<name> granularity so a
# wholly-untracked dataset/ tree is still judged per dataset. Provenance itself
# is decided by dataset_provenance.py — this only produces the candidate list.
dataset_candidates() {
local repo="$1" path depth child
local -a queue found

queue=()
while IFS= read -r path; do
Expand Down Expand Up @@ -122,13 +129,7 @@ simulated_datasets() {
# dataset/imaging/ legitimately holds committed datasets alongside
# simulated ones, so a parent test would protect everything.)
[ -z "$(git -C "$repo" ls-files -- "$path" 2>/dev/null)" ] || continue
type=$(awk -F/ '{print $2}' <<<"$path")
name="${path##*/}"
matched=""
while IFS= read -r hit; do
grep -qF "\"$type\"" "$hit" 2>/dev/null && { matched=1; break; }
done < <(grep -lF "\"$name\"" "${sims[@]}" 2>/dev/null)
[ -n "$matched" ] && found+=("$path")
found+=("$path")
done
[ "${#found[@]}" -gt 0 ] && printf '%s\n' "${found[@]}"
return 0
Expand All @@ -151,30 +152,70 @@ for dir in */; do
[ "$DRY_RUN" = 1 ] || printf '%s\0' "${moddata[@]}" | xargs -0 -r git -C "$repo" checkout --
fi

# 1b. Remove auto-simulated datasets, and flag oversized committed ones.
# 1b. Classify untracked datasets by write site, remove the regenerable
# ones, report orphans, and flag oversized committed ones.
if is_dataset_repo "$repo"; then
# An in-scope repo with no simulator scripts is almost certainly an
# unrecognised layout, not a repo with nothing to simulate — the two
# are otherwise indistinguishable (this is how scripts/simulator/
# went unnoticed in HowToLens). Say so rather than no-op in silence.
if [ "$(simulator_scripts "$repo" | wc -l)" -eq 0 ]; then
warn "WARNING: no simulator scripts found — dataset sweep disabled for this repo"
mapfile -t candidates < <(dataset_candidates "$repo")
if [ "${#candidates[@]}" -gt 0 ]; then
errfile=$(mktemp)
verdicts=$(python3 "$SCRIPT_DIR/dataset_provenance.py" \
--repo "$repo" "${candidates[@]}" 2>"$errfile")
status=$?
# No fallback: guessing provenance is exactly what this replaced.
if [ "$status" -ne 0 ]; then
cat "$errfile" >&2
rm -f "$errfile"
echo "clean_slate: dataset_provenance.py failed for $repo (exit $status)" >&2
exit 1
fi
[ -s "$errfile" ] && cat "$errfile" >&2
rm -f "$errfile"

nsim=0; simkb=0
while read -r verdict rel; do
[ -n "$rel" ] || continue
case "$verdict" in
REGENERABLE)
nsim=$((nsim + 1))
kb=$(du -sk "$repo/$rel" 2>/dev/null | cut -f1)
simkb=$((simkb + ${kb:-0}))
[ "$DRY_RUN" = 1 ] || rm -rf "${repo:?}/${rel:?}"
;;
ORPHAN)
# Never deleted — surfaced so a human can decide
# whether it is real data or forgotten cruft.
warn "orphan dataset (no writer): $rel ($(du -sh "$repo/$rel" 2>/dev/null | cut -f1))"
;;
DOWNLOADED) ;; # real data, cached not redistributed — keep, silently
esac
done <<<"$verdicts"
[ "$nsim" -gt 0 ] && show "remove $nsim simulated dataset(s) ($((simkb / 1024)) MB)"
fi
nsim=0; simkb=0
while IFS= read -r rel; do
[ -n "$rel" ] || continue
nsim=$((nsim + 1))
kb=$(du -sk "$repo/$rel" 2>/dev/null | cut -f1)
simkb=$((simkb + ${kb:-0}))
[ "$DRY_RUN" = 1 ] || rm -rf "${repo:?}/${rel:?}"
done < <(simulated_datasets "$repo")
[ "$nsim" -gt 0 ] && show "remove $nsim simulated dataset(s) ($((simkb / 1024)) MB)"

while IFS= read -r -d '' f; do
kb=$(du -sk "$repo/$f" 2>/dev/null | cut -f1)
[ "${kb:-0}" -gt "$DATASET_WARN_KB" ] || continue
warn "WARNING: committed dataset $f is $((kb / 1024)) MB (>$((DATASET_WARN_KB / 1024)) MB)"
done < <(git -C "$repo" ls-files -z -- 'dataset/*' 2>/dev/null)

# Prune directories the sweep just emptied (and any that were
# already empty). An empty directory is never tracked by git, so
# this can never touch committed content.
nempty=$(find "$repo/dataset" -mindepth 1 -type d -empty 2>/dev/null | wc -l)
if [ "$nempty" -gt 0 ]; then
show "remove $nempty empty dataset director$([ "$nempty" -eq 1 ] && echo y || echo ies)"
[ "$DRY_RUN" = 1 ] || find "$repo/dataset" -mindepth 1 -type d -empty -delete 2>/dev/null
fi

# Bloat warning aggregated per dataset DIRECTORY, not per file: a
# dataset is many .fits files and one line per file buries the
# signal. Tracked bytes only — untracked cruft is not repo bloat.
while IFS=$'\t' read -r dir kb; do
warn "WARNING: committed dataset $dir is $((kb / 1024)) MB (>$((DATASET_WARN_KB / 1024)) MB)"
done < <(git -C "$repo" ls-files -z -- 'dataset/*' 2>/dev/null \
| (cd "$repo" && xargs -0 -r du -k --apparent-size --) 2>/dev/null \
| awk -F'\t' -v warn="$DATASET_WARN_KB" '
NF == 2 {
n = split($2, part, "/")
dir = (n >= 3) ? part[1] "/" part[2] "/" part[3] : $2
total[dir] += $1
}
END { for (d in total) if (total[d] > warn) printf "%s\t%d\n", d, total[d] }' \
| sort)
fi

# 2. Clear output/ and scratch/ dirs (untracked + ignored inside them; tracked kept).
Expand All @@ -193,23 +234,43 @@ for dir in */; do
show "remove $rel"
[ "$DRY_RUN" = 1 ] || rm -f "$f"
done < <(find "$repo" -maxdepth 2 -type f -name test_report.md -not -path '*/.git/*' -print0 2>/dev/null)

# 3b. Remove untracked .ipynb_checkpoints/ — Jupyter autosave copies of
# notebooks, which go stale the moment the real notebook is regenerated.
# __pycache__ is DELIBERATELY left alone: it is an import-speed cache
# that costs seconds of every subsequent run to rebuild.
while IFS= read -r -d '' d; do
rel="${d#"$repo"/}"
n=$(git -C "$repo" clean -ndx -- "$rel" 2>/dev/null | wc -l)
[ "$n" -eq 0 ] && continue
show "remove $rel/"
[ "$DRY_RUN" = 1 ] || git -C "$repo" clean -qfdx -- "$rel"
done < <(find "$repo" -type d -name .ipynb_checkpoints -not -path '*/.git/*' -print0 2>/dev/null)
fi

# 4. Remove ignored, fully-untracked packaging products at managed library
# roots. Keep assistant/workspace build products outside this narrow scope.
# Never match nested domain directories named build, and never clean a
# candidate containing tracked files even though git clean would retain them.
is_packaging_repo "$repo" || continue
while IFS= read -r -d '' d; do
rel="${d#"$repo"/}"
git -C "$repo" check-ignore -q -- "$rel" 2>/dev/null || continue
[ -z "$(git -C "$repo" ls-files -- "$rel" 2>/dev/null)" ] || continue
n=$(git -C "$repo" clean -ndx -- "$rel" 2>/dev/null | wc -l)
[ "$n" -eq 0 ] && continue
show "remove packaging directory $rel/"
[ "$DRY_RUN" = 1 ] || git -C "$repo" clean -qfdx -- "$rel"
done < <(find "$repo" -mindepth 1 -maxdepth 1 -type d \
\( -name '*.egg-info' -o -name build \) -print0 2>/dev/null)
if is_packaging_repo "$repo"; then
while IFS= read -r -d '' d; do
rel="${d#"$repo"/}"
git -C "$repo" check-ignore -q -- "$rel" 2>/dev/null || continue
[ -z "$(git -C "$repo" ls-files -- "$rel" 2>/dev/null)" ] || continue
n=$(git -C "$repo" clean -ndx -- "$rel" 2>/dev/null | wc -l)
[ "$n" -eq 0 ] && continue
show "remove packaging directory $rel/"
[ "$DRY_RUN" = 1 ] || git -C "$repo" clean -qfdx -- "$rel"
done < <(find "$repo" -mindepth 1 -maxdepth 1 -type d \
\( -name '*.egg-info' -o -name build \) -print0 2>/dev/null)
fi

# 5. Compact the object store. `--auto` makes this a no-op until git's own
# loose-object threshold is crossed, so it is cheap to run every morning.
# Housekeeping must never break the sweep: warn and carry on.
if [ "$DRY_RUN" != 1 ]; then
git -C "$repo" gc --auto --quiet 2>/dev/null || warn "WARNING: git gc failed"
fi
done

echo
Expand Down
Loading