Skip to content

feat: HST/ACS reduction pipeline (phase 1) - #3

Merged
Jammy2211 merged 5 commits into
mainfrom
feature/hst-acs-phase1
Jul 8, 2026
Merged

feat: HST/ACS reduction pipeline (phase 1)#3
Jammy2211 merged 5 commits into
mainfrom
feature/hst-acs-phase1

Conversation

@Jammy2211

Copy link
Copy Markdown
Contributor

Implements the HST/ACS reduction pipeline designed under #1 and planned on #2 (autonomous run, --auto supervised; parked at ship sign-off per the autonomy contract).

What this adds

Production autoreduce package (spike → modules), all behind the ACS/WFC instrument adapter:

  • target.py — declarative TargetSpec (YAML-loadable): coords, filter, proposal filter, cutout/PSF shapes, and the user-facing drizzle dials (final_pixfrac default 0.8, final_kernel square, final_scale 0.05″).
  • instruments/ — adapter registry; acs_wfc carries native scale, _flc products, jref + CRDS subpath, default drizzle kwargs, saturation.
  • acquire/ — MAST query with HAP-skycell hygiene + proposal/visit filtering; FLC download; CRDS bestrefs sync (skipped when cached + references present, so offline re-runs work); size-capped transient ExposureCache with provenance manifest (loud on spike-era schemas), per-target eviction sparing shared references.
  • align/ — a-priori WCS provenance recording; TweakReg behind a trigger.
  • drizzle/ — AstroDrizzle wrapper: cps/IVM/north-up, single-exposure branch (SLACS-V caveat), WHT RMS/median uniformity diagnostic, Casertano R recorded; lowercase-relative output root (drizzlepac lowercases paths — case-sensitive FS fix).
  • noise/ — σ = R·√(max(sci,0)/t_exp + 1/W); NaN on bad weights, hard failure if any land inside the packaged cutout.
  • psf/ — tier-1 ePSF: star cuts (point-like, unsaturated, uncrowded, edge-safe) + EPSFBuilder; 21×21 + 61×61 kernels; InsufficientStarsError hard stop with tier-2 as an explicit unimplemented interface (no silent degradation).
  • package/ — WCS/units/EXPTIME-intact cutout headers + reduction.json (full per-stage provenance incl. software versions).
  • pipeline.py — the orchestrator: acquire → align → drizzle → noise → psf → package → evict.

API Changes

New public API (nothing downstream consumes it yet): autoreduce.TargetSpec, autoreduce.reduce_target, autoreduce.instruments.{get,register,InstrumentAdapter}, plus stage modules. No changes to any other PyAuto* repo; autoreduce imports none of them.

Scripts Changed

  • New scripts/reduce_slacs0008.py — integration/acceptance: production reduction of slacs0008-0004 + sub-pixel-registered parity vs the legacy dataset.
  • prototypes/slacs_f814w_spike.py — Casertano fine-grid branch corrected ((p/s) → (s/p); spike results unaffected, it ran s=1).

Gate legs (supervised — parked at sign-off)

  1. Tests: 53/53 pass (numpy/astropy-only; no network/drizzlepac in unit suite). Downstream suites n/a — no repo depends on autoreduce.
  2. Smoke: n/a — no workspace script surface consumes autoreduce yet (stated per contract).
  3. Review: review ran at high effort (8 angles, 3 finder passes + verification). FINDINGS → resolved in-branch: CRDS env now deterministically overrides inherited vars; CRDS subpath moved into the adapter (altitude); ePSF minimum-star contract re-checked post-extraction and edge margin raised to 36; manifest schema mismatches fail loudly; cached re-runs skip the network bestrefs sync; HDUList context-managed; provenance units label sourced from drizzle kwargs; hand-rolled sigma-clip replaced with astropy's. Deliberate non-fixes: tier-2 hard stop and mode="strict" cutouts are the no-silent-guards rule, not bugs. Deferred (noted, low stakes): O(N²) crowding rejection (fine at current N), crude FWHM diagnostic.
  4. Heart: YELLOW — reasons: workspace validation failing (5), 58 stale parked scripts, autolens_assistant version skew, install verification not run, 63 script timing regressions, 9-day-old Mind PR. All pre-existing organism state, none related to this repo; maintainer acknowledged this exact reason set at sign-off (issue feat: HST/ACS reduction pipeline (phase 1) #2), authorizing PR-open.

Validation checklist

  • 53 unit tests green on final state
  • End-to-end integration on slacs0008-0004 (cached re-run after review fixes)
  • Parity vs legacy dataset: data ratio 0.941, noise ratio 0.925 ≈ data ratio (noise recipe confirmed; R=1.364 applied), WHT uniformity 0.089, tier-1 ePSF from 31 stars
  • Residual ~6% global flux scale vs legacy — accepted as documented difference (maintainer decision on feat: HST/ACS reduction pipeline (phase 1) #2); PyAutoMind research prompt filed to chase it if needed
  • Comparative PyAutoLens fits + 2nd/3rd lens parity — deferred to a follow-up prompt (maintainer decision on feat: HST/ACS reduction pipeline (phase 1) #2)

🤖 Generated with Claude Code

https://claude.ai/code/session_01WNhuuvZLgZjHvWBGdhUst3

Closes #2 discussion at ship; merge remains a human act.

Jammy2211 and others added 5 commits July 8, 2026 19:38
TargetSpec, ACS/WFC instrument adapter + registry, MAST acquisition
with HAP-skycell query hygiene, CRDS sync, size-capped transient cache
with provenance manifest, AstroDrizzle wrapper (pixfrac/kernel dials,
single-exposure branch, WHT-uniformity diagnostic), noise map with
Casertano R applied, tier-1 ePSF (star cuts + EPSFBuilder) with loud
tier-2 fallback stop, WCS-intact cutouts + reduction.json, pipeline
orchestrator. 52 numpy/astropy-only unit tests.

Tests caught two real bugs: casertano_r fine-grid branch had p/s
inverted (spike unaffected: it ran s=1 on the other branch), and the
cache size cap counted its own manifest.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNhuuvZLgZjHvWBGdhUst3
…008-0004

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNhuuvZLgZjHvWBGdhUst3
…ost-extraction star check, loud manifest schema, offline cached re-runs, HDU close, units from drizzle kwargs, astropy sigma-clip

Review verdict FINDINGS -> resolved; deliberate non-fixes (tier-2 hard
stop, strict cutout) are the no-silent-guards rule. 53 tests green;
end-to-end re-validated on slacs0008-0004 with identical parity.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNhuuvZLgZjHvWBGdhUst3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: HST/ACS reduction pipeline (phase 1)

1 participant