feat: HST/ACS reduction pipeline (phase 1)

[Jammy2211]

Overview

Implement the HST/ACS reduction pipeline designed and spike-validated under #1, turning prototypes/slacs_f814w_spike.py into production autoreduce code behind the ACS/WFC instrument adapter. docs/design/hst_acs_pipeline.md is the authoritative spec. Autonomous run (--auto, effective level supervised): plan recorded here, mechanical stretches proceed, judgment gates arrive as batched questions, the run parks at ship sign-off — merge stays human.

Routing note: the Brain Feature Agent's heuristic suggested re-homing as research; overridden in-session because the research phase (design docs + parity spike) completed under #1 with user approval — this task is implementation.

Plan

• Target spec + pipeline orchestrator: a per-target declaration (name, coords, filter, cutout size, dials) driving acquire → align → drizzle → noise → psf → package.
• Acquire: filtered MAST query (direct CAL-2 observations, HAP skycells excluded), _flc download, CRDS best-references sync, size-capped transient cache with manifest + eviction.
• Drizzle: AstroDrizzle wrapper with user-facing pixfrac/kernel dials, WHT-uniformity diagnostic + correlated-noise factor R reported per run; single-exposure branch (single-image drizzle, CR handling via DQ + L.A. Cosmic-style masking decision recorded).
• Noise: σ = √(N/W + σ²_sky) with R applied; loud failure on NaN/zero weights inside the cutout.
• PSF: tier-1 empirical ePSF (star selection + photutils EPSFBuilder) with quality diagnostics; tier-2 fallback interface (STScI focus-diverse ePSF grid / TinyTim) — implemented if mechanical, else parked as a batched question.
• Package: WCS-intact cutouts + reduction.json provenance.
• Tests: numpy/astropy-only unit tests (noise math, R factor, cache manifest/eviction, cutout geometry, adapter registry, star filters on synthetic data); network/drizzlepac integration lives in scripts/.
• Acceptance: re-reduce slacs0008-0004 through the production pipeline; parity vs legacy dataset; chase the spike's ~7% flux offset with visit-filtered exposure set + sub-pixel registration. Comparative PyAutoLens fits are expected to park as a sign-off question (long runs).

Detailed implementation plan

Affected Repositories

• PyAutoReduce (primary, only)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoReduce	main	clean

Suggested branch: feature/hst-acs-phase1 (in-place — single-repo task, new repo, no parallel claims; precedent: clone-mitosis-agent)

Implementation Steps

1. autoreduce/target.py — TargetSpec (name, ra, dec, instrument, filter, cutout_shape, pixfrac, kernel, final_scale, proposal filter) loadable from YAML; autoreduce/pipeline.py — stage orchestrator with per-stage provenance accumulation.
2. autoreduce/instruments/acs_wfc.py — adapter: native scale, product suffix (_flc), jref env key, default drizzle params, zero-point/units helpers; instruments/__init__.py registry; nothing outside the adapter names a detector.
3. autoreduce/acquire/mast.py — query (direct calib-level-2 obs only, proposal/visit grouping), FLC product filter, download; acquire/crds.py — bestrefs sync into cache; acquire/cache.py — manifest, size cap, per-target eviction sparing references.
4. autoreduce/align/diagnostics.py — a-priori WCS acceptance check; TweakReg call behind a trigger flag.
5. autoreduce/drizzle/combine.py — AstroDrizzle wrapper (dials, cps, IVM, north-up 0.05″), single-exposure branch, WHT RMS/median diagnostic, R computation (Casertano) recorded.
6. autoreduce/noise/rms.py — sigma map from SCI+WHT+EXPTIME with R applied; NaN/zero-weight loud failure; blank-sky empirical check helper.
7. autoreduce/psf/stars.py + psf/epsf.py — star selection (unsaturated, uncrowded, DQ-clean, point-like) + EPSFBuilder wrapper emitting 21×21 + 61×61 kernels, odd/centred/normalised, diagnostics to provenance; psf/fallback.py — tier-2 interface.
8. autoreduce/package/cutout.py + package/provenance.py — WCS-correct cutouts, reduction.json.
9. test_autoreduce/ — unit tests per module (numpy/astropy only; drizzlepac/astroquery imports guarded so the suite runs without them).
10. scripts/reduce_slacs0008.py — integration: production pipeline end-to-end on slacs0008-0004 + parity report vs legacy dataset.

Key Files

• docs/design/hst_acs_pipeline.md — the spec, incl. deviation table + parity appendix
• prototypes/slacs_f814w_spike.py — validated reference implementation

Original Prompt

Click to expand starting prompt

See PyAutoMind/issued/hst_acs_phase1.md (filed from feature/pyautoreduce/hst_acs_phase1.md) — scope: production implementation of the design-doc pipeline, user-facing pixfrac/kernel dials, SLACS-V single-exposure branch, parity acceptance on 2–3 SLACS lenses including PyAutoLens fits agreeing on both reductions, numpy-only unit tests, ~/venv/PyAuto extension under constraints.

[Jammy2211]

Parked at ship sign-off (supervised --auto — checkpoint-and-continue).

Implementation is complete and validated on feature/hst-acs-phase1 (5 local commits, unpushed — supervised parks before push/PR). What was done, cold-readable:

• All planned modules implemented; 53/53 unit tests green; end-to-end integration on slacs0008-0004 ran twice (fresh + cached/offline) with identical, deterministic parity: data ratio 0.941, noise ratio 0.925 ≈ data ratio (noise recipe confirmed, R=1.364 applied), WHT uniformity 0.089, tier-1 ePSF from 31 stars.
• Review (high effort, multi-angle + verification) returned FINDINGS → all confirmed items fixed in-branch and re-validated (CRDS env determinism, adapter-owned CRDS subpath, ePSF post-extraction star check, loud manifest schema, offline cached re-runs, HDU close, provenance units, astropy sigma-clip). Tier-2 hard stop and strict cutouts kept deliberately (no-silent-guards).
• Heart: YELLOW, reasons all pre-existing organism state unrelated to this standalone repo (workspace validation 5 fails, 58 stale parked scripts, assistant version skew, install verification, timing regressions, 9d-old Mind PR). No launch acknowledgement was given, so per contract this parks rather than ships.

The draft PR body (posted verbatim on push approval):

---

Implements the HST/ACS reduction pipeline designed under #1 and planned on #2 (autonomous run, --auto supervised; parked at ship sign-off per the autonomy contract).

What this adds

Production autoreduce package (spike → modules), all behind the ACS/WFC instrument adapter:

• target.py — declarative TargetSpec (YAML-loadable): coords, filter, proposal filter, cutout/PSF shapes, and the user-facing drizzle dials (final_pixfrac default 0.8, final_kernel square, final_scale 0.05″).
• instruments/ — adapter registry; acs_wfc carries native scale, _flc products, jref + CRDS subpath, default drizzle kwargs, saturation.
• acquire/ — MAST query with HAP-skycell hygiene + proposal/visit filtering; FLC download; CRDS bestrefs sync (skipped when cached + references present, so offline re-runs work); size-capped transient ExposureCache with provenance manifest (loud on spike-era schemas), per-target eviction sparing shared references.
• align/ — a-priori WCS provenance recording; TweakReg behind a trigger.
• drizzle/ — AstroDrizzle wrapper: cps/IVM/north-up, single-exposure branch (SLACS-V caveat), WHT RMS/median uniformity diagnostic, Casertano R recorded; lowercase-relative output root (drizzlepac lowercases paths — case-sensitive FS fix).
• noise/ — σ = R·√(max(sci,0)/t_exp + 1/W); NaN on bad weights, hard failure if any land inside the packaged cutout.
• psf/ — tier-1 ePSF: star cuts (point-like, unsaturated, uncrowded, edge-safe) + EPSFBuilder; 21×21 + 61×61 kernels; InsufficientStarsError hard stop with tier-2 as an explicit unimplemented interface (no silent degradation).
• package/ — WCS/units/EXPTIME-intact cutout headers + reduction.json (full per-stage provenance incl. software versions).
• pipeline.py — the orchestrator: acquire → align → drizzle → noise → psf → package → evict.

API Changes

New public API (nothing downstream consumes it yet): autoreduce.TargetSpec, autoreduce.reduce_target, autoreduce.instruments.{get,register,InstrumentAdapter}, plus stage modules. No changes to any other PyAuto* repo; autoreduce imports none of them.

Scripts Changed

• New scripts/reduce_slacs0008.py — integration/acceptance: production reduction of slacs0008-0004 + sub-pixel-registered parity vs the legacy dataset.
• prototypes/slacs_f814w_spike.py — Casertano fine-grid branch corrected ((p/s) → (s/p); spike results unaffected, it ran s=1).

Gate legs (supervised — parked at sign-off)

1. Tests: 53/53 pass (numpy/astropy-only; no network/drizzlepac in unit suite). Downstream suites n/a — no repo depends on autoreduce.
2. Smoke: n/a — no workspace script surface consumes autoreduce yet (stated per contract).
3. Review: review ran at high effort (8 angles, 3 finder passes + verification). FINDINGS → resolved in-branch: CRDS env now deterministically overrides inherited vars; CRDS subpath moved into the adapter (altitude); ePSF minimum-star contract re-checked post-extraction and edge margin raised to 36; manifest schema mismatches fail loudly; cached re-runs skip the network bestrefs sync; HDUList context-managed; provenance units label sourced from drizzle kwargs; hand-rolled sigma-clip replaced with astropy's. Deliberate non-fixes: tier-2 hard stop and mode="strict" cutouts are the no-silent-guards rule, not bugs. Deferred (noted, low stakes): O(N²) crowding rejection (fine at current N), crude FWHM diagnostic.
4. Heart: YELLOW — reasons: workspace validation failing (5), 58 stale parked scripts, autolens_assistant version skew, install verification not run, 63 script timing regressions, 9-day-old Mind PR. All pre-existing organism state, none related to this repo; no launch-time acknowledgement was given, so the run parks here per contract.

Validation checklist

• 53 unit tests green on final state
• End-to-end integration on slacs0008-0004 (cached re-run after review fixes)
• Parity vs legacy dataset: data ratio 0.941, noise ratio 0.925 ≈ data ratio (noise recipe confirmed; R=1.364 applied), WHT uniformity 0.089, tier-1 ePSF from 31 stars
• Residual ~6% global flux scale vs legacy — understood as calibration-era/kernel question, tracked as acceptance item
• Comparative PyAutoLens fits on both reductions (long runs — deferred, see sign-off question)
• Second/third SLACS lens parity (deferred with the fits)

🤖 Generated with Claude Code

https://claude.ai/code/session_01WNhuuvZLgZjHvWBGdhUst3

---

Questions (batched — answer any subset):

1. Ship? Push feature/hst-acs-phase1 and open this PR against main? Requires your Heart-YELLOW acknowledgement of the exact reason list above. Recommendation: yes — the reasons are organism-wide staleness, none touch PyAutoReduce.
2. The ~6% flux scale vs legacy — investigate now (compare against a legacy-parameter reduction: gaussian kernel, and check FLT-vs-FLC photometric era) or accept as a documented difference given the legacy reduction's provenance is unrecoverable? Recommendation: one bounded experiment (gaussian-kernel re-drizzle) before deciding.
3. Comparative PyAutoLens fits (acceptance leg): run on both reductions of slacs0008-0004 now (hours), or defer to a follow-up prompt alongside the 2nd/3rd lens parity? Recommendation: defer — the pixel-level parity is strong and the fits deserve their own tracked run.

[Jammy2211]

Shipped at PR-open per sign-off answers: PR #3 is up (branch pushed, body carries the plan, gate legs, and your Heart-YELLOW acknowledgement of the exact reason set). The ~6% legacy flux scale is accepted as a documented difference (parity appendix updated; PyAutoMind research prompt filed to chase it if it ever matters). Comparative PyAutoLens fits + 2nd/3rd-lens parity deferred to a follow-up prompt. Merge stays with you. WFC3 phase now starting under --auto per your instruction.

[Jammy2211]

Phase 1 merged via PR #3 (squash). Deferred acceptance items (comparative PyAutoLens fits, 2nd/3rd lens parity) ride the flux-scale research prompt / next phase-1 follow-up.