feat: Keck NIRC2 AO reduction (phases K1-K3, SHARP-grounded)

[Jammy2211]

Overview

Implement the Keck NIRC2 AO reduction designed in #9, on the post-phase-3 consolidated layout (#10, merged 371721f). This makes PyAutoReduce's fourth instrument family — and its first ground-based one — live: raw KOA level-0 NIRC2 frames → modeling-ready al.Imaging.from_fits datasets. Run under --auto at effective level supervised (feature @ large); ship sign-off will park with a batched question per the contract.

Decisions adopted from #9's batched set (user-resolved 2026-07-09): drizzle-package combine backend, B1938+666 (SHARP I) validation anchor, and sequencing resolved by the #10 merge.

Plan

• Ship docs/design/keck_ao.md (the parked research: plan the Keck NIRC2 AO reduction workflow (SHARP context) #9 write leg, adopting post-refactor module names) + roadmap update.
• K1 — acquire + calibrate + sky: KOA/PyKOA acquire backend behind an archive seam; new ground-based calibrate stage (DN→e⁻, dark, flat, bad-pixel) and sky stage (temporally adjacent object-masked running sky).
• K2 — dewarp + combine + noise: Yelda 2010 / Service 2016 distortion solutions as the pixel mapping through the STScI drizzle package; cross-correlation registration; first-principles noise map (sky + dark + read/√coadds + Poisson, coverage-propagated, × Casertano R).
• K3 — PSF tiers + package + validation: provisional-PSF contract (PSF-star candidate products, psf_provisional provenance, AO-conditions block); unchanged output contract; B1938+666 validation per the four acceptance checks on research: plan the Keck NIRC2 AO reduction workflow (SHARP context) #9.

Detailed implementation plan

Work Classification

Library (PyAutoReduce).

Affected Repositories

• PyAutoReduce (primary)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoReduce	main @ 371721f	clean (claim released by refactor-post-phase3)

Suggested branch: feature/keck-ao-reduction
Worktree root: ~/Code/PyAutoLabs-wt/keck-ao-reduction/

Implementation Steps

1. docs/design/keck_ao.md from the draft on research: plan the Keck NIRC2 AO reduction workflow (SHARP context) #9 + docs/design/roadmap.md ground-based section.
2. autoreduce/instruments/nirc2.py adapter (camera plate scales 9.942/39.686 mas, gain, read noise per coadd, distortion-solution selection by date: Yelda 2010 pre-2015-04-13 / Service 2016 after) + acquire-backend seam (koa | mast) with PyKOA TAP queries for science + night calibrations + PSF-star frames.
3. autoreduce/calibrate/ (new stage): DN→e⁻, dark subtraction, flat-field, bad-pixel mask. numpy/astropy only.
4. autoreduce/sky/ (new stage): running median sky from N temporally adjacent object-masked frames, iterative mask update, per-frame sky level recorded for the noise stage.
5. K2 in autoreduce/drizzle/ (post-refactor _common.py seams): dewarp-as-drizzle-mapping, cross-correlation registration, combine at 10 mas (narrow) / 40 mas (wide) with optional 2×2 binning dial; noise construction + Casertano R; blank-sky closure check.
6. K3 in autoreduce/psf/ + package: PSF-star pipeline-identical reduction → candidate products, provenance extensions (psf_provisional, FWHM/epoch/tip-tilt distance, AO-conditions block).
7. Unit tests in test_autoreduce/ (numpy/astropy-only, synthetic frames). Prototype/validation script prototypes/b1938_keck_spike.py + scripts/reduce_b1938.py for the real-data leg (network; results reported on this issue, FITS never committed).
8. Four-leg ship gate → park at sign-off (supervised).

Key Files

• docs/design/keck_ao.md, docs/design/roadmap.md
• autoreduce/instruments/, autoreduce/acquire/, autoreduce/drizzle/_common.py, autoreduce/noise/, autoreduce/psf/, autoreduce/pipeline.py
• autoreduce/calibrate/, autoreduce/sky/ (new)
• test_autoreduce/, prototypes/, pyproject.toml (new deps: pykoa, drizzle)

Original Prompt

Click to expand starting prompt

Original request: "A large refactor of PyAutoReduce just merge in, so combine that with this research to get keck-ao data reduction live --auto"

(Full prompt: PyAutoMind feature/pyautoreduce/keck_ao_reduction.md; design + research context: #9.)

[Jammy2211]

Ship sign-off — keck-ao-reduction parked at checkpoint 2 (supervised)

Implementation is complete, review-hardened, and validated end-to-end on real Keck data. Per the autonomy contract, this run parks here: two local commits on feature/keck-ao-reduction in the task worktree, nothing pushed — your sign-off triggers push → PR-open (merge stays with you as always).

What's ready

Commits 7d8d4d6 (implementation, 21 files, +2078) and 8f3d67e (review-gate fixes, 13 files, +322/−112) deliver the full design from #9: KOA acquire seam (PyKOA; science + night calibrations + PSF-star frames + checksummed distortion solutions), the ground calibrate/sky stages, the nirc2_native drizzle-pixmap combine with IVM weights feeding the unchanged noise recipe, sampling-mode-aware read noise, the provisional-PSF contract with epoch candidates + CR vetting, docs/design/keck_ao.md (closes the #9 write leg), roadmap/AGENTS updates, and keck = ["pykoa", "drizzle"] extras.

Validation — B1938+666 (the actual SHARP I discovery data, program U085N2L)

Acceptance check (keck_ao.md)	Result
Blank-sky noise closure (empirical × R² / predicted)	0.883 ∈ (0.6, 1.6) ✓
Per-frame noise budget (MCDS-32)	62.0 e⁻ predicted vs 62–64 e⁻ empirical ✓
PSF candidate FWHM vs SHARP's ~65–70 mas	72 mas ✓ (4 CR-dominated epochs auto-rejected by the diffraction-floor vetting)
WHT uniformity over cutout	0.084 < 0.2 ✓
Dataset	39 × 180 s lens frames; PSF/tt-star pointings found at 20″ interleaved, exactly as Lagattuta+12 describe

Astrometric parity vs HST and the end-to-end PyAutoLens Einstein-radius check (acceptance items 3–4) need the first reduced dataset in a modelling run — proposed as the follow-up prompt alongside K-implementation items.

Four-leg gate

1. Tests — 110 pass (24 new Keck tests) in the task worktree. Downstream suites n/a: nothing imports autoreduce (stated per gate rule).
2. Smoke — n/a: PyAutoReduce has no workspace script surface in the curated lists.
3. Review — faculty pass, 8 angles → 30 candidates → 10 verified findings; 9 fixed in 8f3d67e (epoch-aware sky grouping, tier-B saturation cut, pointing-coherence guard, portable provenance, distortion-epoch guards, detector-boundary rule, stale-candidate hygiene), 1 deferred as documented open items (orchestrator dispatch generalisation etc.). Post-fix verdict: CLEAN.
4. Heart — YELLOW, six chronic organism-level reasons (none touch PyAutoReduce): workspace validation 5-failed report of 2026-07-08; 58 stale parked scripts; autolens_assistant version pin behind; install verification not run; no release validation for current source; PyAutoMind PR 9d old. Needs your acknowledgement — never acked autonomously.

The question

Reply here (or in-session) with any of:

1. "ship" — I push the branch, open the PR (body ≈ this comment), append the calibration-log row, and stop at PR-open.
2. "ship + merge" — as above, then merge on green CI and close research: plan the Keck NIRC2 AO reduction workflow (SHARP context) #9 + feat: Keck NIRC2 AO reduction (phases K1-K3, SHARP-grounded) #11.
3. Redirections (e.g. adjust the pointing budget, different anchor checks, split the two commits) — say the word.

Boundary disclosure for the review: AGENTS.md's test rule gained one sanctioned exception (the lightweight drizzle resampler behind importorskip; drizzlepac/jwst still excluded), and PR #10's layout is untouched.

[Jammy2211]

Shipped and merged: PR #12 (44a8d1a, squash) — human-directed "ship + merge" at the sign-off checkpoint, Heart YELLOW reason set acknowledged there. Keck NIRC2 AO reduction is live on main: KOA acquire, ground calibrate/sky stages, nirc2_native combine, provisional-PSF contract; validated end-to-end on B1938+666 (closure 0.883, PSF FWHM 72 mas, 110 tests). Follow-up prompts to file separately: acceptance checks 3–4 (HST astrometric parity + PyAutoLens Einstein-radius invariance) and the keck_ao.md open items. Closing.