feat: JWST/NIRCam via the COSMOS-Web ring, four bands (phase 3)

[Jammy2211]

Overview

Phase 3: JWST/NIRCam support, validated on the COSMOS-Web ring (RA 150.10048, Dec +1.89301; Mercier et al. 2024) in all four COSMOS-Web bands — F115W/F150W (SW) and F277W/F444W (LW) — against the autolens_assistant demo dataset's modeling-ready products (SW 0.03″/pix 419², LW 0.06″/pix 209²), the four-band parity anchor. Autonomous run (--auto supervised): plan here, mechanical stretches proceed, judgment gates as batched questions, park at ship sign-off.

Routing note: Feature Agent heuristic suggested research re-home a third time; overridden — the JWST design skeleton is in the roadmap and two adapter phases have proven the architecture.

Plan

• Backend dispatch (the altitude work): stage 3 gains a combine-backend concept — astrodrizzle (HST) vs jwst_image3 (calwebb_image3: tweakreg/skymatch/outlier_detection/resample) — selected by the adapter; likewise the CRDS server (hst-crds vs jwst-crds) moves from a hardcoded constant into the adapter.
• Adapters nircam_sw / nircam_lw: level-2 _cal products (MJy/sr), crds references via jwst-crds, native scales 0.031/0.063, recommended output 0.03/0.06 matching the COSMOS-Web convention.
• Noise: resampled i2d carries ERR/VAR_POISSON/VAR_RNOISE/WHT — stage 4 reads them (+ consistency check vs the empirical sky), per the roadmap; units decision (MJy/sr vs e-/s) settled empirically against the demo products.
• PSF: tier-1 ePSF from mosaic stars where viable; STPSF as the first tier-2 back-end if mechanical under the pinned env, else parked as a question.
• Integration: reduce the ring in all four bands from MAST _cal exposures; sub-pixel-registered parity ratios per band vs the demo dataset.
• Env: jwst 1.14.0 installed into ~/venv/PyAuto under the astropy 6.1.2 constraints (done).
• Design doc docs/design/jwst.md, roadmap tick, numpy/astropy-only unit tests, ACS+WFC3 regression green (62 tests).

Detailed implementation plan

Affected Repositories

• PyAutoReduce (primary, only)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoReduce	feature/wfc3-reduction (pushed, PR #5 awaiting merge)	clean

Suggested branch: feature/jwst-nircam from main; if PR #5 merges first it rebases trivially, else the PR stacks (same pattern as phase 2).

Implementation Steps

1. instruments/adapter.py: add observatory ("hst"/"jwst"), crds_server_url, combine_backend ("astrodrizzle"/"jwst_image3") fields; move the HST server constant out of acquire/crds.py.
2. instruments/nircam_sw.py + nircam_lw.py: MAST names ("NIRCAM/IMAGE" with channel filtering by filter wavelength or detector), _cal suffix, jwst-crds, native/recommended scales, saturation levels.
3. drizzle/jwst_combine.py: calwebb_image3 wrapper honoring the pixel-scale/pixfrac dials (resample.pixel_scale, resample.pixfrac), emitting sci/wht(+var) paths + the same diagnostics fragment (WHT uniformity, R).
4. drizzle/combine.py → thin dispatcher on adapter.combine_backend.
5. noise/jwst_rms.py: noise map from resampled ERR (or VAR sums), consistency check vs empirical sky; loud on missing extensions.
6. pipeline.py: route noise construction by backend; PSF/package stages unchanged.
7. acquire/mast.py: _cal product filtering (calib_level 2), JWST obs_collection ("JWST").
8. Tests: adapter entries, dispatch routing, jwst noise reader on synthetic HDUs, CRDS server per observatory.
9. scripts/reduce_cosmos_web_ring.py --band F444W|F150W|...: acquire→image3→noise→psf→package + parity vs autolens_assistant/dataset/imaging/cosmos_web_ring/wavebands/<band>/.

Key Files

• docs/design/roadmap.md — JWST section being implemented
• autoreduce/instruments/wfc3_ir.py — adapter pattern
• autolens_assistant/dataset/imaging/cosmos_web_ring/wavebands/ — parity anchor

Original Prompt

Click to expand

See PyAutoMind/issued/jwst_nircam_cosmos_web.md.

[Jammy2211]

End-of-day checkpoint (implementation complete; integrations running overnight).

Done on feature/jwst-nircam (7 local commits, stacked on feature/wfc3-reduction/PR #5, unpushed pending sign-off):

• NIRCam SW+LW adapters, combine-backend dispatch (astrodrizzle | jwst_image3), adapter-owned CRDS servers, _i2d normalized to the sci/wht/err contract, noise = R × propagated ERR with sky-consistency check, NaN-safe star finding + finite-window ePSF star cut. 75/75 tests.
• PSF-options literature synthesis in docs/design/jwst.md (user request): COSMOS-Web WL practice (ShOpt/PSFEx/PIFF) + AGN decomposition (Zhuang & Shen) ⇒ LW single ePSF adequate (~0.6% RMS variation), SW/WL-grade needs spatially-varying PSFEx/ShOpt (tier-2 follow-up), STPSF demoted to explicit fallback.
• Env: jwst 2.0.1 (numpy-2 builds; 1.14 wheels crashed on numpy 2.2.6), scipy 1.14→1.17.1 (within autoarray's cap; jax 0.9.2 and autolens verified untouched).
• Traps fixed en route: stpipe .call() needs save_results=True; ePSF star windows must be finite.

Running overnight: scripts/reduce_cosmos_web_ring.py --band all — F444W (cal files cached) then F115W/F150W/F277W (fresh downloads), each with sub-pixel parity vs the assistant demo products. Log: scripts/output/overnight_all_bands.log; per-band validation_summary.json.

Morning resume: read the four validation summaries → parity table → review faculty + Heart → park at ship sign-off here.

[Jammy2211]

Parked at ship sign-off (supervised --auto — checkpoint-and-continue).

Phase 3 is complete and validated on feature/jwst-nircam (14 local commits, unpushed, stacked on feature/wfc3-reduction/PR #5). All four COSMOS-Web ring bands reduced end-to-end from MAST with internal closures passing everywhere; parity vs the demo products is order-unity per band under the agreed "close + internally consistent" bar. En route the run survived and fixed a machine OOM (footprint filtering + on-disk image3 models), the stpipe save_results trap, and NaN-window ePSF stars. Review: 6 findings → 5 fixed, 1 noted; 83/83 tests.

Draft PR body (posted verbatim on approval):

---

Implements JWST/NIRCam support per #6, stacked on feature/wfc3-reduction (PR #5). Autonomous run (--auto supervised), parked at ship sign-off.

What this adds

• Backend dispatch (the altitude work): InstrumentAdapter.combine_backend routes stage 3 to astrodrizzle (HST) or jwst_image3 (calwebb_image3); CRDS server/env shape (observatory, crds_server_url, mast_obs_collection) is adapter-owned.
• instruments/nircam.py — nircam_sw/nircam_lw + filter→channel routing; recommended output scales 0.03″/0.06″ (COSMOS-Web convention).
• drizzle/jwst_combine.py — calwebb_image3 with the lensing dials mapped to resample; save_results=True, in_memory=False (image3 OOM'd this machine otherwise); _i2d normalized to standalone sci/wht/err so downstream stages stay backend-agnostic.
• noise/jwst_rms.py — read propagated ERR × Casertano R + blank-sky consistency check (the roadmap's "read, don't construct").
• acquire/footprint.py — detector-footprint filter (pixel-space containment, immune to RA-wrap/cos-dec box pitfalls): only exposures covering the target enter combination — on COSMOS-Web SW visits this discarded 52 of 64 exposures.
• Bad-pixel policy (noise/rms.py + package/cutout.py split): isolated dead/rejected pixels → masked-by-noise (1e8, the legacy convention), recorded in provenance; loud failure preserved for structured clusters, >0.5% of the cutout, or anything within 1.5″ of the lens.
• PSF options from the WL+AGN literature (docs/design/jwst.md): ShOpt/PSFEx/PIFF (COSMOS-Web WL) + Zhuang & Shen (AGN) ⇒ LW single ePSF adequate (~0.6% RMS spatial variation), SW/WL-grade needs spatially-varying PSFEx/ShOpt (tier-2 follow-up), STPSF an explicit flagged fallback only.

Four-band COSMOS-Web ring validation (vs the assistant demo products)

Acceptance bar per maintainer guidance: close + internally consistent, not reproduction — the demo descends from the bespoke COSMOS-Web team pipeline.

Band	n_exp (skipped)	sky/ERR closure	R	ePSF stars	data ratio	noise ratio
F115W (SW 0.03″)	12 (52)	1.075	1.525	56	1.385	1.476
F150W (SW 0.03″)	12 (52)	1.005	1.525	96	0.812	1.596
F277W (LW 0.06″)	8 (8)	0.956	1.538	302	1.021	0.576
F444W (LW 0.06″)	8 (8)	0.883	1.538	253	1.551	0.733

Internal closures pass everywhere (sky vs ERR floor within ~12%; ≤1 masked pixel/band; units MJy/sr recorded). Parity is order-unity in all bands with per-band scatter in both directions — consistent with the team pipeline's per-band bespoke steps, and inside the agreed bar. Cutout WHT uniformity 0.25–0.31 where measured (sparse COSMOS-Web dithers; reported per run).

API Changes

New adapters + adapter fields (all defaulted — HST adapters unchanged, verified by regression tests); autoreduce.acquire.footprint; noise.jwst_rms; noise.rms.mask_isolated_bad_pixels; package.cutout.make_cutout/write_fits. No other PyAuto* repo touched.

Scripts Changed

New scripts/reduce_cosmos_web_ring.py (four bands + parity). Design docs: jwst.md (deltas, PSF tiering, parity interpretation), roadmap tick.

Environment note

~/venv/PyAuto gained jwst 2.0.1 (1.14 wheels crash under numpy 2) and scipy moved 1.14.0 → 1.17.1 (inside autoarray's declared cap); astropy/numpy/jax unchanged and autolens verified working.

Gate legs (supervised — parked at sign-off)

1. Tests: 83/83 (62 phase-1/2 regression + 21 JWST/footprint/policy). Downstream suites n/a (no dependents).
2. Smoke: n/a — no workspace surface consumes autoreduce (stated per contract).
3. Review: finder+verify on the phase-3 delta → 6 findings: 5 fixed (pixel-space footprint containment replacing the RA-wrap/cos-dec box; jwst path decoupled from EXPTIME; structured-defect loudness added to the masking policy; masked-pixel bleed excluded from parity stats; chdir path resolve), 1 noted (footprint's real-WCS assumption — empirically exercised by all four bands). Two review concerns verified not-bugs (center_xy axes; WHT normalization in the uniformity statistic).
4. Heart: YELLOW — same pre-existing organism reason set as PRs feat: HST/ACS reduction pipeline (phase 1) #3/feat: WFC3 (UVIS + IR) reduction support (phase 2) #5; none related to this repo. Parked for acknowledgement.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WNhuuvZLgZjHvWBGdhUst3

---

Questions (batched — answer any subset):

1. Ship? Push feature/jwst-nircam and open the PR? Requires your Heart-YELLOW acknowledgement of the reason set above (same set as feat: HST/ACS reduction pipeline (phase 1) #3/feat: WFC3 (UVIS + IR) reduction support (phase 2) #5). Base fork: merge PR feat: WFC3 (UVIS + IR) reduction support (phase 2) #5 first and this rebases onto main cleanly (recommended, same as last time); otherwise it opens stacked.
2. Parity table sign-off: the per-band order-unity ratios (0.58–1.60, both directions) are within your stated bar — confirm you're happy treating the demo-product deltas as documented differences (they're recorded in jwst.md), with no further chase?
3. Phase 4 pick (when ready): tier-2 spatially-varying PSF (PSFEx/ShOpt — now the biggest quality lever for SW/WL-grade work), per-exposure _flt/_cal frame products, or a sample-scale driver (many lenses through the ACS path)? No work starts without your pick.

[Jammy2211]

Shipped at PR-open: PR #7 targets main (PR #5 merged first per your answer, 10 commits rebased cleanly, 83/83 re-verified post-rebase). Parity table accepted as documented differences under the close-not-reproduce bar; NIRCam refinements (tier-2 spatially-varying PSF, unit-aware star cut, WHT-uniformity for sparse dithers) stay on the roadmap for when you pick them up. Merge is yours.

[Jammy2211]

Phase 3 merged via PR #7 (squash). NIRCam refinements stay on the roadmap for later pickup.