Skip to content

feat: JWST/NIRCam via the COSMOS-Web ring, four bands (phase 3) - #7

Merged
Jammy2211 merged 10 commits into
mainfrom
feature/jwst-nircam
Jul 9, 2026
Merged

feat: JWST/NIRCam via the COSMOS-Web ring, four bands (phase 3)#7
Jammy2211 merged 10 commits into
mainfrom
feature/jwst-nircam

Conversation

@Jammy2211

Copy link
Copy Markdown
Contributor

Implements JWST/NIRCam support per #6, rebased onto main after PR #5 merged. Autonomous run (--auto supervised); shipped at PR-open on maintainer sign-off.

What this adds

  • Backend dispatch (the altitude work): InstrumentAdapter.combine_backend routes stage 3 to astrodrizzle (HST) or jwst_image3 (calwebb_image3); CRDS server/env shape (observatory, crds_server_url, mast_obs_collection) is adapter-owned.
  • instruments/nircam.pynircam_sw/nircam_lw + filter→channel routing; recommended output scales 0.03″/0.06″ (COSMOS-Web convention).
  • drizzle/jwst_combine.py — calwebb_image3 with the lensing dials mapped to resample; save_results=True, in_memory=False (image3 OOM'd this machine otherwise); _i2d normalized to standalone sci/wht/err so downstream stages stay backend-agnostic.
  • noise/jwst_rms.py — read propagated ERR × Casertano R + blank-sky consistency check (the roadmap's "read, don't construct").
  • acquire/footprint.py — detector-footprint filter (pixel-space containment, immune to RA-wrap/cos-dec box pitfalls): only exposures covering the target enter combination — on COSMOS-Web SW visits this discarded 52 of 64 exposures.
  • Bad-pixel policy (noise/rms.py + package/cutout.py split): isolated dead/rejected pixels → masked-by-noise (1e8, the legacy convention), recorded in provenance; loud failure preserved for structured clusters, >0.5% of the cutout, or anything within 1.5″ of the lens.
  • PSF options from the WL+AGN literature (docs/design/jwst.md): ShOpt/PSFEx/PIFF (COSMOS-Web WL) + Zhuang & Shen (AGN) ⇒ LW single ePSF adequate (~0.6% RMS spatial variation), SW/WL-grade needs spatially-varying PSFEx/ShOpt (tier-2 follow-up), STPSF an explicit flagged fallback only.

Four-band COSMOS-Web ring validation (vs the assistant demo products)

Acceptance bar per maintainer guidance: close + internally consistent, not reproduction — the demo descends from the bespoke COSMOS-Web team pipeline.

Band n_exp (skipped) sky/ERR closure R ePSF stars data ratio noise ratio
F115W (SW 0.03″) 12 (52) 1.075 1.525 56 1.385 1.476
F150W (SW 0.03″) 12 (52) 1.005 1.525 96 0.812 1.596
F277W (LW 0.06″) 8 (8) 0.956 1.538 302 1.021 0.576
F444W (LW 0.06″) 8 (8) 0.883 1.538 253 1.551 0.733

Internal closures pass everywhere (sky vs ERR floor within ~12%; ≤1 masked pixel/band; units MJy/sr recorded). Parity is order-unity in all bands with per-band scatter in both directions — consistent with the team pipeline's per-band bespoke steps, and inside the agreed bar. Cutout WHT uniformity 0.25–0.31 where measured (sparse COSMOS-Web dithers; reported per run).

API Changes

New adapters + adapter fields (all defaulted — HST adapters unchanged, verified by regression tests); autoreduce.acquire.footprint; noise.jwst_rms; noise.rms.mask_isolated_bad_pixels; package.cutout.make_cutout/write_fits. No other PyAuto* repo touched.

Scripts Changed

New scripts/reduce_cosmos_web_ring.py (four bands + parity). Design docs: jwst.md (deltas, PSF tiering, parity interpretation), roadmap tick.

Environment note

~/venv/PyAuto gained jwst 2.0.1 (1.14 wheels crash under numpy 2) and scipy moved 1.14.0 → 1.17.1 (inside autoarray's declared cap); astropy/numpy/jax unchanged and autolens verified working.

Gate legs (supervised — parked at sign-off)

  1. Tests: 83/83 (62 phase-1/2 regression + 21 JWST/footprint/policy). Downstream suites n/a (no dependents).
  2. Smoke: n/a — no workspace surface consumes autoreduce (stated per contract).
  3. Review: finder+verify on the phase-3 delta → 6 findings: 5 fixed (pixel-space footprint containment replacing the RA-wrap/cos-dec box; jwst path decoupled from EXPTIME; structured-defect loudness added to the masking policy; masked-pixel bleed excluded from parity stats; chdir path resolve), 1 noted (footprint's real-WCS assumption — empirically exercised by all four bands). Two review concerns verified not-bugs (center_xy axes; WHT normalization in the uniformity statistic).
  4. Heart: YELLOW — same pre-existing organism reason set as PRs feat: HST/ACS reduction pipeline (phase 1) #3/feat: WFC3 (UVIS + IR) reduction support (phase 2) #5; none related to this repo. Maintainer acknowledged at sign-off (issue feat: JWST/NIRCam via the COSMOS-Web ring, four bands (phase 3) #6), authorizing PR-open.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WNhuuvZLgZjHvWBGdhUst3

Jammy2211 and others added 10 commits July 9, 2026 11:19
…MOS-Web ring integration

Adapter fields observatory/crds_server_url/combine_backend/
mast_obs_collection; nircam_sw+nircam_lw with filter->channel routing;
drizzle/jwst_combine.py (calwebb_image3, dials mapped to resample,
_i2d normalized to sci/wht/err contract); noise = R x propagated ERR
with sky consistency check; CRDS env adapter-owned (jwst-crds,
CRDS_PATH direct); NaN-masked star finding; no full-well cut in
surface-brightness units. 75 tests; docs/design/jwst.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNhuuvZLgZjHvWBGdhUst3
ShOpt/PSFEx/PIFF (COSMOS-Web WL practice), Zhuang & Shen NIRCam PSF
characterization (spatial variation ~5% SW -> 0.6% LW RMS; PSFEx >
photutils; empirical > STPSF for decomposition). Adopted tiering: LW
single ePSF (tier 1, shipped); SW/WL-grade spatially-varying PSFEx or
ShOpt (tier 2, follow-up); STPSF explicit 2b fallback; STARRED/PSFr
tier 3 unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNhuuvZLgZjHvWBGdhUst3
…/DQ holes)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNhuuvZLgZjHvWBGdhUst3
…rovenance, proposal filter

image3 OOM'd the machine (took the session down): only exposures whose
detector footprint covers the target now enter combination, and image3
runs with on-disk models. data_units read from BUNIT (jwst provenance
has no drizzle_kwargs — the cross-backend KeyError that failed F444W
after an otherwise-clean run). Ring script pinned to proposal 1727 so
parity stays COSMOS-Web-pure (5893 exposures were creeping in). 77 tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNhuuvZLgZjHvWBGdhUst3
…matters

Deep resampled stacks routinely carry isolated fully-rejected pixels
(F444W tripped the strict guard on exactly one). Policy: bad noise
pixels -> MASKED_NOISE_VALUE (1e8, the legacy noise-scaling convention)
with data zeroed, recorded in provenance; hard failure preserved for
>0.5% of the cutout or any bad pixel within 1.5" of the lens.
cutout.py split into make_cutout/write_fits for the post-processing
seam. 81 tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNhuuvZLgZjHvWBGdhUst3
…eproduction (maintainer guidance)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNhuuvZLgZjHvWBGdhUst3
…ructured-defect loudness, masked-pixel parity stats, path resolve

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNhuuvZLgZjHvWBGdhUst3
…atches blocks first)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNhuuvZLgZjHvWBGdhUst3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant