diff --git a/active.md b/active.md index 96bc6e3b..ea91fe3f 100644 --- a/active.md +++ b/active.md @@ -62,16 +62,3 @@ - artifacts-are-laptop-only: Actions artifact downloads are blocked from cloud/mobile sessions (egress policy 403s `productionresultssa2.blob.core.windows.net` on CONNECT) — this is what stopped the cloud session finishing the ingest. Both wiki drift reports were captured to `~/.pyauto-heart/release_20260807_wiki_drift/` while on the laptop. - do-not: do NOT use the nightly driver for a manual release — AUTONOMY.md forbids converting a manual release into the scheduled-nightly exception. - repos-none-claimed: this entry claims NO repos — deliberately on one line, NOT as 2-space ` - Repo` bullets, because `worktree_check_conflict` treats any such bullet as a live claim. - -## hst-dq-bits-dial -- issue: https://github.com/PyAutoLabs/PyAutoReduce/issues/65 -- status: issued, UNCLAIMED — filed 2026-08-04 for Fable to pick up later. No worktree, no branch, no local work started. Whoever picks it up runs `/start_dev active/hst_dq_bits_dial.md` first to claim the repo properly. -- root-cause: `drizzle_kwargs_for` never sets `final_bits`/`driz_sep_bits` and no adapter does either (the strings appear nowhere in PyAutoReduce), so every reduction inherits drizzlepac's package default `final_bits="0"` — no DQ bit treated as good, every flagged pixel rejected, including the hot/warm/blob pixels calacs/calwf3 already corrected. -- evidence: read from the cached CRDS MDRIZTAB reference files, so these are STScI's shipped values not recollection — ACS/WFC + WFC3/UVIS `final_bits` 65535 at numimages=1, 336 (=16+64+256) at N>=2; WFC3/IR 65535 at N=1, 528 (=512+16) at N>=2. Three consequences: 336 means hot/warm/saturated are usable and we reject all three; 528 passes exactly the blob bit that caused the PJ011646 hole (5 exposures, so that hole could not occur under standard practice); and the value is exposure-count dependent with single-exposure data at 65535 (the SLACS SNAP regime), so the dial must be N-aware and a flat constant is wrong. -- two-legs: leg 1 = the N-keyed bits dial on the adapters + TargetSpec, recorded in reduction.json, unit-tested at the N boundaries, documented in both design docs. Leg 2 = a LOCAL weight-deficit guard, because both current guards are blind to this defect class: `mask_isolated_bad_pixels` tests `~isfinite | noise <= 0` so a finite reduced-weight stripe passes every check INCLUDING the 1.5" protect_radius_arcsec lens-core protection, and `weight_uniformity` is a global RMS/median (spike measured 0.066 vs the 0.2 limit) that a few columns cannot move. Leg 2 is the detector for leg 1 and should land first so the control test has an objective pass/fail. Two PRs behind one issue. -- control-test: re-drizzle one striped SLACS target at `0` vs the MDRIZTAB value and diff the weight + noise maps BEFORE building the dial. If the stripes do not move the cause is elsewhere (exposure count, dither geometry, genuine bad columns) and leg 1 is not the fix. -- trap: do NOT set `mdriztab=True` to get the bits. MDRIZTAB carries the whole parameter set including final_scale/final_pixfrac/final_kernel/final_rot, and hst_acs_pipeline.md stage 3 deviates from those deliberately (0.05"/pix, pixfrac 0.8, north-up); enabling it risks silently reverting the justified lensing deviations. Read the bits columns and keep our explicit kwargs. -- provenance: rescoped from the WFC3/IR-only prompt filed 2026-07-10 (`wfc3_ir_dq_bits_dial.md`) after a user report that F814W SLACS-gold noise maps carry high-noise stripes through some deflector cores that the legacy SLACS/Amy reductions lack. The MDRIZTAB read showed the WFC3/IR holes and the ACS stripes are one root cause at one fix site, so one prompt and one issue rather than a duplicate. -- repo-unclaimed: PyAutoReduce is the only affected repo, listed on this single line deliberately and NOT as a 2-space ` - PyAutoReduce` sub-bullet, because worktree_check_conflict reads any such bullet as a live claim — this task holds no claim and must not block other PyAutoReduce work. -- ordering: `draft/research/pyautoreduce/acceptance_noise_rebaseline.md` must run AFTER this lands — a bits change moves the IVM weights and therefore the noise maps, so its parity numbers would need redoing otherwise. Related but distinct, do not fold in: #61 (driz_cr flux erosion / LACosmic) and #62 (tier-1 ePSF from the CR-rejected mosaic). -- prompt: active/hst_dq_bits_dial.md diff --git a/active/hst_dq_bits_dial.md b/complete/2026/08/hst-dq-bits-dial.md similarity index 56% rename from active/hst_dq_bits_dial.md rename to complete/2026/08/hst-dq-bits-dial.md index 3b285aeb..d4846c4a 100644 --- a/active/hst_dq_bits_dial.md +++ b/complete/2026/08/hst-dq-bits-dial.md @@ -1,3 +1,117 @@ +- issue: https://github.com/PyAutoLabs/PyAutoReduce/issues/65 +- library-pr: https://github.com/PyAutoLabs/PyAutoReduce/pull/70 (merge 99558128e) +- mind-pr: https://github.com/PyAutoLabs/PyAutoMind/pull/148 +- shipped: 2026-08-07 +- repos: PyAutoReduce +- tests: 281 passed / 15 skipped, up from the 265/15 baseline (+16). CI green on unittest 3.12 + 3.13. + +Every HST reduction the pipeline had ever produced rejected **every** DQ-flagged +pixel. `drizzle_kwargs_for` never set `final_bits`/`driz_sep_bits`, and no +adapter did either — the strings appeared nowhere in the repo — so every run +silently inherited drizzlepac's package default `final_bits="0"` +(`drizzlepac/pars/astrodrizzle.cfg:101`), which treats no bit as good. Hot, warm +and blob pixels that `calacs`/`calwf3` had *already corrected* were thrown away. +An unintended inheritance, never a chosen deviation, and it had been shipping +since phase 1. + +Two visible symptoms, one root cause, one fix site: zero-coverage **holes** on +WFC3/IR (PJ011646, DQ 512 blobs at the same detector pixels in all five +exposures) and high-noise **stripes through deflector cores** on ACS/WFC F814W +SLACS-gold that the legacy SLACS reductions lack. + +## What shipped + +**Leg 2 first, deliberately — it is the detector for leg 1.** +`drizzle/diagnostics.py` gains `local_weight_deficit` / `check_local_weight_deficit`: +inside the same 1.5" radius as `protect_radius_arcsec`, it reports the +science-region median weight and the worst row/column median as fractions of the +cutout median, recorded in `reduction.json` at `drizzle.local_weight_deficit`. +Both axes, because a detector-column defect lands on an image row or column +depending on the frame's orientation on the sky. + +**Leg 1** puts STScI's own MDRIZTAB rows on the adapters as +`(min_exposures, driz_sep_bits, final_bits)` with MDRIZTAB's semantics (the last +row whose `min_exposures <= N`): + + acs_wfc, wfc3_uvis: (1, 65535, 65535), (2, 336, 336) + wfc3_ir: (1, 65535, 65535), (2, 65535, 528), (4, 528, 528) + +`336 = 16+64+256` (hot, warm, saturated); `528 = 512+16` (blob, hot). +`TargetSpec.final_bits`/`driz_sep_bits` override at every N; +`dq_bits_provenance` records each value **and** its source +(`adapter_mdriztab` / `target_spec` / `unset`) so datasets stay re-derivable as +the tables move. + +## Traps and findings worth keeping + +- **The bits are genuinely N-dependent; a flat constant is wrong.** + Single-exposure data uses 65535 — every bit good — because with one exposure + there is nothing to fill a masked pixel with, so the standard recipe keeps + flagged pixels rather than punching holes. This is also an independent + explanation for why the legacy SLACS SNAP maps look clean: SNAP data *is* the + N=1 regime. +- **The two bits columns DIFFER, so rows carry both.** `wfc3_ir` at N=2-3 has + `driz_sep_bits` 65535 with `final_bits` 528 — the separate (median-building) + drizzle still keeps every bit while the final drizzle is already at 528. The + issue's own shorthand ("65535 at N=1, 528 at N>=2") flattened its own + evidence table; the reference rows won. +- **Unset must mean the key is ABSENT, not 0.** `0` is precisely drizzlepac's + "no bit is good" default — writing it explicitly would re-enact the bug. The + non-AstroDrizzle backends (`jwst_image3`, `nirc2_native`) declare no table and + emit no keyword; pinned by test. +- **Do NOT use `mdriztab=True` to get the bits.** It imports the whole parameter + set — `final_scale`, `final_pixfrac`, `final_kernel`, `final_rot` — and would + silently revert the deliberate, justified lensing deviations in + `hst_acs_pipeline.md` stage 3 (0.05"/pix, pixfrac 0.8, north-up). +- **The blindness was structural, not an oversight in one guard.** + `mask_isolated_bad_pixels` tests `~isfinite | <= 0`, so a *degraded* pixel was + never even a candidate — not for the clustering check, and not for the 1.5" + protection whose entire purpose is "the lens itself must be clean"; + `weight_uniformity` is a global RMS/median (slacs0008 measured 0.066 against a + 0.2 limit) that a few columns cannot move. Between the two sat a whole defect + class. `test_the_existing_guards_are_blind_to_the_same_map` pins it: one + synthetic striped map both old guards pass and the new one catches. +- **`star_pass_kwargs_for` was carrying the fingerprint all along.** Its + `int(kwargs.get("final_bits", 0)) | CR_DQ_BIT` only needed a `.get` fallback + because nothing ever set the key. It picked up the fix for free. +- **`reduce_pj011646.py` does not exist** anywhere in PyAutoReduce. The issue's + claim that it "carries a documented monkeypatch workaround until this ships" + was stale — there was nothing to unwind. + +## STILL OWED — the control test gates leg 1's defaults + +**Not run.** It needs `drizzlepac`, the CRDS cache (`scripts/cache/crds/`, which +is gitignored) and archive data; the cloud session that wrote this had none of +them. #65 asked for it *first*, and it remains the gate before these defaults +reach a release: + +> Re-drizzle one striped SLACS target at the old `0` and at the MDRIZTAB value, +> diff the weight and noise maps. If the stripes do not move, the cause is +> elsewhere — exposure count, dither geometry, or genuine bad columns — and +> **revert the adapter defaults rather than shipping the dial.** + +Leg 2's diagnostic now scores that comparison objectively instead of by eye, and +is also what calibrates its own provisional 0.9 limit (derivation: one lost +exposure of N leaves weight `(N-1)/N`, so 0.9 catches a single loss for any +N <= 9 — the regime where `sqrt(N/(N-1))` noise inflation is visible). The +obligation is recorded in `docs/design/hst_acs_pipeline.md` stage 3, not only +here. The leg-2 verdict is **recorded, never raised**, precisely because the +limit is uncalibrated. + +## Follow-ups + +- `draft/research/pyautoreduce/acceptance_noise_rebaseline.md` must run **after** + this — the bits move the IVM weights and therefore the noise maps, so its + SLACS parity numbers would need redoing. +- `autoreduce_workspace` may want to surface the new dial; both `TargetSpec` + fields are additive with `None` defaults, so nothing there breaks. +- Related but deliberately untouched: PyAutoReduce#61 (driz_cr erodes flux at + steep-gradient cores) and #62 (tier-1 ePSF from the CR-rejected mosaic). All + three concern how DQ/CR masking degrades products, but they touch different + stages. + +## Original prompt + # HST needs a DQ-bits dial — we mask every DQ-flagged pixel where STScI keeps most of them Type: bug diff --git a/complete/index.md b/complete/index.md index 6c5eaf9c..2c29ec91 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -930 records across 7 buckets. +931 records across 7 buckets. ## Highlights @@ -31,6 +31,7 @@ _(curate hard-won records here — survives regeneration.)_ - [health-conductor-stale-verdict](2026/08/health-conductor-stale-verdict.md) - [howto-smoke-all-tutorials](2026/08/howto-smoke-all-tutorials.md) - [howtofit-stub-fork-cleanup](2026/08/howtofit-stub-fork-cleanup.md) +- [hst-dq-bits-dial](2026/08/hst-dq-bits-dial.md) - [hygiene-coverage-drift](2026/08/hygiene-coverage-drift.md) - [hygiene-detail-flag](2026/08/hygiene-detail-flag.md) - [hygiene-howto-refs-docstrings](2026/08/hygiene-howto-refs-docstrings.md)