prompt: claim + implement hst-dq-bits-dial (PyAutoReduce#65)

active.md

@@ -62,16 +62,3 @@
 - artifacts-are-laptop-only: Actions artifact downloads are blocked from cloud/mobile sessions (egress policy 403s `productionresultssa2.blob.core.windows.net` on CONNECT) — this is what stopped the cloud session finishing the ingest. Both wiki drift reports were captured to `~/.pyauto-heart/release_20260807_wiki_drift/` while on the laptop.
 - do-not: do NOT use the nightly driver for a manual release — AUTONOMY.md forbids converting a manual release into the scheduled-nightly exception.
 - repos-none-claimed: this entry claims NO repos — deliberately on one line, NOT as 2-space `  - Repo` bullets, because `worktree_check_conflict` treats any such bullet as a live claim.
-
-## hst-dq-bits-dial
-- issue: https://github.com/PyAutoLabs/PyAutoReduce/issues/65
-- status: issued, UNCLAIMED — filed 2026-08-04 for Fable to pick up later. No worktree, no branch, no local work started. Whoever picks it up runs `/start_dev active/hst_dq_bits_dial.md` first to claim the repo properly.
-- root-cause: `drizzle_kwargs_for` never sets `final_bits`/`driz_sep_bits` and no adapter does either (the strings appear nowhere in PyAutoReduce), so every reduction inherits drizzlepac's package default `final_bits="0"` — no DQ bit treated as good, every flagged pixel rejected, including the hot/warm/blob pixels calacs/calwf3 already corrected.
-- evidence: read from the cached CRDS MDRIZTAB reference files, so these are STScI's shipped values not recollection — ACS/WFC + WFC3/UVIS `final_bits` 65535 at numimages=1, 336 (=16+64+256) at N>=2; WFC3/IR 65535 at N=1, 528 (=512+16) at N>=2. Three consequences: 336 means hot/warm/saturated are usable and we reject all three; 528 passes exactly the blob bit that caused the PJ011646 hole (5 exposures, so that hole could not occur under standard practice); and the value is exposure-count dependent with single-exposure data at 65535 (the SLACS SNAP regime), so the dial must be N-aware and a flat constant is wrong.
-- two-legs: leg 1 = the N-keyed bits dial on the adapters + TargetSpec, recorded in reduction.json, unit-tested at the N boundaries, documented in both design docs. Leg 2 = a LOCAL weight-deficit guard, because both current guards are blind to this defect class: `mask_isolated_bad_pixels` tests `~isfinite | noise <= 0` so a finite reduced-weight stripe passes every check INCLUDING the 1.5" protect_radius_arcsec lens-core protection, and `weight_uniformity` is a global RMS/median (spike measured 0.066 vs the 0.2 limit) that a few columns cannot move. Leg 2 is the detector for leg 1 and should land first so the control test has an objective pass/fail. Two PRs behind one issue.
-- control-test: re-drizzle one striped SLACS target at `0` vs the MDRIZTAB value and diff the weight + noise maps BEFORE building the dial. If the stripes do not move the cause is elsewhere (exposure count, dither geometry, genuine bad columns) and leg 1 is not the fix.
-- trap: do NOT set `mdriztab=True` to get the bits. MDRIZTAB carries the whole parameter set including final_scale/final_pixfrac/final_kernel/final_rot, and hst_acs_pipeline.md stage 3 deviates from those deliberately (0.05"/pix, pixfrac 0.8, north-up); enabling it risks silently reverting the justified lensing deviations. Read the bits columns and keep our explicit kwargs.
-- provenance: rescoped from the WFC3/IR-only prompt filed 2026-07-10 (`wfc3_ir_dq_bits_dial.md`) after a user report that F814W SLACS-gold noise maps carry high-noise stripes through some deflector cores that the legacy SLACS/Amy reductions lack. The MDRIZTAB read showed the WFC3/IR holes and the ACS stripes are one root cause at one fix site, so one prompt and one issue rather than a duplicate.
-- repo-unclaimed: PyAutoReduce is the only affected repo, listed on this single line deliberately and NOT as a 2-space `  - PyAutoReduce` sub-bullet, because worktree_check_conflict reads any such bullet as a live claim — this task holds no claim and must not block other PyAutoReduce work.
-- ordering: `draft/research/pyautoreduce/acceptance_noise_rebaseline.md` must run AFTER this lands — a bits change moves the IVM weights and therefore the noise maps, so its parity numbers would need redoing otherwise. Related but distinct, do not fold in: #61 (driz_cr flux erosion / LACosmic) and #62 (tier-1 ePSF from the CR-rejected mosaic).
-- prompt: active/hst_dq_bits_dial.md

active/hst_dq_bits_dial.md → complete/2026/08/hst-dq-bits-dial.md

@@ -1,3 +1,117 @@
+- issue: https://github.com/PyAutoLabs/PyAutoReduce/issues/65
+- library-pr: https://github.com/PyAutoLabs/PyAutoReduce/pull/70 (merge 99558128e)
+- mind-pr: https://github.com/PyAutoLabs/PyAutoMind/pull/148
+- shipped: 2026-08-07
+- repos: PyAutoReduce
+- tests: 281 passed / 15 skipped, up from the 265/15 baseline (+16). CI green on unittest 3.12 + 3.13.
+
+Every HST reduction the pipeline had ever produced rejected **every** DQ-flagged
+pixel. `drizzle_kwargs_for` never set `final_bits`/`driz_sep_bits`, and no
+adapter did either — the strings appeared nowhere in the repo — so every run
+silently inherited drizzlepac's package default `final_bits="0"`
+(`drizzlepac/pars/astrodrizzle.cfg:101`), which treats no bit as good. Hot, warm
+and blob pixels that `calacs`/`calwf3` had *already corrected* were thrown away.
+An unintended inheritance, never a chosen deviation, and it had been shipping
+since phase 1.
+
+Two visible symptoms, one root cause, one fix site: zero-coverage **holes** on
+WFC3/IR (PJ011646, DQ 512 blobs at the same detector pixels in all five
+exposures) and high-noise **stripes through deflector cores** on ACS/WFC F814W
+SLACS-gold that the legacy SLACS reductions lack.
+
+## What shipped
+
+**Leg 2 first, deliberately — it is the detector for leg 1.**
+`drizzle/diagnostics.py` gains `local_weight_deficit` / `check_local_weight_deficit`:
+inside the same 1.5" radius as `protect_radius_arcsec`, it reports the
+science-region median weight and the worst row/column median as fractions of the
+cutout median, recorded in `reduction.json` at `drizzle.local_weight_deficit`.
+Both axes, because a detector-column defect lands on an image row or column
+depending on the frame's orientation on the sky.
+
+**Leg 1** puts STScI's own MDRIZTAB rows on the adapters as
+`(min_exposures, driz_sep_bits, final_bits)` with MDRIZTAB's semantics (the last
+row whose `min_exposures <= N`):
+
+    acs_wfc, wfc3_uvis:  (1, 65535, 65535), (2, 336, 336)
+    wfc3_ir:             (1, 65535, 65535), (2, 65535, 528), (4, 528, 528)
+
+`336 = 16+64+256` (hot, warm, saturated); `528 = 512+16` (blob, hot).
+`TargetSpec.final_bits`/`driz_sep_bits` override at every N;
+`dq_bits_provenance` records each value **and** its source
+(`adapter_mdriztab` / `target_spec` / `unset`) so datasets stay re-derivable as
+the tables move.
+
+## Traps and findings worth keeping
+
+- **The bits are genuinely N-dependent; a flat constant is wrong.**
+  Single-exposure data uses 65535 — every bit good — because with one exposure
+  there is nothing to fill a masked pixel with, so the standard recipe keeps
+  flagged pixels rather than punching holes. This is also an independent
+  explanation for why the legacy SLACS SNAP maps look clean: SNAP data *is* the
+  N=1 regime.
+- **The two bits columns DIFFER, so rows carry both.** `wfc3_ir` at N=2-3 has
+  `driz_sep_bits` 65535 with `final_bits` 528 — the separate (median-building)
+  drizzle still keeps every bit while the final drizzle is already at 528. The
+  issue's own shorthand ("65535 at N=1, 528 at N>=2") flattened its own
+  evidence table; the reference rows won.
+- **Unset must mean the key is ABSENT, not 0.** `0` is precisely drizzlepac's
+  "no bit is good" default — writing it explicitly would re-enact the bug. The
+  non-AstroDrizzle backends (`jwst_image3`, `nirc2_native`) declare no table and
+  emit no keyword; pinned by test.
+- **Do NOT use `mdriztab=True` to get the bits.** It imports the whole parameter
+  set — `final_scale`, `final_pixfrac`, `final_kernel`, `final_rot` — and would
+  silently revert the deliberate, justified lensing deviations in
+  `hst_acs_pipeline.md` stage 3 (0.05"/pix, pixfrac 0.8, north-up).
+- **The blindness was structural, not an oversight in one guard.**
+  `mask_isolated_bad_pixels` tests `~isfinite | <= 0`, so a *degraded* pixel was
+  never even a candidate — not for the clustering check, and not for the 1.5"
+  protection whose entire purpose is "the lens itself must be clean";
+  `weight_uniformity` is a global RMS/median (slacs0008 measured 0.066 against a
+  0.2 limit) that a few columns cannot move. Between the two sat a whole defect
+  class. `test_the_existing_guards_are_blind_to_the_same_map` pins it: one
+  synthetic striped map both old guards pass and the new one catches.
+- **`star_pass_kwargs_for` was carrying the fingerprint all along.** Its
+  `int(kwargs.get("final_bits", 0)) | CR_DQ_BIT` only needed a `.get` fallback
+  because nothing ever set the key. It picked up the fix for free.
+- **`reduce_pj011646.py` does not exist** anywhere in PyAutoReduce. The issue's
+  claim that it "carries a documented monkeypatch workaround until this ships"
+  was stale — there was nothing to unwind.
+
+## STILL OWED — the control test gates leg 1's defaults
+
+**Not run.** It needs `drizzlepac`, the CRDS cache (`scripts/cache/crds/`, which
+is gitignored) and archive data; the cloud session that wrote this had none of
+them. #65 asked for it *first*, and it remains the gate before these defaults
+reach a release:
+
+> Re-drizzle one striped SLACS target at the old `0` and at the MDRIZTAB value,
+> diff the weight and noise maps. If the stripes do not move, the cause is
+> elsewhere — exposure count, dither geometry, or genuine bad columns — and
+> **revert the adapter defaults rather than shipping the dial.**
+
+Leg 2's diagnostic now scores that comparison objectively instead of by eye, and
+is also what calibrates its own provisional 0.9 limit (derivation: one lost
+exposure of N leaves weight `(N-1)/N`, so 0.9 catches a single loss for any
+N <= 9 — the regime where `sqrt(N/(N-1))` noise inflation is visible). The
+obligation is recorded in `docs/design/hst_acs_pipeline.md` stage 3, not only
+here. The leg-2 verdict is **recorded, never raised**, precisely because the
+limit is uncalibrated.
+
+## Follow-ups
+
+- `draft/research/pyautoreduce/acceptance_noise_rebaseline.md` must run **after**
+  this — the bits move the IVM weights and therefore the noise maps, so its
+  SLACS parity numbers would need redoing.
+- `autoreduce_workspace` may want to surface the new dial; both `TargetSpec`
+  fields are additive with `None` defaults, so nothing there breaks.
+- Related but deliberately untouched: PyAutoReduce#61 (driz_cr erodes flux at
+  steep-gradient cores) and #62 (tier-1 ePSF from the CR-rejected mosaic). All
+  three concern how DQ/CR masking degrades products, but they touch different
+  stages.
+
+## Original prompt
+
 # HST needs a DQ-bits dial — we mask every DQ-flagged pixel where STScI keeps most of them
 
 Type: bug

complete/index.md

@@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema:
 only then grep a dated bucket. Curators: edit the band between the CURATED
 markers; everything below GENERATED is rebuilt.
 
-930 records across 7 buckets.
+931 records across 7 buckets.
 
 <!-- CURATED:START -->
 ## Highlights
@@ -31,6 +31,7 @@ _(curate hard-won records here — survives regeneration.)_
 - [health-conductor-stale-verdict](2026/08/health-conductor-stale-verdict.md)
 - [howto-smoke-all-tutorials](2026/08/howto-smoke-all-tutorials.md)
 - [howtofit-stub-fork-cleanup](2026/08/howtofit-stub-fork-cleanup.md)
+- [hst-dq-bits-dial](2026/08/hst-dq-bits-dial.md)
 - [hygiene-coverage-drift](2026/08/hygiene-coverage-drift.md)
 - [hygiene-detail-flag](2026/08/hygiene-detail-flag.md)
 - [hygiene-howto-refs-docstrings](2026/08/hygiene-howto-refs-docstrings.md)