From 3a84b62fde040a787eb868f846380819c656431e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 7 Aug 2026 20:24:01 +0000 Subject: [PATCH 1/3] =?UTF-8?q?prompt:=20claim=20hst-dq-bits-dial=20(#65)?= =?UTF-8?q?=20=E2=86=92=20library-dev=20on=20PyAutoReduce?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- active.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/active.md b/active.md index 96bc6e3b..7655ae5f 100644 --- a/active.md +++ b/active.md @@ -65,13 +65,21 @@ ## hst-dq-bits-dial - issue: https://github.com/PyAutoLabs/PyAutoReduce/issues/65 -- status: issued, UNCLAIMED — filed 2026-08-04 for Fable to pick up later. No worktree, no branch, no local work started. Whoever picks it up runs `/start_dev active/hst_dq_bits_dial.md` first to claim the repo properly. +- status: library-dev — CLAIMED 2026-08-07 by a cloud session (`web-github` environment: no worktree, the harness-designated branch `claude/hst-dq-bits-dial-kl50lb` on the PyAutoReduce clone is the working branch). Baseline suite green before edits: 265 passed, 15 skipped. +- branch: claude/hst-dq-bits-dial-kl50lb (PyAutoReduce) +- decisions-2026-08-07: (a) BOTH legs implemented with the MDRIZTAB-mirroring bits as the shipped adapter defaults — `final_bits="0"` is an unintended inheritance of a drizzlepac package default, not a chosen deviation, so matching STScI is the defensible baseline; (b) leg 2's local weight-deficit guard RECORDS + flags (`acceptable` boolean beside `weight_uniformity_cutout`) and does NOT raise, because its threshold is uncalibrated and a fatal guard could refuse currently-fine reductions; (c) one PR, two commits (leg 2 then leg 1) rather than two stacked PRs, since the harness supplies one designated branch. +- control-test-NOT-RUN-HERE: the cloud session cannot run it — `drizzlepac` is not installed, `scripts/cache/crds/` is gitignored and absent, and there is no archive data. It must be run on the laptop BEFORE the new defaults reach a release; leg 2's diagnostic gives it an objective pass/fail. If the stripes do not move at the MDRIZTAB value, leg 1 is not the fix and the defaults must be reverted rather than shipped. +- corrections-to-the-issue-plan: (1) the issue's shorthand "wfc3_ir 65535 at N=1, 528 at N>=2" flattens its own MDRIZTAB table — the reference rows give IR N=2..3 as driz_sep_bits 65535 WITH final_bits 528 (the two columns differ), so the implementation carries BOTH columns in a threshold table `((1,65535,65535),(2,65535,528),(4,528,528))` mirroring MDRIZTAB row semantics; PJ011646 (5 exposures) lands on 528 either way. (2) `reduce_pj011646.py` does NOT exist in the repo (no `pj011646` reference anywhere) — the issue's "carries a documented monkeypatch workaround until this ships" is stale and there is nothing to unwind. (3) non-HST adapters must stay byte-identical, so the bits table defaults to None = emit no final_bits, leaving the `jwst_image3` and `nirc2_native` backends untouched. +- free-inheritance: `star_pass_kwargs_for` already computes `int(kwargs.get("final_bits", 0)) | CR_DQ_BIT` — that `.get(..., 0)` fallback IS the fingerprint of the missing dial. It picks up the fix automatically; its unit test asserts only the CR bit today and needs extending. +- leg2-hook: `_package` (autoreduce/pipeline.py:~648) already builds `wht_cut` and holds `center_xy`, so the local diagnostic lands there beside the existing `weight_uniformity_cutout` record — no new plumbing. - root-cause: `drizzle_kwargs_for` never sets `final_bits`/`driz_sep_bits` and no adapter does either (the strings appear nowhere in PyAutoReduce), so every reduction inherits drizzlepac's package default `final_bits="0"` — no DQ bit treated as good, every flagged pixel rejected, including the hot/warm/blob pixels calacs/calwf3 already corrected. - evidence: read from the cached CRDS MDRIZTAB reference files, so these are STScI's shipped values not recollection — ACS/WFC + WFC3/UVIS `final_bits` 65535 at numimages=1, 336 (=16+64+256) at N>=2; WFC3/IR 65535 at N=1, 528 (=512+16) at N>=2. Three consequences: 336 means hot/warm/saturated are usable and we reject all three; 528 passes exactly the blob bit that caused the PJ011646 hole (5 exposures, so that hole could not occur under standard practice); and the value is exposure-count dependent with single-exposure data at 65535 (the SLACS SNAP regime), so the dial must be N-aware and a flat constant is wrong. - two-legs: leg 1 = the N-keyed bits dial on the adapters + TargetSpec, recorded in reduction.json, unit-tested at the N boundaries, documented in both design docs. Leg 2 = a LOCAL weight-deficit guard, because both current guards are blind to this defect class: `mask_isolated_bad_pixels` tests `~isfinite | noise <= 0` so a finite reduced-weight stripe passes every check INCLUDING the 1.5" protect_radius_arcsec lens-core protection, and `weight_uniformity` is a global RMS/median (spike measured 0.066 vs the 0.2 limit) that a few columns cannot move. Leg 2 is the detector for leg 1 and should land first so the control test has an objective pass/fail. Two PRs behind one issue. - control-test: re-drizzle one striped SLACS target at `0` vs the MDRIZTAB value and diff the weight + noise maps BEFORE building the dial. If the stripes do not move the cause is elsewhere (exposure count, dither geometry, genuine bad columns) and leg 1 is not the fix. - trap: do NOT set `mdriztab=True` to get the bits. MDRIZTAB carries the whole parameter set including final_scale/final_pixfrac/final_kernel/final_rot, and hst_acs_pipeline.md stage 3 deviates from those deliberately (0.05"/pix, pixfrac 0.8, north-up); enabling it risks silently reverting the justified lensing deviations. Read the bits columns and keep our explicit kwargs. - provenance: rescoped from the WFC3/IR-only prompt filed 2026-07-10 (`wfc3_ir_dq_bits_dial.md`) after a user report that F814W SLACS-gold noise maps carry high-noise stripes through some deflector cores that the legacy SLACS/Amy reductions lack. The MDRIZTAB read showed the WFC3/IR holes and the ACS stripes are one root cause at one fix site, so one prompt and one issue rather than a duplicate. -- repo-unclaimed: PyAutoReduce is the only affected repo, listed on this single line deliberately and NOT as a 2-space ` - PyAutoReduce` sub-bullet, because worktree_check_conflict reads any such bullet as a live claim — this task holds no claim and must not block other PyAutoReduce work. +- repos: + - PyAutoReduce +- claim-note: PyAutoReduce is now a REAL claim (2-space sub-bullet above, which `worktree_check_conflict` reads as live) — it was deliberately held on a single prose line while the task sat UNCLAIMED so it would not block other PyAutoReduce work. `worktree_check_conflict hst-dq-bits-dial PyAutoReduce` exited 0 at claim time: no other active task held the repo. - ordering: `draft/research/pyautoreduce/acceptance_noise_rebaseline.md` must run AFTER this lands — a bits change moves the IVM weights and therefore the noise maps, so its parity numbers would need redoing otherwise. Related but distinct, do not fold in: #61 (driz_cr flux erosion / LACosmic) and #62 (tier-1 ePSF from the CR-rejected mosaic). - prompt: active/hst_dq_bits_dial.md From bac48760f631226390feec1fb13670d6987530e7 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 7 Aug 2026 20:31:23 +0000 Subject: [PATCH 2/3] prompt: hst-dq-bits-dial both legs implemented + pushed (#65) Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014po4zMWnxACBWxatVBMn2f --- active.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/active.md b/active.md index 7655ae5f..34e0cf32 100644 --- a/active.md +++ b/active.md @@ -72,6 +72,12 @@ - corrections-to-the-issue-plan: (1) the issue's shorthand "wfc3_ir 65535 at N=1, 528 at N>=2" flattens its own MDRIZTAB table — the reference rows give IR N=2..3 as driz_sep_bits 65535 WITH final_bits 528 (the two columns differ), so the implementation carries BOTH columns in a threshold table `((1,65535,65535),(2,65535,528),(4,528,528))` mirroring MDRIZTAB row semantics; PJ011646 (5 exposures) lands on 528 either way. (2) `reduce_pj011646.py` does NOT exist in the repo (no `pj011646` reference anywhere) — the issue's "carries a documented monkeypatch workaround until this ships" is stale and there is nothing to unwind. (3) non-HST adapters must stay byte-identical, so the bits table defaults to None = emit no final_bits, leaving the `jwst_image3` and `nirc2_native` backends untouched. - free-inheritance: `star_pass_kwargs_for` already computes `int(kwargs.get("final_bits", 0)) | CR_DQ_BIT` — that `.get(..., 0)` fallback IS the fingerprint of the missing dial. It picks up the fix automatically; its unit test asserts only the CR bit today and needs extending. - leg2-hook: `_package` (autoreduce/pipeline.py:~648) already builds `wht_cut` and holds `center_xy`, so the local diagnostic lands there beside the existing `weight_uniformity_cutout` record — no new plumbing. +- IMPLEMENTED 2026-08-07, both legs pushed to `claude/hst-dq-bits-dial-kl50lb` (PyAutoReduce), NO PR opened (none requested). Suite 281 passed / 15 skipped, up from the 265/15 baseline (+16 tests). Commits: `c514eec` leg 2, `52342b7` leg 1. +- leg2-shipped: `drizzle/diagnostics.py` gains `local_weight_deficit` + `check_local_weight_deficit` — inside the same 1.5" radius as `protect_radius_arcsec`, reports the science-region median weight and the worst row/column median as fractions of the cutout median. BOTH axes, because a detector-column defect lands on an image row or column depending on sky orientation. Recorded in reduction.json at `drizzle.local_weight_deficit`; limit 0.9 (one lost exposure of N leaves (N-1)/N, so 0.9 catches a single loss for any N<=9). RECORDS, NEVER RAISES — per the 2026-08-07 decision. The load-bearing test is `test_the_existing_guards_are_blind_to_the_same_map`: one synthetic striped map that `weight_uniformity` AND `mask_isolated_bad_pixels` both pass clean and the new guard catches. +- leg1-shipped: `InstrumentAdapter.dq_bits_rows` + `dq_bits_for(n)` with MDRIZTAB row semantics (last row whose min_exposures <= N); acs_wfc/wfc3_uvis `((1,65535,65535),(2,336,336))`, wfc3_ir `((1,65535,65535),(2,65535,528),(4,528,528))`. `TargetSpec.final_bits`/`driz_sep_bits` override at every N (validated non-negative int or None). `drizzle_kwargs_for` resolves spec > adapter > emit-nothing; keys stay ABSENT when unset, never 0, since 0 IS the bug. `dq_bits_provenance` records value + per-key source (adapter_mdriztab/target_spec/unset) + the matched row, into reduction.json `drizzle.dq_bits`. Non-AstroDrizzle backends (nircam, nirc2) declare no table and are byte-identical — pinned by test. +- free-fix-confirmed: `star_pass_kwargs_for` now ORs CR_DQ_BIT onto a real 336 instead of the `.get(...,0)` zero; test asserts `336 | 4096`. +- OWED BEFORE RELEASE — the control test. Still not run (no drizzlepac / CRDS cache / archive data in a cloud session) and it is the gate on leg 1's defaults, documented as owed in hst_acs_pipeline.md stage 3. Re-drizzle one striped SLACS target at bits 0 vs the MDRIZTAB value, diff the weight + noise maps, score with leg 2's new `local_weight_deficit`. If the stripes do NOT move, revert the adapter defaults rather than shipping the dial — the cause would then be exposure count, dither geometry or genuine bad columns. +- also-owed: run `draft/research/pyautoreduce/acceptance_noise_rebaseline.md` AFTER this lands (bits move the IVM weights, so the SLACS parity numbers need redoing), and check `autoreduce_workspace` for examples that construct TargetSpec — the two new fields are additive with None defaults so nothing breaks, but the workspace may want to show the dial. - root-cause: `drizzle_kwargs_for` never sets `final_bits`/`driz_sep_bits` and no adapter does either (the strings appear nowhere in PyAutoReduce), so every reduction inherits drizzlepac's package default `final_bits="0"` — no DQ bit treated as good, every flagged pixel rejected, including the hot/warm/blob pixels calacs/calwf3 already corrected. - evidence: read from the cached CRDS MDRIZTAB reference files, so these are STScI's shipped values not recollection — ACS/WFC + WFC3/UVIS `final_bits` 65535 at numimages=1, 336 (=16+64+256) at N>=2; WFC3/IR 65535 at N=1, 528 (=512+16) at N>=2. Three consequences: 336 means hot/warm/saturated are usable and we reject all three; 528 passes exactly the blob bit that caused the PJ011646 hole (5 exposures, so that hole could not occur under standard practice); and the value is exposure-count dependent with single-exposure data at 65535 (the SLACS SNAP regime), so the dial must be N-aware and a flat constant is wrong. - two-legs: leg 1 = the N-keyed bits dial on the adapters + TargetSpec, recorded in reduction.json, unit-tested at the N boundaries, documented in both design docs. Leg 2 = a LOCAL weight-deficit guard, because both current guards are blind to this defect class: `mask_isolated_bad_pixels` tests `~isfinite | noise <= 0` so a finite reduced-weight stripe passes every check INCLUDING the 1.5" protect_radius_arcsec lens-core protection, and `weight_uniformity` is a global RMS/median (spike measured 0.066 vs the 0.2 limit) that a few columns cannot move. Leg 2 is the detector for leg 1 and should land first so the control test has an objective pass/fail. Two PRs behind one issue. From 6cbdda148d24e43aaa3ee6062d3a7d6b40266b00 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 7 Aug 2026 21:05:22 +0000 Subject: [PATCH 3/3] prompt: hst-dq-bits-dial shipped (PyAutoReduce#70 merged) Advances active/hst_dq_bits_dial.md -> complete/2026/08/hst-dq-bits-dial.md via lifecycle.py record, removes the active.md section, refreshes the index. `lifecycle.py check` OK. The record keeps the findings that cost the most to establish: the bits are genuinely N-dependent (single-exposure uses 65535, which independently explains the clean legacy SLACS SNAP maps); the two bits columns DIFFER for wfc3_ir at N=2-3, so rows carry both; unset must leave the key ABSENT rather than 0, because 0 IS drizzlepac's "no bit is good" default and was the bug; and mdriztab=True must not be used to obtain them. Records the standing debt in full: the control test gating leg 1's defaults has NOT been run (no drizzlepac / CRDS cache / archive data in a cloud session) and, if the stripes do not move, the adapter defaults are to be reverted rather than shipped. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014po4zMWnxACBWxatVBMn2f --- active.md | 27 ----- .../2026/08/hst-dq-bits-dial.md | 114 ++++++++++++++++++ complete/index.md | 3 +- 3 files changed, 116 insertions(+), 28 deletions(-) rename active/hst_dq_bits_dial.md => complete/2026/08/hst-dq-bits-dial.md (56%) diff --git a/active.md b/active.md index 34e0cf32..ea91fe3f 100644 --- a/active.md +++ b/active.md @@ -62,30 +62,3 @@ - artifacts-are-laptop-only: Actions artifact downloads are blocked from cloud/mobile sessions (egress policy 403s `productionresultssa2.blob.core.windows.net` on CONNECT) — this is what stopped the cloud session finishing the ingest. Both wiki drift reports were captured to `~/.pyauto-heart/release_20260807_wiki_drift/` while on the laptop. - do-not: do NOT use the nightly driver for a manual release — AUTONOMY.md forbids converting a manual release into the scheduled-nightly exception. - repos-none-claimed: this entry claims NO repos — deliberately on one line, NOT as 2-space ` - Repo` bullets, because `worktree_check_conflict` treats any such bullet as a live claim. - -## hst-dq-bits-dial -- issue: https://github.com/PyAutoLabs/PyAutoReduce/issues/65 -- status: library-dev — CLAIMED 2026-08-07 by a cloud session (`web-github` environment: no worktree, the harness-designated branch `claude/hst-dq-bits-dial-kl50lb` on the PyAutoReduce clone is the working branch). Baseline suite green before edits: 265 passed, 15 skipped. -- branch: claude/hst-dq-bits-dial-kl50lb (PyAutoReduce) -- decisions-2026-08-07: (a) BOTH legs implemented with the MDRIZTAB-mirroring bits as the shipped adapter defaults — `final_bits="0"` is an unintended inheritance of a drizzlepac package default, not a chosen deviation, so matching STScI is the defensible baseline; (b) leg 2's local weight-deficit guard RECORDS + flags (`acceptable` boolean beside `weight_uniformity_cutout`) and does NOT raise, because its threshold is uncalibrated and a fatal guard could refuse currently-fine reductions; (c) one PR, two commits (leg 2 then leg 1) rather than two stacked PRs, since the harness supplies one designated branch. -- control-test-NOT-RUN-HERE: the cloud session cannot run it — `drizzlepac` is not installed, `scripts/cache/crds/` is gitignored and absent, and there is no archive data. It must be run on the laptop BEFORE the new defaults reach a release; leg 2's diagnostic gives it an objective pass/fail. If the stripes do not move at the MDRIZTAB value, leg 1 is not the fix and the defaults must be reverted rather than shipped. -- corrections-to-the-issue-plan: (1) the issue's shorthand "wfc3_ir 65535 at N=1, 528 at N>=2" flattens its own MDRIZTAB table — the reference rows give IR N=2..3 as driz_sep_bits 65535 WITH final_bits 528 (the two columns differ), so the implementation carries BOTH columns in a threshold table `((1,65535,65535),(2,65535,528),(4,528,528))` mirroring MDRIZTAB row semantics; PJ011646 (5 exposures) lands on 528 either way. (2) `reduce_pj011646.py` does NOT exist in the repo (no `pj011646` reference anywhere) — the issue's "carries a documented monkeypatch workaround until this ships" is stale and there is nothing to unwind. (3) non-HST adapters must stay byte-identical, so the bits table defaults to None = emit no final_bits, leaving the `jwst_image3` and `nirc2_native` backends untouched. -- free-inheritance: `star_pass_kwargs_for` already computes `int(kwargs.get("final_bits", 0)) | CR_DQ_BIT` — that `.get(..., 0)` fallback IS the fingerprint of the missing dial. It picks up the fix automatically; its unit test asserts only the CR bit today and needs extending. -- leg2-hook: `_package` (autoreduce/pipeline.py:~648) already builds `wht_cut` and holds `center_xy`, so the local diagnostic lands there beside the existing `weight_uniformity_cutout` record — no new plumbing. -- IMPLEMENTED 2026-08-07, both legs pushed to `claude/hst-dq-bits-dial-kl50lb` (PyAutoReduce), NO PR opened (none requested). Suite 281 passed / 15 skipped, up from the 265/15 baseline (+16 tests). Commits: `c514eec` leg 2, `52342b7` leg 1. -- leg2-shipped: `drizzle/diagnostics.py` gains `local_weight_deficit` + `check_local_weight_deficit` — inside the same 1.5" radius as `protect_radius_arcsec`, reports the science-region median weight and the worst row/column median as fractions of the cutout median. BOTH axes, because a detector-column defect lands on an image row or column depending on sky orientation. Recorded in reduction.json at `drizzle.local_weight_deficit`; limit 0.9 (one lost exposure of N leaves (N-1)/N, so 0.9 catches a single loss for any N<=9). RECORDS, NEVER RAISES — per the 2026-08-07 decision. The load-bearing test is `test_the_existing_guards_are_blind_to_the_same_map`: one synthetic striped map that `weight_uniformity` AND `mask_isolated_bad_pixels` both pass clean and the new guard catches. -- leg1-shipped: `InstrumentAdapter.dq_bits_rows` + `dq_bits_for(n)` with MDRIZTAB row semantics (last row whose min_exposures <= N); acs_wfc/wfc3_uvis `((1,65535,65535),(2,336,336))`, wfc3_ir `((1,65535,65535),(2,65535,528),(4,528,528))`. `TargetSpec.final_bits`/`driz_sep_bits` override at every N (validated non-negative int or None). `drizzle_kwargs_for` resolves spec > adapter > emit-nothing; keys stay ABSENT when unset, never 0, since 0 IS the bug. `dq_bits_provenance` records value + per-key source (adapter_mdriztab/target_spec/unset) + the matched row, into reduction.json `drizzle.dq_bits`. Non-AstroDrizzle backends (nircam, nirc2) declare no table and are byte-identical — pinned by test. -- free-fix-confirmed: `star_pass_kwargs_for` now ORs CR_DQ_BIT onto a real 336 instead of the `.get(...,0)` zero; test asserts `336 | 4096`. -- OWED BEFORE RELEASE — the control test. Still not run (no drizzlepac / CRDS cache / archive data in a cloud session) and it is the gate on leg 1's defaults, documented as owed in hst_acs_pipeline.md stage 3. Re-drizzle one striped SLACS target at bits 0 vs the MDRIZTAB value, diff the weight + noise maps, score with leg 2's new `local_weight_deficit`. If the stripes do NOT move, revert the adapter defaults rather than shipping the dial — the cause would then be exposure count, dither geometry or genuine bad columns. -- also-owed: run `draft/research/pyautoreduce/acceptance_noise_rebaseline.md` AFTER this lands (bits move the IVM weights, so the SLACS parity numbers need redoing), and check `autoreduce_workspace` for examples that construct TargetSpec — the two new fields are additive with None defaults so nothing breaks, but the workspace may want to show the dial. -- root-cause: `drizzle_kwargs_for` never sets `final_bits`/`driz_sep_bits` and no adapter does either (the strings appear nowhere in PyAutoReduce), so every reduction inherits drizzlepac's package default `final_bits="0"` — no DQ bit treated as good, every flagged pixel rejected, including the hot/warm/blob pixels calacs/calwf3 already corrected. -- evidence: read from the cached CRDS MDRIZTAB reference files, so these are STScI's shipped values not recollection — ACS/WFC + WFC3/UVIS `final_bits` 65535 at numimages=1, 336 (=16+64+256) at N>=2; WFC3/IR 65535 at N=1, 528 (=512+16) at N>=2. Three consequences: 336 means hot/warm/saturated are usable and we reject all three; 528 passes exactly the blob bit that caused the PJ011646 hole (5 exposures, so that hole could not occur under standard practice); and the value is exposure-count dependent with single-exposure data at 65535 (the SLACS SNAP regime), so the dial must be N-aware and a flat constant is wrong. -- two-legs: leg 1 = the N-keyed bits dial on the adapters + TargetSpec, recorded in reduction.json, unit-tested at the N boundaries, documented in both design docs. Leg 2 = a LOCAL weight-deficit guard, because both current guards are blind to this defect class: `mask_isolated_bad_pixels` tests `~isfinite | noise <= 0` so a finite reduced-weight stripe passes every check INCLUDING the 1.5" protect_radius_arcsec lens-core protection, and `weight_uniformity` is a global RMS/median (spike measured 0.066 vs the 0.2 limit) that a few columns cannot move. Leg 2 is the detector for leg 1 and should land first so the control test has an objective pass/fail. Two PRs behind one issue. -- control-test: re-drizzle one striped SLACS target at `0` vs the MDRIZTAB value and diff the weight + noise maps BEFORE building the dial. If the stripes do not move the cause is elsewhere (exposure count, dither geometry, genuine bad columns) and leg 1 is not the fix. -- trap: do NOT set `mdriztab=True` to get the bits. MDRIZTAB carries the whole parameter set including final_scale/final_pixfrac/final_kernel/final_rot, and hst_acs_pipeline.md stage 3 deviates from those deliberately (0.05"/pix, pixfrac 0.8, north-up); enabling it risks silently reverting the justified lensing deviations. Read the bits columns and keep our explicit kwargs. -- provenance: rescoped from the WFC3/IR-only prompt filed 2026-07-10 (`wfc3_ir_dq_bits_dial.md`) after a user report that F814W SLACS-gold noise maps carry high-noise stripes through some deflector cores that the legacy SLACS/Amy reductions lack. The MDRIZTAB read showed the WFC3/IR holes and the ACS stripes are one root cause at one fix site, so one prompt and one issue rather than a duplicate. -- repos: - - PyAutoReduce -- claim-note: PyAutoReduce is now a REAL claim (2-space sub-bullet above, which `worktree_check_conflict` reads as live) — it was deliberately held on a single prose line while the task sat UNCLAIMED so it would not block other PyAutoReduce work. `worktree_check_conflict hst-dq-bits-dial PyAutoReduce` exited 0 at claim time: no other active task held the repo. -- ordering: `draft/research/pyautoreduce/acceptance_noise_rebaseline.md` must run AFTER this lands — a bits change moves the IVM weights and therefore the noise maps, so its parity numbers would need redoing otherwise. Related but distinct, do not fold in: #61 (driz_cr flux erosion / LACosmic) and #62 (tier-1 ePSF from the CR-rejected mosaic). -- prompt: active/hst_dq_bits_dial.md diff --git a/active/hst_dq_bits_dial.md b/complete/2026/08/hst-dq-bits-dial.md similarity index 56% rename from active/hst_dq_bits_dial.md rename to complete/2026/08/hst-dq-bits-dial.md index 3b285aeb..d4846c4a 100644 --- a/active/hst_dq_bits_dial.md +++ b/complete/2026/08/hst-dq-bits-dial.md @@ -1,3 +1,117 @@ +- issue: https://github.com/PyAutoLabs/PyAutoReduce/issues/65 +- library-pr: https://github.com/PyAutoLabs/PyAutoReduce/pull/70 (merge 99558128e) +- mind-pr: https://github.com/PyAutoLabs/PyAutoMind/pull/148 +- shipped: 2026-08-07 +- repos: PyAutoReduce +- tests: 281 passed / 15 skipped, up from the 265/15 baseline (+16). CI green on unittest 3.12 + 3.13. + +Every HST reduction the pipeline had ever produced rejected **every** DQ-flagged +pixel. `drizzle_kwargs_for` never set `final_bits`/`driz_sep_bits`, and no +adapter did either — the strings appeared nowhere in the repo — so every run +silently inherited drizzlepac's package default `final_bits="0"` +(`drizzlepac/pars/astrodrizzle.cfg:101`), which treats no bit as good. Hot, warm +and blob pixels that `calacs`/`calwf3` had *already corrected* were thrown away. +An unintended inheritance, never a chosen deviation, and it had been shipping +since phase 1. + +Two visible symptoms, one root cause, one fix site: zero-coverage **holes** on +WFC3/IR (PJ011646, DQ 512 blobs at the same detector pixels in all five +exposures) and high-noise **stripes through deflector cores** on ACS/WFC F814W +SLACS-gold that the legacy SLACS reductions lack. + +## What shipped + +**Leg 2 first, deliberately — it is the detector for leg 1.** +`drizzle/diagnostics.py` gains `local_weight_deficit` / `check_local_weight_deficit`: +inside the same 1.5" radius as `protect_radius_arcsec`, it reports the +science-region median weight and the worst row/column median as fractions of the +cutout median, recorded in `reduction.json` at `drizzle.local_weight_deficit`. +Both axes, because a detector-column defect lands on an image row or column +depending on the frame's orientation on the sky. + +**Leg 1** puts STScI's own MDRIZTAB rows on the adapters as +`(min_exposures, driz_sep_bits, final_bits)` with MDRIZTAB's semantics (the last +row whose `min_exposures <= N`): + + acs_wfc, wfc3_uvis: (1, 65535, 65535), (2, 336, 336) + wfc3_ir: (1, 65535, 65535), (2, 65535, 528), (4, 528, 528) + +`336 = 16+64+256` (hot, warm, saturated); `528 = 512+16` (blob, hot). +`TargetSpec.final_bits`/`driz_sep_bits` override at every N; +`dq_bits_provenance` records each value **and** its source +(`adapter_mdriztab` / `target_spec` / `unset`) so datasets stay re-derivable as +the tables move. + +## Traps and findings worth keeping + +- **The bits are genuinely N-dependent; a flat constant is wrong.** + Single-exposure data uses 65535 — every bit good — because with one exposure + there is nothing to fill a masked pixel with, so the standard recipe keeps + flagged pixels rather than punching holes. This is also an independent + explanation for why the legacy SLACS SNAP maps look clean: SNAP data *is* the + N=1 regime. +- **The two bits columns DIFFER, so rows carry both.** `wfc3_ir` at N=2-3 has + `driz_sep_bits` 65535 with `final_bits` 528 — the separate (median-building) + drizzle still keeps every bit while the final drizzle is already at 528. The + issue's own shorthand ("65535 at N=1, 528 at N>=2") flattened its own + evidence table; the reference rows won. +- **Unset must mean the key is ABSENT, not 0.** `0` is precisely drizzlepac's + "no bit is good" default — writing it explicitly would re-enact the bug. The + non-AstroDrizzle backends (`jwst_image3`, `nirc2_native`) declare no table and + emit no keyword; pinned by test. +- **Do NOT use `mdriztab=True` to get the bits.** It imports the whole parameter + set — `final_scale`, `final_pixfrac`, `final_kernel`, `final_rot` — and would + silently revert the deliberate, justified lensing deviations in + `hst_acs_pipeline.md` stage 3 (0.05"/pix, pixfrac 0.8, north-up). +- **The blindness was structural, not an oversight in one guard.** + `mask_isolated_bad_pixels` tests `~isfinite | <= 0`, so a *degraded* pixel was + never even a candidate — not for the clustering check, and not for the 1.5" + protection whose entire purpose is "the lens itself must be clean"; + `weight_uniformity` is a global RMS/median (slacs0008 measured 0.066 against a + 0.2 limit) that a few columns cannot move. Between the two sat a whole defect + class. `test_the_existing_guards_are_blind_to_the_same_map` pins it: one + synthetic striped map both old guards pass and the new one catches. +- **`star_pass_kwargs_for` was carrying the fingerprint all along.** Its + `int(kwargs.get("final_bits", 0)) | CR_DQ_BIT` only needed a `.get` fallback + because nothing ever set the key. It picked up the fix for free. +- **`reduce_pj011646.py` does not exist** anywhere in PyAutoReduce. The issue's + claim that it "carries a documented monkeypatch workaround until this ships" + was stale — there was nothing to unwind. + +## STILL OWED — the control test gates leg 1's defaults + +**Not run.** It needs `drizzlepac`, the CRDS cache (`scripts/cache/crds/`, which +is gitignored) and archive data; the cloud session that wrote this had none of +them. #65 asked for it *first*, and it remains the gate before these defaults +reach a release: + +> Re-drizzle one striped SLACS target at the old `0` and at the MDRIZTAB value, +> diff the weight and noise maps. If the stripes do not move, the cause is +> elsewhere — exposure count, dither geometry, or genuine bad columns — and +> **revert the adapter defaults rather than shipping the dial.** + +Leg 2's diagnostic now scores that comparison objectively instead of by eye, and +is also what calibrates its own provisional 0.9 limit (derivation: one lost +exposure of N leaves weight `(N-1)/N`, so 0.9 catches a single loss for any +N <= 9 — the regime where `sqrt(N/(N-1))` noise inflation is visible). The +obligation is recorded in `docs/design/hst_acs_pipeline.md` stage 3, not only +here. The leg-2 verdict is **recorded, never raised**, precisely because the +limit is uncalibrated. + +## Follow-ups + +- `draft/research/pyautoreduce/acceptance_noise_rebaseline.md` must run **after** + this — the bits move the IVM weights and therefore the noise maps, so its + SLACS parity numbers would need redoing. +- `autoreduce_workspace` may want to surface the new dial; both `TargetSpec` + fields are additive with `None` defaults, so nothing there breaks. +- Related but deliberately untouched: PyAutoReduce#61 (driz_cr erodes flux at + steep-gradient cores) and #62 (tier-1 ePSF from the CR-rejected mosaic). All + three concern how DQ/CR masking degrades products, but they touch different + stages. + +## Original prompt + # HST needs a DQ-bits dial — we mask every DQ-flagged pixel where STScI keeps most of them Type: bug diff --git a/complete/index.md b/complete/index.md index 6c5eaf9c..2c29ec91 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -930 records across 7 buckets. +931 records across 7 buckets. ## Highlights @@ -31,6 +31,7 @@ _(curate hard-won records here — survives regeneration.)_ - [health-conductor-stale-verdict](2026/08/health-conductor-stale-verdict.md) - [howto-smoke-all-tutorials](2026/08/howto-smoke-all-tutorials.md) - [howtofit-stub-fork-cleanup](2026/08/howtofit-stub-fork-cleanup.md) +- [hst-dq-bits-dial](2026/08/hst-dq-bits-dial.md) - [hygiene-coverage-drift](2026/08/hygiene-coverage-drift.md) - [hygiene-detail-flag](2026/08/hygiene-detail-flag.md) - [hygiene-howto-refs-docstrings](2026/08/hygiene-howto-refs-docstrings.md)