fix(songstats-backfill): backoff on 429 + defer instead of churn (chat#1797 — pacing/backoff + logging)

[sweetmantech]

Addresses chat#1797 bullet 1 (pace + bounded backoff on 429) and bullet 3 (per-step observability logging). Bullet 2 (remove the quota ledger) follows in a stacked PR.

The problem (live prod, full-catalog run)

The drain called Songstats faster than its per-second limit → ~79% 429s sustained for minutes, and backfillTrackStep recorded every 429 as a quota hit → the budget gate filled with phantom hits and would halt the drain with ~400 tracks still pending while the real quota was barely used. The reclaim sweep reset failed → pending and the workflow re-429'd them → a retry storm.

Fix

• fetchSongstatsWithBackoff (new): bounded exponential backoff on 429/408/5xx — retries the same request (~15s total, within a step's duration), honoring the rate limit; returns retriesExhausted when still rejected. 200 and non-retryable statuses (404) return immediately.
• backfillTrackStep: 200 → write/done/hit; 404/other 4xx → done/hit (terminal, real request); backoff exhausted → defer: leave the row pending, record no quota hit (Songstats consumed nothing), and signal the workflow to stop. Songstats is the rate authority — no phantom 429 spend.
• songstatsBackfillWorkflow: stop the run on the first deferral (Songstats saturated; remaining rows stay pending for the next drain) instead of churning failed → reclaim → 429.
• Observability: per-step (song, outcome) and per-batch (claimed N) logs so a drain is traceable in vercel logs (#1797 bullet 3).

Notes

• Rate adaptation is via the backoff (reactive throttling — a 429'd call waits and retries, naturally pacing to Songstats' capacity). An explicit token-bucket is a possible later refinement.
• The budget gate / quota ledger still exist here but no longer inflate from 429s — bullet 2 removes them entirely in the follow-up PR.

Testing

11 new/updated unit tests (backoff: 200/404/retry-then-success/exhausted/5xx/maxMs-cap; step: success/defer/no-data/terminal). Workflows + songstats + research suite 317 green; tsc/lint clean.

---

Summary by cubic

Adds bounded exponential backoff to the Songstats backfill and defers on persistent retryable errors without spending quota, preventing phantom spend and retry storms (chat#1797). Also simplifies queue updates with a single updateSongstatsBackfillQueue(ids[], fields) for single and bulk operations.

• Bug Fixes

  • fetchSongstatsWithBackoff: 4-step exponential backoff (~15s total: 1s, 2s, 4s, 8s cap); retries only 408/429/502–504; returns attempts and retriesExhausted.
  • backfillTrackStep: 200 → write/done/hit; 404/other 4xx → done/hit; backoff exhausted → keep pending, spend 0, return deferred.
  • songstatsBackfillWorkflow: stops on a defer and releases unprocessed claimed rows back to pending; adds per-batch and final summary logs.

• Refactors

  • Extracted retryability to isRetryableStatus and reused the existing delay helper; narrowed retryable 5xx to 502–504.
  • Collapsed queue helpers into updateSongstatsBackfillQueue(ids[], fields); used by releaseClaimedRowsStep and per-row callers (no-op on empty ids).

^{Written for commit e9a1b05. Summary will update on new commits.}

Summary by CodeRabbit

• Bug Fixes
  • Added bounded exponential backoff retries for transient failures (408/429/5xx) and surfaced retry attempt counts.
  • Improved rate-limit handling by deferring excess work and returning it to the queue for later processing.
  • Enhanced quota accounting and logging, including accurate spend reporting and clearer completion vs deferred outcomes.
  • Prevented partial drains from losing claimed rows by releasing them back to pending when processing stops early.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
api	Ready	Preview	Jun 16, 2026 9:36pm

[coderabbitai]

Warning

Review limit reached

@sweetmantech, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 45 minutes and 49 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 40874f0d-9e1b-4f91-a3d2-ec0a42e100e4

📥 Commits

Reviewing files that changed from the base of the PR and between f6eed90 and e9a1b05.

⛔ Files ignored due to path filters (2)

• app/workflows/__tests__/backfillTrackStep.test.ts is excluded by !**/*.test.*, !**/__tests__/** and included by app/**
• lib/supabase/songstats_backfill_queue/__tests__/updateSongstatsBackfillQueue.test.ts is excluded by !**/*.test.*, !**/__tests__/** and included by lib/**

📒 Files selected for processing (3)

• app/workflows/backfillTrackStep.ts
• app/workflows/releaseClaimedRowsStep.ts
• lib/supabase/songstats_backfill_queue/updateSongstatsBackfillQueue.ts

📝 Walkthrough

Walkthrough

Adds fetchSongstatsWithBackoff, a bounded exponential backoff wrapper around fetchSongstats that retries on 408/429/5xx and surfaces retriesExhausted. backfillTrackStep is updated to use this wrapper, deferring queue rows back to pending when retries are exhausted, consuming no quota in that path. songstatsBackfillWorkflow stops draining early on the first deferral signal and returns a new deferred field, while releaseClaimedRowsStep returns unclaimed rows to pending for later retries.

Changes

Songstats backfill: exponential backoff and deferral

Layer / File(s)	Summary
Status classification and queue release helpers lib/songstats/isRetryableStatus.ts, lib/supabase/songstats_backfill_queue/releaseSongstatsBackfillRows.ts	New utilities: isRetryableStatus(status) returns true for 408/429/502/503/504; releaseSongstatsBackfillRows(ids) resets specified queue rows from in_progress to pending via Supabase update.
fetchSongstatsWithBackoff module lib/songstats/fetchSongstatsWithBackoff.ts	Wraps fetchSongstats with injectable-sleep exponential backoff. Retries when isRetryableStatus returns true, stopping on 200 or non-retryable statuses. Returns BackoffResult extending ProxyResult with attempts and retriesExhausted flag.
backfillTrackStep deferral and quota semantics app/workflows/backfillTrackStep.ts	Switches import to fetchSongstatsWithBackoff. Adds retriesExhausted branch that resets the queue row to pending and returns { deferred: true, hitsSpent: 0 }. Simplifies non-200 path to always spend 1 hit and mark row done. Adds success-path log for written history point count. Return type gains optional deferred field.
Workflow drain loop with deferral break and row release app/workflows/songstatsBackfillWorkflow.ts, app/workflows/releaseClaimedRowsStep.ts	Introduces deferred flag and labeled break that exits the drain loop immediately when backfillTrackStep returns result.deferred, leaving remaining claimed rows pending. Adds per-claim batch-size logging and updates summary log for rate-limited condition. Expands return object to { backfilled, failed, deferred }. New releaseClaimedRowsStep durable workflow step calls releaseSongstatsBackfillRows to conditionally return unclaimed rows. Updates header comment to document stop-on-deferral behavior.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related issues

• Songstats backfill: pace + back off on 429 instead of a local quota ledger chat#1797: This PR directly implements the "Pace + bounded exponential backoff on 429" task — fetchSongstatsWithBackoff handles rate-limit retries with configurable bounds, and the deferred: true path leaves rows pending for a later drain run, making Songstats the rate authority.

Possibly related PRs

• recoupable/api#668: Both PRs modify backfillTrackStep's non-200 status handling and queue state/ledger outcome logic.
• recoupable/api#670: Both PRs modify backfillTrackStep for handling non-200 Songstats responses and queue outcome semantics.
• recoupable/api#662: Both PRs modify the backfill drain logic in backfillTrackStep and songstatsBackfillWorkflow, with this PR extending the same code paths with bounded backoff and deferral semantics.

Suggested reviewers

• cubic-dev-ai

Poem

🎵 A 429 walked in, the queue held its breath,
Backoff said "easy now, let's not risk death."
Exponential delays, a sleep and a wait,
retriesExhausted? Then defer — it can wait!
The drain loop breaks early, rows left in pending,
Clean contracts, SOLID code — beauty unending. ✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Solid & Clean Code	⚠️ Warning	PR violates SRP: new function releaseSongstatsBackfillRows added to existing updateSongstatsBackfillQueue.ts instead of its own file. Per check: "each file must export ONE primary function" wit...	Create separate file lib/supabase/songstats_backfill_queue/releaseSongstatsBackfillRows.ts and move the releaseSongstatsBackfillRows function there. Update imports in releaseClaimedRowsStep.ts accordingly.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/songstats-backfill-backoff-pacing

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (2)

app/workflows/backfillTrackStep.ts (1)

24-81: ⚡ Quick win

Split backfillTrackStep into focused helpers.

This function now combines multiple responsibilities (API/backoff outcome handling, queue transitions, quota writes, payload parsing/mapping, and persistence). Please extract branch-specific handlers to keep the orchestration function small and easier to validate.

As per coding guidelines, “Flag functions longer than 20 lines,” “Keep functions small and focused,” and “Write self-documenting code.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/workflows/backfillTrackStep.ts` around lines 24 - 81, The
backfillTrackStep function has grown to 58 lines and combines multiple
responsibilities: handling API/backoff outcomes, managing queue transitions,
writing quota records, parsing payloads, and persisting data. Extract the three
main branches into separate focused helper functions: one for handling
rate-limited results (the retriesExhausted case), one for handling error
responses (non-200 status codes), and one for handling successful data
processing (parsing the historic stats, mapping to measurements, and upserting).
Have each helper function encapsulate its complete logic including logging,
queue updates, quota writes, and return a consistent response shape. Then
refactor backfillTrackStep to simply call the appropriate helper based on the
result status, making it a thin orchestration layer that fits within the 20-line
guideline.

Source: Coding guidelines

app/workflows/songstatsBackfillWorkflow.ts (1)

16-48: ⚡ Quick win

Refactor the workflow function into smaller units.

This workflow function mixes batch claiming, row processing, budget accounting, stop conditions, and summary logging in one block. Extracting row-processing and termination handling into helpers will reduce cognitive load and regression risk.

As per coding guidelines, “Flag functions longer than 20 lines” and “Keep functions small and focused.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/workflows/songstatsBackfillWorkflow.ts` around lines 16 - 48, The
songstatsBackfillWorkflow function exceeds 20 lines and mixes multiple concerns:
batch claiming, row iteration, budget accounting, deferred-state handling, and
logging. Extract the row processing loop (the for loop iterating over rows and
updating budget/backfilled/failed counters) into a separate step function, and
extract the termination and logging logic into another helper function. This
will make the main workflow function shorter and more focused on orchestration
while delegating row processing details to specialized functions.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@app/workflows/songstatsBackfillWorkflow.ts`:
- Around line 29-35: When a row returns deferred and triggers the break drain
statement in the loop, the remaining rows that have been claimed but not yet
processed stay in an in_progress state instead of being reset to pending. Before
executing the break drain in the backfillTrackStep iteration, ensure all
remaining rows in the rows collection that haven't been processed yet are reset
from in_progress back to pending state so they can be properly retried on the
next drain execution instead of waiting for stale-reclaim logic to eventually
process them.

---

Nitpick comments:
In `@app/workflows/backfillTrackStep.ts`:
- Around line 24-81: The backfillTrackStep function has grown to 58 lines and
combines multiple responsibilities: handling API/backoff outcomes, managing
queue transitions, writing quota records, parsing payloads, and persisting data.
Extract the three main branches into separate focused helper functions: one for
handling rate-limited results (the retriesExhausted case), one for handling
error responses (non-200 status codes), and one for handling successful data
processing (parsing the historic stats, mapping to measurements, and upserting).
Have each helper function encapsulate its complete logic including logging,
queue updates, quota writes, and return a consistent response shape. Then
refactor backfillTrackStep to simply call the appropriate helper based on the
result status, making it a thin orchestration layer that fits within the 20-line
guideline.

In `@app/workflows/songstatsBackfillWorkflow.ts`:
- Around line 16-48: The songstatsBackfillWorkflow function exceeds 20 lines and
mixes multiple concerns: batch claiming, row iteration, budget accounting,
deferred-state handling, and logging. Extract the row processing loop (the for
loop iterating over rows and updating budget/backfilled/failed counters) into a
separate step function, and extract the termination and logging logic into
another helper function. This will make the main workflow function shorter and
more focused on orchestration while delegating row processing details to
specialized functions.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 723add2e-9a14-4d01-a244-ac9fa8699753

📥 Commits

Reviewing files that changed from the base of the PR and between 158a1d4 and c250b34.

⛔ Files ignored due to path filters (2)

• app/workflows/__tests__/backfillTrackStep.test.ts is excluded by !**/*.test.*, !**/__tests__/** and included by app/**
• lib/songstats/__tests__/fetchSongstatsWithBackoff.test.ts is excluded by !**/*.test.*, !**/__tests__/** and included by lib/**

📒 Files selected for processing (3)

• app/workflows/backfillTrackStep.ts
• app/workflows/songstatsBackfillWorkflow.ts
• lib/songstats/fetchSongstatsWithBackoff.ts

[cubic-dev-ai]

5 issues found across 5 files

Confidence score: 3/5

• In app/workflows/songstatsBackfillWorkflow.ts, using break drain can leave claimed rows stuck in in_progress, so work may be delayed until stale-row reclaim instead of being promptly retried — switch to the intended defer path so claimed rows are released/requeued before merging.
• In lib/songstats/fetchSongstatsWithBackoff.ts, retrying on every 500 risks treating permanent local/server-side logic failures as transient throttling, which can hide real faults and keep bad rows cycling — narrow retryable statuses to clearly transient upstream gateway/service errors.
• Also in lib/songstats/fetchSongstatsWithBackoff.ts, the default retry/backoff window is ~23s rather than the documented ~15s, so a single deferred row can block workflow progress longer than expected — reduce default retries/delays or update the documented/default budget so step timing is bounded.
• In app/workflows/__tests__/backfillTrackStep.test.ts, the weakened 200-path assertion no longer validates the exact upsert payload, increasing the chance of silent schema/value regressions reaching production — restore argument-level expectations for dates/values/metric/data_source before merging.

Architecture diagram

sequenceDiagram
    participant Workflow as songstatsBackfillWorkflow
    participant Step as backfillTrackStep
    participant Backoff as fetchSongstatsWithBackoff
    participant Songstats as Songstats API
    participant DB as Database

    Note over Workflow,DB: Songstats Backfill Drain (chat#1797 pacing/backoff)

    Workflow->>Workflow: Claim up to BATCH_SIZE rows (SKIP LOCKED)
    Workflow->>Step: backfillTrackStep(row)

    Step->>Backoff: fetchSongstatsWithBackoff(tracks/historic_stats, {isrc, source})

    rect rgb(240, 240, 255)
        Note over Backoff,Songstats: Bounded Exponential Backoff (up to ~15s total)
        Backoff->>Songstats: GET historic_stats
        alt success (200)
            Songstats-->>Backoff: 200 + data
            Backoff-->>Step: {status:200, attempts:1, retriesExhausted:false}
        else transient (408/429/5xx)
            loop up to maxRetries (default 5)
                Backoff->>Backoff: sleep(min(maxMs, baseMs * 2^retry))
                Backoff->>Songstats: retry same request
                alt eventually succeeds
                    Songstats-->>Backoff: 200
                    Backoff-->>Step: {status:200, attempts:N, retriesExhausted:false}
                end
            end
            alt exhausted (still 429/5xx)
                Backoff-->>Step: {status:429/5xx, retriesExhausted:true}
            end
        else definitive (404/other 4xx)
            Songstats-->>Backoff: 404/403/4xx
            Backoff-->>Step: {status:4xx, attempts:1, retriesExhausted:false}
        end
    end

    alt 200 (data written)
        Step->>Step: Parse history to measurements
        Step->>DB: upsertSongMeasurements([{song, platform, metric, value, captured_at}])
        Step->>DB: insertSongstatsQuotaLedger({hits:1, purpose:"backfill <isrc>"})
        Step->>DB: updateSongstatsBackfillQueue(id, {status:"done"})
        Step-->>Workflow: {ok:true, hitsSpent:1}
        Workflow->>Workflow: backfilled += 1
    else retriesExhausted (deferred)
        Step->>DB: updateSongstatsBackfillQueue(id, {status:"pending"})
        Note over Step: No quota hit recorded (Songstats consumed nothing)
        Step-->>Workflow: {ok:false, hitsSpent:0, deferred:true}
        Workflow->>Workflow: break drain loop
    else definitive error (404/other 4xx)
        Step->>DB: insertSongstatsQuotaLedger({hits:1, purpose:"backfill <isrc> (no data/terminal status)"})
        Step->>DB: updateSongstatsBackfillQueue(id, {status:"done"})
        Step-->>Workflow: {ok:false, hitsSpent:1}
        Workflow->>Workflow: failed += 1
    end

    Workflow->>Workflow: Console log per-claim and final summary

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[sweetmantech]

Preview verification of the #1797 "Done when" criteria

Peer test pass on this PR. Heads-up on method: I could not fire a live drain against the preview. The workflow's only two HTTP triggers are both credential-gated and neither is reachable from my side:

• GET /api/internal/playcount-maintenance needs CRON_SECRET, which is a Sensitive Vercel var — vercel env pull returns it blank (the deployment 401s my empty-secret call, confirming the value is set but unreadable).
• POST /api/research/measurement-jobs needs a valid API key, but prod-minted keys 401 on the preview's separate auth env — verified the same RECOUP_API_KEY returns 200 on prod /api/tasks and 401 on the preview. (Preview does share the prod Supabase godremdqwajrwazhbrue, so a trigger would hit the real queue — it's auth, not DB, that blocks me.)

Per the maintainer's call, I verified the criteria against the code paths + unit suite rather than firing a real drain against the shared prod queue. The preview deploy (api-6nwyo51h1-recoup, Ready) builds clean and its runtime log stream is readable (vercel logs … --scope recoup), so the logging below will surface there on the next cron/manual drain.

Unit suite on this branch: 22 passed (fetchSongstatsWithBackoff 6, backfillTrackStep 4, plus existing workflow/step tests) — pnpm test lib/songstats app/workflows.

---

Bullet 1 — Pace + bounded exponential backoff on 429

"Done when"	Status	Evidence
Per-10s 429 rate near-zero	⚠️ Mechanism verified, metric not measured live	The 79% rate came from hammer-then-bounce with zero wait. Every call now routes through fetchSongstatsWithBackoff, which on a 429 sleeps min(8s, 1s·2^n) and retries the same request (≤5×), so a transient 429 is absorbed into a later 200 and never recorded as a reject. Test: "backs off and retries on 429, succeeding on a later attempt" (waits 100ms→200ms, returns 200). The empirical per-10s rate needs a live drain — not captured this pass.
A batch drains without churning rows through failed → reclaim	✅ Verified (code + tests)	backfillTrackStep no longer writes failed at all: 200/terminal-4xx → done, backoff-exhausted → pending (deferred). With no row ever set to failed, the failed → reclaim → immediate re-attempt tight loop is structurally impossible. Tests: 200→done, 404→done (terminal), 403→done (terminal).
A hard quota wall stops the run gracefully (rows stay pending) instead of spinning	✅ Verified (code + tests)	On retriesExhausted, the step returns { deferred: true } and leaves the row pending; the workflow break drains and returns { deferred: true }, so the rest stay pending for the next run. Test: "DEFERS (pending, signals stop) when backoff is exhausted on 429" + the if (result.deferred) break drain path in songstatsBackfillWorkflow.

Bullet 3 — Per-step observability logging

"Done when"	Status	Evidence
A drain run is traceable in Vercel runtime logs	✅ Plumbing verified, live emission not captured	Per-step: [backfill] <isrc> done (N points written) / done (no data 404 | terminal <status>) / deferred (rate-limited <status> after <n> tries). Per-batch: [songstats-backfill] claimed <n> rows and a final done: <x> backfilled, <y> terminal[, deferred (rate-limited)]. These are console.log → Vercel runtime logs; I confirmed the preview's runtime log stream is readable, so they'll appear on the next drain. Not captured live this pass.

---

Note for reviewers

The local quota ledger + budget gate are intentionally still present in this PR (getBackfillBudgetStep, insertSongstatsQuotaLedger) — they're removed in the stacked follow-up #674 (bullet 2), with the table drop in recoupable/database#35. So nothing here should look like a half-removed ledger.

Net: the two structural criteria that don't need live traffic (no-failed churn, graceful defer-on-saturation) are verified by code + tests. The two that are inherently empirical (the per-10s 429 rate, and a real log excerpt) are left as a quick confirmation on the next cron drain — happy to capture them if you trigger one, or I can if you drop the cron call.

[sweetmantech]

SRP

• actual: releaseSongstatsBackfillRows
• required: updateSongstatsBackfillQueue.ts

[sweetmantech]

Done — moved releaseSongstatsBackfillRows into updateSongstatsBackfillQueue.ts (alongside updateSongstatsBackfillQueue + reclaimStaleSongstatsBackfillRows), and relocated its tests into updateSongstatsBackfillQueue.test.ts. Standalone file + test deleted. 805bef4

[sweetmantech]

Review feedback addressed (f6eed90 + 805bef4)

Thanks all — every thread resolved below. Tests stay green (24 in the touched suites; the workflow now has its own test).

Human review (@sweetmantech)

• SRP — isRetryable → own file: extracted to lib/songstats/isRetryableStatus.ts (single export).
• SRP — realSleep → own file: dropped the inline helper and reused the existing lib/time/delay.ts (already documented for retry loops + "use step" code) — no new file needed.
• SRP — releaseSongstatsBackfillRows belongs in updateSongstatsBackfillQueue.ts: moved it there alongside updateSongstatsBackfillQueue + reclaimStaleSongstatsBackfillRows, tests relocated into updateSongstatsBackfillQueue.test.ts, standalone file deleted.

Bots

• P1 (CodeRabbit + cubic) — break drain strands the rest of the claimed batch in in_progress: valid and fixed. New releaseClaimedRowsStep returns the unprocessed remainder to pending before exiting (releaseSongstatsBackfillRows via a single .in("id", ids) update), so the next drain retries them immediately instead of waiting on the 1-hour stale-reclaim. Covered by a new songstatsBackfillWorkflow.test.ts (asserts the remainder is released and the loop stops at the defer).
• P1 (cubic) — retrying 500 re-tries permanent local failures: valid. fetchSongstats maps a missing API key and any fetch failure to 500, so retryable 5xx is now narrowed to the transient gateway codes 502/503/504 (plus 408/429). New isRetryableStatus test pins this (500/501 → not retried).
• P2 (cubic) — default backoff budget is 23s, not ~15s: valid. Dropped the default to 4 retries → 1+2+4+8 = 15s, matching the doc.
• P2 (cubic) — 200 test lost the exact-payload assertion: restored — the test asserts the full measurement row (date→captured_at, value, metric, data_source, raw_ref).
• P3 (cubic) — deferred log says "rate-limited" but covers 408/5xx too: changed to deferred (retryable <status> …).

The strand fix and the SRP co-location are also propagated into the stacked #674 (kept the branch consistent).

[sweetmantech]

KISS

Suggested change

	export async function releaseSongstatsBackfillRows(ids: string[]): Promise<void> {
	export async function updateSongstatsBackfillQueue(ids: string[]): Promise<void> {

[sweetmantech]

Done (e9a1b05) — collapsed to one updateSongstatsBackfillQueue(ids: string[], fields) using .in("id", ids) (works for a single id or many). Per-row callers pass [row.id]; the deferred-batch release calls updateSongstatsBackfillQueue(ids, { status: "pending" }). No-op on an empty list. Separate releaseSongstatsBackfillRows removed. Propagated to #674 too.

[cubic-dev-ai]

0 issues found across 11 files (changes from recent commits).

<sub>Requires human review: Core changes to rate-limiting and data pipelines introduce new workflow orchestration and per-row logging, increasing complexity and risk to data integrity.

Re-trigger cubic</sub>