Skip to content

fix: /v1/tracks/{id}/stems 500s hid contest stems — harden NULL handling (stems invisible since ~Jul 12 cutover) - #995

Merged
dylanjeffers merged 1 commit into
mainfrom
fix/stems-endpoint-null-orig-filename
Jul 27, 2026
Merged

fix: /v1/tracks/{id}/stems 500s hid contest stems — harden NULL handling (stems invisible since ~Jul 12 cutover)#995
dylanjeffers merged 1 commit into
mainfrom
fix/stems-endpoint-null-orig-filename

Conversation

@dylanjeffers

Copy link
Copy Markdown
Contributor

TL;DR

/v1/tracks/{id}/stems fails the entire request when any selected column scans NULL into a non-pointer Go field, and every stem indexed by the Go ETL since the ~July 12 Python→Go cutover has NULL orig_filename. The result was a silent 500 that made the stems section / ContestStemsCard blank for post-cutover uploads — most visibly Andrew Lux's "Alone" remix contest (parent JBKl7R6, 43 stems uploaded 7/20, all correct on-chain). #973 fixed the orig_filename case on main (June 23), but this PR closes the remaining NULL-scan path (stem_of wiped → category/parent_track_id NULL) and adds regression coverage.

Root cause chain

  1. The Go ETL never persists orig_filename — the entity-manager create handler doesn't extract it, so only legacy Python-indexed rows carry a value. Verified against the raw on-chain txs (core blocks 28480416–28480440): the metadata contains orig_filename and a correct stem_of, and replaying the exact tx through the vendored ETL persists stem_of fine.
  2. The endpoint scanned NULLs into non-pointer string/int fields (pgx.RowToStructByName), so one ETL-shaped row 500s the whole response — every stem on the parent disappears, not just the affected one.
  3. Fix stems endpoint with null original filenames #973 (June 23) fixed the orig_filename scan via COALESCE(orig_filename, title, ''). Prod is running 915c37d today (pods restarted 2026-07-26 08:07 UTC), which includes it — but the incident behavior on 7/20–7/25 is consistent with the pods running a pre-Fix stems endpoint with null original filenames #973 build until that 7/26 restart. Worth confirming from deploy history (kubectl --context prod -n api rollout history / image tags).
  4. Still broken until this PR: a stems join row can outlive tracks.stem_of. An explicit "stem_of": null in a client update wipes the column (clients send the full track object on edit — same failure class as the CID wipe fixed in fix(etl): don't let track updates wipe CIDs via explicit null OpenAudio/go-openaudio#410) while the stems row survives. category and parent_track_id then come back NULL from the jsonb and the request 500s again.

Fix

  • COALESCE(t.stem_of->>'category', ''), COALESCE(t.track_cid, '')
  • parent_track_id now read from the stems join key (never NULL) instead of the jsonb
  • Regression test TestGetTrackStemsWipedStemOf: seeds a stems row whose track has no stem_of/orig_filename — 500 before, 200 after

Blast radius

Every stem uploaded after the ~July 12 cutover was invisible in the UI while the NULL-scan bug was live (any parent with ≥1 ETL-shaped stem row 500'd). The DB rows themselves are intact — once the endpoint tolerates NULLs, existing stems reappear with no reindex.

Andrew Lux follow-up (support)

Andrew deleted all of his stems while debugging (one on 7/25 13:38 UTC after 9 retries, a fresh batch uploaded 7/25 14:08–14:50, then a bulk on-chain Track/Delete of everything on 7/27 12:35 UTC). Deleted tracks can't be un-deleted on-chain, so after this deploys he needs to re-upload the stems once via Edit Track → stems on "Alone" — the same flow he used before; nothing was wrong with his uploads.

Go ETL follow-ups (OpenAudio/go-openaudio — patches staged locally, separate PR)

  • Persist orig_filename on track create/update (Python parity; never-clear like the CIDs) so new rows don't rely on the title fallback
  • Ignore explicit "stem_of": null in track updates (same guard as the Fix quote math #410 CID fix) so client edits can't unlink stems
  • Upsert the stems join row when an update carries stem_of (also gives us an on-chain repair path for lost links)

🤖 Generated with Claude Code

A stems join row can outlive its track's stem_of jsonb (an explicit
"stem_of": null in a client update wipes the column without touching the
stems table — same class of bug as the CID wipe in
OpenAudio/go-openaudio#410). category and parent_track_id then come back
NULL, the row scan fails, and the endpoint 500s — hiding every stem on
the parent, not just the wiped one. This is the remaining NULL-scan
hazard after #973 covered orig_filename.

COALESCE category/track_cid and read parent_track_id from the stems join
key, which is never NULL.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dylanjeffers added a commit to OpenAudio/go-openaudio that referenced this pull request Jul 27, 2026
…#421)

Three parity/robustness fixes to entity-manager track writes, found while
investigating stems that never appeared on a remix contest page
(AudiusProject/api#995):

- Persist orig_filename on create and update (never-clear, like the
  CIDs). Only legacy Python-indexed rows carried it, and downstream
  consumers (/v1/tracks/{id}/stems) read it for stem download filenames.
- Ignore an explicit "stem_of": null on update. Clients send the full
  track object on edit, so a carried null permanently unlinked the stem
  from its parent — same failure class as the CID wipe fixed in #410. No
  edit flow legitimately detaches a stem; removing one means deleting
  the stem track.
- Upsert the stems join row when an update carries stem_of, keeping the
  join table in sync and giving an on-chain repair path for stems whose
  link was lost.

The regression test replays the exact on-chain create tx from the
incident (core block 28480422) and covers all three behaviors.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@dylanjeffers
dylanjeffers merged commit d9ca34c into main Jul 27, 2026
2 checks passed
@dylanjeffers
dylanjeffers deleted the fix/stems-endpoint-null-orig-filename branch July 27, 2026 21:54
dylanjeffers added a commit that referenced this pull request Jul 27, 2026
…audio#421) (#996)

Bumps both `github.com/OpenAudio/go-openaudio` and `.../pkg/etl` pins
from `b4a5ebe` (2026-07-16) to `1d9f697` — the merge of
**OpenAudio/go-openaudio#421**, same flow as #994 did for the #410 CID
fix.

## What the new pin brings (ETL / entity manager)

- Persist `orig_filename` on track create/update (never-clear, like the
CIDs) — new stem rows no longer rely on the `/stems` endpoint's title
fallback
- Ignore explicit `"stem_of": null` on track updates so client edits
can't unlink a stem from its parent (#410 failure class)
- Upsert the `stems` join row when an update carries `stem_of` (on-chain
repair path for lost links)

Intervening upstream commits also included (library-only for api):
mediorum transcode/retry fixes (#338, #417#419) and core/logging spam
guards (#420). Nothing release- or mainnet-config-related.

## Relation to #995

#995 is the unblocking change (endpoint tolerates NULLs already in the
DB); this bump prevents recurrence at the write path. Both trace back to
the July stems-invisible incident (Andrew Lux's "Alone" remix contest).

Verified locally: `go build ./...`, stems endpoint tests, and `go test
./indexer/...` all pass on the new pin.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant