fix: /v1/tracks/{id}/stems 500s hid contest stems — harden NULL handling (stems invisible since ~Jul 12 cutover) - #995
Merged
Conversation
A stems join row can outlive its track's stem_of jsonb (an explicit "stem_of": null in a client update wipes the column without touching the stems table — same class of bug as the CID wipe in OpenAudio/go-openaudio#410). category and parent_track_id then come back NULL, the row scan fails, and the endpoint 500s — hiding every stem on the parent, not just the wiped one. This is the remaining NULL-scan hazard after #973 covered orig_filename. COALESCE category/track_cid and read parent_track_id from the stems join key, which is never NULL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dylanjeffers
added a commit
to OpenAudio/go-openaudio
that referenced
this pull request
Jul 27, 2026
…#421) Three parity/robustness fixes to entity-manager track writes, found while investigating stems that never appeared on a remix contest page (AudiusProject/api#995): - Persist orig_filename on create and update (never-clear, like the CIDs). Only legacy Python-indexed rows carried it, and downstream consumers (/v1/tracks/{id}/stems) read it for stem download filenames. - Ignore an explicit "stem_of": null on update. Clients send the full track object on edit, so a carried null permanently unlinked the stem from its parent — same failure class as the CID wipe fixed in #410. No edit flow legitimately detaches a stem; removing one means deleting the stem track. - Upsert the stems join row when an update carries stem_of, keeping the join table in sync and giving an on-chain repair path for stems whose link was lost. The regression test replays the exact on-chain create tx from the incident (core block 28480422) and covers all three behaviors. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
dylanjeffers
added a commit
that referenced
this pull request
Jul 27, 2026
…audio#421) (#996) Bumps both `github.com/OpenAudio/go-openaudio` and `.../pkg/etl` pins from `b4a5ebe` (2026-07-16) to `1d9f697` — the merge of **OpenAudio/go-openaudio#421**, same flow as #994 did for the #410 CID fix. ## What the new pin brings (ETL / entity manager) - Persist `orig_filename` on track create/update (never-clear, like the CIDs) — new stem rows no longer rely on the `/stems` endpoint's title fallback - Ignore explicit `"stem_of": null` on track updates so client edits can't unlink a stem from its parent (#410 failure class) - Upsert the `stems` join row when an update carries `stem_of` (on-chain repair path for lost links) Intervening upstream commits also included (library-only for api): mediorum transcode/retry fixes (#338, #417–#419) and core/logging spam guards (#420). Nothing release- or mainnet-config-related. ## Relation to #995 #995 is the unblocking change (endpoint tolerates NULLs already in the DB); this bump prevents recurrence at the write path. Both trace back to the July stems-invisible incident (Andrew Lux's "Alone" remix contest). Verified locally: `go build ./...`, stems endpoint tests, and `go test ./indexer/...` all pass on the new pin. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
/v1/tracks/{id}/stemsfails the entire request when any selected column scans NULL into a non-pointer Go field, and every stem indexed by the Go ETL since the ~July 12 Python→Go cutover has NULLorig_filename. The result was a silent 500 that made the stems section / ContestStemsCard blank for post-cutover uploads — most visibly Andrew Lux's "Alone" remix contest (parentJBKl7R6, 43 stems uploaded 7/20, all correct on-chain). #973 fixed theorig_filenamecase on main (June 23), but this PR closes the remaining NULL-scan path (stem_ofwiped →category/parent_track_idNULL) and adds regression coverage.Root cause chain
orig_filename— the entity-manager create handler doesn't extract it, so only legacy Python-indexed rows carry a value. Verified against the raw on-chain txs (core blocks 28480416–28480440): the metadata containsorig_filenameand a correctstem_of, and replaying the exact tx through the vendored ETL persistsstem_offine.string/intfields (pgx.RowToStructByName), so one ETL-shaped row 500s the whole response — every stem on the parent disappears, not just the affected one.orig_filenamescan viaCOALESCE(orig_filename, title, ''). Prod is running915c37dtoday (pods restarted 2026-07-26 08:07 UTC), which includes it — but the incident behavior on 7/20–7/25 is consistent with the pods running a pre-Fix stems endpoint with null original filenames #973 build until that 7/26 restart. Worth confirming from deploy history (kubectl --context prod -n api rollout history/ image tags).stemsjoin row can outlivetracks.stem_of. An explicit"stem_of": nullin a client update wipes the column (clients send the full track object on edit — same failure class as the CID wipe fixed in fix(etl): don't let track updates wipe CIDs via explicit null OpenAudio/go-openaudio#410) while thestemsrow survives.categoryandparent_track_idthen come back NULL from the jsonb and the request 500s again.Fix
COALESCE(t.stem_of->>'category', ''),COALESCE(t.track_cid, '')parent_track_idnow read from thestemsjoin key (never NULL) instead of the jsonbTestGetTrackStemsWipedStemOf: seeds a stems row whose track has nostem_of/orig_filename— 500 before, 200 afterBlast radius
Every stem uploaded after the ~July 12 cutover was invisible in the UI while the NULL-scan bug was live (any parent with ≥1 ETL-shaped stem row 500'd). The DB rows themselves are intact — once the endpoint tolerates NULLs, existing stems reappear with no reindex.
Andrew Lux follow-up (support)
Andrew deleted all of his stems while debugging (one on 7/25 13:38 UTC after 9 retries, a fresh batch uploaded 7/25 14:08–14:50, then a bulk on-chain
Track/Deleteof everything on 7/27 12:35 UTC). Deleted tracks can't be un-deleted on-chain, so after this deploys he needs to re-upload the stems once via Edit Track → stems on "Alone" — the same flow he used before; nothing was wrong with his uploads.Go ETL follow-ups (OpenAudio/go-openaudio — patches staged locally, separate PR)
orig_filenameon track create/update (Python parity; never-clear like the CIDs) so new rows don't rely on the title fallback"stem_of": nullin track updates (same guard as the Fix quote math #410 CID fix) so client edits can't unlink stemsstemsjoin row when an update carriesstem_of(also gives us an on-chain repair path for lost links)🤖 Generated with Claude Code