store: Create postponed indexes in the background#6608
Open
lutter wants to merge 2 commits into
Open
Conversation
The runner used to block block processing while postponed indexes were created. Since these are built with CREATE INDEX CONCURRENTLY and can take a long time, spawn the work in a background task so indexing continues. Errors are only logged; the postponed_indexes_created flag is set only on success, so a failed run is retried the next time the deployment starts.
A CREATE INDEX CONCURRENTLY that is interrupted (e.g. by a node restart) leaves an invalid index behind. Because postponed indexes are created with 'if not exists', such a leftover was skipped on retry and never rebuilt, silently leaving the deployment without that index. Before creating each postponed index, drop any invalid remnant so the retry rebuilds it, mirroring what create_manual_index already does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a deployment is first created, creation of some attribute indexes is postponed until the subgraph nears the chain head. That work was previously run inline on the block-processing path: the runner
awaitedcreate_postponed_indexes, so indexing stalled while the (potentially long-running)CREATE INDEX CONCURRENTLYstatements executed.This PR moves index creation into a background task so block processing continues uninterrupted, and hardens the retry path against invalid indexes left behind by interrupted runs.
A
CREATE INDEX CONCURRENTLYthat is interrupted (e.g. by a node restart while the build is in flight) leaves behind an invalid index in Postgres. Because postponed indexes are created withIF NOT EXISTS, that leftover would be skipped on retry and never rebuilt, silently leaving the deployment without the index once the flag eventually flips totrue.Index-creation failures are now logged rather than propagated. Previously a failure errored out the subgraph (forcing a restart-and-retry); now the subgraph keeps indexing and the work is retried on the next deployment start (the flag stays
falseuntil success).