fix(webapp,core): retry run resume through transient database outages

Resuming a run after a wait calls the engine's continue endpoint. When the
database was briefly unreachable, that route caught the Prisma infrastructure
error and returned a non-retryable 422, so the worker aborted the run with
TASK_EXECUTION_ABORTED over a transient blip.

The continue route now lets infrastructure errors propagate to the generic 500
handler (scrubbed and retryable), matching how the trigger path already treats
them. The worker's continue call also retries with a longer, jittered backoff
so it can ride out an outage lasting tens of seconds without stampeding the
database on recovery. Genuine validation errors still return 422.

Author: matt-aitken

.changeset/resume-retry-transient-db.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/core": patch
+---
+
+Runs resuming after a wait now survive a transient platform database outage instead of failing with `TASK_EXECUTION_ABORTED`. The worker retries the resume call generously with jittered backoff, so a brief blip while the run is being continued no longer aborts it.

apps/webapp/app/routes/engine.v1.worker-actions.runs.$runFriendlyId.snapshots.$snapshotFriendlyId.continue.ts

@@ -4,7 +4,7 @@ import type { WorkerApiContinueRunExecutionRequestBody } from "@trigger.dev/core
 import { z } from "zod";
 import { logger } from "~/services/logger.server";
 import { createLoaderWorkerApiRoute } from "~/services/routeBuilders/apiBuilder.server";
-import { clientSafeErrorMessage } from "~/utils/prismaErrors";
+import { clientSafeErrorMessage, isInfrastructureError } from "~/utils/prismaErrors";
 
 export const loader = createLoaderWorkerApiRoute(
   {
@@ -31,7 +31,21 @@ export const loader = createLoaderWorkerApiRoute(
 
       return json(continuationResult);
     } catch (error) {
-      logger.warn("Failed to suspend run", { runFriendlyId, snapshotFriendlyId, error });
+      logger.warn("Failed to continue run execution", {
+        runFriendlyId,
+        snapshotFriendlyId,
+        error,
+      });
+
+      // A Prisma infrastructure error (e.g. P1001 "Can't reach database
+      // server") means the DB was transiently unreachable while resuming. A 422
+      // is non-retryable, so the worker would permanently abort a run over a
+      // blip. Let it propagate to the generic 500 handler, which scrubs the
+      // message and is retried by the worker's HTTP client.
+      if (isInfrastructureError(error)) {
+        throw error;
+      }
+
       if (error instanceof Error) {
         throw json({ error: clientSafeErrorMessage(error) }, { status: 422 });
       }

packages/core/src/v3/runEngineWorker/workload/http.ts

@@ -132,6 +132,20 @@ export class WorkloadHttpClient {
           headers: {
             ...this.defaultHeaders(),
           },
+        },
+        {
+          // Resuming after a wait is idempotent (guarded server-side by the
+          // snapshot id), so retry generously to ride out a transient database
+          // outage rather than aborting the run. `randomize` jitters the delay
+          // so a fleet of runs resuming at once doesn't stampede the DB the
+          // moment it recovers.
+          retry: {
+            minTimeoutInMs: 500,
+            maxTimeoutInMs: 10_000,
+            maxAttempts: 8,
+            factor: 2,
+            randomize: true,
+          },
         }
       )
     );