[pull] canary from vercel:canary - #1409
Merged
Merged
Conversation
## Summary Make deploy mode follow the existing setup/start lifecycle: skipStart prepares the fixture without deploying, and next.start() performs the deployment. Failed starts retain CLI and build logs before rejecting, so tests can assert expected build failures in their own bodies. Default automatic startup still fails its setup hook on an uncaught deployment error. Remove the expectDeploymentFailure option. Refactor app-invalid-revalidate into fixed fixtures and use the same start/error assertions locally and on Vercel, without a deployment exclusion. ## Verification - All four invalid-revalidate cases passed against real Vercel preview builds with Turbopack. Each test caught the rejection from `next.start()` and asserted the expected diagnostic in `next.cliOutput`; no skipped tests. - All four cases passed locally in dev and start modes with both Turbopack and webpack. - Seventeen lifecycle unit tests passed, covering the real `nextTestSetup` hooks, deferred/default startup, failed-build logs, retries, repeated starts, custom scripts, existing deployments, and cleanup. - `pnpm test-deploy-turbo test/e2e/test-utils-tests/basic/basic.test.ts`: the default automatic deployment reached READY and loaded its build/deployment IDs, but the HTTP assertion failed because the temporary project's protection redirected to Vercel SSO. Project protection settings were left unchanged. The suite's separate, existing Cache Components manifest exclusion is unchanged. <!-- NEXT_JS_LLM -->
### What? Keep a task alive across the restore-waiter handoff: from the moment a caller observes another thread restoring a category until that caller acquires the restored task for use. This is the first open layer in stack #98709. The construction and restore-owner races have already landed in `canary` through #98688 and #98700. ### Why? The `restoring` bits elect exactly one thread to perform backing-storage I/O. A waiting caller observes that ownership under the task lock, drops the lock, and then holds only a `TaskId` while waiting for notification. That temporary in-session reference is not represented by a parent or aggregation edge, so GC may otherwise collect the task after the restorer clears its bit but before the waiter acquires a guard. Eviction is different: it is safe for eviction to clear the completed restore in this window as long as the waiter does not mistake that state for an I/O failure. The waiter can claim the now-unowned category and restore it again. Batch preparation has the same task-id-outside-the-graph interval between classification and callback. Pair access needs both task IDs kept alive until both guards can be acquired together. ### How? Each waiting caller increments the existing `transient_ref_count` before dropping the task lock and decrements exactly one ref under the guard at its actual use boundary. This uses the GC-root mechanism and invariant checks that already protect construction, transient-parent, and detached-handle references. The transient ref does not block value eviction. If a waiter observes a category that is neither restored nor currently restoring, it keeps its GC pin, claims the category, and performs another restore. A replacement restorer retains its acquired guard through notification and returns that same still-pinned guard to the caller. Successful pin release belongs entirely to the caller: - single-task access releases after receiving the restored guard; - pair access releases after acquiring both guards together; - batch preparation releases immediately before passing the guard to its callback. The helper decrements the ref itself only on `Err`, where no guard is returned. The already-restored fast path remains and avoids listener registration. This removes the dedicated handoff field and eviction blocker. It also keeps `gc_maybe_collectible()` and `gc_debug_assert_root_held_by_transient_pin()` aligned on the same existing transient-ref predicate. ### Verification - Latest `origin/canary` (`389f164d`) in a detached worktree - `cargo test -p turbo-tasks-backend --lib` (102 passed) - `cargo test -p turbo-tasks-backend --test eviction --test gc_resurrection --test gc_collection --test gc_interrupt --test gc_stress` (19 passed) - Eviction-permitting stress: 0 failures across 1,200 runs using 8 parallel workers - Sequential stress: 0 failures across 150 runs - `cargo fmt -- --check` - `cargo clippy -p turbo-tasks-backend --tests -- -D warnings` Followed by #98729. <!-- NEXT_JS_LLM --> <!-- fleet 6aa6acec-6286-4b03-86f4-13f4c7e2b2fa --> --------- Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com> Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
…th (#98855) #98440 unintentionally caused the rspack `Cargo.lock` to become outdated because it added a `smallvec` dependency to `turbo-unix-path`, which gets used by `next-taskless`. We only really need `smallvec`'s `serde` and `bincode_impl` features when used with turbo-tasks, so move the feature declaration there.
### What? Ensure a GC-resurrected persistent task is persisted again as a new task. This is the final layer in stack #98709 and depends on the restore-waiter lifetime fix in #98728. ### Why? GC soft-deletes a disconnected task, then the following snapshot persists a tombstone that removes its durable task record and task-type lookup entry. Normal operations resume after the snapshot's logical state is frozen, so a concurrent connection can still find the resident soft-deleted task and resurrect it before post-snapshot eviction removes the in-memory entry. Resurrection already cleared `deleted`, marked the task dirty, and scheduled recomputation. It did not, however, mark the task as newly publishable. If the tombstone had already committed, later snapshots could persist recomputed data without re-establishing the complete persistent identity of the resurrected task. Once in-memory cache state was evicted, future lookups could no longer recover that otherwise-live task by type. A resident deleted task is guaranteed to retain its type: GC restores all categories before marking it deleted, and post-snapshot eviction handles deleted tasks by removing the whole entry rather than partially dropping the type. ### How? When the connect handshake wins the resurrection race, mark the task as `new_task` again and mark its Data category modified. The next snapshot therefore persists it using the normal new-task path, including recreating its task-type lookup entry and the Data record used to verify that lookup. The task remains dirty so its cells and graph edges are rebuilt through the existing recomputation path. The update is unconditional for a resident deleted task; a missing type remains an invariant violation rather than being silently skipped. ### Verification - Top-of-stack stress: 0 failures across 1,200 runs using 8 parallel workers - Sequential stress: 0 failures across 150 runs - `cargo test -p turbo-tasks-backend --lib` (102 passed) - `cargo test -p turbo-tasks-backend --test eviction --test gc_resurrection --test gc_collection --test gc_interrupt --test gc_stress` (19 passed) - `cargo fmt -- --check` - `cargo clippy -p turbo-tasks-backend --tests -- -D warnings` Depends on #98728. <!-- NEXT_JS_LLM --> <!-- fleet 6aa6acec-6286-4b03-86f4-13f4c7e2b2fa --> Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
Implements the following extension:
```typescript
export interface NftFileList {
// File paths relative to the directory containing the `.nft.json` file, or
// the current root (if inside of `NftAdditionalRoot`).
//
// When using webpack, these paths may exist outside of the tracing root. The
// [`@vercel/next` package ignores these paths][vc-next].
//
// When using Turbopack, these paths are all guaranteed to exist within the
// `turbopack.root` specified or inferred from the `next.config.js` file.
//
// [vc-next]: https://github.com/vercel/vercel/blob/%40vercel/next%404.20.5/packages/next/src/server-build.ts#L1022-L1026
files: string[]
// Turbopack extension: A parallel array to `files` (same indices and length)
// with file content hashes. For symlinks, the hash of the path of the target
// is stored.
fileHashes?: string[]
// Turbopack extension: Explicit symlink mapping information.
//
// If included, it's safe to assume that if a file is not in `symlinks` that
// it is not a symlink. If this is an empty array, there are no symlinks.
//
// If omitted, the processor of the nft file must call `read_link` on every
// file to determine if it is a symlink and determine the target path.
//
// This field is always included if `NftJson` includes `additionalRoots`, and
// it is always included on `NftAdditionalRoot`.
symlinks?: NftSymlink[]
}
export interface NftJson extends NftFileList {
version: 1
// Turbopack extension: A hash of the entrypoint that refers to these traced
// files. This hash only depends on the content of the entrypoint file, and
// not all of its traced dependencies.
entryHash?: string
// Turbopack extension: Paths stored with different base paths, typically
// outside of the tracing root.
additionalRoots?: NftAdditionalRoot[]
}
// Turbopack extension: A collection of paths stored with a different base path.
export interface NftAdditionalRoot extends NftFileList {
// Stable unique identifier provided in the `next.config.js`. This can be used
// to generate the output path where these files are copied to (e.g.
// `.additionalRoots/$[name}`).
//
// This is guaranteed to use a character set that is valid on most
// filesystems, and the identifiers are guaranteed to not have overlaps on
// case-insensitive filesystems.
name: string
// A source path on the build machine that the paths in `files` are relative
// to. The final build output directory should not depend on this path.
absolutePath: string
// Always specified on NftAdditionalRoot.
symlinks: NftSymlink[]
}
// Turbopack extension: Information on a symlink, including which additional
// root it maps to. Symlinks that do not cross root boundaries (the common case)
// omit the index into `additionalRoots`.
//
// It is often complicated to transform raw symlink targets to root-relative
// paths, and including this information here ensures that the NFT reader gets
// the same result that Turbopack's tracing system expects.
//
// Because the link target type is unspecified, on Windows the reader needs to
// call `stat` to determine if a link target is a directory or file.
export type NftSymlink =
| [
// Index of `files` that refers to a symlink.
number,
// The target path of the link. In `NftJson`, this path is relative to the
// directory containing the `.nft.json` file. In `NftAdditionalRoot`, this
// is relative to the current root.
string,
]
| [
// Index of `files` that refers to a symlink.
number,
// The target path of the link relative to the specified root.
string,
// An index into `additionalRoots`, -1 if the target path is relative to
// the `.nft.json` file's directory,
//
// If the symlink target is relative to the same root as the symlink
// itself (the "current root"), this field is omitted.
number,
]
```
Nothing currently populates `additionalRoots`.
#98003 will do it.
Full motivation and plan here: https://app.notion.com/p/vercel/Turbopack-pnpm-Global-Virtual-Store-383e06b059c480579403ddfd71cc2d40?source=copy_link The goal is to allow `DiskFileSystem` to traverse outside of it's own root to other configured `DiskFileSystem`s when following symlinks. We may allow traversal in other situations in the future, but this is limited to symlink resolution for now. ## Global Virtual Store The motivation for this is to enable [pnpm's Global Virtual Store feature](https://pnpm.io/global-virtual-store) (and there are other package managers doing this, including nub and bun). We'd expose the ability to manually configure this in `next.config.js`, but we should also auto-configure ourselves for popular package managers (or at least make a best effort to do so, the `PNPM_HOME` semantics can be complicated). The `ignoreIfMissing` option is provided for this situation: We can configure a bunch of roots automatically, and they only actually get set up if they exist, the check for directory existence is cheap. ## NFT changes This requires a couple extensions to the `*.nft.json` file format: #98469 ## Related Issues - #93556 - pnpm/pnpm#14972
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )