sea: add vfsArchive to serve the assets from a ZIP archive

When "vfsArchive" names a ZIP archive in the SEA configuration (with
"useVfs": true), --build-sea embeds the archive verbatim as one
reserved asset, and at runtime the SEA virtual file system mounts the
existing ZipProvider over a zero-copy view of the embedded archive
instead of the SEAProvider. The main script is injected into the
in-memory archive index as a stored entry, so the mounted tree looks
the same as with "assets".

The archive can be produced with any ZIP tool or with the ZIP support
in node:zlib; no ZIP serialization logic is added to the build side.
This trades asset read speed (entries are inflated when opened) for a
substantially smaller executable when the assets are compressible.

Also make FindSingleExecutableResource() return the deserialized
SeaResource by reference instead of by value: the copy included the
whole assets map, and getAsset() calls it once per asset read, so
every fs operation served by the SEA virtual file system paid a cost
linear in the number of bundled assets - with 8192 assets, reading
each of them took seconds instead of milliseconds.

Signed-off-by: Matteo Collina <hello@matteocollina.com>
PR-URL: #65810
Reviewed-By: Paolo Insogna <paolo@cowtech.it>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>

Author: mcollina

doc/api/single-executable-applications.md

@@ -117,6 +117,7 @@ The configuration currently reads the following top-level fields:
   "useSnapshot": false,  // Default: false
   "useCodeCache": true, // Default: false
   "useVfs": true, // Default: false
+  "vfsArchive": "/path/to/assets.zip", // Optional
   "execArgv": ["--no-warnings", "--max-old-space-size=4096"], // Optional
   "execArgvExtension": "env", // Default: "env", options: "none", "env", "cli"
   "assets": {  // Optional
@@ -260,6 +261,41 @@ Module format detection works the same way as on the real file
 system: name bundled ES modules with the `.mjs` extension (or provide the
 relevant `package.json` files as assets) so they are interpreted as ESM.
 
+#### Serving the assets from a ZIP archive with `"vfsArchive"`
+
+Instead of listing individual `"assets"`, the configuration can point
+`"vfsArchive"` at a prebuilt ZIP archive. The archive is embedded into the
+executable as-is, and the virtual file system serves the files inside it,
+inflating each one when it is read. When the assets are compressible (such
+as JavaScript, JSON, or other text), a deflate-compressed archive can
+substantially reduce the size of the generated executable.
+
+The archive can be built with any ZIP tool, or with the ZIP support in
+[`node:zlib`][]:
+
+```mjs
+import { zipFiles } from 'node:zlib';
+import { createWriteStream } from 'node:fs';
+import { pipeline } from 'node:stream/promises';
+
+await pipeline(
+  zipFiles([
+    ['./dist/config.json', 'config.json'],
+    ['./dist/data.txt', 'data/data.txt'],
+  ]),
+  createWriteStream('assets.zip'),
+);
+```
+
+The mounted file tree looks the same as with `"assets"`: the entries appear
+under the mount point using their archive names, the main script is placed
+at the mount point root, and access through `__dirname`-relative paths,
+`require()`, and `import` is unchanged. However, `sea.getAsset()` and
+`sea.getAssetAsBlob()` do not serve the individual files, because the
+executable only embeds the archive; read the files through the file system
+APIs instead. `"vfsArchive"` requires `"useVfs": true` and cannot be
+combined with `"assets"`.
+
 #### Snapshot and code caching limitations
 
 `"useVfs": true` cannot be used together with `"useSnapshot": true` or
@@ -751,6 +787,7 @@ to help us document them.
 [Using native addons in the injected main script]: #using-native-addons-in-the-injected-main-script
 [VFS documentation]: vfs.md
 [Windows SDK]: https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/
+[`node:zlib`]: zlib.md
 [`process.execPath`]: process.md#processexecpath
 [`require()`]: modules.md#requireid
 [`require.main`]: modules.md#accessing-the-main-module

doc/api/vfs.md

@@ -518,6 +518,11 @@ is loaded from inside the mount through the ESM loader, and
 `"useVfs"` cannot be used together with `"useSnapshot"` or `"useCodeCache"`.
 The SEA configuration parser will error if either combination is detected.
 
+Instead of listing individual `"assets"`, the SEA configuration can point
+`"vfsArchive"` at a prebuilt ZIP archive; the mount is then backed by a
+[`ZipProvider`][] over the embedded archive, and each file is inflated when
+it is read. See [Serving the assets from a ZIP archive][] for details.
+
 See the [Single Executable Application][] documentation for more information
 on creating SEA builds with assets.
 
@@ -693,6 +698,7 @@ fields use synthetic but stable values:
 [CommonJS resolution algorithm]: modules.md#all-together
 [ES modules resolution algorithm]: esm.md#resolution-algorithm
 [Explicit Resource Management]: https://github.com/tc39/proposal-explicit-resource-management
+[Serving the assets from a ZIP archive]: single-executable-applications.md#serving-the-assets-from-a-zip-archive-with-vfsarchive
 [Single Executable Application]: single-executable-applications.md
 [`--import`]: cli.md#--importmodule
 [`--require`]: cli.md#-r---require-module

lib/internal/vfs/sea.js

@@ -1,6 +1,15 @@
 'use strict';
 
-const { isSea, isVfsEnabled } = internalBinding('sea');
+const {
+  ObjectKeys,
+} = primordials;
+
+const {
+  isSea,
+  isVfsEnabled,
+  isVfsArchiveEnabled,
+  getAsset,
+} = internalBinding('sea');
 const { kEmptyObject } = require('internal/util');
 const {
   codes: {
@@ -38,9 +47,14 @@ function initSeaVfs(options = kEmptyObject) {
   }
 
   const { VirtualFileSystem } = require('internal/vfs/file_system');
-  const { SEAProvider } = require('internal/vfs/providers/sea');
 
-  const provider = new SEAProvider({ extraFiles: options.extraFiles });
+  let provider;
+  if (isVfsArchiveEnabled()) {
+    provider = createZipProvider(options.extraFiles);
+  } else {
+    const { SEAProvider } = require('internal/vfs/providers/sea');
+    provider = new SEAProvider({ extraFiles: options.extraFiles });
+  }
   // The SEA warning already covers the feature; don't emit the
   // VirtualFileSystem experimental warning for the implicit SEA mount.
   const vfs = new VirtualFileSystem(provider, {
@@ -51,6 +65,43 @@ function initSeaVfs(options = kEmptyObject) {
   return vfs;
 }
 
+// The reserved asset key under which --build-sea stores the ZIP archive
+// named by "vfsArchive". Must match kVfsArchiveAssetName in src/node_sea.cc.
+const kVfsArchiveAssetName = 'node:sea:vfs.zip';
+
+/**
+ * Creates a ZipProvider over the ZIP archive embedded by `"vfsArchive"`.
+ * The archive bytes are used in place (a zero-copy view over the SEA
+ * blob); entries are inflated on demand when they are opened.
+ * The extra files (the SEA main script) are added to the in-memory archive
+ * index as stored entries, leaving the embedded bytes untouched.
+ * @param {Record<string, string|Buffer>} [extraFiles] Additional files to
+ *   serve alongside the assets (used for the SEA main script)
+ * @returns {ZipProvider}
+ */
+function createZipProvider(extraFiles) {
+  const { Buffer } = require('buffer');
+  const { ZipBuffer } = require('internal/zip');
+  const { ZipProvider } = require('internal/vfs/providers/ziparchive');
+
+  // getAsset returns a zero-copy ArrayBuffer over the (possibly read-only)
+  // SEA blob; ZipBuffer only reads from it, and decompressed contents are
+  // fresh buffers, so the view can be used without copying the archive.
+  const archive = getAsset(kVfsArchiveAssetName);
+  const zip = new ZipBuffer(Buffer.from(archive));
+
+  if (extraFiles !== undefined) {
+    const names = ObjectKeys(extraFiles);
+    for (let i = 0; i < names.length; i++) {
+      const content = extraFiles[names[i]];
+      const data = typeof content === 'string' ? Buffer.from(content) : content;
+      zip.addSync(names[i], data, { __proto__: null, method: 'store' });
+    }
+  }
+
+  return new ZipProvider(zip);
+}
+
 /* c8 ignore stop */
 
 module.exports = {

src/module_wrap.cc

@@ -367,7 +367,7 @@ void ModuleWrap::New(const FunctionCallbackInfo<Value>& args) {
       // For embedder ESM in a SEA, use the bundled code cache if available.
       if (id_symbol == realm->isolate_data()->embedder_module_hdo() &&
           sea::IsSingleExecutable()) {
-        sea::SeaResource sea = sea::FindSingleExecutableResource();
+        const sea::SeaResource& sea = sea::FindSingleExecutableResource();
         if (sea.use_code_cache()) {
           std::string_view data = sea.code_cache.value();
           user_cached_data = new ScriptCompiler::CachedData(

src/node.cc

@@ -330,7 +330,7 @@ MaybeLocal<Value> StartExecution(Environment* env,
 #ifndef DISABLE_SINGLE_EXECUTABLE_APPLICATION
   // Snapshot in SEA is only loaded for the main thread.
   if (sea::IsSingleExecutable() && env->is_main_thread()) {
-    sea::SeaResource sea = sea::FindSingleExecutableResource();
+    const sea::SeaResource& sea = sea::FindSingleExecutableResource();
     // The SEA preparation blob building process should already enforce this,
     // this check is just here to guard against the unlikely case where
     // the SEA preparation blob has been manually modified by someone.
@@ -986,7 +986,7 @@ static ExitCode InitializeNodeWithArgsInternal(
       !(flags & ProcessInitializationFlags::kDisableNodeOptionsEnv);
 #ifndef DISABLE_SINGLE_EXECUTABLE_APPLICATION
   if (sea::IsSingleExecutable()) {
-    sea::SeaResource sea_resource = sea::FindSingleExecutableResource();
+    const sea::SeaResource& sea_resource = sea::FindSingleExecutableResource();
     if (sea_resource.exec_argv_extension != sea::SeaExecArgvExtension::kEnv) {
       should_parse_node_options = false;
     }
@@ -1603,7 +1603,7 @@ bool LoadSnapshotData(const SnapshotData** snapshot_data_ptr) {
 #ifndef DISABLE_SINGLE_EXECUTABLE_APPLICATION
   if (sea::IsSingleExecutable()) {
     is_sea = true;
-    sea::SeaResource sea = sea::FindSingleExecutableResource();
+    const sea::SeaResource& sea = sea::FindSingleExecutableResource();
     if (sea.use_snapshot()) {
       std::unique_ptr<SnapshotData> read_data =
           std::make_unique<SnapshotData>();

src/node_contextify.cc

@@ -1768,7 +1768,7 @@ static void CompileFunctionForCJSLoader(
   ScriptCompiler::CachedData* cached_data = nullptr;
 #ifndef DISABLE_SINGLE_EXECUTABLE_APPLICATION
   if (is_sea_main) {
-    sea::SeaResource sea = sea::FindSingleExecutableResource();
+    const sea::SeaResource& sea = sea::FindSingleExecutableResource();
     // Use the "main" field in SEA config for the filename.
     Local<Value> filename_from_sea;
     if (!ToV8Value(context, sea.code_path).ToLocal(&filename_from_sea)) {

src/node_sea.cc

@@ -44,6 +44,10 @@ using v8::Value;
 namespace node {
 namespace sea {
 
+// The reserved asset key under which the ZIP archive named by "vfsArchive"
+// is stored. Must match the name used by lib/internal/vfs/sea.js.
+constexpr std::string_view kVfsArchiveAssetName = "node:sea:vfs.zip";
+
 namespace {
 
 SeaFlags operator|(SeaFlags x, SeaFlags y) {
@@ -246,7 +250,7 @@ bool SeaResource::use_code_cache() const {
   return static_cast<bool>(flags & SeaFlags::kUseCodeCache);
 }
 
-SeaResource FindSingleExecutableResource() {
+const SeaResource& FindSingleExecutableResource() {
   static const SeaResource sea_resource = []() -> SeaResource {
     std::string_view blob = FindSingleExecutableBlob();
     per_process::Debug(DebugCategory::SEA,
@@ -266,12 +270,21 @@ void IsSea(const FunctionCallbackInfo<Value>& args) {
 void IsVfsEnabled(const FunctionCallbackInfo<Value>& args) {
   bool enabled = false;
   if (IsSingleExecutable()) {
-    SeaResource sea_resource = FindSingleExecutableResource();
+    const SeaResource& sea_resource = FindSingleExecutableResource();
     enabled = static_cast<bool>(sea_resource.flags & SeaFlags::kEnableVfs);
   }
   args.GetReturnValue().Set(enabled);
 }
 
+void IsVfsArchiveEnabled(const FunctionCallbackInfo<Value>& args) {
+  bool enabled = false;
+  if (IsSingleExecutable()) {
+    const SeaResource& sea_resource = FindSingleExecutableResource();
+    enabled = static_cast<bool>(sea_resource.flags & SeaFlags::kVfsArchive);
+  }
+  args.GetReturnValue().Set(enabled);
+}
+
 void IsExperimentalSeaWarningNeeded(const FunctionCallbackInfo<Value>& args) {
   bool is_building_sea =
       !per_process::cli_options->experimental_sea_config.empty();
@@ -285,7 +298,7 @@ void IsExperimentalSeaWarningNeeded(const FunctionCallbackInfo<Value>& args) {
     return;
   }
 
-  SeaResource sea_resource = FindSingleExecutableResource();
+  const SeaResource& sea_resource = FindSingleExecutableResource();
   args.GetReturnValue().Set(!static_cast<bool>(
       sea_resource.flags & SeaFlags::kDisableExperimentalSeaWarning));
 }
@@ -300,7 +313,7 @@ std::tuple<int, char**> FixupArgsForSEA(int argc,
     static std::vector<std::string> exec_argv_storage;
     static std::vector<std::string> cli_extension_args;
 
-    SeaResource sea_resource = FindSingleExecutableResource();
+    const SeaResource& sea_resource = FindSingleExecutableResource();
 
     new_argv.clear();
     exec_argv_storage.clear();
@@ -466,6 +479,16 @@ std::optional<SeaConfig> ParseSingleExecutableConfig(
       if (use_vfs) {
         result.flags |= SeaFlags::kEnableVfs;
       }
+    } else if (key == "vfsArchive") {
+      std::string_view archive_path;
+      if (field.value().get_string().get(archive_path)) {
+        FPrintF(stderr,
+                "\"vfsArchive\" field of %s is not a string\n",
+                config_path);
+        return std::nullopt;
+      }
+      result.vfs_archive_path = archive_path;
+      result.flags |= SeaFlags::kVfsArchive;
     } else if (key == "assets") {
       simdjson::ondemand::object assets_object;
       if (field.value().get_object().get(assets_object)) {
@@ -597,6 +620,26 @@ std::optional<SeaConfig> ParseSingleExecutableConfig(
     }
   }
 
+  if (static_cast<bool>(result.flags & SeaFlags::kVfsArchive)) {
+    if (!static_cast<bool>(result.flags & SeaFlags::kEnableVfs)) {
+      FPrintF(stderr, "\"vfsArchive\" requires \"useVfs\" to be true\n");
+      return std::nullopt;
+    }
+    if (!result.assets.empty()) {
+      FPrintF(stderr,
+              "\"vfsArchive\" cannot be used together with \"assets\"\n");
+      return std::nullopt;
+    }
+    if (result.vfs_archive_path.empty()) {
+      FPrintF(stderr,
+              "\"vfsArchive\" field of %s is not a non-empty string\n",
+              config_path);
+      return std::nullopt;
+    }
+    // The archive is embedded as a single reserved asset.
+    result.flags |= SeaFlags::kIncludeAssets;
+  }
+
   if (result.main_path.empty()) {
     FPrintF(stderr,
             "\"main\" field of %s is not a non-empty string\n",
@@ -808,6 +851,27 @@ ExitCode GenerateSingleExecutableBlob(
   if (!config.assets.empty() && BuildAssets(config.assets, &assets) != 0) {
     return ExitCode::kGenericUserError;
   }
+  if (static_cast<bool>(config.flags & SeaFlags::kVfsArchive)) {
+    std::string archive;
+    int r = ReadFileSync(&archive, config.vfs_archive_path.c_str());
+    if (r != 0) {
+      const char* err = uv_strerror(r);
+      FPrintF(stderr,
+              "Cannot read vfsArchive %s: %s\n",
+              config.vfs_archive_path,
+              err);
+      return ExitCode::kGenericUserError;
+    }
+    // Only a signature sanity check; the archive is parsed by the ZIP
+    // support in JS when the executable starts.
+    if (archive.size() < 4 || archive[0] != 'P' || archive[1] != 'K') {
+      FPrintF(stderr,
+              "vfsArchive %s is not a ZIP archive\n",
+              config.vfs_archive_path);
+      return ExitCode::kGenericUserError;
+    }
+    assets.emplace(std::string(kVfsArchiveAssetName), std::move(archive));
+  }
   std::unordered_map<std::string_view, std::string_view> assets_view;
   for (auto const& [key, content] : assets) {
     assets_view.emplace(key, content);
@@ -868,7 +932,7 @@ void GetAsset(const FunctionCallbackInfo<Value>& args) {
   CHECK_EQ(args.Length(), 1);
   CHECK(args[0]->IsString());
   Utf8Value key(args.GetIsolate(), args[0]);
-  SeaResource sea_resource = FindSingleExecutableResource();
+  const SeaResource& sea_resource = FindSingleExecutableResource();
   if (sea_resource.assets.empty()) {
     return;
   }
@@ -892,7 +956,7 @@ void GetAsset(const FunctionCallbackInfo<Value>& args) {
 void GetAssetKeys(const FunctionCallbackInfo<Value>& args) {
   CHECK_EQ(args.Length(), 0);
   Isolate* isolate = args.GetIsolate();
-  SeaResource sea_resource = FindSingleExecutableResource();
+  const SeaResource& sea_resource = FindSingleExecutableResource();
 
   Local<Context> context = isolate->GetCurrentContext();
   LocalVector<Value> keys(isolate);
@@ -914,7 +978,7 @@ MaybeLocal<Value> LoadSingleExecutableApplication(
   // env->context() is entered.
   Environment* env = info.env();
   Local<Context> context = env->context();
-  SeaResource sea = FindSingleExecutableResource();
+  const SeaResource& sea = FindSingleExecutableResource();
 
   CHECK(!sea.use_snapshot());
   // TODO(joyeecheung): this should be an external string. Refactor UnionBytes
@@ -936,7 +1000,7 @@ bool MaybeLoadSingleExecutableApplication(Environment* env) {
     return false;
   }
 
-  SeaResource sea = FindSingleExecutableResource();
+  const SeaResource& sea = FindSingleExecutableResource();
 
   if (sea.use_snapshot()) {
     // The SEA preparation blob building process should already enforce this,
@@ -962,7 +1026,7 @@ void Initialize(Local<Object> target,
   Isolate* isolate = env->isolate();
 
   if (IsSingleExecutable()) {
-    SeaResource sea_resource = FindSingleExecutableResource();
+    const SeaResource& sea_resource = FindSingleExecutableResource();
     // Expose the main script path recorded in the SEA config so the VFS
     // integration can place the main script at the mount point root.
     if (static_cast<bool>(sea_resource.flags & SeaFlags::kEnableVfs)) {
@@ -983,6 +1047,7 @@ void Initialize(Local<Object> target,
 
   SetMethod(context, target, "isSea", IsSea);
   SetMethod(context, target, "isVfsEnabled", IsVfsEnabled);
+  SetMethod(context, target, "isVfsArchiveEnabled", IsVfsArchiveEnabled);
   SetMethod(context,
             target,
             "isExperimentalSeaWarningNeeded",
@@ -994,6 +1059,7 @@ void Initialize(Local<Object> target,
 void RegisterExternalReferences(ExternalReferenceRegistry* registry) {
   registry->Register(IsSea);
   registry->Register(IsVfsEnabled);
+  registry->Register(IsVfsArchiveEnabled);
   registry->Register(IsExperimentalSeaWarningNeeded);
   registry->Register(GetAsset);
   registry->Register(GetAssetKeys);