Skip to content

gh-156780: Emscripten: support building CPython with static linking - #156781

Open
clementperon wants to merge 4 commits into
python:mainfrom
clementperon:emscripten-static-linking
Open

gh-156780: Emscripten: support building CPython with static linking#156781
clementperon wants to merge 4 commits into
python:mainfrom
clementperon:emscripten-static-linking

Conversation

@clementperon

@clementperon clementperon commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Fixes #156780.

CPython's Emscripten glue was written for the -sMAIN_MODULE configuration it is built and tested in. Linking libpython.a into a program without that flag failed at startup because the glue relied on things only that configuration provides: resolveGlobalSymbol() from libdylink.js, a growable function table for addFunction(), and _PyRuntime being exported through CPython's own -sEXPORTED_FUNCTIONS.

Changes

  • Move main() wrapping out of libpython. Running main() under WebAssembly.promising only makes sense when Python is the program. The wrapper moves from Python/emscripten_syscalls.c to Programs/emscripten_beforemain.c, which is linked into the interpreter (and installed next to python.o) but not into libpython. It rebinds _main, and wasmImports.main under -sMAIN_MODULE, so one code path covers both configurations. When main() completes it exits through exitJS(), so atexit handlers run and buffered output is flushed, and the exit status is preserved. FS.createAsyncInputDevice() stays in libpython.
  • Gate the suspending syscalls on the wrapper. fd_read and poll only suspend when Module.Py_EmscriptenStackSwitching is set, which the interpreter's wrapper does on entry to main(). An embedder running its own promising entry point sets the flag itself; otherwise the calls keep their synchronous behaviour. The flag is documented in emscripten_syscalls.c.
  • Support static linking. The call trampoline exports _PyRuntime's address itself and takes over a placeholder's function table slot instead of calling addFunction(), so it needs neither -sEXPORTED_FUNCTIONS nor -sALLOW_TABLE_GROWTH. __PyRuntime is dropped from the interpreter's export list since nothing reads it any more.
  • Add an embedding smoke test. Platforms/emscripten/web_embed_test links libpython.a into a program whose main() is not Python's, without -sMAIN_MODULE or the interpreter's link flags, and runs a script that imports from the stdlib, calls through the trampoline and polls. run_test.sh builds and runs it and is wired into the Emscripten CI job.

Testing

configure was regenerated with make regen-configure.

  • Dynamic-linking interpreter (the CI configuration): stack switching active, exit codes propagate, test_capi.test_emscripten, test_select, test_builtin, test_json, test_sys, test_functools pass.
  • Static-linking interpreter (--disable-wasm-dynamic-linking, not covered by CI): builds and behaves the same, including stack switching on Node 24 with JSPI.
  • Embedding smoke test passes against libpython.a; the syscall-override object is pulled into the link. make Programs/_testembed and make libainstall work.

@clementperon

Copy link
Copy Markdown
Contributor Author

@hoodmane 👀 maybe ?

@hoodmane

hoodmane commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Maybe let's start with a PR that adds the EM_JS_DEPS() everywhere? That seems like an easy improvement.

I think rather than using Module.Py_EmscriptenPromisingMain as a runtime flag, it would be more morally upstanding to move that logic into python.o or some other object file we only link into the interpreter program, that way embedders wouldn't get it at all. Maybe we can rename _emscripten_promising_main_js to _PyEmscripten_BeforeMain() or something and move the following to python.c or to its own object file next to it:

#ifdef __EMSCRIPTEN__
__attribute__((constructor)) void _PyEmscripten_Constructor(void) {
   _PyEmscripten_BeforeMain();
}

python.c is currently very short, so I'm not sure how people will feel about adding this there but we should find out.

And of course as traditional we need much less prose since Claude is as rambly as usual. Please rewrite all the comments and the commit message by hand. I find that helps cut them down to a more reasonable length.

Of course, it's likely that any of this will bit rot without test coverage.

cc @freakboy3742 WDYT?

@clementperon

Copy link
Copy Markdown
Contributor Author

@hoodmane as you suggest move to a dedicated object file + reword commit + add a test

Comment thread Misc/NEWS.d/next/Build/2026-09-01-23-32-22.gh-issue-156780.Qw3nRt.rst Outdated
Comment thread Python/emscripten_syscalls.c Outdated
@freakboy3742

Copy link
Copy Markdown
Contributor

Maybe let's start with a PR that adds the EM_JS_DEPS() everywhere? That seems like an easy improvement.

+1 to this as an approach to landing the changes.

I think rather than using Module.Py_EmscriptenPromisingMain as a runtime flag, it would be more morally upstanding to move that logic into python.o
...
python.c is currently very short, so I'm not sure how people will feel about adding this there but we should find out.

I agree that linking into the mainline is a preferable approach to a runtime flag.

As for where to put it - I can definitely see the argument for using python.c, but the fact that it is so minimal at present is at least a cause for a raised eyebrow. Is Modules/main.c perhaps a better candidate?

And of course as traditional we need much less prose since Claude is as rambly as usual. Please rewrite all the comments and the commit message by hand. I find that helps cut them down to a more reasonable length.

+1 to this as well. Claude's predilection for extraneous verbositude is usually contrary to the underlying goals of effective communication :-)

@clementperon
clementperon force-pushed the emscripten-static-linking branch from fa3261b to ee90cfa Compare September 8, 2026 09:00
@clementperon

clementperon commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

@hoodmane EM_JS_DEPS has been merged thanks to @freakboy3742.

I will rework a bit this PR + clean commit message & comment.

@clementperon
clementperon marked this pull request as draft September 8, 2026 10:23
@clementperon
clementperon force-pushed the emscripten-static-linking branch 3 times, most recently from 5417e3e to 7c1f523 Compare September 8, 2026 14:30
@clementperon
clementperon marked this pull request as ready for review September 8, 2026 14:30
@clementperon
clementperon force-pushed the emscripten-static-linking branch from 7c1f523 to 18d7011 Compare September 8, 2026 14:32
@clementperon

clementperon commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

@hoodmane new version after discussion with AI

  • No more Py_EMSCRIPTEN_DYNAMIC_LINKING or -sALLOW_TABLE_GROWTH. The wrapper rebinds _main and, when present, wasmImports.main, so a single code path covers both configurations. The trampoline now takes over a placeholder's table slot instead of calling addFunction(), so the table never grows. The previous static branch also wrapped wasmExports["main"], which does not exist (the export is __main_argc_argv); fixed.
  • Cleaner exit. After main() completes the wrapper goes through exitJS() instead of Module.onExit directly, so atexit handlers run and trailing output is flushed.
  • Module.Py_EmscriptenStackSwitching is set when main() is actually entered rather than in the constructor, has its own commit, and is documented in emscripten_syscalls.c for embedders.
  • FS.createAsyncInputDevice() stays in libpython, so _testinternalcapi keeps working for any libpython consumer.
  • Build fixes: emscripten_beforemain.o depends on the headers, is linked into _testembed, and is installed by libainstall.
  • Embed test links libpython.a directly, is built by run_test.sh instead of every make, checks node's exit status, and is linked with -O2 -g0 (wasm went from 41 MB to 8 MB).

Tested with the CI configuration, a --disable-wasm-dynamic-linking interpreter, and the embed test. PR description updated.

@hoodmane

hoodmane commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

The changes to Python/emscripten_trampoline.c look good to me, let's next split out those into a separate PR.

clementperon and others added 4 commits September 10, 2026 16:34
The trampoline exports _PyRuntime's address itself and reuses a
placeholder's table slot, so libpython no longer needs
-sEXPORTED_FUNCTIONS=__PyRuntime or a growable function table.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Running main() under WebAssembly.promising only makes sense when Python
is the program. Move the wrapper into Programs/emscripten_beforemain.c,
linked into the interpreter but not into libpython, and let the runtime
shut down normally afterwards so atexit handlers run and buffered output
is flushed. FS.createAsyncInputDevice() stays in libpython.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…() wrapper

Suspending in fd_read or poll needs a promising entry point, which
libpython no longer sets up itself. Check Module.Py_EmscriptenStackSwitching,
set by the interpreter's wrapper, instead of WebAssembly.promising alone.
An embedder with its own promising entry point sets the flag.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Link libpython.a into a program whose main() is not Python's, without
-sMAIN_MODULE or the interpreter's link flags, and run a script that
imports from the stdlib, calls through the trampoline and polls.
Platforms/emscripten/web_embed_test/run_test.sh builds and runs it in CI.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@clementperon
clementperon force-pushed the emscripten-static-linking branch from 18d7011 to 1eaadc6 Compare September 10, 2026 14:36
@clementperon

Copy link
Copy Markdown
Contributor Author

Rebase on top of #157238

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Emscripten: static (non-MAIN_MODULE) builds fail at startup on dynamic-linking-only JS helpers

3 participants