Add interrupts, memory caps, async streaming, ETF callback results - #70
Merged
Conversation
Close four gaps that blocked running untrusted Python: no way to stop
running code, no per-context memory bound, no async-generator streaming,
and a lossy callback result encoding.
py:interrupt/1 raises KeyboardInterrupt in the thread executing a context,
and py_context:call/eval/exec now interrupt on timeout instead of only
abandoning the reply while the thread kept running.
py_context:new(#{mode => owngil, memory_limit => Bytes}) caps memory per
context, accounted from obmalloc arenas and enforced with MemoryError.
Opt-in via enable_memory_limits, since the allocator is hooked before
Python starts.
py:stream_start/3,4 drives async generators again, which the docs had
claimed since 3.0.0 without the code behind it.
Callback results now cross as external term format instead of Python repr
strings parsed with ast.literal_eval, fixing binaries with backslashes,
quotes or newlines, empty lists, float precision, and the base64 pid/ref
round-trip. Breaking: an Erlang string returned from a callback arrives as
a list of integers, matching call arguments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes four gaps that blocked running Python you do not control.
Interrupt running Python
py:interrupt/1raisesKeyboardInterruptin the thread executing a context;the in-flight call returns
{error, interrupted}.py_context:call/eval/execinterrupt automatically when their timeout expires, so
{error, timeout}nowstops the Python code instead of abandoning the reply while the thread kept
burning CPU and the context stayed wedged.
The BEAM is the watchdog: no C-side timer thread, one new NIF. Works in both
workerandowngilmode (the latter attaches a freshPyThreadStateto thecontext's subinterpreter), and is callable while the context process is blocked
in a NIF. Async exceptions land at bytecode boundaries, so code blocked in a C
call is interrupted once that call returns.
Per-context memory caps
py_context:new(#{mode => owngil, memory_limit => Bytes}). Accounting comesfrom obmalloc arena traffic, which is the only place a free hook receives the
block size, so no per-object header is needed. Opt-in via
{enable_memory_limits, true}, since the allocator is hooked before Pythonstarts.
Enforcement raises
MemoryErrorasynchronously rather than returning NULL fromthe arena allocator: obmalloc treats a failed arena as a reason to fall back to
PyMem_RawMalloc, so a NULL would let the allocation silently succeedoff-arena and the cap would stop counting instead of stopping the code. Measured
during development: a 64 MB cap let usage reach 82 MB with no error.
Limits are documented: allocations over 512 bytes bypass obmalloc and are not
counted, granularity is one 1 MB arena, and
workermode returns{error, memory_limit_requires_owngil}because those contexts share the maininterpreter.
Async generator streaming
py:stream_start/3,4drives async generators on a private event loop. The docshave claimed this since 3.0.0 with no code behind it;
py:stream/4with kwargsand
py:stream_eval/1,2stay sync-only, now stated explicitly.Callback results as external term format
Results returned from an Erlang callback now cross as
term_to_binaryand aredecoded by the same
term_to_pyconverter used for call arguments, replacing aPython repr string parsed with
ast.literal_eval. Fixes binaries containingbackslashes, quotes, newlines or tabs (which produced an unparseable literal and
were silently delivered to Python as the raw repr text),
[]arriving as'',float precision loss, and the base64 round-trip for pids and refs, which now
cross as native
Pid/Refobjects. Removes ~200 lines of duplicated encoders.Breaking: an Erlang string returned from a callback (
"abc", a list ofintegers) now reaches Python as
[97, 98, 99]rather than'abc', matching howcall arguments have always converted. Return a binary for a Python
str.Performance
Callback round-trips get faster; hot paths are unchanged. The added per-request
cost is two uncontended mutex pairs, below the noise floor of a 9-15 us
operation.