Skip to content

Guard against underflow when sysconf(_SC_PAGESIZE) returns 0#3031

Merged
soutaro merged 1 commit into
ruby:masterfrom
paracycle:fix/wasi-page-size-clamp
Jul 27, 2026
Merged

Guard against underflow when sysconf(_SC_PAGESIZE) returns 0#3031
soutaro merged 1 commit into
ruby:masterfrom
paracycle:fix/wasi-page-size-clamp

Conversation

@paracycle

@paracycle paracycle commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Problem

On WASI in a Rust-linked wasm32 module, sysconf(_SC_PAGESIZE) returns 0 instead of -1. The existing fallback only checked for -1, so a 0 result flowed through to rbs_allocator_init where:

allocator->default_page_payload_size = system_page_size - sizeof(rbs_allocator_page_t);

This underflowed to near SIZE_MAX (unsigned wraparound: 0 - 12 = 0xFFFFFFF4). The subsequent malloc(sizeof(header) + 0xFFFFFFF4) wrapped to malloc(0), allocating a zero-byte arena page. Arena sub-allocations overflowed into adjacent heap memory, corrupting the lexer's string.start/string.end pointers and causing parse failures.

The resulting corruption is memory-layout-dependent; in the Rust-linked cdylib it manifested when the constant pool's allocation overlapped the undersized arena page.

Fix

Extract rbs_allocator_normalize_page_size(long) which falls back to 4096 when the raw value is <= 0 or smaller than sizeof(rbs_allocator_page_t). get_system_page_size routes through it on both POSIX and Windows paths.

Regression test

The existing rbs_wasm_selftest (run via rake wasm:check) now calls rbs_allocator_normalize_page_size with synthetic inputs -1, 0, 1, and 65536, verifying both fallback and pass-through paths.

Mutation-verified (rake wasm:check):

  • Green (patched): WebAssembly selftest passed.
  • Red (both guards removed → identity function): WebAssembly selftest failed: rbs_wasm_selftest returned "0" (expected "1") + rake aborted!
  • Green (restored): WebAssembly selftest passed.

Scope

  • Upstream's standalone reactor build returns 65536 from sysconf, so the bug does not reproduce there. The fix is defensive and preserves behavior on all platforms with a valid page size.
  • The bug was discovered and verified in a Rust cdylib linking the RBS C extension for wasm32-wasip1 (Rubydex playground).

@paracycle
paracycle force-pushed the fix/wasi-page-size-clamp branch from 5e2268b to d9196a4 Compare July 24, 2026 00:22
On WASI in a Rust-linked wasm32 module, `sysconf(_SC_PAGESIZE)` returns
0 instead of -1. The existing fallback only checked for -1, so a 0 result
flowed through to `rbs_allocator_init` where
`payload_size = page_size - sizeof(rbs_allocator_page_t)` underflowed
to near SIZE_MAX (unsigned wraparound). The subsequent
`malloc(header_size + payload_size)` wrapped to `malloc(0)`, allocating
a zero-byte arena page. Arena sub-allocations overflowed into adjacent
heap memory, corrupting the lexer's string pointers and causing parse
failures.

The resulting corruption is memory-layout-dependent; in the Rust-linked
cdylib it manifested when the constant pool's allocation overlapped the
undersized arena page.

Fix: extract `rbs_allocator_normalize_page_size(long)` which falls back
to 4096 when the raw value is <= 0 or smaller than
`sizeof(rbs_allocator_page_t)`. The WASM selftest
(`rake wasm:check`) now calls it with synthetic inputs -1, 0, 1, and
65536 to verify both fallback and pass-through paths.

Note: upstream's standalone reactor build returns 65536 from sysconf, so
the bug does not reproduce there. The fix is defensive and preserves
behavior on all platforms with a valid page size.
@paracycle
paracycle force-pushed the fix/wasi-page-size-clamp branch from d9196a4 to 33fb933 Compare July 24, 2026 00:22
@soutaro soutaro added this to the RBS 4.1 milestone Jul 27, 2026
@soutaro
soutaro enabled auto-merge July 27, 2026 02:24

@soutaro soutaro left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@soutaro
soutaro merged commit b9b76c1 into ruby:master Jul 27, 2026
27 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants