[bug] runtime rotation app helpers leak past their idle timeout; all helpers trample one shared status file
Impact observed (macOS, 16 GB M-series, codex-multi-auth 2.8.3→2.8.4, codex-cli 0.147.0)
The machine ground into swap thrash. The single largest memory consumer:
183 processes node … scripts/codex.js --codex-multi-auth-runtime-app-helper
5.58 GB RSS (~31 MB each)
182 of 183 orphaned (ppid 1) — by design (detached), but
41 of 183 past the 12h idle timeout, the oldest at 33 hours
accumulation 10–28 new helpers/hour under ordinary mcodex/TUI use
Alongside: 579 owner files (runtime-rotation-app-helper-owner.<pid>.json)
against those 183 live helpers (later measured at 701 against 3), and one
shared runtime-rotation-app-helper.json being atomically rewritten ~183
times per second.
All defective sites date to commit 1bc40eb ("Add runtime rotation proxy")
and are present on current main (v2.8.4).
Defect 1 — the idle reaper's owner check cannot survive PID reuse
isProcessAlive (scripts/codex.js) is a bare kill(pid, 0). It answers
"does a process hold this integer", never "is this still my launcher". The
reaper tick treats a live owner PID as activity:
if (ownerPid && isRuntimeRotationAppHelperOwnerAlive(ownerPid)) {
lastActivityAt = currentTime;
}macOS recycles PIDs aggressively; a busy machine churns hundreds of
short-lived node/bun processes per hour. One recycled hit at one tick pushes
the helper's deadline out another 12 hours — and the deadline only ever moves
forward, so a single false "alive" is never corrected by later true "dead"s.
Repeat and the helper is effectively immortal, which is what 41 helpers past
their timeout demonstrate. Ruled out on the live processes: IDLE_MS
overrides (none set), proxy traffic (totalRequests: 0), a wedged cleanup
(healthy idle event loop under sample), a dead timer (steady ~0.1 % CPU
from the 1 Hz tick). One leaked helper's decoded status showed
lastActivityAt bumped 5.5 h after startedAt with zero requests — only the
owner-liveness branch can have done that, and its owner was long dead.
Defect 2 — nothing else bounds a helper's life
Any bug in activity accounting produces an unbounded leak, because the idle
check is the only exit. There is no absolute lifetime ceiling.
Defect 3 — one status file, N writers, 1 Hz
resolveRuntimeRotationAppHelperStatusPath returns a single shared path with
no PID component, and publishStatus rewrites it from every tick of every
helper. Two independent problems: the file describes an arbitrary helper
(last writer wins — rotation status and the current-account resolution in
lib/runtime/runtime-current-account.ts report a random or dead helper when
several run, e.g. one account-pinned app-server per account), and it is
real I/O load that scales linearly with the leak. Noted as known follow-up
№ 1 in #662.
Defect 4 — owner files are written at spawn and never removed
writeRuntimeRotationAppHelperOwner creates
runtime-rotation-app-helper-owner.<pid>.json per helper; nothing deletes
them. 579→701 files accumulated on one machine.
Reproduction
Deterministic (identity mismatch stands in for a recycled PID — a live
process holding the dead launcher's integer):
CODEX_MULTI_AUTH_APP_ROTATION_IDLE_MS=250- Spawn the helper directly (
scripts/codex.js --codex-multi-auth-runtime-app-helper <token>) with
CODEX_MULTI_AUTH_APP_ROTATION_OWNER_PID=<pid of any live process>.
- The helper never idles out while that PID lives, regardless of who it is —
kill(pid, 0) succeeds, the deadline ratchets forward every tick.
Organic: ordinary mcodex use on a busy machine; each short-lived invocation
detaches a helper; PID churn does the rest. The 33-hour helpers above are
that repro, run by accident, at scale.
Fix
PR to follow: owner identity = PID plus the launcher's kernel start time
(recycled PID ⇒ later start time ⇒ mismatch ⇒ owner correctly dead),
re-verified at most once a minute, degrading to bare liveness where ps is
unavailable; an absolute lifetime ceiling
(CODEX_MULTI_AUTH_APP_ROTATION_MAX_LIFETIME_MS, default 24 h) so any future
accounting bug leaks bounded instead of unbounded; per-PID status files with
change-only publishing plus heartbeat, readers preferring the newest live
helper and still reading the legacy path; owner-file removal on helper exit
plus a launcher-side sweep of metadata whose helper PID is dead.
[bug] runtime rotation app helpers leak past their idle timeout; all helpers trample one shared status file
Impact observed (macOS, 16 GB M-series, codex-multi-auth 2.8.3→2.8.4, codex-cli 0.147.0)
The machine ground into swap thrash. The single largest memory consumer:
Alongside: 579 owner files (
runtime-rotation-app-helper-owner.<pid>.json)against those 183 live helpers (later measured at 701 against 3), and one
shared
runtime-rotation-app-helper.jsonbeing atomically rewritten ~183times per second.
All defective sites date to commit
1bc40eb("Add runtime rotation proxy")and are present on current
main(v2.8.4).Defect 1 — the idle reaper's owner check cannot survive PID reuse
isProcessAlive(scripts/codex.js) is a barekill(pid, 0). It answers"does a process hold this integer", never "is this still my launcher". The
reaper tick treats a live owner PID as activity:
macOS recycles PIDs aggressively; a busy machine churns hundreds of
short-lived node/bun processes per hour. One recycled hit at one tick pushes
the helper's deadline out another 12 hours — and the deadline only ever moves
forward, so a single false "alive" is never corrected by later true "dead"s.
Repeat and the helper is effectively immortal, which is what 41 helpers past
their timeout demonstrate. Ruled out on the live processes:
IDLE_MSoverrides (none set), proxy traffic (
totalRequests: 0), a wedged cleanup(healthy idle event loop under
sample), a dead timer (steady ~0.1 % CPUfrom the 1 Hz tick). One leaked helper's decoded status showed
lastActivityAtbumped 5.5 h afterstartedAtwith zero requests — only theowner-liveness branch can have done that, and its owner was long dead.
Defect 2 — nothing else bounds a helper's life
Any bug in activity accounting produces an unbounded leak, because the idle
check is the only exit. There is no absolute lifetime ceiling.
Defect 3 — one status file, N writers, 1 Hz
resolveRuntimeRotationAppHelperStatusPathreturns a single shared path withno PID component, and
publishStatusrewrites it from every tick of everyhelper. Two independent problems: the file describes an arbitrary helper
(last writer wins —
rotation statusand the current-account resolution inlib/runtime/runtime-current-account.tsreport a random or dead helper whenseveral run, e.g. one account-pinned
app-serverper account), and it isreal I/O load that scales linearly with the leak. Noted as known follow-up
№ 1 in #662.
Defect 4 — owner files are written at spawn and never removed
writeRuntimeRotationAppHelperOwnercreatesruntime-rotation-app-helper-owner.<pid>.jsonper helper; nothing deletesthem. 579→701 files accumulated on one machine.
Reproduction
Deterministic (identity mismatch stands in for a recycled PID — a live
process holding the dead launcher's integer):
CODEX_MULTI_AUTH_APP_ROTATION_IDLE_MS=250scripts/codex.js --codex-multi-auth-runtime-app-helper <token>) withCODEX_MULTI_AUTH_APP_ROTATION_OWNER_PID=<pid of any live process>.kill(pid, 0)succeeds, the deadline ratchets forward every tick.Organic: ordinary
mcodexuse on a busy machine; each short-lived invocationdetaches a helper; PID churn does the rest. The 33-hour helpers above are
that repro, run by accident, at scale.
Fix
PR to follow: owner identity = PID plus the launcher's kernel start time
(recycled PID ⇒ later start time ⇒ mismatch ⇒ owner correctly dead),
re-verified at most once a minute, degrading to bare liveness where
psisunavailable; an absolute lifetime ceiling
(
CODEX_MULTI_AUTH_APP_ROTATION_MAX_LIFETIME_MS, default 24 h) so any futureaccounting bug leaks bounded instead of unbounded; per-PID status files with
change-only publishing plus heartbeat, readers preferring the newest live
helper and still reading the legacy path; owner-file removal on helper exit
plus a launcher-side sweep of metadata whose helper PID is dead.