Allocator pressure relief is macOS-only; long-lived Linux daemon accumulates 15GB of swap

[iceteaSA]

memory.rs has an allocator-reclamation path that Linux never gets:

#[cfg(target_os = "macos")]
pub fn relieve_allocator_pressure() -> AllocatorPressureRelief { … malloc_zone_pressure_relief … }  // :616

#[cfg(target_os = "macos")]
pub fn spawn_allocator_slack_relief_if_due(…) -> bool { … }                                          // :584

There is no Linux counterpart — malloc_trim, mallopt, and M_ARENA_MAX have zero occurrences in the crate.

The doc comment on spawn_allocator_slack_relief_if_due describes the exact failure this causes:

one long-lived chatty session used to block reclamation for the process lifetime (observed: 5.1 GB RSS over ~600 MB of live data)

That diagnosis was correct and the macOS fix works. Linux just never got the symmetric one, because until the subc daemon nothing ran long enough to need it — standalone spawned a worker per session and exited.

Observed

An 18-hour aft --subc instance on Linux (glibc, 16 cores, 7 project roots):

VmSwap   15.3 GB          ← 2.5× peak RSS
VmHWM     6.0 GB
VmRSS     2.3 GB

VmSwap far exceeding VmHWM is the allocate-touch-once-never-again signature: pages get touched once, land in a per-thread arena that is never reused, and get swapped out permanently. Box swap reached 99.8%; killing the process returned ~18 GB.

Region-level detail on that instance: 55 rw-p anon regions of 32-132 MB, 2348 MB of them swapped (95% of process swap), all but one ~100% cold. Anonymous was 97.6% of RSS and the mapping count was low and static (38 map_files), which rules out the pread disk index and mmap growth — this is allocator retention, not a mapping leak.

Confirmation

Running the same binary with MALLOC_ARENA_MAX=2, at equal 9-minute uptime:

uncapped :  ~50 cold regions, 2348 MB touched-then-swapped
capped   :   27 regions, 1891 MB RSS, 0 MB swapped
             of which 1698 MB is one fully-resident heap (the real working set)

Caveat stated honestly: the capped arm has more free RAM available than the baseline did, so the swap difference alone is confounded. The pressure-independent evidence is the touched footprint and the residency breakdown, which agree. A longer arm is still running.

Note when reading region counts: glibc reserves 64 MB of address space per arena eagerly and commits lazily, so 18 arena-sized regions can appear under a cap of 2 with 17 of them at ~0% resident. Counting map entries measures reservations; only RSS measures what can swap. (I misread this myself first time.)

Suggested fix

malloc_trim(0) on the same idle-sweep gate the macOS path already uses — spawn_allocator_slack_relief_if_due's trigger conditions look directly reusable. That keeps the "caller owns the gate because relief adds latency" contract intact.

MALLOC_ARENA_MAX as a spawn-env default is the cheaper mitigation, but it trades swap for malloc lock contention on a process that runs ~195 threads, and aft has paths asserting absolute wall-clock bounds (the storm rigs). I would not set it low without a latency guard.

Related but separate: the thread count itself is inflated by per-root inspect pools — filed separately, since the fixes are independent.

[ualtinok]

Accepted — both the finding and the priority. The macOS-only gating is exactly what you diagnosed: relieve_allocator_pressure is a malloc_zone_pressure_relief binding, and Linux never got an equivalent because the dev fleet is Mac-heavy — a gap, not a decision. For glibc the right tool is malloc_trim(0) (returns free arena pages to the OS; jemalloc/musl need different handling, which is why it can't be a one-liner), wired to the same slack-triggered cadence the macOS path uses (relief when allocator slack ≥ 1 GiB, off the dispatch loop). Your 15 GB swap number also suggests we should report allocator slack in health.check on Linux the way phys_footprint disambiguates it on macOS, so the fleet can see the pressure before it becomes swap. Queued for the next train.

[ualtinok]

Fixed on main (198d3f13), CI green. Linux glibc daemons now get allocator pressure relief on the same slack-triggered cadence as macOS: malloc_trim(0) runs off the dispatch loop when allocator slack exceeds 1 GiB (at most every 5 minutes), with the primitive runtime-resolved so musl and jemalloc environments are an honest no-op rather than a pretend-relief. Slack is also now visible before it becomes swap: health.check reports allocator_slack_bytes with a sibling allocator_slack_measured flag on both platforms (absent-vs-zero disambiguated). Reachable from both the daemon maintenance tick and the standalone drain loop. Ships in the next release — and thank you for the measurement discipline in the report; the 15 GB swap number with the arena breakdown made the priority case unambiguous.

[iceteaSA]

Correcting my own report: the fix is correct code, and it does not reclaim anything on this workload. My diagnosis named the right subsystem and the wrong shape.

Deployed 198d3f13 on the Linux daemon that produced the original 15.3 GB number — 7 roots, glibc 2.44, ~150 threads. The relief pass fires exactly as designed. It releases zero:

21:57:11  released=4337664  slack_before=9.71GB   slack_after=9.74GB
22:02:12  released=0        slack_before=15.06GB  slack_after=15.06GB
22:07:12  released=0        slack_before=15.50GB
22:12:12  released=0        slack_before=15.53GB
22:17:12  released=0        slack_before=15.74GB

Gating works — 5-minute spacing, above the 1 GiB threshold, mallinfo2 resolving on 2.33+. One small release, then nothing, while reported slack climbs 6 GB.

malloc_trim(0) only returns free pages from the top of a heap. The free space here is interleaved beneath live allocations, so there is nothing at the top to return. The hoard is real and it is fragmentation-pinned, which trim structurally cannot address.

Worth stating what the slack metric rules out, since I got this wrong first: retained_slack_bytes is arena - uordblks, so it is held-but-free by construction, not live. The arithmetic confirms it — slack 14.6 GB against RSS 1.11 GB plus 3.55 GB swapped is 4.66 GB actually committed. A live set that size is impossible. This is not a leak.

The overnight series moderates the severity, and it also confirms the mechanism. Twelve hours on one instance:

• VmSwap 4.12 GB at 22:52, peak 4.19, fell to 2.20 by 02:52, plateaued 2.8–2.9 through morning
• Large anon regions 159 → 241 during the evening workload, then flat at 241 across six idle hours
• VmData grew 13 → 20 GB under load, then sat at ~20 GB all night — never shrank, never grew

The hoard tracks allocation churn, not time. It expands with work and then sits. A leak grows while idle; this does not. That is exactly the fragmentation signature and it is a much better outcome than the original incident: 2.9 GB at 12 hours versus 15.3 GB at 18 hours.

What I cannot tell you is which change earned that. Three candidates landed or changed at once — trim helping at the margin, 8bd35ff4 cutting inspect threads 88 → 8 and shrinking churn with them, and a lighter workload. Observation can't separate them.

So: no ask on this issue, and nothing that needs reopening. The remaining question is whether the free-space distribution is ordinary fragmentation or one pathological allocation pattern in aft, and malloc_info(3) per-arena output answers it directly. I'm building an isolated harness for that — deterministic churn, since an idle process won't reproduce it — and will file separately if it turns up something actionable rather than speculate here.

One note for the record on 8bd35ff4, since it shipped alongside: inspect threads went 88 → 8 and process threads 211 → 150 on the same 7-root daemon. That one did exactly what it said.

[ualtinok]

Appreciated — and the correction is as useful as the original report. Trim releasing ~zero against a climbing slack number is exactly what a fragmentation-pinned heap looks like: malloc_trim can only return the contiguous free run at the top of each heap, and free space interleaved beneath live allocations is structurally unreachable. Your slack arithmetic (held-but-free by construction, impossible live set) settles the not-a-leak question cleanly, and the overnight series — growth tracking churn, flat across idle hours — is the confirming signature.

One mechanism link that bears on which of your three confounded candidates did the work: glibc creates arenas per contending thread (up to 8×cores by default), and each arena maintains its own heaps with its own fragmentation. 8bd35ff4 cutting inspect threads 88 → 8 (process 211 → 150) doesn't just reduce churn — it reduces the number of arenas interleaving allocations, which attacks the pinning mechanism directly rather than at the margin. That makes the thread cut the structurally-favored candidate, and it gives your malloc_info(3) harness a concrete axis: compare arena count and per-arena free distribution across the two thread regimes. If the hoard turns out to be spread across many low-occupancy arenas, the actionable lever on our side is a bounded arena count (M_ARENA_MAX via mallopt at daemon startup) — measurable and easily reversible, and we'd take that conversation if your per-arena data points there.

No action needed on this issue; looking forward to what the harness finds.

[iceteaSA]

The arena-per-thread link breaks an assumption in my design, and I'd have run the experiment without noticing.

I had MALLOC_ARENA_MAX=2 in as a negative control — the idea being that since it measured ineffective in production, a rig that reproduces production should also show it ineffective. But the production arm that measured it ran the pre-8bd35ff4 binary at 211 threads, and the patched instance runs 150 with 8 inspect workers instead of 88. If arena count follows thread contention, those two arms sit in different arena regimes and my control was comparing across the variable you've just identified as most likely causal. Downgraded in the design from "clean control" to "rough sanity check, cannot validate the rig on its own."

Your axis is now an arm. Stock allocator, thread regime as the independent variable:

arm
0	scaffolding only, zero workload
A	stock, current regime (8 inspect threads)
B	MALLOC_ARENA_MAX=2
C	decay-purging allocator
D	stock, inspect pool forced wide (approximating pre-8bd35ff4)

A-vs-D isolates thread count from allocator behaviour, which is exactly the confound the overnight series couldn't touch — three candidates, one observation. And it's the arm with the cheapest remedy attached: if the hoard is spread across many low-occupancy arenas, mallopt(M_ARENA_MAX) at startup is in-tree, measurable, reversible, and doesn't require anyone to accept a #[global_allocator] change. I've reordered the design so that outcome gets checked before the allocator-swap one.

One implementation note, since it's a small change to your code rather than my test: dispatch.rs::default_pool_size() is available_parallelism().min(8) with no override, so arm D needs an env knob to force the pool wide. I'd add it test-side, but a pool-size override looks generally useful — for reproducing thread-regime issues, and for constrained hosts where 8 workers is too many. Say the word if you'd rather have it as a config field than an env var, and I'll shape it that way.

Also worth noting what this does to the ceiling: 8×cores on this box is 128 arenas against 150 threads. Not far from saturating, which makes the arena-multiplication story mechanically plausible rather than merely available.

Open questions are on #205 — mainly whether a non-default allocator has been considered and rejected before I produce evidence for one.