Add MADV_PAGEOUT support for eager eviction of compressed spill

[antiguru]

Motivation

On the column pager's lz4 + swap spill path, the compressed bytes are kept resident in the process address space (CompressedInner::Memory). Today they receive no madvise at all — they're plain anonymous memory the kernel reclaims only lazily under LRU pressure. So while the byte budget bounds the logical resident working set, actual RSS drifts up toward the cgroup cap and is only relieved at the kernel's pressure cliff, with no headroom.

This PR teaches the pager to issue MADV_PAGEOUT over those bytes, evicting them eagerly at spill time so RSS is held at the budget. It is gated behind a new, default-off dynamic config, so behavior is unchanged until explicitly enabled.

Why MADV_PAGEOUT, and why only on the lz4 path

This is one half of a combined spill strategy (lz4 + MADV_PAGEOUT) that benchmarking converged on for the swap backend; full data and reasoning live in CLU-108. In short:

• Eager vs lazy. Leaving the bytes unmanaged (today) reclaims only at the pressure cliff, so RSS pins near the cap with no headroom. An MADV_COLD hint would have the same lazy character. MADV_PAGEOUT is proactive: it evicts at spill time and holds RSS at the budget (measured no-cap peak RSS: lz4+PAGEOUT 0.40 GiB vs an MADV_COLD-on-compressed variant 0.97 GiB — the full compressed working set).
• Why it is coupled to lz4. MADV_PAGEOUT's cost is that a later read must fault the pages back. lz4 shrinks the byte volume ~5.6×, so the re-fault is cheap — that is what makes eager eviction pay off. On the uncompressed path it is a measured net loss (raw+PAGEOUT 77s vs raw+COLD 72s): the full-size re-swap penalty has nothing to cancel it. That is why the flag only takes effect when lz4 is on, and why the raw pageout_swap path is left on its existing MADV_COLD hint.

(Note: MADV_COLD-on-the-compressed-bytes was only ever a benchmark comparison point, never a shipped code path — the compressed-memory variant has always been un-advised. This PR adds the first eviction hint it gets.)

Description

Core pager (src/ore/src/pager/swap.rs):

• Extracted the page-alignment logic shared by the existing MADV_COLD hint into a madvise_aligned(ptr, len, advice) helper.
• Added a public advise_pageout(&[u8]) that issues MADV_PAGEOUT over a byte slice without transferring ownership (unlike pageout_swap, which moves the Vec<u64> into a handle and hints MADV_COLD). The caller keeps the allocation addressable; a later read re-faults the pages.
• No-op on non-Linux targets. Unit tests cover data preservation and the empty/sub-page edge cases.

Column pager (src/timely-util/src/column_pager.rs):

• Process-global SWAP_PAGEOUT atomic, set by apply_tiered_config, mirroring how the backend/codec are configured.
• The lz4 + swap spill path calls advise_pageout on the compressed bytes when the flag is set. Added round_trip_swap_lz4_pageout covering correctness with it enabled.

Configuration (src/compute-types/src/dyncfgs.rs, src/compute/src/compute_state.rs):

• New column_paged_batcher_swap_pageout dynamic config (default false), read in apply_worker_config and threaded to apply_tiered_config. Registered with parallel-workload and mzcompose's system-parameter lists.

Public API (src/ore/src/pager.rs): re-exported advise_pageout.

Verification

• Unit + integration tests as above; skipped under Miri (no madvise).
• MADV_PAGEOUT is a non-mutating reclaim hint — pages stay readable and re-fault on access, so data integrity is preserved.
• Flag defaults to false; existing behavior is unchanged until enabled.

https://claude.ai/code/session_012Ji6Nd26mZVJCRGTxbUBfn

[DAlperin]

 The idea is good, but I have some layering concerns. CompressedInner::Memory probably isn't the right place to hold explicitly non resident memory. I think its probably acceptable to use the same pad_u8_to_u64 in the File branch and make the madvise property of the swap backend. We could delete the ::Memory variant entirely if we decide its not needed.

Do you have benchmarks?

[DAlperin]

This isn't true though right? Currently the memory variant of compressed gets no madvise at all

[DAlperin]

Same issue in the PR description, but does point to a middle option (that I assume you tested but worth saying anyway) of unconditionally madvise cold-ing the compressed bytes?

[antiguru]

Yeah, Claude got confused with data from benchmarks, sorry for that.

[DAlperin]

Pedantic but maybe something like column_paged_batcher_swap_eager_reclaim

[antiguru]

I think pageout is the right term, because it's not reclaiming memory (or, only in a very specific sense). pageout is the actual mechanism we use, so it removes ambiguity.

[antiguru]

Thanks for the review! I agree, the PR was lacking details, so trying to correct this here.

I ran measurements and they landed on lz4 + swap benefitting from MADV_PAGEOUT. Just swap wouldn't benefit because it'd be bandwidth limited, but once the data was smaller, we started to see breakevens. Additionally, in a redlined cluster, always doing MADV_PAGEOUT seems sometimes better than not doing it. The problem with the policy as implemented is that we only start paging after 50% (or so) memory utilization, at which point we've put swap into a position to need to work through half of our memory, consuming I/O bandwidth when we need it most.

I ran benchmarks, but not end-to-end, which is why I'm not yet publishing numbers. I'd say let's merge this and run some e2e benches and see where we land.

[DAlperin]

Happy to merge and test. Still slightly partial to pushing the decision down to the pager level or changing the Enum variants to make it less confusing but I won't block on it