s3parquet destination + OOM tuning (final stack layer) — fixes 3 pre-existing test failures#29

Merged

randomizedcoder merged 36 commits into

mainfrom

s3parquet-destination-pr

Jun 15, 2026

Commits on Jun 15, 2026

xtcp: s3parquet destination — direct Parquet → MinIO (retires Vector)

randomizedcoder
and
claude
committed
microvm: s3parquet-long runner — long-soak parquet test with per-minute heartbeats

randomizedcoder
and
claude
committed
microvm: s3parquet-long → 63 MiB production flush threshold

randomizedcoder
and
claude
committed
xtcp + microvm: Pyroscope continuous-profiling agent + in-VM server

randomizedcoder
and
claude
committed
xtcp ns: fix OS-thread leak in netNamespaceInstance under heavy ns churn

randomizedcoder
and
claude
committed
xtcp2.service: grant CAP_SYS_ADMIN — root cause of the ns thread leak

randomizedcoder
and
claude
committed
xtcp ns: regression test + forbidigo lint guard for the LockOSThread/Setns pattern

randomizedcoder
and
claude
committed
xtcp2: fail-early capability check with per-cap diagnostics + capcheck-fail microvm flavor

randomizedcoder
and
claude
committed
nix/checks: lightweight capability-check derivations (no microvm needed)

randomizedcoder
and
claude
committed
microvm: 3-knob aggressive soak workload — exercises the full parquet pipeline

randomizedcoder
and
claude
committed
microvm: fix ordering cycle in xtcp2-soak-ns-traffic systemd unit

randomizedcoder
and
claude
committed
nsTest: -traffic flag — in-process loopback connection per new ns

randomizedcoder
and
claude
committed
nsTest: -conns N flag — N persistent loopback conns per ns with varied io profiles

randomizedcoder
and
claude
committed
microvm: clickhouse-pipeline-parquet flavor — kafka + parquet side by side

randomizedcoder
and
claude
committed
microvm: forward + open :9089 :8890 for the parquet xtcp2 instance; lower flush for smoke

randomizedcoder
and
claude
committed
microvm: bump clickhouse-pipeline-parquet memory budgets — 12 GiB VM + 8 GB ClickHouse

randomizedcoder
and
claude
committed
clickhouse-pipeline-parquet: disable chatty internal log tables to tame OOMs

randomizedcoder
and
claude
committed
clickhouse-kafka: cap per-poll memory with kafka_poll_max_batch_size=256

randomizedcoder
and
claude
committed
clickhouse-pipeline: schema-warm barrier + lower kafka_poll_max_batch_size

randomizedcoder
and
claude
committed
Revert: clickpipe-up schema-warm barrier

randomizedcoder
and
claude
committed
docs/SQL: correct the kafka MV "halt" framing from the prior commits

randomizedcoder
and
claude
committed
clickhouse-pipeline-parquet: container 12000m→14000m + batch_size 64→16

randomizedcoder
and
claude
committed
docs: note that bigger ClickHouse memory does NOT reduce OOMs

randomizedcoder
and
claude
committed
clickhouse + redpanda: explicit memory caps

randomizedcoder
and
claude
committed
clickhouse kafka_engine: shrink Block to one envelope — eliminates OOMs

randomizedcoder
and
claude
committed
docs: document the actual kafka_max_block_size root cause

randomizedcoder
and
claude
committed
microvm prometheus: forward :19090 + scrape both xtcp2 instances

randomizedcoder
and
claude
committed
build/scripts: add Prometheus probe + stability-summary tooling

randomizedcoder
and
claude
committed
microvm clickpipe: bound redpanda + persist CH data across restarts

randomizedcoder
and
claude
committed
clickhouse kafka: extend max.poll.interval.ms to tolerate slow MV inserts

randomizedcoder
and
claude
committed
clickhouse: MALLOC_CONF to return jemalloc-retained chunks to OS

randomizedcoder
and
claude
committed
microvm clickpipe-parquet: bigger docker disk + dedicated MinIO disk

randomizedcoder
and
claude
committed
gofmt + nixfmt: format s3parquet destination + microvm files

randomizedcoder
and
claude
committed
pkg/xtcp: capability-check test seam so Init-driven tests run unprivileged

randomizedcoder
and
claude
committed
cmd/xtcp2: init s3/pyroscope flag fields in TestPrintFlags + TestBuildConfig

randomizedcoder
and
claude
committed
pkg/xtcp: de-flake TestS3ParquetDest_corner_queueFull (2s → 30s deadline)

randomizedcoder
and
claude
committed