ci: stop losing releases to things that are allowed to fail - #121
Merged
Conversation
Nine of the last twenty-three test runs failed. Most were real defects CI had never been able to see until #108 turned the suites on, and those are fixed. These are the rest — each one tied to a failure that took a release with it. - `bun install` retried on every platform, not only Windows. A registry that answered "Failed to install 1 package" once is why 0.14.0 did not publish, and this action runs in every job of every workflow. - `playwright install-deps` is apt against Ubuntu's mirrors. It now runs only when the browser cache missed, and retries when it does run. - Spawned CLI processes get the file-watcher kill switch. The environment there replaces the parent's, so the flag CI sets never arrived, and Bun segfaulted inside that watcher on a Windows runner. - `packages/core` tests no longer write to the real home directory. They were creating `~/.red/redcode`, sharing a binary cache across concurrent jobs, and letting an ambient `~/.npmrc` reach the installer under test. The binary cache stays shared across processes on purpose: per-pid would isolate correctly and make every suite re-download ripgrep. - `packages/core` runs at 30s per test like the other heavy packages, instead of bun's 5s default while it spawns git and node. Claude-Session: https://claude.ai/code/session_014XwJPDhq1ahcm3rd454WzQ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The number
Nine of the last twenty-three
test.ymlruns failed — 39% — and 0.15.0 has been sitting unreleasedthrough five of them.
That sounds like chaos and is not. Most of today's failures were real defects CI had never been
able to see until #108 turned on the suites it had been skipping since the rename: Windows path
and CRLF assumptions, a component that mishandled mutation-record ordering, a test racing its own
request. Those are fixed. What is left is a short list of external dependencies that fail a few
percent of the time and take a release with them each time.
What this changes
bun installis retried on every platform, not only Windows. The action already retried therefor a known bun issue; Linux and macOS got one attempt. That asymmetry is exactly where the 0.14.0
publish died —
Failed to install 1 packageon the darwin-arm64 sidecar. This composite action runsin every job of every workflow, so one transient refusal anywhere costs a build.
playwright install-deps chromiumis apt against Ubuntu's mirrors. It ran on every Linux e2erun, gated only on the OS, and when the mirrors refused (
Connection failed [IP: 91.189.92.22 80])the job died before a test executed. It now runs only when the browser cache missed — the deps come
with the cache — and retries when it does run.
Spawned CLI processes get the file-watcher kill switch. The unit job sets
REDCODE_EXPERIMENTAL_DISABLE_FILEWATCHERon Windows, butcli-process.tsbuilds a replacementenvironment, so every
cliIt.livetest started a redcode that loaded@parcel/watcheranyway —the module Bun segfaulted in. A test pins it, because the failure mode is silent.
packages/coretests stop writing to the real home directory.global.tsresolves the app homeat module scope from
REDCODE_TEST_HOME ?? os.homedir(), and core's preload — five lines — never setit. So every core run created
~/.red/redcodeon the runner, shared one binary cache acrossconcurrent jobs, and let an ambient
~/.npmrcreach the installer under test. The preload nowisolates home and XDG and clears provider keys, the way
packages/redcode's already does.One deliberate wrinkle: the isolated home is shared across core test processes rather than per-pid.
Global.Path.binhangs off it and is where ripgrep lands when the machine has none onPATH; per-pidwould isolate perfectly and make every suite download it again.
packages/coreruns at 30 s per test. It was on bun's 5 s default — the only heavyweight packagestill there — while spawning
git, spawningnode, and resolving ripgrep across 149 files on ashared 4-vCPU runner.
Deliberately not done
No blanket retry on the test jobs. Retrying setup steps is right: they are I/O against someone
else's server. Retrying
bun turbo testwould have hidden every real defect found today.Verification
packages/core1143 pass with the isolated home;packages/redcode/test/cli378 pass with thewatcher off in subprocesses. The workflow changes prove themselves over the next runs — that is what
a flake fix is.
https://claude.ai/code/session_014XwJPDhq1ahcm3rd454WzQ
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.