flake: TestRunStandaloneGateway_ContextCanceled

[flake-investigator]

CI failure

• CI run: https://github.com/coder/coder/actions/runs/30606104460
• Failed job: https://github.com/coder/coder/actions/runs/30606104460/job/91078547153 (test-go-pg (macos-latest))
• Failure date: 2026-07-31 05:53 UTC
• Commit: 79724ab0baef2e9f5565e8c335bfb7c214e918f6
• Commit author: Jake Howell
• Run attempt: 1

The job failed within seconds of the Slack alert. All 8,999 lines of the failed job log were available and reviewed. This is an actual test failure on attempt 1, not a matrix cancellation/rerun artifact. The Windows matrix job passed.

Failure evidence

=== FAIL: enterprise/cli TestRunStandaloneGateway_ContextCanceled (0.03s)
    aigatewaystart_internal_test.go:339:
        Error Trace: /Users/runner/work/coder/coder/enterprise/cli/aigatewaystart_internal_test.go:527
                     /Users/runner/work/coder/coder/enterprise/cli/aigatewaystart_internal_test.go:339
        Error:       Received unexpected error:
                     listen tcp 127.0.0.1:58565: bind: address already in use
        Test:        TestRunStandaloneGateway_ContextCanceled
        Messages:    HTTP listener must be closed before run returns

DONE 28764 tests, 78 skipped, 1 failure in 1356.462s

This was one assertion failure in enterprise/cli, not a package-process crash.

Error analysis and root cause

Classification: flaky test — ephemeral-port reuse/collision on macOS.

The test runs in parallel and obtains its address through:

address := fmt.Sprintf("127.0.0.1:%d", testutil.RandomPort(t))

testutil.RandomPort asks the OS for 127.0.0.1:0, closes that listener, and returns the released port. Its own documentation warns that the OS may reallocate the port very quickly. After cancellation and runStandaloneGateway returning, the test tries to prove shutdown by binding the same address again:

func requireListenerAvailable(t *testing.T, address, message string) {
    listener, err := net.Listen("tcp", address)
    require.NoError(t, err, message)
    require.NoError(t, listener.Close())
}

That final bind is not exclusive evidence about the gateway: once the gateway listener closes, another parallel test or process can claim the released ephemeral port before this assertion runs.

The production shutdown path does not show evidence of a leaked listener. standaloneGateway.serve:

1. starts http.Server.Serve with defer listener.Close();
2. cancels and waits for provider reload;
3. calls httpServer.Shutdown, falling back to Close on error;
4. waits for the HTTP serving goroutine with serveWG.Wait() before returning.

Therefore the best assessment is that the availability probe raced with unrelated port allocation, rather than that runStandaloneGateway returned while its own listener remained open. A more deterministic test should retain ownership of a listener/port reservation or verify server-goroutine shutdown directly instead of re-binding a released ephemeral port.

The failing commit only migrates site dialogs and is unrelated to the AI Gateway test, so it was not used for assignment.

Race, panic, OOM, and resource checks

The complete log contains no:

• WARNING: DATA RACE or race detected during execution of test;
• panic or runtime-error trace;
• OOM, signal: killed, or allocation failure;
• disk, file-descriptor, or other resource-exhaustion indicator.

This is not a data race report or process crash.

Duplicate search

Searched open and closed coder/internal issues, including issues closed in the last 30 days, for:

• TestRunStandaloneGateway_ContextCanceled and TestRunStandaloneGateway;
• aigatewaystart_internal_test.go and standalone AI Gateway flakes;
• HTTP listener must be closed before run returns;
• bind: address already in use, listener timing, RandomPort, and port collisions;
• related panic, process-crash, OOM, unknown-failure, and data-race signatures.

No issue describes this test and failure mode.

Related but not duplicate:

• flake: TestStandaloneGatewayServe_ShutdownOrder #1633 is another test in the same file, but it times out waiting for initial provider loading under -race; this occurrence fails a post-shutdown port re-bind on non-race macOS.
• flake: TestWorkspaceProxy_Server_PrometheusEnable #92 contains the same generic bind error in a workspace-proxy Prometheus test from 2024, a different component and root cause.

Assignment analysis

Ownership commands/line ranges:

git blame -L 322,340 enterprise/cli/aigatewaystart_internal_test.go
git log --oneline -10 --follow enterprise/cli/aigatewaystart_internal_test.go
git log --oneline -20 -- enterprise/cli

The exact failing function, its newStandaloneGatewayTestParams setup using testutil.RandomPort, and requireListenerAvailable were added together in 468b1a27 (fix: remove standalone AI Gateway http listener dependency on loading providers, PR #27303) by Paweł Banaszewski (pawbana). Recent history of this test file is also predominantly Paweł's standalone AI Gateway work (bab8ce9d, ccba3969).

Assigning pawbana based on exact test-function ownership, not the failing run's commit author.

Reproduction

The collision is timing-dependent and most likely under macOS/full-suite port churn:

go test ./enterprise/cli -run '^TestRunStandaloneGateway_ContextCanceled$' -count=100

Running the full package/suite with parallel tests on macOS may be required. A focused stress harness that repeatedly allocates 127.0.0.1:0 listeners in parallel should increase the chance of reproducing the final re-bind collision.

[linear-code]

ENG-3115