Repository navigation
flake: TestProxy_HotReloadRouting (multiple variants, Windows EOF) #1564
Description
This issue tracks intermittent Windows EOF failures from routed HTTPS requests in the enterprise/aibridgeproxyd hot-reload test family. The routing decision succeeds, but the loopback proxy-to-aibridged round trip returns no response; goproxy closes the MITM connection and the test client surfaces EOF.
Original occurrence:
CI Run: https://github.com/coder/coder/actions/runs/26933139214
Job: https://github.com/coder/coder/actions/runs/26933139214/job/79456825080
Commit: becc858fa85182ed55a45d1a317bc7e60da8f8bb (author: Ethan)
Failure:
=== FAIL: enterprise/aibridgeproxyd TestProxy_HotReloadRoutingInvalidProviders/RefreshErrorPreservesPreviousSnapshot
reload_test.go:574:
Error: Received unexpected error:
Post "https://alpha.invalid/v1/messages": EOF
Messages: request to routed host must succeed
Error analysis:
- The proxy routed request to aibridged returned EOF during the "refresh error preserves previous snapshot" assertion.
- The test expects the previous router snapshot to continue serving requests after a reload error; the EOF suggests a transient proxy/MITM connection failure.
Root cause classification: flaky test / Windows loopback infrastructure pressure.
Race/Panic/OOM evidence: none found in the original logs.
Assignment analysis:
git log --oneline -10 --follow enterprise/aibridgeproxyd/reload_test.go12520ee964"feat: add ai provider status and reload freshness metrics" (Danny Kopping)a9f5ed7644"fix: re-validate provider per request and classify reloads" (Danny Kopping)79e007cf30"feat: hot-reload aibridged and aibridgeproxyd providers on DB changes" (Danny Kopping; introduced this test file and CRUD test)
Assigned to Danny based on test/component ownership, not the failing commit author.
Reproduction:
go test ./enterprise/aibridgeproxyd -run 'TestProxy_HotReloadRouting' -count=100- The original occurrence could not be reproduced on Linux after 5,000+ iterations; the observed failures are on Windows under full-suite loopback connection churn.
Activity
linear-code commentedon Jun 4, 2026
ENG-2828
dannykopping commentedon Jun 19, 2026
I couldn't see anything obviously wrong here, and I got Opus 4.8 to look closer:
Root cause: transient network blip on the Windows CI runner, not a product bug
The failure is on the
test-go-pg (windows-2022)runner. The CI slog output shows both routed requests in the subtest:routing MITM request to aibridged ... provider=alpha→received response from aibridged status=200✅routing MITM request to aibridged ... provider=alpha→ no response line →Post "https://alpha.invalid/v1/messages": EOF❌Key takeaways
The routing logic worked correctly. Request 2 was re-validated against the live router, matched
provider=alpha, and was forwarded to aibridged. It did not hit the pass-through branch, so the "refresh error preserves snapshot" behavior and the per-request re-validation (commitsa9f5ed7644/12520ee964) are fine.The EOF came from the proxy→aibridged hop. The missing
received response from aibridgedline meansproxy.Tr.RoundTrip()to the loopback aibridged backend returned an error. goproxy handles an upstream round-trip error by silently closing the client's MITM TLS connection with no response, so the client surfaces a bareEOF.Not a stale-connection reuse race. I instrumented the backend: every request arrives on a fresh source port with
Connection: close(the test client'sDisableKeepAlivespropagates through the MITM), so each request opens and tears down fresh TCP connections. The two requests are structurally identical; one just hiccupped.Environmental trigger. ~24k tests churning fresh loopback connections on Windows (ephemeral-port /
TIME_WAITpressure) makes a one-off fresh-connect/round-trip failure plausible. Could not reproduce on Linux (5000+ iterations under-race, lowGOMAXPROCS, high parallelism = all green).Why the test fails on a blip
expectRoutedTotreats any request error as fatal (require.NoError(..., "request to routed host must succeed")). A single transient TCP/TLS error doesn't disprove "this host is routed to aibridged."[-]flake: TestProxy_HotReloadRoutingInvalidProviders/RefreshErrorPreservesPreviousSnapshot[/-][+]flake: TestProxy_HotReloadRouting (multiple variants, Windows EOF)[/+]flake-investigator commentedon Aug 4, 2026
New occurrence of the same Windows routed-request
EOFfailure mode.1f946bb50bc5d6eed5c156b920093ba3fcb9b43b(ci(.github): retry Go module downloads, author Ethan)test-go-pg (windows-2022), run attempt 1TestProxy_HotReloadRoutingCRUDKey evidence:
This ties directly to the original occurrence: the router successfully reloaded and selected
alpha, CONNECT authentication succeeded, and the request entered the MITM-to-gateway path, but no backend response was logged before the client receivedEOF. That matches the previously analyzed transient Windows loopback proxy-to-aibridged round-trip failure rather than incorrect hot-reload routing.Root cause classification: flaky test / Windows loopback infrastructure pressure. The full 2,711-line job log contains one test failure and no process crash. Race detection was disabled for this job; there are no
WARNING: DATA RACE/race detectedindicators. There are also nopanic:, runtime fatal, OOM,signal: killed, disk, file-descriptor, or resource-exhaustion indicators.Assignment trace:
git blame -L 391,473 enterprise/aibridgeproxyd/reload_test.gopoints the CRUD test/harness ownership to the hot-reload work introduced in79e007cf30by Danny Kopping.git log --oneline -10 --follow enterprise/aibridgeproxyd/reload_test.goshows the latest meaningful test/component changes in12520ee964,a9f5ed7644, and79e007cf30, all by Danny.Suggested reproduction/stress command on Windows:
dannykopping commentedon Aug 4, 2026
Same issue, closing
fix: disable keep-alives on mock aibridged to prevent stale-conn EOF (#…