Skip to content

flake: test-go-pg (macos) job timeout/cancelled #1365

Open
Open
flake: test-go-pg (macos) job timeout/cancelled#1365
Assignees
BobbyHo
Labels
flake

Description

@flake-investigator
flake-investigator
bot
opened on Feb 23, 2026

Current understanding (updated 2026-10-07)

The macOS failures on this ticket are memory exhaustion inside the cli test binary, not a runner pause. On 2026-10-07 the process allocated about 6 GiB in 8 seconds, grew to 47 GB resident on a 17 GiB runner, pushed swap to 34 GB, and exited silently before the Go test timeout could dump goroutines. The per-second runner sampler never gapped, so the host did not stall. The "runner suspension / severe host stall" classification in the 2026-08-06, 2026-09-27, and 2026-10-07 comments, and the earlier host-side theory, are superseded. See the 2026-10-07 analysis comment.

The allocation site is not yet known. coder/coder#30487 adds a memory guard that fails the process at 6 GiB with a heap profile and a goroutine dump. On the next macOS occurrence: download the heap-profiles-test-go-pg-macOS artifact and run go tool pprof -top -sample_index=alloc_space on it. The panic line in the job log names the file.

Scope: this ticket tracks the macOS cli.test memory burst. The 2026-09-02 and 2026-09-07 Linux occurrences show short scheduling gaps with flat memory and are not shown to share this cause. The 2026-10-06 macOS hang in TestAgent_SSHConnectionEnvVars/SSH_CLIENT is also separate. New occurrences of either should get their own ticket.

The original report below (February 2026, GOCACHEPROG shutdown hang after all tests passed) was a different failure and is kept for history.

CI Run Link: https://github.com/coder/coder/actions/runs/22303873776
Failed Job: test-go-pg (macos-latest) https://github.com/coder/coder/actions/runs/22303873776/job/64518157527
Aggregator failure: required job failed because test-go-pg was cancelled

Commit Info:

  • SHA: b776a14b46fc14a972d15d59473daa5f150d452c
  • Author: Thomas Kosiewski
  • Message: fix(coderd): harden OAuth2 provider security (#22194)

Failure evidence:

  • required job output:
- test-go-pg: cancelled
One of the required checks has failed or has been cancelled
  • macOS job log (test step):
2026-02-23T11:24:37Z ok 	github.com/coder/coder/v2/cli 100.025s
2026-02-23T11:44:25Z ##[error]The operation was canceled.

Timeline / timeout context:

  • test-go-pg job has timeout-minutes: 25 in .github/workflows/ci.yaml.
  • Job started at ~11:19:13Z and was cancelled at 11:44:25Z (≈25 minutes).
  • No further test output appeared between 11:24:37Z and cancellation, suggesting the make test step hung or stalled and hit the job timeout.

Root cause classification: Infrastructure / test suite hang

  • The macOS test-go-pg job exceeded the 25-minute job timeout during make test (gotestsum over ./cli/... ./enterprise/cli/... ./agent/...).
  • No WARNING: DATA RACE, panic, or OOM indicators found in the job log; cancellation is consistent with the workflow timeout rather than a specific test failure.

Assignment analysis (component / infra ownership):

  • No specific test function identified (gotestsum standard-quiet output stops before hang), so test-level blame isn’t possible.
  • Using infra ownership fallback:
    • git log --oneline -10 .github/actions/test-go-pg/action.yaml
      • 7d558e7 fix: make make test runnable again (Danny Kopping)
      • 84b7a03 chore: abstract pg test logic and double runner sizes (Danny Kopping)
  • Assigning to @dannykopping as the most recent meaningful maintainer of the test-go-pg action and parallelism logic.

Related issues search:

  • Searched coder/internal for: "test-go-pg macos cancelled", "operation was canceled", "test-go-pg timeout", "job timeout" — no matching open issues found.

Reproduction:

  • Re-run the CI workflow on main and watch test-go-pg (macos-latest).
  • Locally on macOS: TEST_PACKAGES="./cli/... ./enterprise/cli/... ./agent/..." make test (observe for hang or unusually long runtime).

Activity

  1. flake-investigator
    assigned
    dannykopping
    on Feb 23, 2026
  2. flake-investigator
    added
    flake
    on Feb 23, 2026

  3. zedkipp commented on Feb 23, 2026

    @zedkipp
    zedkipp
    on Feb 23, 2026

    I took a first pass at this failure. Seems like all tests passed, but something got stuck after all tests ran.

    Some interesting logs from 16_test-go-pg (macos-latest).txt

    11:19:48 GOCACHEPROG='/usr/local/bin/depot gocache'
<snip>
11:24:37  ok  github.com/coder/coder/v2/cli  100.025s      <- last test package passes
<20 minutes of zero output>
11:44:25  ##[error]The operation was canceled.             <- 25 min job timeout fires

    Then at cleanup:

      11:44:28Z  Terminate orphan process: pid (3969) (depot)
  11:44:28Z  Terminate orphan process: pid (3968) (go)
  11:44:28Z  Terminate orphan process: pid (3913) (postgres)

    During cleanup the depot process (pid 3969) and its parent go process (pid 3968) were both still alive 20 minutes after all tests passed. It seems like go test couldn't exit because depot gocache (spawned via GOCACHEPROG) didn't shut down.

    I did some digging on how GOCACHEPROG works. go essentially tells the subprocess to exit over stdin/stdout with a protocol specific to GOCACHEPROG.

    Happy path expectation:

    1. go sends a close message on the subprocess's stdin
    2. The subprocess replies with a response, then exits
    3. go detects the exit by reading EOF on the subprocess's stdout

    Given that the depot and go process were both still alive, the hang could be:

    • depot gocache never received or processed the close command
    • depot gocache received close but hung during its cleanup/flush
    • go never sent the close command

    I didn't find anything in the Go issue tracker for any known issues with the cleanup/shutdown pathway. I am going to send a message to Depot and see if this is a known issue.

  4. dannykopping commented on Feb 24, 2026

    @dannykopping
    dannykopping
    on Feb 24, 2026
    Collaborator

    @zedkipp thanks for taking a look! I'll close this from our side in the interim, but please post any updates you get.

  5. dannykopping
    closed this as not plannedon Feb 24, 2026
  6. flake-investigator
    reopened this on May 15, 2026

  7. flake-investigator commented on May 15, 2026

    @flake-investigator
    flake-investigatorBot
    on May 15, 2026
    Author

    Recurrence on main:

    Evidence from job log:

    2026-05-15T20:57:51.9884830Z ok  	github.com/coder/coder/v2/enterprise/cli	56.900s
2026-05-15T21:18:44.0097770Z ##[error]Step canceled by GitHub (see: https://depot.dev/docs/github-actions/troubleshooting#error-step-canceled-by-github)

    Notes:

    • Job timeout is still timeout-minutes: 25 in .github/workflows/ci.yaml (test-go-pg).
    • Job started at ~20:53:29Z and was canceled at 21:18:44Z (~25m), consistent with timeout during make test (packages: ./cli/... ./enterprise/cli/... ./agent/...).
    • No WARNING: DATA RACE, panic traces, or OOM indicators found in the retrieved logs.

    Root cause remains: macOS test-go-pg job timed out/hung during tests; required check failed due to cancellation.

  8. f0ssel
    unassigned
    dannykopping
    on May 28, 2026

  9. flake-investigator commented on Jun 25, 2026

    @flake-investigator
    flake-investigatorBot
    on Jun 25, 2026
    Author

    New occurrence (same failure mode):

    CI Run: https://github.com/coder/coder/actions/runs/28139812015
    Failed Job: https://github.com/coder/coder/actions/runs/28139812015/job/83336219257 (test-go-pg (macos-latest))
    Aggregator failure: required job failed because test-go-pg was cancelled.

    Commit Info:

    Failure date: 2026-06-25

    Key evidence:

    • required job output:
    - test-go-pg: cancelled
One of the required checks has failed or has been cancelled
    • macOS job log (test step):
    2026-06-25T01:42:51.8824110Z ##[error]Step canceled by GitHub (see: https://depot.dev/docs/github-actions/troubleshooting#error-step-canceled-by-github)

    Timeout context:

    • Job ran from ~01:17:38Z to 01:42:51Z (~25m) which matches the workflow timeout-minutes: 25 for test-go-pg in .github/workflows/ci.yaml.

    No WARNING: DATA RACE, panic, or OOM indicators found in the retrieved logs.

  10. BobbyHo
    self-assigned this
    on Jul 10, 2026
  11. BobbyHo
    added a commit that references this issue on Jul 15, 2026

    fix(testutil/expecter): remove nested pipe from output logging path (#…

    511c3cd
  12. BobbyHo
    closed this as completedon Jul 15, 2026
  13. linear-code
    reopened this on Aug 6, 2026
  14. BobbyHo
    added a commit that references this issue on Aug 10, 2026

    fix: raise test-go-pg timeout budget, lower macOS parallelism (#27997)

    bd693ad

  15. BobbyHo commented on Aug 25, 2026

    @BobbyHo
    BobbyHo
    on Aug 25, 2026 – with Linear Code

    Verification: no recurrence since the last fix (checked 2026-08-25)

    Note on which fix is "the last one": the PR attached to this ticket (coder/coder#27204, merged 2026-07-15) was only a partial fix and said so. A second change landed after the 2026-08-06 recurrence reported above:

    PR Merged Change
    coder/coder#27204 2026-07-15 testutil/expecter: removed the nested io.Pipe in the debug-logging path that could deadlock a command's writes
    coder/coder#27997 2026-08-10 go test -timeout 20m to 25m, job timeout-minutes 25 to 30, macOS test-parallelism-tests 16 to 8

    The 2026-08-06 occurrence therefore predates the current mitigation and is not evidence against it.

    Result

    No recurrence of the timeout/hang signature in the 15 days since #27997.

    • 233 of 233 test-go-pg (macos-latest) jobs on main succeeded since 2026-08-10T19:29Z. Method: listed all 637 ci.yaml runs on main since 2026-07-15, then pulled job-level results for every run in the post-fix window (288 runs; the rest of the gap is cancelled runs that were superseded by a later push before any job started).
    • Zero cancelled macOS jobs. The original 25-minute job-cancellation signature is absent.
    • Job duration 4.7m min / 5.8m median / 7.2m max against a 30m job budget and 25m Go timeout. Pre-fix, healthy runs of the same job ranged 5m to 12.2m, so the slow tail tightened rather than growing under the lower parallelism.

    The only three macOS test-go-pg failures in the whole window since #27204 all predate #27997:

    Date Duration Classification
    2026-07-28 14.0m job, logs expired; same run also failed test-js and Storybook
    2026-08-06 24.4m job, the package timeout reported above (593 failures, 572 (unknown))
    2026-08-06 8.2m job, failed in Upload Test Cache after tests passed. Unrelated to this ticket

    Related but not this flake

    nightly-gauntlet test-go-pg (macos-latest) failed on 2026-08-11, 2026-08-19, and 2026-08-20, after the fix. These are not this failure mode: each ended with DONE ~29,700 tests, 1 to 3 failures, that is, a small number of genuine per-test flakes. This ticket's signature is a single process death that marks hundreds of in-flight subtests (unknown), which is what 2026-08-06 showed and these do not.

    No new flake-bot report has landed on this issue since 2026-08-06.

    Scope and caveat

    The scan covered main ci.yaml runs and nightly-gauntlet. It did not cover PR-branch runs, where failures are dominated by genuine test failures and the API result cap makes an exhaustive sweep impractical. Flake alerting for this ticket fires on main, so this matches how the issue has been tracked.

    #27997 is a mitigation, not a confirmed root cause: the underlying macOS runner scheduling stall is still unexplained, and the Depot GOCACHEPROG shutdown hang from the original report lives outside this repo. The 25m/30m split is deliberate so that a future hang produces a goroutine dump from go test instead of a silent runner kill. Recommend closing on the evidence above and reopening if the flake bot files another occurrence, at which point the goroutine dump should be available.

  16. BobbyHo
    closed this as completedon Aug 25, 2026
  17. linear-code
    reopened this on Sep 2, 2026
  18. BobbyHo
    added a commit that references this issue on Sep 8, 2026

    ci: move Linux Go temp dir to tmpfs and stop Postgres statement loggi…

    d69e235
  19. linear-code
    closed this as completedon Sep 28, 2026
  20. BobbyHo
    reopened this on Sep 28, 2026
  21. linear-code
    closed this as completedon Sep 29, 2026
  22. linear-code
    reopened this on Oct 7, 2026
  23. linear-code
    closed this as completedon Oct 7, 2026
  24. BobbyHo
    reopened this on Oct 7, 2026

  25. BobbyHo commented on Oct 7, 2026

    @BobbyHo
    BobbyHo
    on Oct 7, 2026 – with Linear Code

    Correction, 2026-10-07: the macOS failures are memory exhaustion inside the cli test binary, not a runner pause. The telemetry from coder/coder#30050 shows the process allocating about 6 GiB in 8 seconds and going deep into swap while the runner's per-second sampler never gapped. The "runner suspension / severe host stall" classification and the earlier host-side theory are superseded.

    The allocation site is still unknown. coder/coder#30487 adds a memory guard that fails the process at 6 GiB with a heap profile and a goroutine dump, and uploads the profile. Full details are in the 2026-10-07 analysis comment on the Linear issue.

  26. BobbyHo
    added a commit that references this issue on Oct 14, 2026

    ci: fail the test process with a heap profile when memory passes a pe…

    11f1ee7
  27. BobbyHo
    closed this as completedon Oct 14, 2026
  28. linear-code
    reopened this on Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

flake

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions