Skip to content

Test Suite (ubuntu-latest) intermittently crashes with tinypool ERR_IPC_CHANNEL_CLOSED, unrelated to PR content #4307

Open

Description

@ruvnet

Problem

node scripts/ci-test-ratchet.mjs (the `Test Suite (ubuntu-latest)` job in `.github/workflows/ci.yml`) intermittently crashes before writing its report:

```
⎯⎯⎯⎯ Unhandled Rejection ⎯⎯⎯⎯⎯
Error: Channel closed
❯ target.send node:internal/child_process:753:16
❯ ProcessWorker.send node_modules/tinypool/dist/index.js:100:41
...
Serialized Error: { code: 'ERR_IPC_CHANNEL_CLOSED' }

CI test ratchet: report was not produced: /home/runner/work/ruflo/ruflo/.test-results/vitest.json


Observed 3 times out of 5 consecutive runs on PR #4306 (a release PR whose own diff is three package.json version bumps, a changelog, and two one-line test fixes — nothing that touches vitest config, worker pools, or the file below). The other 2/5 runs got further and surfaced two unrelated, genuinely pre-existing issues (now fixed in #4306): a sql.js 1.14 exports-map regression in `fts5.test.ts`, and a stale `timeout-minutes` assertion from #3950.

## Evidence / pattern

The crash consistently lands right after (or very near) the last test in `plugins/ruflo-business-pods/scripts/pod-tick.test.mjs`:

Subtest: CLI: envelope is written to room JSONL backing store

ok 21 - CLI: envelope is written to room JSONL backing store
[crash follows]


That test (and several siblings in the same file) use `spawnSync('node', [POD_TICK, ...])` — a **blocking, synchronous** child-process spawn inside a vitest test. No `vitest.config.*` exists at the repo root, so vitest's defaults apply: no explicit `pool`/`maxForks`, default `testTimeout` (5s). A hypothesis worth checking: if a `spawnSync` call runs long enough under concurrent CI load, vitest's timeout-driven forced termination of that test could kill the whole forked worker process hosting it (not just the hung test), which would manifest as exactly this `ERR_IPC_CHANNEL_CLOSED` when the orchestrator next tries to hand that worker another file.

## Why this matters

The repo currently has **no branch protection on `main`** (`gh api repos/ruvnet/ruflo/branches/main/protection` → 404), so nothing technically blocks a merge on this — but it burns CI minutes and reviewer attention repo-wide on every PR that happens to hit it, and it's the kind of flake that erodes trust in a red check.

## Suggested next steps (not yet attempted, flagging for someone with more CI-runner-resource context)

- Add an explicit, conservative `--pool=forks --poolOptions.forks.maxForks=2` (or similar) to the `vitest run` invocation in `scripts/ci-test-ratchet.mjs`, trading some wall-clock speed for stability, if resource pressure is the cause.
- Or raise `testTimeout`/`hookTimeout` for the suite so a slow `spawnSync` child doesn't trigger a timeout-driven kill.
- Or move the heaviest `spawnSync`-based suites (`plugins/ruflo-business-pods/scripts/pod-tick.test.mjs` and similar) to their own isolated CI step/job, separate from the main vitest ratchet run.

🤖 Generated with [RuFlo](https://github.com/ruvnet/ruflo)

Activity

  1. ruvnet commented on Oct 20, 2026

    @ruvnet
    OwnerAuthor

    Update: my pod-tick.test.mjs hypothesis was wrong (or at least incomplete). #4308 fixes a real, independently-justified miscategorization (that file imports node:test, not vitest, and belongs in ci-test-excluded.txt) — but after merging that fix into the release branch, the crash recurred a 7th time, at a completely different point in the suite (a desktop/runner-heartbeat test file, unrelated to ruflo-business-pods).

    So the crash location is not fixed to one file — it's moved at least twice now. This points away from "one specific file corrupts the worker" and back toward generic resource pressure (likely memory) building up over the ~70-90s a worker runs before getting killed, with the exact failure point depending on which file happens to be running when the threshold is crossed. The suggested mitigations in the issue body (reduce maxForks, raise timeouts, or isolate heavy suites) are still worth trying, but no single-file fix is going to resolve this — it needs someone to actually profile memory usage of the Test Suite (ubuntu-latest) job over its run, or just reduce vitest's fork pool size as a first cheap experiment.

    Observed so far: 7 runs on PR #4306, 5 crashed with ERR_IPC_CHANNEL_CLOSED at three different points in the suite, 2 surfaced genuine (now-fixed) content bugs once they got far enough to finish. #4308 (the pod-tick categorization fix) still stands on its own merits and should still merge, just isn't the full fix for this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions