Skip to content

fix(runner): run the runner under tini so orphaned children get reaped - #7915

Merged
mmabrouk merged 1 commit into
release/v0.127.2from
agenta-/runner-tini
Oct 29, 2026
Merged

mmabrouk merged 1 commit into
release/v0.127.2from
agenta-/runner-tini

Conversation

@mmabrouk

Copy link
Copy Markdown
Member

Problem

On Kubernetes, runner pods collect zombie (<defunct>) processes. A runner pod showed six zombies after 29 minutes: three pi and three pi-acp, about two for each local session. The zombies hold no memory. They do take process table slots, and they stay until the pod restarts.

The cause is PID 1. The image runs tsx src/server.ts, so tsx is PID 1. When a process in the chain sandbox-agent → pi-acp → pi exits, its children are handed to PID 1. Node only reaps children it started itself, so those processes stay as zombies. Docker compose already avoids this with init: true (#5557). The Helm chart and the Railway wrapper use the image default, so they never got the fix.

Change

Both runner images (Dockerfile.gh, Dockerfile.dev) install Debian's tini package (about 730 KB) and set ENTRYPOINT ["tini", "-s", "--"]. The CMD is unchanged, so a compose command: override still works.

  • tini forwards SIGTERM to its direct child only. The runner still gets the signal first and drains its turns, as before.
  • -s makes tini a subreaper. It still reaps when compose's init: true makes Docker's own init PID 1, and it prints no warning there.

This fixes the Helm chart, Railway, and plain docker run together, with no chart change.

Tests

On a local build of Dockerfile.gh, a Node parent starts sh -c "sleep 1 & exit 0", and ps runs 3 seconds later:

Setup PID 1 Orphaned sleep
Old behavior (--entrypoint node) node stays a zombie (Z)
New image default tini reaped
New image with --init (compose) docker-init, then tini reaped

docker stop on the real runner, with and without --init: the runner logs received SIGTERM, draining and cleaning up sandboxes and exits 0.

I did not build Dockerfile.dev locally. Its change is the same two lines.

What to QA

After a stage deploy, run a few local sessions. Then check that kubectl exec <runner-pod> -- ps -eo stat,comm | grep -c '^Z' stays at 0 and that a rollout still drains turns.

On Kubernetes nothing adds an init process, so tsx is PID 1 and every pi or
pi-acp whose parent exits stays a zombie for the life of the pod. Install tini
in both runner images and make it the ENTRYPOINT. -s keeps it working under
compose's init: true.

Entire-Checkpoint: 01M64JG63C6FPQW2N3PWCG2CGB
@coderabbitai

coderabbitai Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 34713b8 and ae9621a.

Files selected for processing: 2
  • services/runner/docker/Dockerfile.dev
  • services/runner/docker/Dockerfile.gh

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 6 remain after this review.

📝 Summary

Summary by CodeRabbit

  • Chores
    • Updated development and CI runner containers to manage child processes more reliably while keeping their existing commands unchanged.
    • This provides more consistent process handling across both container environments.

Both runner Docker images now install tini and use tini -s -- as the entrypoint. Their existing CMD instructions remain in place and run as tini's child.

Changes: Runner container startup

Layer / File(s) Summary
Install and configure tini
services/runner/docker/Dockerfile.dev, services/runner/docker/Dockerfile.gh
Both images install tini and set tini -s -- as the entrypoint. The existing CMD instructions remain unchanged.

Priority: Normal

Estimated code review effort: 2 (Simple) | ~5 minutes

Merge Risk: Minimal · up to ae962

The runner images retain their existing startup and shutdown behavior while adding zombie-process reaping.

Pre-merge checks: Passed 5
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: running the runner under tini to reap orphaned child processes.
Description check Passed The description directly explains the zombie-process problem, the tini-based fix, affected images, compatibility considerations, test results, and QA steps.
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (2 skipped: 2 unsupported.)

Finishing Touches
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Railway Preview Environment

Item Value
Status stopped
Commit ae9621a8d53885baf9feafa061eb13d18d7174a9
Manual expiry (UTC) Not granted
CI deadline (UTC) Not granted
Workflow https://github.com/Agenta-AI/agenta/actions/runs/40941451030

Dependent tests finished. Post /preview for manual review. Environment absence verified; retained storage requires separate provider accounting.

Post /preview as a new comment to request a one-hour preview. Repeating it while active does not extend the hour.
Preview data is disposable. Expiry is enforced by the next successful cleanup sweep, not an exact timer.
Environment deletion stops its compute. Provider-retained storage and permanent services may still be billed.

@mmabrouk mmabrouk added lgtm This PR has been approved by a maintainer merge queue In the release merge queue; an agent is landing it labels Oct 29, 2026
@mmabrouk
mmabrouk merged commit 5bb7cef into release/v0.127.2 Oct 29, 2026
75 of 77 checks passed
@mmabrouk mmabrouk removed the merge queue In the release merge queue; an agent is landing it label Oct 29, 2026
@mmabrouk mmabrouk mentioned this pull request Oct 29, 2026
23 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm This PR has been approved by a maintainer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant