Repository navigation
Description
User Story
I run shared OpenShell gateways where agents create short-lived sandboxes all day (create, run a few commands, delete), and the gateways are expected to stay up for weeks. In HA runs, each gateway held 0.7–1.3 GiB with 300 sandboxes on a 5-node Kubernetes cluster, and 2.7 GiB with 1,080 sandboxes on a larger HA run: about 2–3.6 MiB per sandbox on every gateway. After sandboxes are deleted, gateway memory doesn't go back down, so we can't tell freed memory from leaked memory or size gateways from measurements.
Description
The gateway uses the default glibc allocator (distroless/cc-debian13 base image; no #[global_allocator] in the gateway binary). Freed blocks, such as the ~1.7 MiB per-sandbox buffers, often stay mapped in the process, so RSS does not drop after sandboxes are deleted. Evaluate jemalloc, or tuning such as MALLOC_ARENA_MAX, so freed memory returns to the OS.
Context
This is not the leak itself; the leaks are tracked in #5399, #5400, #5401 and #5402. Once those land, there is less memory to return, so measure before switching.
Definition of Done
- Gateway RSS is measured after a create/delete cycle, with the fixes for bug(gateway): per-sandbox log/event buffers pre-allocate ~3.4 MiB even when idle #5399–bug(gateway): sandbox log/event buffers are re-created after delete and leak #5402 in place, on the default allocator and on jemalloc (or tuned glibc).
- Switch only if RSS returns meaningfully closer to baseline without a CPU or latency regression; otherwise close with the numbers.
Activity