Template layer-cache artifacts leak: build writes one dir per layer, only the final build is in env_builds (GC missing) #3694

Summary

Template deletion (#3239) and its fix PR #3343 only reclaim the final build artifact. A template build writes one storage directory per layer (base, user, each step, ensure-free-disk, optimize), each under its own fresh build ID, but only the final build has a row in env_builds. Every intermediate layer directory — plus the content-addressed layer cache ({scope}/files/{hash}.tar) and its index ({scope}/index/{hash}) — has no DB row and no garbage collector, so it leaks permanently.

This was surfaced by @eren23 on #3239 (self-hosted Embed, local-disk template storage): deleting a template freed only ~13% of the build's disk; the other layers stayed and were reused from the layer cache on the next build of the same recipe.

Why PR #3343 cannot cover this

GetExclusiveBuildsForTemplateDeletion joins env_build_assignments → env_builds:

JOIN "public"."env_builds" eb ON eb.id = eba.build_id
WHERE eba.env_id = @template_id

Only the final build is ever inserted into env_build_assignments/env_builds. The intermediate layer build IDs are minted per phase and never recorded there:

• packages/orchestrator/pkg/template/build/phases/base/builder.go:340 — BuildID: uuid.New().String()
• packages/orchestrator/pkg/template/build/phases/steps/builder.go:140 — BuildID: uuid.NewString()
• packages/orchestrator/pkg/template/build/phases/ensurefreedisk/builder.go:91 — BuildID: uuid.NewString()

Each produces a metadata.Template{BuildID: <uuid>} whose artifacts upload to {buildID}/ (packages/shared/pkg/storage/paths.go, StorageDir() returns BuildID). So the exclusivity query — and therefore PR #3343 — is structurally blind to every layer except the last.

The layer cache is content-addressed and cross-template

The layer cache is a hash→buildID index decoupled from env_builds:

• packages/orchestrator/pkg/template/build/storage/cache/cache.go — SaveLayerMeta / LayerMetaFromHash write/read {scope}/index/{hash} → {buildID}
• packages/orchestrator/pkg/template/build/storage/paths/*.go — layer files at {scope}/files/{hash}.tar (GetLayerFilesCachePath), index at {scope}/index/{hash} (HashToPath)

A rebuild of the same recipe under a different template resolves the same hashes and reuses the same layer build IDs. So a layer cannot be deleted "because template X was deleted" — it can only go once no build of any template references it. That is a reference-counting / mark-sweep problem, not a cascade-on-delete problem.

No GC exists

DeleteObjectsWithPrefix on build artifacts is called in exactly two places:

• build-failure rollback (packages/orchestrator/pkg/template/build/builder.go:212)
• the TemplateBuildDelete gRPC (build cancel / timeout, and now PR fix(api): delete GCS/S3 artifacts when template is deleted #3343 for the final build)

There is no sweep over {scope}/files/ or {scope}/index/, and no reference table for layers. Intermediate layer directories and unreferenced cache blobs are pure orphans. On GCS/S3 this is silent cost growth; on self-hosted local-disk storage (Embed) it fills the node's disk — one template build per commit of a 4.2 GB image adds ~3.7 GiB that never comes back (per @eren23).

Impact

• Storage leak, ~87% uncovered by fix(api): delete GCS/S3 artifacts when template is deleted #3343. The dominant share of every build (all non-final layers + cache blobs) leaks regardless of template deletion.
• Local-disk nodes fill up (Embed): a build-per-commit workload exhausts the disk.
• Existing GCS/S3 deployments accumulate orphaned files//index/ blobs with no mechanism to ever reclaim them.

Proposed direction

Layer reclamation needs a GC keyed on reachability / reference count, not on template ownership:

1. Mark-sweep GC (recommended): periodically walk {scope}/index/ and {scope}/files/, mark everything reachable from the currently-live set of builds and their layer chains, delete the unreachable. Works for both object storage and local disk, and reclaims pre-existing orphans (the only thing that helps a node whose disk is already full).
2. Layer reference table: record, at build time, every layer hash/build a build depends on; delete a layer only when its refcount reaches zero. More precise, larger change.
3. Local-disk LRU / capacity cap (stopgap for Embed): evict least-recently-used files//index/ entries under a size bound to stop the bleeding quickly.

Recommend 1 + 3: mark-sweep for correctness and existing-orphan reclamation, local-disk LRU as immediate relief for self-hosted nodes.

Relationship to #3239 / #3343

• Template deletion does not remove build artifacts from storage (GCS/S3) #3239 = artifacts not deleted on template delete.
• fix(api): delete GCS/S3 artifacts when template is deleted #3343 fixes that for the final build only.
• This issue = the layer-cache orphans (the remaining majority), which is a distinct GC problem and should not block fix(api): delete GCS/S3 artifacts when template is deleted #3343.

cc @eren23