Repository navigation
Bump Substrate to f03327bd: follow the current API and stop sending ate-target-actor twice - #77
Merged
Merged
Conversation
…l API env pinned Substrate 0b3d2d07 (2026-09-09). Against current Substrate the pinned client fails to decode any actor that has a snapshot: ActorStatus field 11 was a string and is now the repeated Snapshot list, so the second suspend and every DeleteEnvironment (which suspends first) failed with "string field contains invalid UTF-8" and released environments leaked as SUSPENDED. The bump (zero ateapipb changes between the c2052e45 deployed for testing and f03327bd) breaks only the manifest generator and the fake control server: - ActorTemplate: readyz is now wakeupProbe, snapshotsConfig is now snapshotConfig. Set preferredFidelity MEMORY explicitly: it is the server default, but the schema marks the field required, and environments must come back from suspend with their process state. Golden snapshots stay enabled by omitting goldenSnapshotConfig. - WorkerPool: spec.sandboxClasses is required (exactly one entry). Emit gvisor with configRef gvisor-default, the SandboxConfig a stock `ate-setup deploy ate-system` installs and the one the template already named; both now share one constant. - fakecontrol: WorkerPodIp is WorkerPodIps; a suspended actor now carries a Snapshot entry instead of the removed ExternalSnapshot. README: state the Substrate revision the manifests and client need.
Guest calls appended ate-target-actor twice: once in forwardOutgoingContext (copied from the caller's metadata and set again) and once in the guestConn interceptors. Since agent-substrate/substrate#2308 the router rejects a request that carries the header more than once, so every exec and file operation failed with "invalid actor reference". Nothing in the guest reads caller metadata, so drop the forwarding rather than filter it: the guestConn interceptors are the only writer of the routing header and a client can no longer pass routing headers through the proxy. The caller's context still flows for deadline and cancellation. The fake router now rejects a duplicated header like the real one, which makes the existing proxy test fail without this fix, and a new test sends a client-supplied ate-target-actor over both the unary and the stream path.
This was referenced Oct 13, 2026
Collaborator
Author
|
Live run on
|
Tomer Glottmann (tomergee)
marked this pull request as ready for review
October 13, 2026 02:34
Collaborator
Author
|
/assign Sunny (@sunnylovestiramisu) |
Jaana Dogan (rakyll)
approved these changes
Oct 13, 2026
Tomer Glottmann (tomergee)
added a commit
that referenced
this pull request
Oct 13, 2026
After the rebase onto #77 the fixtures and assertions in imagetemplate_test.go and server_test.go still named the old template fields. Use wakeupProbe / ContainerWakeupProbe and snapshotConfig / SnapshotConfig, and set preferredFidelity on the base fixture so the derived-template equality check also covers the field the server now requires.
This was referenced Oct 13, 2026
Closed
Tomer Glottmann (tomergee)
added a commit
that referenced
this pull request
Oct 13, 2026
After the rebase onto #77 the fixtures and assertions in imagetemplate_test.go and server_test.go still named the old template fields. Use wakeupProbe / ContainerWakeupProbe and snapshotConfig / SnapshotConfig, and set preferredFidelity on the base fixture so the derived-template equality check also covers the field the server now requires.
Tomer Glottmann (tomergee)
added a commit
that referenced
this pull request
Oct 13, 2026
…plus second-runtime layers (#70) * Run unmodified task images: guest injected as an OCI image volume Until now a template's container image had to be the ate-env-guest image itself, so running a task image meant rebaking it with the guest as the entrypoint, once per image. Substrate's image volumes allow the other split: the task image stays unmodified and the guest is mounted into it read-only and started from the mount. Two ways to get such a template: - ate-env manifest template --task-image <repo@sha256:...> writes a template whose container is the task image, with the guest image as a volume at /ate and the command re-rooted to /ate/ko-app/ate-env-guest. --workspace passes -workspace to the guest in either mode. - CreateEnvironment gains an image field. ate-env-api takes the request's template (default-template by default) as the base, derives "<base>-<12 hex digits of the digest>" from it on first use and reuses it for later environments on the same image; the response reports it. A derived name already taken by a template that runs something else is a FailedPrecondition, as is a missing base; an unpinned image, a derived name over 63 characters, or a base command that is not an absolute path to the guest is an InvalidArgument. Concurrent creates racing on one derived template both succeed. The derivation lives in internal/apiservice/imagetemplate.go and accepts either kind of base, guest-as-image or already injected, so a second runtime can reuse it. internal/ate gains Get/CreateActorTemplate and the fake control plane stores templates. The CLI gains create --image, the Python client create(image=), and the full-stack e2e suite a create-from-image test gated on ATE_ENV_TASK_IMAGE that checks the guest comes from the volume, the rootfs is the image's, and a second environment reuses the derived template. docs/task-images/README.md is the how-to (both flows, what an image needs, errors, cleanup) and docs/task-images/DESIGN.md the design (derivation rules, naming and idempotency, failure semantics, security, alternatives, future work). Stubs regenerated: Go with buf and protoc-gen-go v1.36.11, Python with grpcio-tools 1.61.3, keeping the committed license headers. * Inject a second runtime as a layer: guest sidecars and template layers A task-image template carries the guest as a read-only image volume; an agent framework's own in-sandbox daemon, such as OpenSandbox's execd, is one more volume plus a process that has to start next to the guest. The guest is the container's only process, so it is the one that starts it. ate-env-guest gains -sidecar (repeatable: an absolute binary path and its arguments, run without a shell, output logged with the binary's name as prefix, restarted with a backoff that doubles from one second to thirty and resets after a minute of stable running) and -sidecar-readyz (repeatable: URLs that must answer 2xx once before /readyz reports ready, so Substrate's wakeup probe admits traffic only when every runtime is serving). Bad -sidecar values fail at start rather than loop. ate-env manifest template gains --layer name=image@sha256:...=/mount, --sidecar and --sidecar-readyz, which become image volumes, mounts and guest flags on the template. Template derivation recognizes the guest volume by name rather than as "an image volume", so a base with layers still gets the guest added or kept correctly and environments created on demand with --image inherit the second runtime. Only a command that starts with the guest binary is re-rooted under the mount. docs/task-images/RUNTIMES.md describes the model, the flags, who can reach the second runtime from where (processes inside the environment now; outside callers still only reach port 80 through the router, which is the next piece), suspend and restart behavior, and the limits. examples/runtime-layers is a runnable demo with a static busybox as the stand-in runtime: httpd on 44772 as the sidecar under both a python and a debian task image, the second one created from an image on the layered base. The first attempt with the dynamically linked busybox failed on the debian image with a glibc version error, which is why the docs make the self-contained-layer rule and its failure signature explicit. * Address review: document CreateEnvironment defaults and prioritize derived-template GC Spell out on CreateEnvironment how id, template and atespace default, that one atespace resolves the template and places the actor, how an image request derives and reuses its template, and which inputs yield InvalidArgument versus FailedPrecondition. Move derived-template garbage collection to the top of the design's future work and record why it is not tied to environment deletion: the template is shared by every environment on the image and holds the golden snapshot. * Rename CreateEnvironmentRequest.image to task_image Review asked for a less ambiguous name now that runtime layers are also images on the template. The field is new in this branch, so the rename is free: proto field task_image (same number), Go GetTaskImage, CLI `ate-env create --task-image` to match `manifest template --task-image`, Python Client.create(task_image=), and the docs and examples. Generated Go and Python stubs regenerated; the Go file keeps main's descriptor path. * Follow Substrate f03327bd in the task-image tests After the rebase onto #77 the fixtures and assertions in imagetemplate_test.go and server_test.go still named the old template fields. Use wakeupProbe / ContainerWakeupProbe and snapshotConfig / SnapshotConfig, and set preferredFidelity on the base fixture so the derived-template equality check also covers the field the server now requires. * Rename the rootfs image field: task_image -> rootfs_image "Task" is overloaded (review on #70): in AI systems it names a unit of work, and at Google an execution boundary. The image is the environment's root filesystem, which the design doc already calls it, so name it that on the wire and everywhere the name shows: - proto: CreateEnvironmentRequest.rootfs_image (field 4, same number as the unreleased task_image, so wire-compatible) - CLI: `ate-env create --rootfs-image`, `ate-env manifest template --rootfs-image`, ROOTFS_IMAGE in the runtime-layers example - Python: Client.create(rootfs_image=...); stubs regenerated - Go: RootfsImage / GetRootfsImage; docs and comments follow guest_image was suggested as well, but that name already means the injected ate-env-guest image (--guest-image, the `guest` volume) and would invert the vocabulary. * README: image digests built from this branch * e2e: expect FailedPreconditionError for a missing base template The client maps FAILED_PRECONDITION to FailedPreconditionError since #65, a sibling of RpcError rather than a subclass, so the negative test written before that change no longer matched the exception it got. Assert the typed error and the message; drop the unused grpc and RpcError imports. * README: second runtime as a layer in the quickstart * gofmt after the rootfs_image rename
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
env pins Substrate
0b3d2d07(2026-09-09). Against current Substrate main three things break, all reproduced on a cluster runningc2052e45(v0.4.0-83):ActorStatusfield 11 wasstring current_actor_template_uidand is nowrepeated Snapshot snapshots. The first suspend works; the second suspend and everyDeleteEnvironment(which suspends first) fail withstring field contains invalid UTF-8, so released environments leak asSUSPENDED.invalid actor reference. ate-env-api sentate-target-actortwice (forwardOutgoingContextand theguestConninterceptors); the router rejects duplicates since Reject requests with duplicate ate-target-actor headers across HTTP and CONNECT substrate#2308.ate-env manifestoutput no longer parses. The WorkerPool needsspec.sandboxClasses(exactly one entry), the template'ssnapshotsConfigis nowsnapshotConfig, and the containerreadyzis nowwakeupProbe.What
Pin
f03327bd(2026-10-12).pkg/proto/ateapipbhas no changes betweenc2052e45andf03327bd, so the client is wire-identical to what that cluster speaks. Only the manifest generator and the fake control server failed to compile; production code built unchanged.cmd/ate-env/manifest.go:wakeupProbe;snapshotConfigwith an explicitpreferredFidelity: MEMORY(the server default, but the schema marks the field required, and environments must survive suspend with their process state); WorkerPoolsandboxClasses: [{gvisor, configRef: gvisor-default}], sharing one constant with the template'sconfigName. Golden snapshots stay enabled by omittinggoldenSnapshotConfig.internal/apiservice/server.go: stop forwarding caller metadata to the guest. Nothing in the guest reads it; theguestConninterceptors are the only writer ofate-target-actor, and a client can no longer pass routing headers through the proxy. The caller's context still carries deadline and cancellation.internal/internaltest/fakerouter: rejects a duplicatedate-target-actorlike the real router (which comma-joins repeats and rejects even identical copies). With that, the existing proxy test fails without the fix, and a new test sends a client-supplied header over both the unary and the stream path.internal/internaltest/fakecontrol:WorkerPodIps; a suspended actor now carries aSnapshotentry.make images.Testing
make build vet test verify-boilerplate; Python clientpytest: 70 passed.TestProxyGuestServices40/40.server.gomakesTestProxyGuestServicesandTestProxyGuestServicesClientTargetActorHeaderfail.nodeSelector, and the newpreferredFidelityline differ.c2052e45cluster: passed. ate-env-api rolled to this branch's image,default-templaterecreated fromate-env manifest templateoutput throughkubectl-ate(strict parse accepted, golden baked in ~10 s), Python e2etest_full_stack.py2/2 including the final delete, and create → suspend → shell (implicit resume) → suspend → delete through the CLI with nothing left behind. Details in the first comment.Follow-ups, not in this PR
--ingress-auth-mode, substrateab6dd775): the default is stilldeprecated-insecure;guestConndials the router with insecure credentials and breaks when that flips.f03327bd) is opt-in and off by default.manifest.goandserver.goand should rebase after this merges.