Skip to content

Add an opt-in agent backend over a native sandbox library - #418

Merged
Pedro Henrique Penna (ppenna) merged 5 commits into
devfrom
esaurez/nvxhost-edge-backend
Oct 8, 2026
Merged

Pedro Henrique Penna (ppenna) merged 5 commits into
devfrom
esaurez/nvxhost-edge-backend

Conversation

@esaurez

@esaurez Enrique Saurez (esaurez) commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Replaces #395, and now also carries #419, #420, #421, and #422, which this revision folds in and closes. The sandbox lifecycle moves out of this repository: the opt-in agent backend becomes a thin client of a separately supplied native library, aci_edge_agent, that owns the lifecycle, and the data model moves into its own crate so that both sides of the library's ABI compile the same definitions. See "Changes since the previous revision".

Summary

  • Model crate. The edge sandbox crate's serializable requests, results, capabilities, errors, sandbox IDs, and request validation move into a model crate, in the crate's model directory, with no backend. The crate re-exports every type, so its public API is unchanged, and its CI checks now cover both packages.
  • Setup, sandbox spec, and native ABI types. A native host takes a SetupConfig once, when it opens: the runtime bundle or files, the state root, the hypervisor, the CPU profile, timeouts, the guest session policy, flow control of streamed executions, diagnostics, and the defaults that sandboxes inherit. A SandboxSpec carries the per-sandbox settings that the provision request does not define: the image, resources, guest network, hostname, and host loopback forwards. The provision request gains runtimeConfig.networkProxy. The client and Backend gain provision_with, and capabilities report which spec fields and network features a backend honors, so validation rejects what a backend does not support. A backend that advertises no spec support, such as the direct OpenVMM backend, accepts only an empty spec.
  • agent backend (opt-in). A thin client of aci_edge_agent.dll (libaci_edge_agent.so on Linux), which the caller supplies with an independently approved SHA-256. It loads the library once per process after checking that digest and the sandbox ABI version, and it loads the bytes that it checked, so a writer cannot swap the file in between. It opens a host with a SetupConfig, and forwards each Backend operation through the sandbox ABI, whose aci_edge_* exports pass the model's JSON. The library owns the state root and its records, the OpenVMM processes and boot consoles, the guest sessions, and the image registry. Later backends for a loaded library and setup open without touching a file, so a caller may create one per operation.
  • Streamed executions. Output arrives as raw bytes while the command runs, with no size limit; one pump thread copies it into the sinks, and a failed sink discards only its own stream. Piped stdin, cancellation, and timeouts in whole milliseconds up to an hour are supported, and cancellation and timeouts end the whole process tree. A stream that the caller holds without reading cannot delay a timeout. If reading the caller's stdin fails, the workload is killed and the execution fails. Up to 16 executions run side by side on one sandbox.
  • Linux process liveness. A zombie OpenVMM main thread counts as running until the process's other threads exit, as a Windows process runs until its last thread exits, so a stop no longer returns while those threads still hold OpenVMM's descriptors.
  • Direct backend. Its guest network check is now the model's, which also rejects the network, broadcast, and gateway addresses that OpenVMM refuses. Otherwise its behavior is unchanged.
  • Tests and docs. Guest tests run the backend end to end against a built library. Unit tests run executions against a test double of the library, including a failed event, a failed stdin read, and an undrained stream at a timeout. An ignored library test checks, without a hypervisor, that a separately built library binds the sandbox ABI. An example walks through the lifecycle, and the README documents the backend, its setup, images, host paths and network policies, the proxy and forwarded ports, working directories and environments, and streamed executions.

Commits

  1. Pin OpenVMM with RAM-backed microVM overlays without scratch (microVM: opt in to read-only distro with RAM overlay upper nanvix/openvmm#121).
  2. Treat a Linux process as running until its last thread exits.
  3. Move the data model into its own crate.
  4. Add the setup, sandbox spec, and native ABI types to the model.
  5. Add an opt-in agent backend over a native sandbox library.

Dependencies

  • Requires microVM: opt in to read-only distro with RAM overlay upper nanvix/openvmm#121, now rebased onto 96e2363, the OpenVMM revision that dev pins and nanvix/openvmm's main. This branch pins f12dcf5e8f2ea60cc15899dd16852320530670bd.
  • The native library and its guest are built separately and are not included in this public repository. A caller-supplied digest alone is not an authenticated installation: callers must approve and protect the installed library independently.

Changes since the previous revision

  • Rebased onto dev. The openvmm pin moves to nvx: prototype nested KVM guests #121 rebased onto dev's pin, and the crate's CI action keeps dev's toolchain selection while checking the whole workspace, which now includes the model crate. The crate's tree is unchanged at every commit.
  • The lifecycle moves out of the crate into the native library: the state store, the OpenVMM processes and consoles, the guest sessions, the image registry, host paths, egress rules, the proxy, forwarded ports, and CPU profiles. The direct backend's files return to their dev form, apart from the liveness fix, the guest network check, and Hypervisor, which moves into the model.
  • Map host paths and apply egress rules in the native-host backend #419–Let the native-host backend boot on a host CPU profile #422 are folded in. Their features are now model types that the library implements (runtimeConfig.networkProxy, host loopback forwards, the guest network, and the CPU profile), and their guest tests are part of the end-to-end suite.
  • AgentConfig holds the setup, the library path, and the library's approved SHA-256, instead of the OpenVMM configuration and the image.
  • Executions stream instead of returning buffered output. Piped stdin, cancellation, and millisecond timeouts are supported, and the output limit is gone.
  • Review fixes since 7ed52d8:
    • A failed event or stdin read now kills the workload before the input worker is joined, so the execution ends.
    • Flow-control windows whose sum overflows are rejected.
    • {"resources": {}} is an empty spec.
    • Image references reject any whitespace or control character.
    • The example stops a sandbox whose start failed.
    • The README states that resources.vcpus may only be 1.
  • Since 6c5d614, the backend and its library are renamed, with no change in behavior or in the ABI's JSON:
    • The feature and module are agent, the types AgentBackend and AgentConfig, and the constructor AciEdgeSandbox::agent.
    • The library is aci_edge_agent.dll (libaci_edge_agent.so), and the backend binds its aci_edge_* exports.
    • The guest tests and the example are agent_guest and agent_lifecycle, and they read EDGE_AGENT_TEST_* variables.
    • A guest test now checks the library's actual sandbox directory after deprovision.
  • Since 6bebbe1, the example checks the guest log for the exec: line that a streamed execution writes. It looked for an execute: line that the guest no longer writes, so it reported an error after the command had run.
  • Review fixes since fc06c99:
    • The backend loads the bytes that it checked. On Linux it loads them from a sealed memfd, and the load fails if the kernel refuses to create or execute one. On Windows, a handle that denies writes and deletes keeps the file and its directories in place until the library is loaded. Other hosts cannot load the library.
    • If a timed execution's outcome has not arrived when its timeout elapses, wait closes the output streams, as cancellation does. A stream that the caller holds without reading therefore cannot delay the outcome, and an execution that loses output this way ends as TimedOut. The deadline uses the whole milliseconds that the library receives.
    • A malformed sandbox token from the library is a backend_error, not the caller's malformed_id.
    • The provision and exec envelopes borrow the request, so they copy nothing. They stop compiling when the model's wire inputs gain a field, and a test checks that each one deserializes as the model's type.
  • Review fixes since 199aa7a:
    • exec_next returns a backend_error when the library reports an execution event with a nonzero length and a null data pointer, instead of building a slice from it.
    • A capability set without network.networkProxy parses as it does without spec, with the proxy unsupported, so older payloads still deserialize.
    • The openvmm pin moves to microVM: opt in to read-only distro with RAM overlay upper nanvix/openvmm#121 at f12dcf5, which also rejects a bare nvx_overlay_upper token and documents the RAM-backed overlay layout in the OpenVMM Guide.
    • The Copilot Autofix commit d66e6db is folded into commit 4 in the form that cargo fmt expects, which its separate commit did not use. image.path must now be valid UTF-8, as well as absolute and free of NUL characters.

Validation

  • Each commit that changes the crate (2–5) passes its CI checks on Windows with Rust 1.95.0, the release that dev pins: fmt; Clippy with -D warnings on all targets with --all-features, and on the library with --no-default-features; tests with --all-features (163, 163, 183, and 193 tests across commits 2–5) and with default features; docs with -D warnings; and cargo +1.89 check.
  • The head also passes the bundled artifact tests on Windows, and on Linux the same checks plus the macOS build check.
  • The ignored library test passes against freshly built aci_edge_agent.dll and libaci_edge_agent.so.
  • The guest suite (25 tests, --release --test-threads=1) passes at the head against a freshly built library and an OpenVMM built from microVM: opt in to read-only distro with RAM overlay upper nanvix/openvmm#121:
    • Windows/WHP on an AMD EPYC 7763, which amd.milan.v1 serves: in 125 s.
    • Linux/MSHV on an AMD EPYC 9V74, with the host CPU profile: in 105 s.
  • openvmm_e2e (7 tests) passes with dev's release guest (v0.1.0-dev.a399b7b192b7) on WHP, and on MSHV. On MSHV it ran with a local launch patch for the host CPU profile, because auto selects amd.genoa.v1, which that hypervisor cannot present; and as root with a world-writable umask, because the suite expects an ordinary caller and /dev/mshv is root-only on that host.
  • The example runs a command end to end with the same builds: on WHP, and on MSHV with --hypervisor mshv --cpu-profile host. On MSHV, its process maps the library only from the sealed memfd.
  • Interleaved benchmark rounds on MSHV against fc06c99, 30 lifecycles a side, pass the performance gate. Medians: /bin/true exec 1.48 ms against 1.43 ms, and stop 168 ms against 161 ms; stop varies by about ±40 ms with the OpenVMM process's exit. The first backend of a process still takes about 0.73 s.
  • Interleaved benchmark rounds against the previous revision's in-crate lifecycle show no regression:
    • Median /bin/true exec latency drops from 5.1 to 2.6 ms on WHP and from 1.8 to 1.3 ms on MSHV.
    • Creating a backend for each call adds about 18 µs on WHP and 118 µs on MSHV.
    • 100 MiB of output streams in 3.0–3.9 s on WHP and in about 1.9 s on MSHV.

Notes

  • AgentConfig is #[non_exhaustive]: construct it with AgentConfig::new and its builder methods.
  • The library does not honor image references or hostnames in a SandboxSpec yet, and Capabilities::spec reports them as unsupported.
  • Every guest has one virtual processor, so the library accepts resources.vcpus only as 1.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Stop deadlines can be exceeded, binary output is not preserved, and failed example startup can leak OpenVMM.

Review effort: Balanced
Findings: 3 Medium severity

Open (3)
What changed in this PR

Adds an opt-in image-backed nvxhost lifecycle backend while preserving direct OpenVMM as the default.

Changes:

  • Adds native-library bindings and ttrpc lifecycle operations.
  • Adds content-addressed image registration, artifact sealing, and crash recovery.
  • Adds documentation, an example, and native guest tests.
File Description
aci_edge_sandboxes/​Cargo.toml Defines the feature, dependencies, and example.
aci_edge_sandboxes/​Cargo.lock Locks new dependencies.
aci_edge_sandboxes/​README.md Documents configuration, trust, and lifecycle behavior.
aci_edge_sandboxes/​examples/​nvxhost_lifecycle.rs Demonstrates native lifecycle usage.
aci_edge_sandboxes/​src/​client.rs Adds the client constructor.
aci_edge_sandboxes/​src/​lib.rs Gates the native binding module.
aci_edge_sandboxes/​src/​nvxhost.rs Implements the checked native ABI and ttrpc transport.
aci_edge_sandboxes/​src/​openvmm/​config.rs Generalizes Unix socket-path validation.
aci_edge_sandboxes/​src/​openvmm/​images.rs Implements image registration and file sealing.
aci_edge_sandboxes/​src/​openvmm/​launch.rs Updates state fixtures.
aci_edge_sandboxes/​src/​openvmm/​mod.rs Exports the backend and extends launch recovery.
aci_edge_sandboxes/​src/​openvmm/​native.rs Implements native lifecycle operations.
aci_edge_sandboxes/​src/​openvmm/​platform/​linux.rs Adds Linux seals and launch-log claims.
aci_edge_sandboxes/​src/​openvmm/​platform/​mod.rs Defines portable file seals.
aci_edge_sandboxes/​src/​openvmm/​platform/​other.rs Adds unsupported-platform stubs.
aci_edge_sandboxes/​src/​openvmm/​platform/​windows.rs Adds Windows seals and launch-log claims.
aci_edge_sandboxes/​src/​openvmm/​state.rs Adds backend-aware native state.
aci_edge_sandboxes/​tests/​nvxhost_guest.rs Adds ignored guest lifecycle tests.
openvmm Pins the required scratchless-overlay revision.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI balanced review requested due to automatic review settings October 7, 2026 18:37
@esaurez Enrique Saurez (esaurez) changed the title Add opt-in native-host image-backed edge lifecycle Add an opt-in nvxhost backend over a native sandbox library Oct 7, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread aci_edge_sandboxes/src/nvxhost/mod.rs Resolved
Comment thread aci_edge_sandboxes/model/src/setup.rs Resolved
Comment thread aci_edge_sandboxes/model/src/spec.rs Resolved
Comment thread aci_edge_sandboxes/model/src/spec.rs Resolved
Comment thread aci_edge_sandboxes/README.md Resolved
Copilot AI balanced review requested due to automatic review settings October 7, 2026 18:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread aci_edge_sandboxes/src/nvxhost/mod.rs Resolved
Comment thread aci_edge_sandboxes/README.md Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread aci_edge_sandboxes/src/agent/library.rs Resolved
Comment thread aci_edge_sandboxes/src/agent/mod.rs Resolved
Copilot AI balanced review requested due to automatic review settings October 8, 2026 00:52
@esaurez Enrique Saurez (esaurez) changed the title Add an opt-in nvxhost backend over a native sandbox library Add an opt-in agent backend over a native sandbox library Oct 8, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The large native ABI and security-sensitive loader depend on a separately built library unavailable for complete verification.

0 open findings

2 resolved since last review

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Copilot AI balanced review requested due to automatic review settings October 8, 2026 15:43
@esaurez
Enrique Saurez (esaurez) marked this pull request as ready for review October 8, 2026 15:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new trusted native-library boundary and concurrent streaming lifecycle warrant final human review despite extensive tests.

0 open findings

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Copilot AI balanced review requested due to automatic review settings October 8, 2026 16:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The native event binding has an undefined-behavior path, and capability deserialization breaks older payloads.

1 open finding

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Comment thread aci_edge_sandboxes/src/agent/library.rs Resolved
The edge runtime boots a sandbox from one read-only image with a RAM-backed
overlay upper and no scratch disk. OpenVMM accepts that layout from
nanvix/openvmm#121, rebased here onto the previous pin.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
On Linux the main thread of OpenVMM can exit and leave a zombie while its other
threads still exit. Liveness checks treated that zombie as gone, so a stop
could return while those threads still held OpenVMM's descriptors, and an
immediate restart could collide with them. A zombie now counts as running
until its other threads exit, as a Windows process runs until its last thread
exits.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The edge sandbox crate's serializable requests, results, capabilities,
errors, sandbox IDs, and request validation move into a model crate with
no backend. A backend implemented outside the crate can then read and
write the same JSON from the same definitions. The crate re-exports every
type, so its public API is unchanged, and it becomes a workspace whose
checks now cover both packages.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 8, 2026 17:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Image paths with invalid UTF-8 currently pass validation but cannot cross the JSON ABI.

1 open finding
1 resolved since last review

🧠 Review effort: Balanced

Comment thread aci_edge_sandboxes/model/src/spec.rs Resolved
Copilot AI balanced review requested due to automatic review settings October 8, 2026 17:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The lifecycle example incorrectly rejects supported IPv6 egress rules.

0 open findings

1 resolved since last review
Previously missed (1)

In code that hasn't changed since last review

Medium severity IPv6 CIDR egress rules are rejected by left-to-right parsing

aci_edge_sandboxes/​examples/​agent_lifecycle.rs:307

Parsing from the left treats every IPv6 colon as a field separator, so a documented rule such as --egress-allow 2001:db8::/32 is rejected even though the policy model supports IPv6. Parse an optional protocol/port suffix from the right and otherwise keep the full CIDR.

🧠 Review effort: Balanced

A native host takes a SetupConfig once, when it opens: the runtime bundle
or files, the state root, the hypervisor, the CPU profile, timeouts, the
guest session policy, flow control of streamed executions, diagnostics,
and the defaults that sandboxes inherit. A SandboxSpec carries the
per-sandbox creation settings that the provision request does not define:
the image, resources, guest network, hostname, and host loopback forwards.

The provision request gains runtimeConfig.networkProxy, which routes the
guest's traffic through a proxy on host loopback. The client and the
Backend trait gain provision_with, which takes the spec, and capabilities
report which spec fields and network features a backend honors, so
validation rejects a request or spec that the backend does not support.
The image types, execution outcomes, and Hypervisor move into the model
with serde forms, and the wire module defines the envelopes of the native
ABI.

The direct OpenVMM backend checks its guest network with the model's check,
which also rejects the network, broadcast, and gateway addresses that
OpenVMM refuses.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The agent feature adds a backend that is a thin client of a separately
supplied native library, aci_edge_agent, which owns the whole sandbox
lifecycle: the state root and its records, the OpenVMM processes and boot
consoles, the guest sessions, and the image registry. The backend loads the
library once per process after checking its SHA-256 and sandbox ABI version,
opens a host with a SetupConfig, and forwards each Backend operation through
the sandbox ABI, whose aci_edge_* exports pass the JSON of the model crate.
It loads the bytes that it checked: on Linux from a sealed in-memory copy,
and on Windows while its handle keeps the file from being changed, renamed,
or deleted; other hosts cannot load it. A later backend for a loaded library
finds it by its requested path without touching the file.

Executions stream: output arrives as raw bytes while the command runs, with
no size limit, and one pump thread copies it into the sinks; a failed sink
discards only its own stream. A stdin worker feeds piped input once the
library accepts it; a failed read of that input kills the workload and fails
the execution. Cancellation and timeouts in whole milliseconds end the whole
process tree. Once a timeout elapses, waiting closes the output streams, so
that a stream that the caller holds without reading cannot delay the
outcome; an execution that loses output that way ends as TimedOut.
AgentBackend takes AgentConfig {setup, library, library_sha256}, and images,
guest logs, and diagnostics stay available through it.

Guest tests run the backend end to end against a built library, and unit
tests drive executions through input and library failures and an undrained
stream at a timeout with a test double of the library. An ignored library
test checks that a separately built library binds the sandbox ABI, and an
example walks through the lifecycle.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 8, 2026 17:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unsafe cross-platform library loading and lifecycle behavior delegated to a separately built private component require final human review.

0 open findings

🧠 Review effort: Balanced

@ppenna
Pedro Henrique Penna (ppenna) merged commit c90c60b into dev Oct 8, 2026
32 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the esaurez/nvxhost-edge-backend branch October 8, 2026 21:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants