Skip to content

Map host paths and apply egress rules in the native-host backend - #419

Closed
Enrique Saurez (esaurez) wants to merge 8 commits into
esaurez/nvxhost-edge-backendfrom
esaurez/nvxhost-edge-fsnet
Closed

Enrique Saurez (esaurez) wants to merge 8 commits into
esaurez/nvxhost-edge-backendfrom
esaurez/nvxhost-edge-fsnet

Conversation

@esaurez

Copy link
Copy Markdown
Contributor

Replaces #407 on the rebased stack. Stacked on #418. Gives the opt-in native-host backend the same filesystem and network policies as the direct OpenVMM backend.

Summary

  • Planning in the host library. Provision serializes the request's filesystem and network sections and passes them to the library's new nvx_plan_sandbox export. The library resolves mapped and denied paths, chooses the virtio-fs export (the deepest common directory), pins the identity of every object inside a read-write mapping, and expands egress rules exactly. Validation uses the same export with a validate-only flag that touches nothing on the host; unenforceable policies fail with policy_validation.
  • The plan travels with the sandbox. The sandbox record keeps the plan's JSON as the library wrote it. Start hands it to the new nvx_build_ramfs_launch_arguments_with_plan export, which reapplies every planning rule that needs no host access and refuses when a pinned object was replaced; start then fails with backend_error and asks for a new provision. The old nvx_build_ramfs_launch_arguments binding is gone.
  • The state root stays private. It holds every sandbox's plan, which decides what the next start exports and which egress it allows. Provision refuses a mapping inside the state root, and one that contains it (for example a home directory that holds the default state root) unless a denied path inside the mapping hides it.
  • Guest contract. The edge guest binds each mapping into its RAM overlay (read-only where requested), never exposes the export itself, configures the virtio-net device and, when the policy allows the gateway's DNS port, /etc/resolv.conf. Workloads stay root but keep only the default container capabilities, with no_new_privs and without user namespaces. The runtime ABI name moves to microvm-abi-v2-edge-ramfs-v2.
  • Capabilities. The backend now declares egress allow/deny and rules, ingress deny, host-loopback deny, and read-only, read-write, and denied paths. Ingress and host-loopback access are still rejected.
  • Linux process exits. On Linux, OpenVMM's main thread can exit before its other threads, which still hold its descriptors. Liveness checks treated that zombie as gone, so a stop returned early and an immediate restart could not claim the OpenVMM log. A zombie now counts as running until its last thread exits, as a Windows process does.
  • Guest network configuration. OpenVmmConfig::guest_network now rejects the network, broadcast, and gateway addresses, which OpenVMM refuses, when a backend is created (backend_unavailable), instead of when a sandbox first attaches a network device. This applies to both backends.
  • Tests and example. The ignored guest tests take NVXHOST_TEST_HYPERVISOR (default whp on Windows) and drop their whp_ prefix. New tests cover mappings, the state root, denied and unmapped paths, workload containment, replaced host objects, the kernel command-line budget, and egress rules against host services. The example takes --readonly, --readwrite, --denied, --egress, --egress-allow, and --egress-deny.

Dependencies

  • Requires the matching private host library and edge guest (new exports and runtime ABI v2); an older library fails to load because the new exports are missing.
  • OpenVMM is unchanged: the pinned revision combines --mount, --mount-deny, and --net with the scratchless RAM-overlay topology.

Changes since #407

  • OpenVmmConfig::guest_network keeps dev's canonical prefix parsing (cidr::parse_prefix_length) and adds the network, broadcast, and gateway checks to it.

Validation

  • Each PR head in this stack passes cargo test --all-features and Clippy with -D warnings on all targets with --all-features, on Windows with Rust 1.93 (135, 137, 139, 139, and 140 unit tests).
  • At the top of the stack (5b0a315), on Windows and on Linux with Rust 1.93: the crate's CI checks, which are fmt, Clippy with --all-features and with --no-default-features, tests with --all-features and with default features, docs with -D warnings, cargo +1.89 check, and, on Linux, the macOS build check. The pinned library test passes against freshly built nvxhost.dll and libnvxhost.so.
  • Guest suite at the top of the stack, 20 tests, --release --test-threads=1, each checking that no OpenVMM process outlives its sandbox, with dev's Linux 6.18.38 kernel:
    • Windows/WHP on an AMD EPYC 7763, which amd.milan.v1 serves: all pass, in about 100 s, and again with a host profile;
    • Linux/MSHV on an AMD EPYC 9V74, which no built-in profile serves: all pass with a host profile, in about 82 s.

Notes

Load a separately supplied, caller-approved native host library for scratchless image-backed provision/start/exec/stop/deprovision. Keep the direct backend as default, isolate persisted state, fence unsupported capabilities, and pin the RAM-overlay OpenVMM change.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Bound guest-session disposal by each operation deadline, preserve zero as an unbounded command timeout while rejecting sub-second precision, keep native-only files outside direct-backend cleanup, and validate both derived Unix socket paths before loading the library.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Show a concrete WHP invocation, the separately supplied artifacts and trust requirement, and the unverified MSHV path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
OpenVMM serves one boot-console client at a time and retains guest output
while none is connected. The native backend keyed its console listener by
sandbox only, so a restart after an OpenVMM crash reused the finished
listener of the crashed launch; with no listener, a verbose guest stalled
until start timed out. A second backend instance that reattached to a running
guest could not connect while the first one held the console, and stop then
failed after the guest had already stopped.

Key listeners by the OpenVMM process identity and retire stale ones before a
launch. A listener that finds the console held elsewhere waits, without the
start deadline, to take it over and ends quietly when OpenVMM exits; a reset
after OpenVMM exits ends the log. Stop and deprovision report capture
failures as consoleError metadata instead of failing.

Extend the opt-in WHP suite with exec semantics, lifecycle errors and
restarts, reattachment from a new backend, a crashed guest, a guest that
outlives the process that started it, starts killed at several points,
parallel sandboxes, and concurrent commands. List OpenVMM processes in a way
that fails loudly, so the leak checks cannot pass vacuously.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Hash the image once when it is registered, or record a digest that the
caller already verified, and refer to it by content ID. Provision and
start compare cheap file seals instead of hashing again, and fail closed
when a file changed. Hash OpenVMM, the kernel, and the initramfs once per
backend, optionally against approved digests, and keep full re-hashing
as an opt-in diagnostic. Starts now cost about the guest boot time.

Windows seals omit the change time: writers can set it, and the system
updates it when it caches a file hash in an extended attribute, which
failed an unchanged OpenVMM executable closed during testing.

Claim the OpenVMM log before a native launch. OpenVMM inherits the
claim, so recovery can clear an interrupted start that recorded no
process identity once nothing holds the log, instead of leaving the
sandbox unusable. Markers of the default backend are unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The native-host backend accepts the same filesystem and network policies as the
direct OpenVMM backend. Provision passes the policy to the host library's
nvx_plan_sandbox export, which resolves mapped and denied paths, chooses the
virtio-fs export, pins objects inside read-write mappings, and expands egress
rules exactly. The sandbox record keeps the plan, and start hands it back through
nvx_build_ramfs_launch_arguments_with_plan, which checks it again and fails with
backend_error when a pinned object was replaced.

The state root holds every sandbox's plan, which decides what the next start
exports and which egress it allows. Provision therefore refuses a mapping inside
the state root, and one that contains it unless a denied path hides it.

The edge guest binds each mapping into its RAM overlay, configures the
virtio-net device, and runs workloads as root with only the default container
capabilities, no_new_privs, and no user namespaces, so the guest runtime ABI
moves to microvm-abi-v2-edge-ramfs-v2.

The opt-in guest tests run under the hypervisor that NVXHOST_TEST_HYPERVISOR
names and cover mappings, the state root, denied and unmapped paths,
containment, replaced host objects, the kernel command-line budget, and egress
rules. The example accepts host-path and egress options.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
On Linux the main thread of OpenVMM can exit and leave a zombie while its other
threads still exit. Liveness checks treated that zombie as gone, so a stop
returned while those threads still held OpenVMM's descriptors, and an
immediate restart failed to claim the OpenVMM log that they kept locked. A
zombie now counts as running until its other threads exit, as a Windows
process runs until its last thread exits.

The forced-stop guest test now uses a one-nanosecond stop timeout: on a fast
host a graceful shutdown could complete within the previous millisecond.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
OpenVMM refuses a guest address that is its network's own address, its
broadcast address, or its first address, which is the gateway. The
configuration accepted them, so a backend failed only when a sandbox first
attached a network device, and the native backend reported its planning error
as the request's policy_validation rather than as an unusable configuration.
Creating a backend now rejects such an address with backend_unavailable.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@esaurez

Copy link
Copy Markdown
Contributor Author

Folded into #418, which now carries the whole change. The native library implements this feature behind the model types that #418 adds, and its guest tests are part of the end-to-end suite there.

@esaurez
Enrique Saurez (esaurez) deleted the esaurez/nvxhost-edge-fsnet branch November 13, 2026 17:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant