Repository navigation
openvmm: pin the policy check of a restored host CPU profile - #1585
Conversation
A snapshot of a host profile carries the only copy of the profile's document, and the digest that it records proves nothing about what the document says, since whoever crafts a snapshot writes both. OpenVMM now also holds the document to its derivation policy when it restores the snapshot (E_PROFILE_DIGEST), so that an edited host profile cannot give the guest what every derived profile withholds, such as VMX or SVM on KVM, and refuses a profile that the policy would change. No version is created or bumped, and OpenVMM applies the policy of its own build whatever policy a profile's provenance names. A snapshot of a host profile that an OpenVMM with another policy derived therefore restores only if the restoring OpenVMM's policy leaves it unchanged. Host profiles are for development, are not pinned, and a build that computes another effective CPUID already rejects their snapshots, so the documents that describe a host profile's snapshot say that it can need a recapture: the usage guide's limits of host profiles, the CPU profile selection and verification of the design, and the restore compatibility of the compatibility model, whose bullet for host profiles now names the policy. The verification's report of every violation notes that a profile outside the policy names its first eight differences. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A snapshot of a host CPU profile (--cpu-profile host, or the fallback of auto) carries the only copy of the profile's document, and restoring it accepted any document that decodes to a valid host profile with the recorded ID and digest, which whoever crafts a snapshot writes along with the effective CPUID that the document gives. Nothing compared the document with the derivation's policy, so a document with VMX (CPUID.1:ECX[5]) or SVM (CPUID.0x80000001:ECX[2]) restored, and on KVM, where the time ABI partition's CPUID is the profile's effective CPUID, the guest could enable VMX or SVM and reach KVM's nested-virtualization code, which nested_virt false keeps closed (#1357). Pin 7b205c0a0fe7a40cc08fd46951d724bc293bc703, the head of microsoft/openvmm#5094, one commit on efe6bc267, the head of feature/microvm/dev: - 7b205c0a0 cpu_profile: refuse a restored host profile outside the derivation policy host_profile_for_restore now also refuses a host profile that the policy would change (E_PROFILE_DIGEST), naming at most eight of the differences, and both the controller's restore preflight and the VM worker's profile selection call it. The check applies the derivation's own rules to the profile's CPUID table, so a derived profile passes it, and it checks the XSAVE features against the policy's allowances. The derivation, the pinned profiles, and the policy label are unchanged, and no version is created or bumped: restore applies the rules of its own build, so a host profile's snapshot that an OpenVMM with another policy captured restores only if this policy leaves its profile unchanged, and otherwise needs a recapture, as the error says. Tests: with OpenVMM built from the pin's tree, a host profile's snapshot restores, and a copy of it whose profile has VMX set, recorded with its own digest and effective CPUID, is refused with E_PROFILE_DIGEST on nvx-kvm-test, nvx-mshv-test, and nvx-whp-test. The unchanged OpenVMM restores that copy on nvx-kvm-test, where the guest reads IA32_VMX_BASIC through /dev/cpu/0/msr without a #GP, and refuses it on nvx-mshv-test and nvx-whp-test with E_PROFILE_UNSUPPORTED, since their partitions lack VMX. The microVM scenarios guest-boot, cpu-profile-fallback, and snapshot-core, which capture and restore a real host profile, pass on all three. The revision's three tests of the restore fail at efe6bc267, since each restore succeeds, and its 132 cpu_profile, 138 openvmm_helpers, and 130 openvmm_core tests pass on Windows. NVX needs no coordinated change: the control-session vectors, the source excerpts, the contract versions, the activation bound, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NVX pin moves to 7b205c0a0, position 276, 152 commits past the mapped tip. Position 276, the one commit of microsoft/openvmm#5094 on efe6bc267, the head of feature/microvm/dev, is not yet assigned to a PR. Position 276 makes the restore of a host CPU profile refuse a profile that the derivation policy would change, which let a snapshot's own copy of its profile give the guest the VMX or SVM that every derived profile withholds. It adds policy_violations to the cpu_profile crate's derivation, which applies the policy of position 131 to the profile, and has host_profile_for_restore of position 142 refuse a profile that differs. It extends position 142's host profile tests of the restore preflight and of the worker's selection of the profile, and describes the refusal on the Guide's snapshots page. It changes nothing that a mapped PR adds. Replaying it on upstream main conflicts in all seven of its files, since upstream main lacks the cpu_profile crate, the time ABI's modules of openvmm_helpers and openvmm_core, and the snapshots page's section on time and CPU compatibility, so it follows the time ABI series upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
The security-sensitive snapshot-policy change needs maintainer review because exact-pin platform CI is not fully verified and the required openvmm#5094 change remains unmerged.
No open issues 🎉
What changed in this PR
This PR pins OpenVMM’s restored host CPU profile policy check to address #1357 and documents its compatibility implications in NVX. The check rejects snapshot profiles that violate the restoring build’s derivation policy, including profiles that enable VMX or SVM.
Changes:
- Document policy validation and the
E_PROFILE_DIGESTrejection. - Explain when snapshots derived under another policy require recapture.
- Map OpenVMM commit
7b205c0a0at roadmap position 276.
| File | Description |
|---|---|
| doc/usage.md | Explains host-profile restore restrictions and recapture requirements. |
| doc/openvmm-upstream-roadmap.md | Updates the pin, chain counts, and position 276 mapping. |
| doc/design/2.2-cpu-and-topology.md | Describes policy validation and bounded violation reporting. |
| doc/design/1.5-compatibility-and-versioning-model.md | Adds policy compliance to host-profile restore requirements. |
Review effort: Balanced
|
No change needed here. Copilot's overview, which has no findings, notes that exact-pin platform CI was not yet verified and that microsoft/openvmm#5094 was unmerged when it reviewed. Both are now settled, or will be before this merges:
|
Fixes #1357
What and why
This pins microsoft/openvmm#5094 at
7b205c0a0fe7a40cc08fd46951d724bc293bc703. With the pin, restoring a snapshot of a host CPU profile (--cpu-profile host, or the fallback ofauto) refuses a profile document that the derivation's policy would change, withE_PROFILE_DIGEST. Before, a snapshot that carried a document with VMX or SVM set restored, and on KVM its guest could use them.At NVX's previous pin,
efe6bc267:host_profile_for_restoreaccepted any document that decodes to a valid host profile with the ID and digest that the snapshot recorded. Whoever crafts a snapshot writes the digest of an edited document and the effective CPUID that it gives. Nothing compared the document with the derivation's policy (apply_policy,clear_stateless_features, and theALLOWED_*XSAVE constants ofvmm_core/cpu_profile/src/derive.rs), which clears the features that a VM must not see. A document with CPUID.1:ECX[5], VMX, or CPUID.0x80000001:ECX[2], SVM, therefore restored: the effective CPUID, the backend's support check, and VP 0's CPUID all follow the document.new_partitionadds for!nested_virtdoes not reach it. The guest could enable VMX or SVM and reach KVM's nested-virtualization code, whichnested_virtfalse keeps closed. On MSHV and WHP, the backend's support check refuses the edited profile (E_PROFILE_UNSUPPORTED), since those partitions lack VMX. Only a crafted or foreign snapshot carries such a document: OpenVMM derives the profile of its own captures with the same policy, so they never do.What the OpenVMM pull request changes
microsoft/openvmm#5094 adds one signed commit on
efe6bc267, the head offeature/microvm/dev. It has a good signature (openvmm_pin.py source).7b205c0a0derive::policy_violationsapplies the derivation's ownapply_policyandclear_stateless_featuresto a profile's CPUID table, with the profile's XCR0, compares the result with the table, and checks the XSAVE features againstALLOWED_XCR0,ALLOWED_XSS, andALLOWED_XSAVE_EXTENSIONS.host_profile_for_restorereturnsE_PROFILE_DIGESTfor a profile that it finds outside the policy, naming at most eight of the differences, and both the controller's restore preflight (preflight_time_abi_restore) and the VM worker's profile selection (select_on_host) call it, so a crafted snapshot fails before the worker starts, and again if a worker is ever handed one. A profile that the derivation produced is a fixed point of its policy, which only sets and clears bits and zeroes leaves, so the check accepts it, and a rule that the derivation gains is checked without a second copy of it. The derivation, the pinned profiles, the policy labelPOLICY, and the error codes are unchanged. Three tests of the restore fail atefe6bc267, since each restore succeeds, and thederivetests check that every pinned profile, the host profile of every recorded host, and the host profile of random mutations of each host's CPUID follow the policy (an offline run of 18,000 mutations found no violation), and that each feature that the policy withholds is a violation alone.The patch that Ruize Tang (@tangruize) attached to #1357 proposes a
cpu_profile::policymodule of declarative rule tables thatCpuProfile::effective_cpuidchecks at every start. This takes a smaller route to the same refusal: it reuses the derivation's own rules, so no second copy of them can fall out of step with the derivation, and it checks only where a profile enters from a snapshot, so a cold boot and a pinned profile, which are constants that OpenVMM verifies, pay nothing.What this pull request changes in NVX
dca32976276documents the check where the documentation describes a host profile's snapshot: the limits of host profiles indoc/usage.md, the CPU profile selection and verification ofdoc/design/2.2-cpu-and-topology.md, and the restore compatibility ofdoc/design/1.5-compatibility-and-versioning-model.md.0928ba504moves theopenvmmgitlink to the pin, and nothing else.106dcd449maps position 276 in the OpenVMM roadmap.The consequence for older host-profile snapshots
The issue flags that a host-profile snapshot that an OpenVMM with another policy captured can no longer be restored once the policy changes. No version is created or bumped to handle it: no policy version, profile revision, schema or format version, capability or generation, or error code. Restore applies the rules of its own build, and does not read the policy label that a profile's provenance names. A host-profile snapshot that another policy derived therefore restores only if this build's policy leaves its profile unchanged, and otherwise fails with
E_PROFILE_DIGEST, whose message says that the snapshot needs a recapture. Host profiles are for development and are not pinned, and a build that computes another effective CPUID already rejects their snapshots (E_CPU_SURFACE), so the documentation states the behavior and the recapture rather than a compatibility promise.Pin details
7d90c4104, lacks thecpu_profilecrate and the time ABI's modules ofopenvmm_helpersandopenvmm_core, which its change and its tests are in, and the section of the snapshots page that it edits. Replaying it there conflicts in all seven of its files, so it follows the time ABI series upstream. It changes nothing that a mapped PR adds.efe6bc267, is NVX's previous pin and the head offeature/microvm/dev.nvx-rscopies from OpenVMM all match the revision's (openvmm_vectorsandopenvmm_cpu_profiles, below).Scope and limits
check_generation; and the brand and provenance are informative only.71b358bce, an earlier version of the pinned commit whose tree,fee5fcc1c, is that of7b205c0a0(they differ in the message only), with NVX at6e7a13200, before the rebase onto507ceeea4. The rebase moves no OpenVMM orcpu_profilecode, so the exact head's CI below shows the same scenarios on the rebased base.Landing order
microsoft/openvmm#5094 landed first, by fast-forwarding
feature/microvm/devfromefe6bc267to7b205c0a0(python scripts\openvmm_pin.py land 7b205c0a0fe7a40cc08fd46951d724bc293bc703 --remote origin --push), and it shows as merged at that commit. This pull request merged after that, through the merge queue. OpenVMM's merge button would squash the commit and leave the pinned revision off the branch, so it wasn't used.Validation
NVX
Run on Windows:
python scripts\openvmm_pin.py source 7b205c0a0fe7a40cc08fd46951d724bc293bc703 --pr 5094: passed. The revision is the pull request's head, with a good signature.python scripts\openvmm_pin.py check-roadmap --worktree: passed, with 276 positions and 287 references.python scripts\openvmm_pin.py check-branch --topic: passed, with three signed commits and no coordinated files.python scripts\check_markdown_links.pyandgit diff --check: passed.python scripts\nvx.py verify: passed.cargo test --locked --manifest-path nvx-rs/Cargo.toml --no-default-features --test openvmm_vectors -- --ignored(4 passed) and--test openvmm_cpu_profiles -- --ignored(1 passed).python scripts\local_gates.py --base origin/devselects seven gates:test-changes,markdown-links,test-layout,openvmm-vectors,openvmm-cpu-profiles,nvx-cli, andopenvmm-ci. I ran the first six, and all passed;nvx-cli, aspython scripts\local_gates.py --base origin/dev --run --only nvx-cliwithNVX_RSset to a releasenvx-rsbuilt from this branch, ran 2199 tests (88 skipped), thenscripts\test_adversarial.pyran 84 (11 skipped).test-changesfound no changes to existing tests, so the pull request needs notests-modifiedlabel. The seventh,openvmm-ci, isnvx.py check-openvmm-ci, under CI below.OpenVMM
For
cpu_profile,openvmm_helpers, andopenvmm_core, with this change, with Rust 1.97.1 and cargo-nextest 0.9.133:cargo clippy --all-targets,cargo doc --no-deps, andcargo nextest run --profile agent(132, 138, and 130 passed) passed, andcargo xtask fmt --fixchanged nothing.cargo nextest run --profile agentpassed with 132cpu_profiletests and 276openvmm_helpersandopenvmm_coretests.efe6bc267, with the new tests, the restore test ofcpu_profile, the preflight test ofopenvmm_helpers, and the selection test ofopenvmm_corefail, since each restore succeeds. With the pinned revision, disabling the stateless clear, the XSAVE check, or the derivation's VMX clear makes tests fail.Hosts
The checkouts were temporary worktrees. On each host,
nvx-rs downloadinstalled the release of the nearest first-parent ancestor of the checkout,v0.1.0-dev.6e7a13200472, whose OpenVMM is the packagedefe6bc267binary. OpenVMM was then built from71b358bcein a release build without LTO or debug information: on each Linux host in its worktree, and for WHP on Windows x64, from whereopenvmm.exewas copied to the host.A scratch script,
issue1357_repro.py, which is not part of this change, cold boots a microVM with--cpu-profile host, captures its snapshot with a post-restore probe that readsIA32_VMX_BASICthrough/dev/cpu/0/msr, restores the snapshot as captured (the control), then restores a copy whose host profile document has CPUID.1:ECX[5] set, recorded with the edited document's own digest and the effective CPUID record that it gives. The control restores on every host, and its guest's read faults, since the guest has no VMX.efe6bc267(packaged)IA32_VMX_BASIC,0x0098100011e57ed0, without a #GP[E_PROFILE_DIGEST]: "CPUID 0x1 ECX bit 5 is set, which the policy clears"E_PROFILE_UNSUPPORTEDE_PROFILE_DIGESTE_PROFILE_UNSUPPORTEDE_PROFILE_DIGESTOn KVM, a build of
efe6bc267that I built on the host restores it the same way as the packaged one.nvx.py test-microvm --backend BACKEND --scenario cpu-profile-fallback --scenario guest-boot --scenario snapshot-coreran these scenarios, which capture and restore a real host profile, on each host with the pin's tree, and all passed.CI
7b205c0a0: run 43565576664. The six jobs that NVX requires passed, aspython scripts\nvx.py check-openvmm-ci --revision 7b205c0a0fe7a40cc08fd46951d724bc293bc703reports:unit tests [x64-linux]and[x64-windows], andrun vmm-testsforx64-linux-amd-kvm,x64-linux-intel-mshv,x64-windows-intel, andx64-windows-amd. 33 of the run's 34 jobs passed. The one that failed,run vmm-tests [x64-windows-amd-snp], isn't one that NVX requires, and it fails at the base too: run 43527602221 ofefe6bc267failed that job and no other. It wasn't rerun.106dcd449succeeded: 38 checks passed and 8 were skipped,Publish development release,Publish nvx crate,Package nvx crate,Record performance results,Persist performance baselines,Cancel superseded merge group runs, and the KVM and WHPDebug kerneljobs. It ran what I left to it:NVX CLI / Linux, whose gate I ran on Windows only, andNVX CLI / Windows.NVX microVM testson KVM, MSHV, and WHP, with their shards, their guests and sandboxes, and WHP's runtime adapter, with the OpenVMM that the jobs built from the pin.Platform / Linux / KVM,Platform / Linux / MSHV, andPlatform / Windows / WHP, all on virtual machines, andPerformance regression gate.OpenVMM CI of the pinned revision, which waited 1 h for OpenVMM's jobs above, andRequired status check.507ceeea4, succeeded: 35 jobs passed and 8 were skipped.