Skip to content

openvmm: pin the XSAVE restore's component check - #1589

Merged
Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
agents/issue-1380-xstate-bv-caps
Nov 14, 2026
Merged

Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
agents/issue-1380-xstate-bv-caps

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1380

What and why

This pins microsoft/openvmm#5101 at b723cbed35f253a7b97e9e7f32f3d7d3502cae36. With the pin, a KVM restore refuses a VP's XSAVE area whose XSTATE_BV selects a component that the partition's caps lack: it fails with KvmError::InvalidState, which restore_all reports as the StateError of the xsave element, before KVM_SET_XSAVE changes anything, so the VP's XSAVE state stays as it was. Before, the restore succeeded: KVM loaded the component and marked it in use, and the next save dropped it, so the VP's state after the restore was not the state that the snapshot held.

At NVX's previous pin, c5509cde2:

  • Cause. KvmVpStateAccess::set_xsave in vmm_core/virt_kvm/src/arch/x86_64/vp_state.rs passed the area's header into KVM_SET_XSAVE through Xsave::write_standard, unchecked. KVM takes any user component that it supports, whether or not the guest's CPUID leaf 0xD enumerates it, but KVM_GET_XSAVE returns only the enumerated ones. A save never holds such a component, since KVM returns only components within the guest's supported XCR0, which the caps' features bound, so only a crafted snapshot, or one that other caps saved, reaches this. MSHV and WHP pass the area to the hypervisor as it is, and are not affected.

  • Reproduction. On nvx-kvm-test, whose KVM supports XCR0 0xe7, a scratch test that is not part of the change builds the one-VP time ABI partition of amd.milan.v1, whose caps' features are 0x7 (x87, SSE, and AVX, without AVX-512), restores areas with one XSTATE_BV each through set_xsave, and reads the VP's XSTATE_BV back with KVM_GET_XSAVE:

    Restored XSTATE_BV At c5509cde2 With the pin
    0x3, 0x7 (within the caps) Ok, reads back 0x3, 0x7 Ok, reads back 0x3, 0x7
    0x20 (AVX-512 opmask) Ok, reads back 0x0 InvalidState, stays 0x7
    0xe0 (AVX-512) Ok, reads back 0x0 InvalidState, stays 0x7
    0x27 (AVX and AVX-512 opmask) Ok, reads back 0x7 InvalidState, stays 0x7

    The partition of intel.emeraldrapids.v1, whose features are 0xe7, still restores its own components, 0x20, 0xe0, and 0xe7, and a save_all followed by a restore_all succeeds in both partitions. The reproduction drives set_xsave itself, since a save never holds such a component: reaching it through nvx-rs takes a snapshot whose XSAVE area was edited.

The fix, and the decisions that the issue left open

The issue's fix.patch, written against 3cf58a5234e, adds check_xstate_bv and has set_xsave call it before it writes anything. The pin keeps its rule and adapts it to the current virt_kvm:

  1. The rule. xsave::restore refuses an area whose XSTATE_BV has a bit outside the caps' XSAVE features other than x87 and SSE (X86X_XSAVE_LEGACY_FEATURES, in place of the patch's literal 3), before write_standard and KVM_SET_XSAVE, so a refused restore changes nothing.
  2. A module of its own. The check and the restore that calls it live in a new xsave.rs module of virt_kvm's x86-64 module, and set_xsave is now one call to xsave::restore. The ignored KVM-backed test then drives the function that set_xsave runs; the patch has no KVM-backed test.
  3. The existing error. KvmError::InvalidState with a static message, as the patch has it, and no new variant or type.
  4. XSTATE_BV only. The check reads XSTATE_BV (word 64) with get, so an area too short to hold it selects nothing here, and write_standard still panics on it, which openvmm: restoring a short or malformed XSAVE area panics the VM worker on KVM #1381 covers. It checks neither XCOMP_BV nor the area's layout.
  5. Supervisor components. The mask holds no supervisor bit, so an area that selects one is refused too. KVM_SET_XSAVE takes user state only and refuses a supervisor bit with EINVAL itself, so such a restore failed before as well, and now fails with the check's error and before any write.
  6. No virt_backend_tests contract. Its partitions take KVM's supported CPUID, so their caps hold every component that KVM supports, and KVM itself refuses a component that the caps lack: a contract test of the restore passes without this change. The ignored test below covers what the harness cannot.

Why the check cannot refuse a legitimate restore

It refuses only an XSTATE_BV bit outside the caps' features and x87 and SSE. A legitimate restore holds an area that KVM_GET_XSAVE returned, or a reset's, and neither has such a bit:

  • KVM returns only components within the guest's supported XCR0, which it takes from leaf 0xD of the CPUID that bind programs into it, and partition_capabilities derives the caps' features from the partition's CPUID, the same one.
  • A reset's area has XSTATE_BV 0.
  • The ignored test a_restore_gives_the_vp_its_component_or_writes_nothing restores a one-component area through xsave::restore for every component from 2 to 63, into a real VP whose CPUID enumerates the host's XCR0, its AVX subset, or x87 and SSE only. It first sees what KVM does with the area alone, and requires that the restore succeeds for every component that KVM keeps, fails with the check's error for one that KVM takes and drops, and fails for one that KVM refuses, and that a failed restore leaves the VP's XSAVE state as it was.

What the OpenVMM pull request changes

microsoft/openvmm#5101 adds one signed commit on c5509cde2, the head of feature/microvm/dev. It has a good signature (openvmm_pin.py source).

Position Commit Change
281 b723cbed3 set_xsave restores through the new xsave::restore, which calls check_xstate_bv before it writes anything and refuses an XSTATE_BV outside the caps' features, with x87 and SSE, with KvmError::InvalidState. Five native tests in xsave::tests cover the accepted and the refused XSTATE_BV of the Milan caps, a component that only caps with it accept, the x87 and SSE state of a partition without XSAVE, and areas shorter than the header; the ignored xsave::kvm_tests::a_restore_gives_the_vp_its_component_or_writes_nothing covers a real VP. With the check disabled, three native tests and the ignored test fail.

The issue's patch is data: its rule is the commit's, and its test is the model of the native ones.

What this pull request changes in NVX

  • d9f5568bebe moves the openvmm gitlink to the pin, and nothing else.
  • 0e38517ada3 maps position 281 in the OpenVMM roadmap.

No documentation or nvx-rs change: nothing of NVX names set_xsave or XSTATE_BV, and the Guide and doc/ describe the CPU profiles' XSAVE surface, not what a restore checks. No version is created or bumped, and no snapshot, saved-state, or wire format changes.

Compatibility of existing snapshots

A snapshot that a VP saved restores as before, since a save's XSTATE_BV is within the caps. Only an area that selects a component outside the caps, which a save never holds, is refused, and it never restored faithfully: the next save dropped the component.

Pin details

  • Roadmap. The pin is 157 commits past the mapped tip. Position 281 is not yet assigned to a PR and changes nothing that a mapped PR adds. It changes code that upstream main, 7d90c4104, has and no commit of the chain changes, and replaying it there applies without conflicts. virt_kvm type-checks there with its tests, which pass, the ignored one included, so it can go upstream on its own.
  • Prerequisites. None. The pull request's base, c5509cde2, is NVX's previous pin and the head of feature/microvm/dev.
  • Equivalence. None claimed; this pin replaces no reviewed revision.
  • Coordinated changes. None. The control-session vectors, the source excerpts, the contract versions, the activation bound, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's (openvmm_vectors and openvmm_cpu_profiles, below).

Landing order

microsoft/openvmm#5101 lands first, by fast-forwarding feature/microvm/dev from c5509cde2 to b723cbed35f253a7b97e9e7f32f3d7d3502cae36, which python scripts/openvmm_pin.py land b723cbed35f253a7b97e9e7f32f3d7d3502cae36 --remote REMOTE --push does. This pull request merges after that, through the merge queue. OpenVMM's merge button squashes, which would leave the pinned revision off the branch.

Validation

NVX

Run on Windows:

  • python scripts/openvmm_pin.py source b723cbed35f253a7b97e9e7f32f3d7d3502cae36 --pr 5101: passed. The revision is the pull request's head, with a good signature.
  • python scripts/openvmm_pin.py check-roadmap --worktree: passed, with 281 positions and 292 references.
  • python scripts/openvmm_pin.py check-branch --topic: passed, with two signed commits and no coordinated files.
  • python scripts/check_markdown_links.py and git diff --check: passed.
  • python scripts/nvx.py verify: passed.
  • cargo test --locked --manifest-path nvx-rs/Cargo.toml --no-default-features --test openvmm_vectors -- --ignored (4 passed) and --test openvmm_cpu_profiles -- --ignored (1 passed).
  • python scripts/check_test_changes.py --base origin/dev reports no change to an existing test, so the pull request needs no tests-modified label.
  • python scripts/local_gates.py --base origin/dev --run --keep-going: test-changes, markdown-links, test-layout, openvmm-vectors, openvmm-cpu-profiles, and nvx-cli passed; nvx-cli ran 2199 scripts/test_*.py tests (88 skipped), the 84 adversarial tests (11 skipped), and the help commands of nvx.py. openvmm-ci listed OpenVMM's six required jobs as pending, so the pull request's own openvmm-ci job decides it.

OpenVMM

With Rust 1.97.1 and cargo-nextest 0.9.133, on nvx-kvm-test, for the tree of the pinned commit:

  • Reproduction. The table above: at c5509cde2, set_xsave of XSTATE_BV 0x20, 0xe0, or 0x27 returns Ok and KVM reads 0x0, 0x0, or 0x7; with the commit, each returns InvalidState and the VP's XSTATE_BV stays 0x7.
  • Linux x64: cargo nextest run --profile agent -p virt_kvm passed 57 tests, cargo nextest run --profile agent --run-ignored all -p virt_kvm -p kvm passed 83, the ignored /dev/kvm ones included, cargo clippy --all-targets and cargo doc --no-deps on virt_kvm have no warnings, and cargo xtask fmt --fix changes nothing.
  • Mutations. Without the check, three native tests and the ignored test fail, the latter with "XCR0 mask 0x7, component 5: Dropped: Ok(())". With a restore that skips the check, or runs it after the write, only the ignored test fails. Without the x87 and SSE allowance, a native test fails.
  • Upstream main. In a temporary worktree of 7d90c4104 with the commit's tree change merged in (git merge-tree, as the roadmap tool prints), cargo check --tests -p virt_kvm passed, and cargo test -p virt_kvm xsave -- --include-ignored passed its 8 tests.

Hosts

All three are virtual machines. On each, OpenVMM was built from the pin's tree in a release build, the artifacts of v0.1.0-dev.ae35cbd896e0 came from nvx-rs download, and python scripts/nvx.py test-microvm --scenario smp --scenario smp-snapshot --scenario snapshot-core --scenario snapshot-rollback --scenario restore-downtime --scenario restore-processors --scenario snapshot-tiers --scenario time-abi-conformance --processors 1 2 --timeout 180 passed, with --backend kvm, mshv, and whp.

Host Platform Host type Before the fix With the pin
nvx-kvm-test (Intel Xeon Platinum 8573C, Linux 6.18.57.1-1.azl3) linux-kvm virtual-machine The reproduction above: Ok, and the component is dropped. InvalidState, the VP unchanged; the eight scenarios pass.
nvx-mshv-test (Intel Xeon Platinum 8573C, Linux 6.18.34.mshv3-1.azl3) linux-mshv virtual-machine Not affected: MSHV passes the area to the hypervisor as it is. The eight scenarios pass.
nvx-whp-test (Intel Xeon Platinum 8370C, Windows Server 2025) windows-whp virtual-machine Not affected, as on MSHV. The eight scenarios pass.

I left the rest of the KVM, MSHV, and WHP microVM suites, the NVX CLI gates on Linux, the runtime adapter, the platform and performance gates, and OpenVMM's own CI of the pin to CI.

CI

I left the rest of the KVM, MSHV, and WHP microVM suites, the NVX CLI gates on Linux, the runtime adapter, the platform and performance gates, and OpenVMM's own CI of the pin to CI, which passed:

On KVM, restoring a VP whose saved XSAVE area selected a component that
the partition's caps lack succeeded when the host's KVM supports the
component: KVM_SET_XSAVE takes any user component that it supports,
whether or not the guest's CPUID leaf 0xD enumerates it, but
KVM_GET_XSAVE returns only the enumerated ones, so the next save dropped
the component and the restored state was not the snapshot's (#1380).

Pin b723cbed35f253a7b97e9e7f32f3d7d3502cae36, the head of
microsoft/openvmm#5101, one commit on c5509cde2, the head of
feature/microvm/dev:

- b723cbed3 virt_kvm: refuse a restored XSAVE component outside the caps

set_xsave now checks the area's XSTATE_BV against the caps' XSAVE
features, with x87 and SSE, before it writes anything, and refuses a
component outside them with KvmError::InvalidState, so the restore fails
with the xsave element's StateError and leaves the VP's XSAVE state as
it was. A save never holds such a component, so a snapshot that a VP
saved restores as before, and only a crafted snapshot, or one that other
caps saved, is refused. MSHV and WHP pass the area to the hypervisor as
it is, and are unchanged. No version is created or bumped, and no
snapshot or wire format changes.

Tests: on nvx-kvm-test (Intel Xeon Platinum 8573C, KVM XCR0 0xe7, Linux
6.18.57), a scratch test builds the one-VP time ABI partition of
amd.milan.v1, whose features are 0x7. At c5509cde2, set_xsave of an area
whose XSTATE_BV is 0x20, 0xe0, or 0x27 returns Ok and KVM_GET_XSAVE then
reads XSTATE_BV 0x0, 0x0, or 0x7; at the pin, each returns InvalidState
and leaves XSTATE_BV at 0x7, and the partition of intel.emeraldrapids.v1
still restores its own components. With the check disabled, three of the
revision's native tests and its ignored /dev/kvm test fail; with it, its
57 tests of virt_kvm pass with cargo nextest, and its 83 tests of
virt_kvm and kvm pass with the ignored ones. With OpenVMM built from the
pin's tree and the artifacts of v0.1.0-dev.ae35cbd896e0, the microVM
scenarios smp, smp-snapshot, snapshot-core, snapshot-rollback,
restore-downtime, restore-processors, snapshot-tiers, and
time-abi-conformance pass at 1 and 2 processors on nvx-kvm-test,
nvx-mshv-test, and nvx-whp-test.

NVX needs no coordinated change: the control-session vectors, the source
excerpts, the contract versions, the activation bound, and the CPU
profiles that nvx-rs copies from OpenVMM match the revision's.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NVX pin moves to b723cbed3, position 281, 157 commits past the
mapped tip. Position 281, the one commit of microsoft/openvmm#5101 on
c5509cde2, the head of feature/microvm/dev, is not yet assigned to a PR.

Position 281 makes the KVM backend's set_xsave refuse a restored XSAVE
area whose XSTATE_BV selects a component that the partition's caps
lack, which KVM_SET_XSAVE took when the host's KVM supports it, and the
next save dropped. It restores through xsave::restore, in a new module
of virt_kvm's x86-64 module, which checks XSTATE_BV against the caps'
features, with x87 and SSE, before it writes anything, and adds five
unit tests and an ignored test that restores each component into a real
VP. It changes code that upstream main has and no commit of the chain
changes, in files that PR10's positions 019 and 022, PR17's position
070, and twelve positions that are not yet assigned also change, and it
changes nothing that a mapped PR adds. Replaying it on upstream main,
7d90c4104, applies without conflicts, and virt_kvm type-checks there
with its tests, which pass, the ignored one included.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The KVM XSAVE restore change needs maintainer validation of a hardware-backed regression ignored by CI, completion of required exact-pin checks, and confirmation that prerequisite OpenVMM #5101 has landed.

No open issues 🎉

What changed in this PR

Updates NVX’s OpenVMM pin to include the KVM XSAVE restore check for #1380, rejecting unsupported state components before they can be silently lost on a later save.

Changes:

  • Advances the pin to b723cbed3 from microsoft/openvmm#5101.
  • Maps position 281 and updates the roadmap’s chain counts and ranges.
  • Documents the component check, its tests, and its independent upstream applicability.
File Description
doc/​openvmm-upstream-roadmap.md Records the new pin and position 281, including behavior, test coverage, and upstream applicability.

Review effort: Balanced

@ppenna

Copy link
Copy Markdown
Contributor Author

Answering the three notes of the review summary at 0e38517ada3:

  1. The hardware-backed regression that CI ignores. The new /dev/kvm test, xsave::kvm_tests::a_restore_gives_the_vp_its_component_or_writes_nothing, is #[ignore]d and runs in no OpenVMM CI job, as the Validation section says, so I ran it where /dev/kvm exists: on nvx-kvm-test (Intel Xeon Platinum 8573C, KVM XCR0 0xe7), cargo nextest run --profile agent --run-ignored all -p virt_kvm -p kvm passed 83 tests with it, and with the check disabled it fails ("XCR0 mask 0x7, component 5: Dropped: Ok(())"). A reproduction through the one-VP partition of amd.milan.v1 shows what it guards: at c5509cde2, set_xsave of XSTATE_BV 0x20, 0xe0, or 0x27 returns Ok and KVM reads back 0x0, 0x0, or 0x7; with the pin, each returns InvalidState and the VP's XSTATE_BV stays 0x7.
  2. The exact-pin checks. They have completed. OpenVMM's run 43604358597 at b723cbed35f253a7b97e9e7f32f3d7d3502cae36 passed the six jobs that openvmm-ci requires, and this pull request's run 43606821137 passed at the current head, Required status check and OpenVMM CI of the pinned revision included.
  3. [DO NOT REVIEW] virt_kvm: refuse a restored XSAVE component outside the caps openvmm#5101 has not landed. That is the landing order: it lands first, by fast-forwarding feature/microvm/dev from c5509cde2 to the pinned revision, and this pull request is queued only after that.

@ppenna
Pedro Henrique Penna (ppenna) added this pull request to the merge queue Nov 14, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 14, 2026
@ppenna
Pedro Henrique Penna (ppenna) added this pull request to the merge queue Nov 14, 2026
@ppenna

Copy link
Copy Markdown
Contributor Author

The first merge group run, 43611241037, failed in NVX CLI / Windows on one test, scripts.test_host_telemetry.AzureMetadataRequestTests.test_a_connect_still_in_progress_is_shut_down_at_the_deadline (self.assertTrue(blocked.is_set())), the IMDS deadline test that #1239 tracks: it needs the request's worker to reach connect within 0.2 s, so a runner stall fails it. The other 2198 tests of that job passed, so did every other job of the run apart from Required status check, which reports it, and the same job passed on this head's run, 43606821137. This change touches no telemetry code: it moves the openvmm gitlink and edits the roadmap. I queued the pull request again once.

Merged via the queue into dev with commit 9e41ebd Nov 14, 2026
48 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the agents/issue-1380-xstate-bv-caps branch November 14, 2026 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openvmm: a restored XSAVE component outside the caps loads, then is lost

2 participants