Repository navigation
openvmm: restore the init components of an XSAVE area as init - #1641
Merged
Pedro Henrique Penna (ppenna) merged 2 commits intoNov 15, 2026
Merged
Conversation
A KVM_SET_XSAVE of an area that marks PKRU, MXCSR, or x87 and SSE state as init did not always make the guest's registers init: it kept a VP's old PKRU on a host with PKU, its old MXCSR on a kernel that uses neither XSAVES nor XSAVEC, and loaded the legacy bytes that XSTATE_BV marks init on a host without XSAVE. So a VM reset, a restore onto a VP that has run, or a crafted snapshot left state that KVM's own reports hid (#1360). Pin 23fb678876c9b2a1125664fc599e5b261b325d8c, the head of microsoft/openvmm#5118, one commit on 85b93bbd2, the head of feature/microvm/dev: - 23fb67887 virt_kvm: restore the init components of an XSAVE area as init xsave::restore now writes the legacy components that the area's XSTATE_BV leaves out as init bytes and, where KVM would keep PKRU or MXCSR, first a seed that makes KVM write their init values, then the area itself, so that the VP's XSAVE state is the area's own. The single forced buffer that the issue proposed would leave PKRU and SSE marked in use, which the debug assertions of reset_all and restore_all, and a guest's XGETBV(1), would see, so it is not used. No saved state, saved-state format, or version changes, and none is created or bumped. A restore onto a new VP, which is what NVX does, behaves as before. Tests: unit tests of virt_kvm, which OpenVMM's CI runs, among them a model of the hosts' kernels, and one KVM test, ignored without /dev/kvm, whose PKRU part runs only on a host with PKU. None of the three cases occurs on the hardware that was available, so the model and byte tests cover them, and fail when any one of the conversions is removed; the PKRU part of the KVM test has not run. NVX needs no coordinated change: the control-session vectors, the source excerpts, the contract versions, the activation bound, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NVX pin moves to 23fb67887, position 301, 177 commits past the mapped tip. Position 301, the one commit of microsoft/openvmm#5118 on 85b93bbd2, the head of feature/microvm/dev, is not yet assigned to a PR. Position 301 makes a restore of an XSAVE area write the init state that the area names, on every KVM host: KVM_SET_XSAVE keeps a VP's PKRU on a host with PKU and its MXCSR on a kernel that uses neither XSAVES nor XSAVEC when the area's XSTATE_BV leaves the component out, and a kernel without XSAVE loads the legacy bytes that XSTATE_BV marks init. xsave::restore now writes the legacy components that XSTATE_BV leaves out as init bytes, a seed first where KVM would keep PKRU or MXCSR, and the area itself last. It changes a file that only position 281 changes, and it changes only lines that position 281 added, which no PR has yet, so it changes no line that a mapped PR adds. Replaying it on upstream main conflicts in xsave.rs, which does not exist there, since position 281 adds it. So it follows position 281, which applies there on its own. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The host-dependent XSAVE/PKRU change requires maintainer assessment of model fidelity because none of the three affected host configurations was exercised and the PKRU hardware path has never run.
No open issues 🎉
What changed in this PR
This PR records the OpenVMM revision that fixes restoration of XSAVE components marked as init on affected KVM hosts, addressing #1360 without changing the saved-state format.
Changes:
- Updates the documented NVX pin to
23fb67887, position 301. - Updates chain counts and maps the commit to microsoft/openvmm#5118.
- Documents the fix, its tests, and its upstream dependency on position 281.
| File | Description |
|---|---|
doc/openvmm-upstream-roadmap.md |
Updates the pin and commit mapping, and explains the XSAVE restore fix and upstream ordering. |
Review effort: Balanced
Pedro Henrique Penna (ppenna)
deleted the
agents/issue-1360-set-xsave-reset
branch
November 15, 2026 18:34
This was referenced Nov 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1360
What and why
On KVM,
KVM_SET_XSAVEdoes not always read a clear XSTATE_BV bit as "this component is in its init state" (fpu_copy_uabi_to_guest_fpstateandcopy_uabi_to_xstatein the kernel'sarch/x86/kernel/fpu/, checked in Linux v6.18 and v5.15), and what it keeps depends on the host.xsave::restorehandedwrite_standard's buffer to it unchanged, so a VM reset (VmRpc::Reset, which a guest's reset request causes too) or a restore onto a VP that has run left state that KVM's own reports hid:KVM_GET_XSAVEreported 0.KVM_GET_XSAVEreported 0x1f80.A restore onto a new VP, which is what NVX does (a microVM never resets its VPs), starts from init state and is not affected by the first two; the third reaches a restore through a crafted snapshot.
The fix is in OpenVMM's
virt_kvm(microsoft/openvmm#5118, one commit onfeature/microvm/dev, head23fb678876c9b2a1125664fc599e5b261b325d8c):xsave::restorenow writes the buffers that the newxsave::Buffersbuilds.init_cleared_legacyzeroes the x87 and XMM bytes of a legacy component that XSTATE_BV leaves out.Why this is not the issue's
fix.patchfix.patchforces the SSE and PKRU bits in the one buffer that it writes. That leaves the forced bits set in the VP's XSAVE state, which two things see:Xsave::normalizedoes not clear it. The debug assertions ofreset_allandrestore_all, which compare a save with the area, would then fail on a host with PKU.Writing a seed first and the area last gets the same init values into KVM without either side effect.
With the PKRU raise of #1359
Since microsoft/openvmm#5117 (#1359), a VP's thread has a nonzero PKRU where the host has PKU, which XSAVE always marks in use, so a save after a
KVM_RUNsets XSTATE_BV bit 9, the PKRU component, and reports the guest's PKRU, which is 0 after a reset: the same state as the component's init. A save right after a restore reads the area back, which is what the debug assertions check. No test of this change assumes a particular thread PKRU: the model has none, and the/dev/kvmtest has its guest set and read its own PKRU.No version
Nothing about saved state changes: no field, format, schema, capability, or version, and none is created or bumped.
restorewrites what the area already says. A restore of an area that has x87, SSE, or AVX, and that leaves no PKRU out, writes the same buffer as before, but for the init bytes of a legacy component that its XSTATE_BV leaves out; a snapshot saved before this change restores as it did on hosts that read a clear bit as init, and on the others it now restores the init state that the area names.#1389
Hardware note for #1389, which is being closed as a kernel issue, not planned: on the Intel Xeon Platinum 8573C of the test host, XRSTOR in the compacted form loads MXCSR only with XSTATE_BV bit 1 set (XSTATE_BV 0, x87, or AVX alone give 0x1f80), where the standard form loads it whatever XSTATE_BV says. So an area with AVX, no SSE, and a non-default MXCSR, which only a standard-format host's save can produce (
KVM_GET_XSAVEcopies MXCSR when SSE or AVX is set), loses MXCSR when it is restored onto a compacted host. This change does not address that save-side loss, and #1381 (a short area that panicswrite_standard) is not covered either.Validation
None of the three host cases could be reproduced on the hardware that is available.
nvx-kvm-test(Intel Xeon Platinum 8573C, Linux 6.18.57, a virtual machine) has XSAVE, XSAVEC, and XSAVES, and no PKU (its KVM XCR0 is 0xe7, without PKRU), so it never keeps PKRU, MXCSR, or legacy bytes, and no host with PKU, or without XSAVEC and XSAVES, or without XSAVE, was at hand. Each case is covered by tests of the conversion instead:seed_components_are_sse_without_x87_sse_and_avx_and_pkru_where_the_caps_have_it(the seed adds PKRU where the caps have it and the area does not),a_reset_restores_init_components_as_buffers_that_say_so(the reset's two buffers, byte for byte), and the model tests below for the two PKU hostsinit_cleared_legacy_zeroes_the_legacy_components_that_xstate_bv_leaves_outand the model's no-XSAVE hostThe model tests restore into a model of a host's kernel, written from the kernel functions above and the XRSTOR behavior measured on the test host (standard-form XRSTOR loads the MXCSR in memory, 0x7f80 over 0x1fa0, with XSTATE_BV 0 or SSE; the compacted form gives 0x1f80 with XSTATE_BV 0, x87, or AVX, and 0x7f80 with SSE), for hosts with XSAVEC or XSAVES, without them, with PKU, and without XSAVE:
a_reset_gives_every_host_the_init_state: a reset gives each host the init state, and a save reads back the reset area; restoring the buffer as it was keeps stale PKRU, MXCSR, and legacy bytes on exactly the hosts that the issue names.an_area_restores_as_the_state_that_it_means_on_every_host: every area of x87, SSE, AVX, and PKRU, with non-init bytes in the components that XSTATE_BV leaves out, restores as what the architecture says it means, and a save reads it back.With each conversion removed in turn (
cargo nextest run --profile agent -p virt_kvm --run-ignored all -E 'test(xsave)', 19 tests, onnvx-kvm-test, at the rebased tree), tests fail: withoutinit_cleared_legacy, 1; without SSE in the seed, 4; without PKRU in the seed, 4; with none of the three (restore as it was), 4; with the seed written last instead of the area, 2.The ignored test
a_reset_gives_the_guest_the_init_statehas a real-mode guest set MXCSR, FCW, an x87 register, XMM0, and, on a host with PKU, PKRU, restores the reset area, and reads them back. It passes onnvx-kvm-testwith and without this change, since that host keeps no state, so it guards against a regression there and can fail only on a host that keeps state. Its PKRU part has never run. On a host with PKU, a maintainer can run it, fromopenvmm/, with:cargo nextest run --run-ignored all -p virt_kvm -E 'test(xsave)'All 19 tests should pass there, and the
/dev/kvmtest should fail without the change: after the reset, its guest would read back the PKRU that it set before.Checks that ran
python scripts/nvx.py verifypython scripts/local_gates.py --base origin/dev --run --keep-goingtest-changes,markdown-links,test-layout,openvmm-vectors,openvmm-cpu-profiles, andopenvmm-cipassed,openvmm-cias a local false pass, since the tool reports it as passed although it cannot check the pin's OpenVMM runs from here, so CI does. Thenvx-cligate ran in the same command, and its per-test details are not quoted here: CI's NVX CLI jobs cover them (the change touches neitherscripts/nornvx-rs/)python scripts/openvmm_pin.py check-roadmap --worktree,python scripts/openvmm_pin.py check-branch --topic,python scripts/check_markdown_links.py,python scripts/check_test_layout.py,python scripts/check_test_changes.py --base origin/dev(0 changes),git diff --checkcargo test --locked --manifest-path nvx-rs/Cargo.toml --no-default-features --test openvmm_vectors -- --ignored, and--test openvmm_cpu_profiles -- --ignoredopenvmm/at23fb67887:cargo clippy --locked --all-targets -p virt_kvm --target x86_64-unknown-linux-musl -- -D warnings, and the same for--target aarch64-apple-darwinnvx-kvm-testopenvmm/at85b93bbd2plus the commit:cargo check --locked -p virt_kvm,cargo clippy --locked --all-targets -p virt_kvm -- -D warnings(glibc andx86_64-unknown-linux-musl),RUSTDOCFLAGS='-D warnings' cargo doc --locked --no-deps -p virt_kvm,cargo xtask fmt --fixnvx-kvm-testcargo nextest run --locked --profile agent -p virt_kvm --no-fail-fast --run-ignored all/dev/kvmtests included; the PKRU part of the new one returns early on this host)nvx-kvm-testnvx-kvm-testpython3 scripts/nvx.py test-microvm --backend kvm --processors 1 2with--scenariosnapshot-core,restore-processors,smp-snapshot,snapshot-rollback,restore-memory, andrestore-downtime, which restore a guest's XSAVE area throughrestore, with theopenvmmrelease binary built at the pin (source_revision23fb678876c9b2a1125664fc599e5b261b325d8c, clean) in a checkout of this branch, with the release artifacts ofv0.1.0-dev.f7f4cde335c7The
cargorows ran in a tree of85b93bbd2with the commit applied, whose content is the pinned tree's; thetest-microvmrow ran in the checkout of the pin itself.Left to CI:
openvmm-ciand OpenVMM's own CI of the pinned revision, whichopenvmm-ciwaits for; the MSHV and WHP jobs (the change is confined tovirt_kvm, which compiles only on Linux); and the benchmarks.CI of this head,
53352a22c74, whose OpenVMM is23fb67887:OpenVMM CI of the pinned revision, the microVM tests on KVM, MSHV, and WHP, the performance regression gate, andRequired status checkincluded, with no rerun.23fb67887: the six jobs thatopenvmm-cirequires passed, the unit tests on x64-linux and x64-windows, and the vmm-tests onx64-linux-amd-kvm,x64-linux-intel-mshv,x64-windows-intel, andx64-windows-amd.x64-windows-amd-snpfailed and is not required. No job was rerun.53352a22c74) has no thread and no open issue. Its one note is the one that this description states under Validation: none of the three host cases was exercised on hardware, and the PKRU hardware path has never run, which only a host with PKU can settle.Tests modified
python scripts/check_test_changes.py --base origin/devlists no change to an existing test asset, so this pull request has notests-modifiedlabel. The tests are OpenVMM's, listed in microsoft/openvmm#5118, and all of them are new: five unit tests and the/dev/kvmtest inxsave.rs.OpenVMM pin
23fb678876c9b2a1125664fc599e5b261b325d8c, a signed commit.85b93bbd2, the head offeature/microvm/dev, which holds [DO NOT REVIEW] virt_kvm: raise a VP thread's PKRU of 0 so that saves keep the guest's openvmm#5117 (openvmm: with task PKRU 0 on a PKU host, KVM snapshots can lose guest PKRU #1359). It changes one file,vmm_core/virt_kvm/src/arch/x86_64/xsave.rs. No prerequisites.xsave.rs, which position 281 adds, so it follows position 281 upstream.cargo test --locked --manifest-path nvx-rs/Cargo.toml --no-default-features --test openvmm_vectors -- --ignoredand--test openvmm_cpu_profiles -- --ignored) pass at the pin.python scripts/openvmm_pin.py check-roadmap --worktreeandcheck-branch --topicpass.feature/microvm/devto the pinned revision23fb678876c9b2a1125664fc599e5b261b325d8c(OpenVMM's merge button squashes and would leave the pin off the branch), then queue this pull request withgh pr merge --merge.