Skip to content

openvmm: restore the init components of an XSAVE area as init - #1641

Merged
Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
agents/issue-1360-set-xsave-reset
Nov 15, 2026
Merged

Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
agents/issue-1360-set-xsave-reset

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1360

What and why

On KVM, KVM_SET_XSAVE does not always read a clear XSTATE_BV bit as "this component is in its init state" (fpu_copy_uabi_to_guest_fpstate and copy_uabi_to_xstate in the kernel's arch/x86/kernel/fpu/, checked in Linux v6.18 and v5.15), and what it keeps depends on the host. xsave::restore handed write_standard's buffer to it unchanged, so a VM reset (VmRpc::Reset, which a guest's reset request causes too) or a restore onto a VP that has run left state that KVM's own reports hid:

  • PKRU, on a host with PKU: KVM keeps the guest's PKRU in its vCPU, apart from the guest's FPU state, and writes it only from an area whose XSTATE_BV bit 9, the PKRU component, is set, so each VP kept its old PKRU while KVM_GET_XSAVE reported 0.
  • MXCSR, on a kernel that uses neither XSAVES nor XSAVEC (Linux 5.16 and later): KVM writes it only from an area with x87, SSE, or AVX in XSTATE_BV, but the standard form of XRSTOR, which the guest then runs with, loads it whatever XSTATE_BV says, so each VP kept its old MXCSR while KVM_GET_XSAVE reported 0x1f80.
  • x87 and SSE state, on a host without XSAVE: the kernel copies the whole 512-byte legacy area whatever XSTATE_BV says, so a crafted snapshot whose area leaves a legacy component out, with bytes that are not its init state, restored them as live registers.

A restore onto a new VP, which is what NVX does (a microVM never resets its VPs), starts from init state and is not affected by the first two; the third reaches a restore through a crafted snapshot.

The fix is in OpenVMM's virt_kvm (microsoft/openvmm#5118, one commit on feature/microvm/dev, head 23fb678876c9b2a1125664fc599e5b261b325d8c): xsave::restore now writes the buffers that the new xsave::Buffers builds.

  • init_cleared_legacy zeroes the x87 and XMM bytes of a legacy component that XSTATE_BV leaves out.
  • Where KVM would keep PKRU or MXCSR through the area, a seed is written first: the area with SSE marked in use where XSTATE_BV has none of x87, SSE, and AVX (so that KVM takes MXCSR, 0x1f80, from it), and with PKRU marked in use, its bytes zero, where the caps have PKRU and the area leaves it out.
  • The area itself is always written last, so the VP's XSAVE state is exactly the area's. KVM refuses a buffer before it changes the VP and takes the area whenever it took the seed, whose XSTATE_BV has all of the area's bits, so a refused restore still leaves the VP's XSAVE state as it was.

Why this is not the issue's fix.patch

fix.patch forces the SSE and PKRU bits in the one buffer that it writes. That leaves the forced bits set in the VP's XSAVE state, which two things see:

  • PKRU: a save reads XSTATE_BV bit 9 back set where the reset area (and the area of any restore that leaves PKRU init) has it clear, and Xsave::normalize does not clear it. The debug assertions of reset_all and restore_all, which compare a save with the area, would then fail on a host with PKU.
  • SSE: a guest's XGETBV(1), and its first XSAVE, could see SSE marked in use after a reset, where it was init.

Writing a seed first and the area last gets the same init values into KVM without either side effect.

With the PKRU raise of #1359

Since microsoft/openvmm#5117 (#1359), a VP's thread has a nonzero PKRU where the host has PKU, which XSAVE always marks in use, so a save after a KVM_RUN sets XSTATE_BV bit 9, the PKRU component, and reports the guest's PKRU, which is 0 after a reset: the same state as the component's init. A save right after a restore reads the area back, which is what the debug assertions check. No test of this change assumes a particular thread PKRU: the model has none, and the /dev/kvm test has its guest set and read its own PKRU.

No version

Nothing about saved state changes: no field, format, schema, capability, or version, and none is created or bumped. restore writes what the area already says. A restore of an area that has x87, SSE, or AVX, and that leaves no PKRU out, writes the same buffer as before, but for the init bytes of a legacy component that its XSTATE_BV leaves out; a snapshot saved before this change restores as it did on hosts that read a clear bit as init, and on the others it now restores the init state that the area names.

#1389

Hardware note for #1389, which is being closed as a kernel issue, not planned: on the Intel Xeon Platinum 8573C of the test host, XRSTOR in the compacted form loads MXCSR only with XSTATE_BV bit 1 set (XSTATE_BV 0, x87, or AVX alone give 0x1f80), where the standard form loads it whatever XSTATE_BV says. So an area with AVX, no SSE, and a non-default MXCSR, which only a standard-format host's save can produce (KVM_GET_XSAVE copies MXCSR when SSE or AVX is set), loses MXCSR when it is restored onto a compacted host. This change does not address that save-side loss, and #1381 (a short area that panics write_standard) is not covered either.

Validation

None of the three host cases could be reproduced on the hardware that is available. nvx-kvm-test (Intel Xeon Platinum 8573C, Linux 6.18.57, a virtual machine) has XSAVE, XSAVEC, and XSAVES, and no PKU (its KVM XCR0 is 0xe7, without PKRU), so it never keeps PKRU, MXCSR, or legacy bytes, and no host with PKU, or without XSAVEC and XSAVES, or without XSAVE, was at hand. Each case is covered by tests of the conversion instead:

Case Covered by
PKRU kept on a PKU host seed_components_are_sse_without_x87_sse_and_avx_and_pkru_where_the_caps_have_it (the seed adds PKRU where the caps have it and the area does not), a_reset_restores_init_components_as_buffers_that_say_so (the reset's two buffers, byte for byte), and the model tests below for the two PKU hosts
MXCSR kept on a kernel without XSAVEC and XSAVES the same tests, for the two standard-format hosts, and the XRSTOR measurement below
Legacy bytes loaded on a host without XSAVE init_cleared_legacy_zeroes_the_legacy_components_that_xstate_bv_leaves_out and the model's no-XSAVE host

The model tests restore into a model of a host's kernel, written from the kernel functions above and the XRSTOR behavior measured on the test host (standard-form XRSTOR loads the MXCSR in memory, 0x7f80 over 0x1fa0, with XSTATE_BV 0 or SSE; the compacted form gives 0x1f80 with XSTATE_BV 0, x87, or AVX, and 0x7f80 with SSE), for hosts with XSAVEC or XSAVES, without them, with PKU, and without XSAVE:

  • a_reset_gives_every_host_the_init_state: a reset gives each host the init state, and a save reads back the reset area; restoring the buffer as it was keeps stale PKRU, MXCSR, and legacy bytes on exactly the hosts that the issue names.
  • an_area_restores_as_the_state_that_it_means_on_every_host: every area of x87, SSE, AVX, and PKRU, with non-init bytes in the components that XSTATE_BV leaves out, restores as what the architecture says it means, and a save reads it back.

With each conversion removed in turn (cargo nextest run --profile agent -p virt_kvm --run-ignored all -E 'test(xsave)', 19 tests, on nvx-kvm-test, at the rebased tree), tests fail: without init_cleared_legacy, 1; without SSE in the seed, 4; without PKRU in the seed, 4; with none of the three (restore as it was), 4; with the seed written last instead of the area, 2.

The ignored test a_reset_gives_the_guest_the_init_state has a real-mode guest set MXCSR, FCW, an x87 register, XMM0, and, on a host with PKU, PKRU, restores the reset area, and reads them back. It passes on nvx-kvm-test with and without this change, since that host keeps no state, so it guards against a regression there and can fail only on a host that keeps state. Its PKRU part has never run. On a host with PKU, a maintainer can run it, from openvmm/, with:

cargo nextest run --run-ignored all -p virt_kvm -E 'test(xsave)'

All 19 tests should pass there, and the /dev/kvm test should fail without the change: after the reset, its guest would read back the PKRU that it set before.

Checks that ran

Profile Backend Host type Check Result
local Windows python scripts/nvx.py verify pass
local Windows python scripts/local_gates.py --base origin/dev --run --keep-going exit 0; test-changes, markdown-links, test-layout, openvmm-vectors, openvmm-cpu-profiles, and openvmm-ci passed, openvmm-ci as a local false pass, since the tool reports it as passed although it cannot check the pin's OpenVMM runs from here, so CI does. The nvx-cli gate ran in the same command, and its per-test details are not quoted here: CI's NVX CLI jobs cover them (the change touches neither scripts/ nor nvx-rs/)
local Windows python scripts/openvmm_pin.py check-roadmap --worktree, python scripts/openvmm_pin.py check-branch --topic, python scripts/check_markdown_links.py, python scripts/check_test_layout.py, python scripts/check_test_changes.py --base origin/dev (0 changes), git diff --check pass
local Windows cargo test --locked --manifest-path nvx-rs/Cargo.toml --no-default-features --test openvmm_vectors -- --ignored, and --test openvmm_cpu_profiles -- --ignored pass (4 tests and 1): no coordinated change
local Windows in openvmm/ at 23fb67887: cargo clippy --locked --all-targets -p virt_kvm --target x86_64-unknown-linux-musl -- -D warnings, and the same for --target aarch64-apple-darwin pass (the second builds none of this change's code, which only x86-64 Linux compiles)
nvx-kvm-test kvm virtual-machine in openvmm/ at 85b93bbd2 plus the commit: cargo check --locked -p virt_kvm, cargo clippy --locked --all-targets -p virt_kvm -- -D warnings (glibc and x86_64-unknown-linux-musl), RUSTDOCFLAGS='-D warnings' cargo doc --locked --no-deps -p virt_kvm, cargo xtask fmt --fix pass: no warnings, nothing to change
nvx-kvm-test kvm virtual-machine cargo nextest run --locked --profile agent -p virt_kvm --no-fail-fast --run-ignored all 121 pass, 0 skipped (the /dev/kvm tests included; the PKRU part of the new one returns early on this host)
nvx-kvm-test kvm virtual-machine the mutation sweep above 19 pass unmutated; 1, 4, 4, 4, and 2 fail with the conversions removed
nvx-kvm-test kvm virtual-machine python3 scripts/nvx.py test-microvm --backend kvm --processors 1 2 with --scenario snapshot-core, restore-processors, smp-snapshot, snapshot-rollback, restore-memory, and restore-downtime, which restore a guest's XSAVE area through restore, with the openvmm release binary built at the pin (source_revision 23fb678876c9b2a1125664fc599e5b261b325d8c, clean) in a checkout of this branch, with the release artifacts of v0.1.0-dev.f7f4cde335c7 pass (exit 0; no CPU-profile fallback warning)

The cargo rows ran in a tree of 85b93bbd2 with the commit applied, whose content is the pinned tree's; the test-microvm row ran in the checkout of the pin itself.

Left to CI: openvmm-ci and OpenVMM's own CI of the pinned revision, which openvmm-ci waits for; the MSHV and WHP jobs (the change is confined to virt_kvm, which compiles only on Linux); and the benchmarks.

CI of this head, 53352a22c74, whose OpenVMM is 23fb67887:

  • NVX run 43740586732: every job passed or was skipped (35 passed, 8 skipped), OpenVMM CI of the pinned revision, the microVM tests on KVM, MSHV, and WHP, the performance regression gate, and Required status check included, with no rerun.
  • OpenVMM run 43738991166 for [DO NOT REVIEW] virt_kvm: restore the init components of an XSAVE area as init openvmm#5118 at 23fb67887: the six jobs that openvmm-ci requires passed, the unit tests on x64-linux and x64-windows, and the vmm-tests on x64-linux-amd-kvm, x64-linux-intel-mshv, x64-windows-intel, and x64-windows-amd. x64-windows-amd-snp failed and is not required. No job was rerun.
  • Copilot's review of this head (53352a22c74) has no thread and no open issue. Its one note is the one that this description states under Validation: none of the three host cases was exercised on hardware, and the PKRU hardware path has never run, which only a host with PKU can settle.

Tests modified

python scripts/check_test_changes.py --base origin/dev lists no change to an existing test asset, so this pull request has no tests-modified label. The tests are OpenVMM's, listed in microsoft/openvmm#5118, and all of them are new: five unit tests and the /dev/kvm test in xsave.rs.

OpenVMM pin

A KVM_SET_XSAVE of an area that marks PKRU, MXCSR, or x87 and SSE state
as init did not always make the guest's registers init: it kept a VP's
old PKRU on a host with PKU, its old MXCSR on a kernel that uses neither
XSAVES nor XSAVEC, and loaded the legacy bytes that XSTATE_BV marks init
on a host without XSAVE. So a VM reset, a restore onto a VP that has
run, or a crafted snapshot left state that KVM's own reports hid
(#1360).

Pin 23fb678876c9b2a1125664fc599e5b261b325d8c, the head of
microsoft/openvmm#5118, one commit on 85b93bbd2, the head of
feature/microvm/dev:

- 23fb67887 virt_kvm: restore the init components of an XSAVE area as
  init

xsave::restore now writes the legacy components that the area's
XSTATE_BV leaves out as init bytes and, where KVM would keep PKRU or
MXCSR, first a seed that makes KVM write their init values, then the
area itself, so that the VP's XSAVE state is the area's own. The single
forced buffer that the issue proposed would leave PKRU and SSE marked in
use, which the debug assertions of reset_all and restore_all, and a
guest's XGETBV(1), would see, so it is not used. No saved state,
saved-state format, or version changes, and none is created or bumped. A
restore onto a new VP, which is what NVX does, behaves as before.

Tests: unit tests of virt_kvm, which OpenVMM's CI runs, among them a
model of the hosts' kernels, and one KVM test, ignored without /dev/kvm,
whose PKRU part runs only on a host with PKU. None of the three cases
occurs on the hardware that was available, so the model and byte tests
cover them, and fail when any one of the conversions is removed; the
PKRU part of the KVM test has not run.

NVX needs no coordinated change: the control-session vectors, the source
excerpts, the contract versions, the activation bound, and the CPU
profiles that nvx-rs copies from OpenVMM match the revision's.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NVX pin moves to 23fb67887, position 301, 177 commits past the
mapped tip. Position 301, the one commit of microsoft/openvmm#5118 on
85b93bbd2, the head of feature/microvm/dev, is not yet assigned to a PR.

Position 301 makes a restore of an XSAVE area write the init state that
the area names, on every KVM host: KVM_SET_XSAVE keeps a VP's PKRU on a
host with PKU and its MXCSR on a kernel that uses neither XSAVES nor
XSAVEC when the area's XSTATE_BV leaves the component out, and a kernel
without XSAVE loads the legacy bytes that XSTATE_BV marks init.
xsave::restore now writes the legacy components that XSTATE_BV leaves
out as init bytes, a seed first where KVM would keep PKRU or MXCSR, and
the area itself last. It changes a file that only position 281 changes,
and it changes only lines that position 281 added, which no PR has yet,
so it changes no line that a mapped PR adds. Replaying it on upstream
main conflicts in xsave.rs, which does not exist there, since position
281 adds it. So it follows position 281, which applies there on its own.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The host-dependent XSAVE/PKRU change requires maintainer assessment of model fidelity because none of the three affected host configurations was exercised and the PKRU hardware path has never run.

No open issues 🎉

What changed in this PR

This PR records the OpenVMM revision that fixes restoration of XSAVE components marked as init on affected KVM hosts, addressing #1360 without changing the saved-state format.

Changes:

  • Updates the documented NVX pin to 23fb67887, position 301.
  • Updates chain counts and maps the commit to microsoft/openvmm#5118.
  • Documents the fix, its tests, and its upstream dependency on position 281.
File Description
doc/​openvmm-upstream-roadmap.md Updates the pin and commit mapping, and explains the XSAVE restore fix and upstream ordering.

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openvmm: KVM's set_xsave keeps old PKRU or MXCSR after a VM reset on some hosts

2 participants