Skip to content

openvmm: keep undelivered portb output - #1405

Merged
Pedro Henrique Penna (ppenna) merged 3 commits into
devfrom
agents/issue-1370-portb-write-error
Nov 9, 2026
Merged

Pedro Henrique Penna (ppenna) merged 3 commits into
devfrom
agents/issue-1370-portb-write-error

Conversation

@ppenna

Copy link
Copy Markdown
Contributor

Fixes #1370

What and why

The byte channel (portb, the microVM's raw output console at port 0xe9) lost
guest output that a snapshot should keep, in two ways:

  • A write to its host endpoint that failed with any error other than
    BrokenPipe, such as ConnectionReset, cleared all the output that portb had
    buffered, up to 1 MiB. A snapshot's console flush and the device's stop both
    write that output, so such a failure there saved an empty buffer, and the
    restored VM never wrote those bytes, although the controller logs that
    undelivered bytes stay in the snapshot. The output was lost for the
    endpoint's next peer too.
  • The device's stop wrote to its endpoint whether or not a peer was connected.
    The socket, named-pipe, and file backends drop their connection at any failed
    write and then discard writes while reporting them as written, so stop
    emptied the buffer of an endpoint that no peer had joined, or that a failed
    write had disconnected, even after a broken pipe, which openvmm: a console write error drops microVM output a snapshot should keep #1370 took for safe.
    NVX's own endpoint, the pair that OpenVMM relays to its stderr, is such a
    backend: once that relay is gone, a snapshot lost the output that the guest
    wrote meanwhile (see Validation).

microsoft/openvmm#4989, chipset: keep undelivered microVM portb output, keeps
the output that the endpoint has not accepted, whatever the error, for the next
write, the next peer, and the snapshot, and its stop writes only to a connected
endpoint. Of the retention policies that #1370 weighs, it takes the first,
keeping the bytes on every error:

  • The serial backends drop their connection on any write failure precisely so
    that a frontend such as portb keeps what a write fails to send, for the next
    peer.
  • Keeping the bytes only while a flush or stop runs would keep only the output
    written since the last failure, and still lose it for the next peer.
  • Keeping the drop, with corrected messages, would contradict the byte
    channel's specification (design 3.1), the CLI guide, and the controller's
    warning, under which pending output is saved.

The 1 MiB bound and its overflow policy are unchanged: a full buffer drops the
newest bytes, and guest writes never block. A failed write is retried at the
next guest write, drain request, or peer, never in a loop. A failure other than
a broken pipe now logs a rate-limited warning rather than an error, since no
output is lost.

Commits:

  • openvmm: pin keeping undelivered portb output moves the gitlink from
    99aac19b3, the head of feature/microvm/dev, to
    881619911d1a6ec2fec7bebdad7cfe69d09018f1, the head of
    [DO NOT REVIEW] chipset: keep undelivered microVM portb output openvmm#4989, one commit on 99aac19b3. The protocol vectors and
    the CPU profiles that nvx-rs copies are unchanged.
  • doc: say when the byte channel's output stays pending says, in design 3.1,
    which output stays pending, and that the restored VM writes it.
  • doc: map keeping undelivered portb output in the OpenVMM roadmap maps
    position 226, unassigned, 95 commits past the mapped tip, with the mapped
    work that it revisits: the output of PR15's portb console (position 060),
    PR25's sentence on it in the CLI guide (position 125), and position 208's
    test endpoint. It conflicts with upstream main in both of its files, so it
    follows PR15 and PR25 upstream.

881619911 is patch-equivalent to 65a8e2c50, the same commit on 90f9e6180,
which the reproduction below tested (openvmm_pin.py equivalence: identical
range-diff).

OpenVMM's test changes (this pull request changes no NVX test):

  • The Listener test endpoint now discards writes while no peer is connected
    and reports them as written, as the socket and named-pipe backends do, where
    it recorded them as delivered, and it can report a reset as
    ConnectionReset. The existing tests that use it never write without a peer.
  • portb_output_error_preserves_the_next_write_wakeup now expects the failed
    write's bytes before the next write's (b"kept next write"), where it
    expected them dropped; it still checks that the next guest write wakes the
    device.

Landing order: land microsoft/openvmm#4989 first, by fast-forwarding
feature/microvm/dev to 881619911d1a6ec2fec7bebdad7cfe69d09018f1 (OpenVMM's
merge button squashes, which would leave the pin off the branch); then merge
this pull request.

Validation

Reproduction, before and after:

Check Profile Backend Host type Before (90f9e6180) After (65a8e2c50) After the rebase (881619911)
cargo nextest run --profile agent -p chipset nvx-kvm-test Linux virtual machine 5 of 100 fail 100 of 100 pass 100 of 100 pass
cargo nextest run --profile agent -p chipset local workstation Windows - 5 of 100 fail 100 of 100 pass 100 of 100 pass
Relay gone, then snapshot and restore, 3 runs nvx-kvm-test kvm virtual machine lost 3/3 kept 3/3 kept 3/3
Relay gone, then snapshot and restore, 3 runs nvx-mshv-test mshv virtual machine lost 3/3 kept 3/3 kept 3/3
Relay gone, then snapshot and restore, 3 runs nvx-whp-test whp virtual machine lost 3/3 kept 3/3 kept 3/3

The end-to-end check boots the guest of release v0.1.0-dev.733a7031e655 with
its boot console on OpenVMM's standard streams and closes OpenVMM's stderr, so
the byte channel's relay fails at the guest's next write, NVX-1370-A, and
exits. The guest then writes NVX-1370-B to /dev/hvc0, which finds the relay
gone, requests a snapshot, and, once restored, writes NVX-1370-C. The
release's own OpenVMM (90f9e6180) restored a VM that wrote only C; with the
change, the restored VM wrote the tail of A that the relay never read, then
B, then C.

With this branch's head (OpenVMM built at the pin on each host),
python3 scripts/nvx.py test-microvm --backend BACKEND --scenario console-exit --scenario console-snapshot --scenario snapshot-core --scenario snapshot-rollback --scenario lifecycle --scenario run-console-restore --scenario smp-snapshot --processors 1 2 passed on nvx-kvm-test (kvm),
nvx-mshv-test (mshv), and nvx-whp-test (whp), all virtual machines, as it did
before the rebase.

Three of the first five runs of console-exit with 2 vCPUs on nvx-whp-test, a
host that other sessions share, timed out: after 60 s, the restored guest had
written only 8-33 KiB of its 64 KiB to hvc0, and the 1-vCPU console-exit run
just before the first of them took 29 s, against 11 s later. Every later run
passed: 5 runs of the scenario with the change, 6 at origin/dev (99aac19b3),
and 10 replays of its restore with the change. The change alters only what a
failed write keeps and what stop writes to an endpoint without a peer; a
failed write there would end the restore with an exit error, within the exit
drain's 5 s, not stall it.

OpenVMM package checks for chipset, on Linux and Windows: cargo check,
cargo clippy --all-targets, and cargo doc --no-deps pass without warnings,
and nextest passes 100 of 100; cargo xtask fmt --fix passes.

Local gates (python scripts/local_gates.py --base origin/dev --run, Windows):
test-changes (no changes to existing tests), markdown-links, test-layout,
openvmm-vectors, and openvmm-cpu-profiles pass, and
python scripts/openvmm_pin.py check-branch --topic and check-roadmap pass.
nvx-cli passes (2187 and 84 tests). openvmm-ci waits for OpenVMM's CI of the
pin, the run of microsoft/openvmm#4989.

Left to CI: the full microVM matrix, the other scenarios and processor counts,
benchmarks, and OpenVMM's CI of the pin.

Pins microsoft/openvmm#4989 at 881619911d1a6ec2fec7bebdad7cfe69d09018f1,
one commit on 99aac19b3, the head of feature/microvm/dev:

- 881619911 chipset: keep undelivered microVM portb output

The byte channel (portb) dropped all the guest output that it had
buffered when a write to its host endpoint failed with an error other
than a broken pipe, such as a connection reset. Its stop then wrote the
output that it had kept to an endpoint without a peer, which the
socket, named-pipe, and file backends discard and report as written. A
snapshot taken after such a failure, or while no peer was connected,
therefore saved none of that output, and the restored VM never wrote
it, although the byte channel's specification saves pending output so
that capture does not silently lose it (#1370). NVX's own endpoint, the
pair that OpenVMM relays to stderr, also disconnects at its first
failed write, so once that relay was gone, a snapshot lost the output
that the guest wrote meanwhile.

The pinned revision keeps the output that the endpoint has not
accepted, whatever the error, for the next write, the next peer, and
the snapshot, and its stop writes only to a connected endpoint. The
protocol vectors, the excerpts and contract versions that nvx-rs copies
from OpenVMM, and its CPU profiles match the revision's.

On nvx-kvm-test, nvx-mshv-test, and nvx-whp-test (KVM, MSHV, and WHP
virtual machines), the guest of release v0.1.0-dev.733a7031e655 wrote a
marker to hvc0 after OpenVMM's stderr was closed, so that the byte
channel's relay had exited, then requested a snapshot, which was
restored. The release's own OpenVMM, at 90f9e6180, lost the marker in
3 of 3 runs on each backend. The change on 90f9e6180, 65a8e2c50, and
the pinned revision both had the restored VM write it, in 3 of 3 runs
each on each backend.

Tests: OpenVMM's chipset tests pass, 100 of 100, on Linux and Windows,
and the 5 new and changed ones fail without the change. The
openvmm_vectors and openvmm_cpu_profiles comparisons pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The byte channel's specification saves pending output, so that capture
does not silently lose it, but did not say which output is pending. The
pinned OpenVMM revision keeps the output that the host endpoint has not
accepted, while no peer is connected and after a write to the endpoint
fails, whatever the error, where it used to drop the output after most
write failures and to lose it at the device's stop when no peer was
connected (#1370).

Say so in 3.1, with the restored VM writing the saved output to its
endpoint.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Map position 226, 881619911, the commit of microsoft/openvmm#4989 that
the previous commit pins, after position 225, 99aac19b3, the head of
feature/microvm/dev. It keeps the output that portb's host endpoint has
not accepted, after a failed write with any error and at the device's
stop while no peer is connected, for the next peer and the snapshot.

It changes the output of the portb console of PR15's position 060, the
CLI guide's sentence on that output, which PR25's position 125 adds,
and position 208's test endpoint, so it is unassigned, as the other
commits after the mapped tip are, and the remarks say which mapped work
it revisits. Replaying it on upstream main conflicts in both of its
files, which upstream main lacks or lacks the changed part of, so it
follows PR15 and PR25 upstream.

The pin is 95 commits past the mapped tip.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block approval of the documentation updates.

No open issues 🎉

What changed in this PR

The reviewed documentation accompanies the OpenVMM fix for #1370, clarifying how undelivered microVM byte-channel output survives snapshots and recording the fix’s upstream placement.

Changes:

  • Clarifies that output remains pending without a connected peer or after any write failure, and is saved for delivery after restore.
  • Updates the roadmap to pin 881619911 at position 226 and records its dependencies on earlier upstream work.
File Description
doc/​openvmm-upstream-roadmap.md Records the new pin, commit position, and upstream dependencies.
doc/​design/​3.1-platform-control-device.md Clarifies output retention and delivery after restore.

Review effort: Balanced

@ppenna
Pedro Henrique Penna (ppenna) added this pull request to the merge queue Nov 9, 2026
Merged via the queue into dev with commit 94584ae Nov 9, 2026
88 of 90 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the agents/issue-1370-portb-write-error branch November 9, 2026 03:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openvmm: a console write error drops microVM output a snapshot should keep

2 participants