Repository navigation
Description
Summary
OpenVMM vmm-tests fails on every CI runner whose stable Rust is 1.99.0. Before any test runs, flowey builds OpenVMM's guest_test_uefi for x86_64-unknown-uefi, and the link fails with rust-lld: error: undefined symbol: wcslen. Runners with Rust 1.98.1 build the same OpenVMM revision and pass.
This is the upstream regression rust-lang/rust#163614 (open, P-high, regression-from-stable-to-stable). Rust 1.99.0 ships LLVM 23.1.1, which can replace a loop that measures a null-terminated UTF-16 string with a call to wcslen. rust-lang/compiler-builtins#1297 added a wcslen builtin for the earlier report, rust-lang/rust#160827. The 1.99 beta backport, rust-lang/rust#162987, did not include it, so the 1.99.0 UEFI link has no wcslen.
NVX picks up the new compiler because it doesn't pin one. The runner setup scripts install the floating stable channel, and run_openvmm_tests sets RUSTUP_TOOLCHAIN=stable.
*-scus-*runners:azure-mshv-scus-1,azure-mshv-scus-z1-1,azure-windows-scus-0, andazure-windows-scus-1all reportrustc 1.99.0 (b940084d7 2026-09-28). They took their first CI job at 2026-10-06T01:15Z.- Older runners: the
azure-kvm-*,azure-mshv-*, andazure-windows-*runners checked here reportrustc 1.98.1 (48a229cea 2026-09-01).
Both groups match the vmm-tests jobs' runs-on labels, so whether a run passes depends on where its vmm-tests jobs land.
Incidents
| Run (attempt) | Pull request | Failed job | Runner | OpenVMM |
|---|---|---|---|---|
| 37398037527 (1) | #318 | OpenVMM vmm-tests / Linux / MSHV |
azure-mshv-scus-1 |
b7696e7b |
| 37413122102 (1) | #397 | OpenVMM vmm-tests / Linux / MSHV |
azure-mshv-scus-1 |
b7696e7b |
| 37413122102 (1) | #397 | OpenVMM vmm-tests / Windows / WHP |
azure-windows-scus-1 |
b7696e7b |
Through run 37413727664, these were the only vmm-tests jobs that ran on a *-scus-* runner. All three ran Rust 1.99.0 and failed with the same error. On 1.98.1 runners the same jobs built guest_test_uefi and passed, including at the same OpenVMM revision:
| Run | Head | OpenVMM | vmm-tests runners, all Rust 1.98.1 | Result |
|---|---|---|---|---|
| 37362885159 | dev at bce2e919 |
b7696e7b |
azure-kvm-5, azure-mshv-7, azure-windows-5 |
Passed |
| 37409832176 | #394 at 33aa16ae |
4cb5777f |
azure-kvm-4, azure-mshv-1, azure-windows-5 |
Passed |
| 37413727664 | dev at 02ba24ba |
4cb5777f |
azure-kvm-1, azure-mshv-5, azure-windows-4 |
Passed |
Latest incident
-
Run: workflow
CI, run 37413122102, attempt 1. -
Event:
pull_requestfor collect-alpine-sources: preserve recipe line endings #397. The pull request is Ready, and the run's head,c50900aa14ae0f37cc9fcef311a1deb59abca8da, is still current. The tested merge ref merges that head intodevat794f29b00ad2eebca5d2f7c8f1c578ac1b92e350. -
OpenVMM source:
b7696e7bac7344b9f415c64fbd6372d3959076c1. -
Failed steps:
Run OpenVMM tests on MSHVinOpenVMM vmm-tests / Linux / MSHV.Run OpenVMM tests on WindowsinOpenVMM vmm-tests / Windows / WHP.
Both failed in flowey's
cargo build guest_test_uefistep before any test ran, so neither job uploaded test results. -
Runners: both reported
rustc 1.99.0 (b940084d7 2026-09-28)with LLVM 23.1.1.azure-mshv-scus-1: machineesaurez-nvx-ci-backend-scus-rg-mshv-vmss000000, job labelsself-hosted,linux,mshv,virtual-machine.azure-windows-scus-1: machinevmss6grqr000001, job labelsself-hosted,windows,whp,virtual-machine.
-
Impact:
Required status checkfailed only because of these two jobs. Every other job that ran passed, includingOpenVMM vmm-tests / Linux / KVMonazure-kvm-3. -
Classification: a failure on the current head, caused by the runners' toolchain. collect-alpine-sources: preserve recipe line endings #397 changes only
scripts/nvx_tools/collect_alpine_sources.pyand its test.
$ rustup run stable-x86_64-unknown-linux-gnu cargo build --message-format=json-render-diagnostics -p guest_test_uefi --target x86_64-unknown-uefi --profile release --bin guest_test_uefi --target-dir /opt/nvx-runner/_work/nvx/nvx/openvmm/target/guest_test_uefi
error: linking with `rust-lld` failed: exit status: 1
= note: rust-lld: error: undefined symbol: wcslen
>>> referenced by /opt/nvx-runner/_work/nvx/nvx/openvmm/guest_test_uefi/src/uefi/mod.rs:19
>>> referenced by /opt/nvx-runner/_work/_temp/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/uefi-0.35.0/src/system.rs:28
>>> ...:(uefi::system::firmware_vendor)
error: could not compile `guest_test_uefi` (bin "guest_test_uefi") due to 1 previous error
guest_test_uefi/src/uefi/mod.rs:19 prints system::firmware_vendor(), which reads the firmware's null-terminated UCS-2 vendor string through uefi 0.35.0. The Windows job reports the same two references.
Where NVX chooses the toolchain
- Runner setup:
scripts/setup/setup-linux-runner.shandscripts/setup/setup-linux-mshv.shsetRUST_TOOLCHAIN=stable, andscripts/setup/setup-windows-whp.ps1sets$RustToolchain = "stable". They require only Rust 1.95.0 or newer, so each runner gets whateverstablewas when it was provisioned. - Test environment:
OpenVMMBuildConstants.RUST_TOOLCHAINinscripts/nvx_tools/build_constants.pyis"stable"._prepare_openvmm_test_environmentinscripts/nvx_tools/ci.pyexports it asRUSTUP_TOOLCHAINforcargo xflowey vmm-tests-run. - OpenVMM's own pin: OpenVMM pins its pipelines to Rust 1.95.0 in
flowey/flowey_lib_hvlite/src/cfg_rustup_version.rs, both atb7696e7band upstream, and declaresrust-version = "1.95". NVX runs flowey with the runner's toolchain instead, so NVX CI is the first place OpenVMM meets a new compiler.
Next steps
- Pin the Rust release that NVX CI uses for OpenVMM to a known-good version, such as 1.98.1, in the runner setup scripts and in
OpenVMMBuildConstants.RUST_TOOLCHAIN, then update the*-scus-*runners. Before it installs anything,_prepare_openvmm_test_environmentrunsrustup target list --installed --toolchain <name>, which fails for a toolchain that isn't installed. Either install the pinned toolchain first, or make the query handle a missing toolchain. - Make the
Validate Linux toolchainandValidate Windows toolchainsteps require the pinnedrustcversion. Toolchain drift then fails with a clear error rather than a link error inside flowey. - Follow Regression in Rust 1.99: missing symbol
wcslenrust-lang/rust#163614. Before moving the pin to a fixed release, confirm thatguest_test_uefilinks on a*-scus-*runner. - Optionally, as a stopgap until the pin lands, apply the workaround suggested upstream for
no_stdbuilds such asguest_test_uefi:-C llvm-args=-disable-loop-idiom-wcslen. It would have to be scoped to thex86_64-unknown-uefibuild and to toolchains whose LLVM accepts the option, so pinning is simpler.
Until this is fixed, a rerun passes only if its vmm-tests jobs land on 1.98.1 runners.
Related: #318, whose run 37398037527 hit this first, and #397, whose run 37413122102 hit it after the pull request was marked Ready.
Note
Runner naming update (2026-10-07): Names in linked historical jobs and dated tables remain exactly as GitHub reported them. Current sequential aliases and runner registrations are mapped by VM instance in #211's rename note. Use the current name when connecting to or scheduling a runner.
Activity
ppenna commentedon Oct 6, 2026
Local reproduction
I reproduced the failure on Windows with the
x86_64-pc-windows-msvchost, the same host triple as the WHP job. I built OpenVMMb7696e7bac7344b9f415c64fbd6372d3959076c1with flowey's build command, minus--message-format:1.99.0(b940084d7 2026-09-28, LLVM 23.1.1)undefined symbol: wcslen1.98.1(48a229cea 2026-09-01, LLVM 22.1.8)guest_test_uefi.efi1.99.0-C llvm-args=-disable-loop-idiom-wcslenguest_test_uefi.efi1.98.1-C llvm-args=-disable-loop-idiom-wcslenguest_test_uefi.efiRust 1.99.0 fails with the same references as CI:
I passed the workaround as
CARGO_TARGET_X86_64_UNKNOWN_UEFI_RUSTFLAGS, which Cargo joins with OpenVMM'starget.'cfg(all())'.rustflags. The verboserustcinvocation forguest_test_uefishowed-Cforce-frame-pointers=yes,-Csymbol-mangling-version=v0, and the workaround flag, so the flags that OpenVMM sets were kept.LLVM 22.1.8 accepts the option too. The workaround therefore only needs to be scoped to the
x86_64-unknown-uefibuild; it doesn't need to depend on whether the runner has 1.98.1 or 1.99.0.ppenna commentedon Oct 7, 2026
Update: 10 more failures since the description, all on Rust 1.99.0
*-scus-*runnersSince the incident table in the description,
OpenVMM vmm-testshas failed 10 more times. Each failure ran on a*-scus-*runner that reportsrustc 1.99.0 (b940084d7 2026-09-28), and each stopped in flowey'scargo build guest_test_uefiwithrust-lld: error: undefined symbol: wcslenbefore any test ran. They span eight pull requests and seven OpenVMM revisions.azure-windows-scus-0b7696e7bazure-windows-2azure-mshv-scus-z1-1e3708600azure-mshv-scus-107850d27azure-mshv-1azure-mshv-scus-1aa9f6c4fazure-mshv-scus-1aa9f6c4fazure-mshv-1azure-mshv-scus-z1-16a3855e6azure-mshv-scus-z1-13951c841azure-mshv-5azure-mshv-scus-1b7696e7bazure-mshv-scus-13e1f5bd4azure-mshv-scus-1againazure-mshv-scus-13e1f5bd4These figures cover vmm-tests jobs started from 2026-10-06 00:00Z to 2026-10-07 16:13Z, including the three incidents in the description:
*-scus-*runner failed this way: 11 MSHV jobs onazure-mshv-scus-1andazure-mshv-scus-z1-1, and 2 WHP jobs onazure-windows-scus-0andazure-windows-scus-1. All 13 logs report Rust 1.99.0.azure-kvm-1,-3,-4,-5, and-6;azure-mshv-1,-5,-6, and-7;azure-windows-1to-5) reportsrustc 1.98.1 (48a229cea 2026-09-01).*-scus-*runner. No WHP vmm-tests job has run on anazure-windows-scus-*runner since 2026-10-06 06:56Z.Latest incident
CI, run 37642981509, attempt 1.pull_requestfor openvmm: pin the WHP fix for nested UEFI timer interrupts #384. The pull request is Ready, and the run's head,ba5bc7ffec17cea2eedbf65193ff86ac7c76b6f1, is still current. The tested merge ref,0e313fd, merges that head intodevatbe859aa.virt_whpcommit.3e1f5bd48392eaeb90962b4ae191a908c859a0ab.Run OpenVMM tests on MSHVinOpenVMM vmm-tests / Linux / MSHV.azure-mshv-scus-1, machineesaurez-nvx-ci-backend-scus-rg-mshv-vmss000000. The link error has the same two references as in the description:guest_test_uefi/src/uefi/mod.rs:19anduefi-0.35.0/src/system.rs:28.OpenVMM vmm-tests / Linux / KVMonazure-kvm-3(58 tests passed) andOpenVMM vmm-tests / Windows / WHPonazure-windows-2(29 passed) both ran Rust 1.98.1. Both builtguest_test_uefifrom the same OpenVMM revision.azure-mshv-scus-1again, and the job failed the same way.Required status check. The WHP platform job also failed in attempt 1, but that was a ci: investigate recurring WHP mapped-memory-flush stalls #169 incident, and it passed in attempt 2.Status
devbe859aa:OpenVMMBuildConstants.RUST_TOOLCHAINis still"stable".stablechannel, with a 1.95.0 floor.validate-runnerprintsrustc --versionbut doesn't check it.1.99.0 (b940084d7 2026-09-28).wcslenrust-lang/rust#163614 is still open atP-high.*-scus-*runners to the currentstabletherefore wouldn't help. The pin, or the scoped-C llvm-args=-disable-loop-idiom-wcslenworkaround from the reproduction comment, is still needed.azure-mshv-scus-*runners.ppenna commentedon Oct 7, 2026
Update: same-commit comparison on #362, and
devnow fails#362: one commit, two outcomes
Attempts 2 and 3 of run 37367403124 tested the same merge commit,
932d7ca(#362's heade15b613merged intodevatbce2e919), with OpenVMMb7696e7b. Only the MSHV job's runner changed:azure-mshv-6azure-mshv-scus-1undefined symbol: wcslenincargo build guest_test_uefiIn attempt 3, vmm-tests on
azure-kvm-6andazure-windows-3ran Rust 1.98.1 and passed. #362 changes onlyscripts/nvx_tools/azurelinux.py,common.py, andubuntu.py. The MSHV job is the only failure behind attempt 3'sRequired status check, so the same commit went from green to red because of the runner's toolchain alone.devnow fails toopushtodevat8e1b611with OpenVMM762bc1c7, the eighth affected revision.OpenVMM vmm-tests / Linux / MSHVran onazure-mshv-scus-z1-1with Rust 1.99.0 (LLVM 23.1.1) and failed the same way. It was the run's only failed job. vmm-tests passed onazure-kvm-4andazure-windows-5.devpush among the incidents here; every earlier one was a pull-request run.Persist performance baselineandPublish development releaserequireopenvmm-vmm-teststo succeed or be skipped, so this push persisted no performance history and published no development release.How flowey picks the compiler
vmm_tests_run.rsconfigures flowey withignore_rust_version: true(b7696e7b, unchanged at762bc1c7).install_rusttherefore ignores OpenVMM's 1.95.0 pin. Itsdetect active toolchainstep takes the first word ofrustup show active-toolchain(install_rust.rs), andrun_cargo_buildruns each build asrustup run <toolchain> cargo build ....rustup show active-toolchaingivesRUSTUP_TOOLCHAINprecedence over arust-toolchain.tomlfile._prepare_openvmm_test_environmentexportsRUSTUP_TOOLCHAIN=stable, which is why the failed log showsrustup run stable-x86_64-unknown-linux-gnu. A toolchain file alone therefore wouldn't change the compiler that vmm-tests uses.Pinning pitfalls in the current code
stablelacks one of the vmm-tests targets andRUNNER_TEMPis set,_prepare_openvmm_test_environmentinstallsstableinto$RUNNER_TEMP/openvmm-rustup. Today that resolves to 1.99.0, so this path would fail the same way even on a 1.98.1 runner. Run 37367403124's failed job didn't take it: its log shows no toolchain or target installation. Pinning only the setup scripts wouldn't cover this path; changingOpenVMMBuildConstants.RUST_TOOLCHAIN, as the first next step proposes, would.stable. A runner provisioned by them has no1.98.1-<host>toolchain, even when itsstableis 1.98.1. A1.98.1pin therefore hits the missing-toolchain case from the first next step on every self-hosted runner until they're reprovisioned, not only on the*-scus-*ones.jay23606 commentedon Oct 16, 2026
Fixed by #429, which pins Rust 1.95.0 in rust-toolchain.toml, OpenVMMBuildConstants.RUST_TOOLCHAIN, and the setup tool manifest. Since it merged, OpenVMM vmm-tests pass on MSHV, KVM and WHP in dev runs 38926212043 and 38942383387.
jay23606 commentedon Oct 16, 2026
Fixed by #429, which pins Rust 1.95.0 in rust-toolchain.toml, OpenVMMBuildConstants.RUST_TOOLCHAIN, and the setup tool manifest. Since it merged, OpenVMM vmm-tests pass on MSHV, KVM and WHP in dev runs 38926212043 and 38942383387.
ppenna commentedon Oct 28, 2026
Closing: #429 fixed this, as the comments above say, and the remaining steps are done or no longer apply.
rust-toolchain.tomlpins Rust 1.95.0, and the runner setup installs it.azure-mshv-scus-1, reimaged on 2026-10-28, installed 1.95.0.validate-runnerfails a job whose runner lacks the pinned release or runs another one.OpenVMM vmm-testsjobs passed: 786 on MSHV, 741 on KVM, and 940 on WHP. None of the 19 that failed reportedwcslen.wcslenrust-lang/rust#163614 closed on 2026-10-14. Before the pin moves to 1.99 or later, confirm thatguest_test_uefilinks, as the third step above says.*-scus-*runners that reported 1.99.0, the MSHV ones no longer receive jobs (ci: measure each performance series on one benchmark runner #524, ci: dedicate the benchmark runners to the platform jobs #649), andvalidate-runnerholds every runner that does to the pinned release.