Skip to content

ci: OpenVMM vmm-tests fail on runners with Rust 1.99.0 because guest_test_uefi cannot link wcslen #402

Closed

Description

Summary

OpenVMM vmm-tests fails on every CI runner whose stable Rust is 1.99.0. Before any test runs, flowey builds OpenVMM's guest_test_uefi for x86_64-unknown-uefi, and the link fails with rust-lld: error: undefined symbol: wcslen. Runners with Rust 1.98.1 build the same OpenVMM revision and pass.

This is the upstream regression rust-lang/rust#163614 (open, P-high, regression-from-stable-to-stable). Rust 1.99.0 ships LLVM 23.1.1, which can replace a loop that measures a null-terminated UTF-16 string with a call to wcslen. rust-lang/compiler-builtins#1297 added a wcslen builtin for the earlier report, rust-lang/rust#160827. The 1.99 beta backport, rust-lang/rust#162987, did not include it, so the 1.99.0 UEFI link has no wcslen.

NVX picks up the new compiler because it doesn't pin one. The runner setup scripts install the floating stable channel, and run_openvmm_tests sets RUSTUP_TOOLCHAIN=stable.

  • *-scus-* runners: azure-mshv-scus-1, azure-mshv-scus-z1-1, azure-windows-scus-0, and azure-windows-scus-1 all report rustc 1.99.0 (b940084d7 2026-09-28). They took their first CI job at 2026-10-06T01:15Z.
  • Older runners: the azure-kvm-*, azure-mshv-*, and azure-windows-* runners checked here report rustc 1.98.1 (48a229cea 2026-09-01).

Both groups match the vmm-tests jobs' runs-on labels, so whether a run passes depends on where its vmm-tests jobs land.

Incidents

Run (attempt) Pull request Failed job Runner OpenVMM
37398037527 (1) #318 OpenVMM vmm-tests / Linux / MSHV azure-mshv-scus-1 b7696e7b
37413122102 (1) #397 OpenVMM vmm-tests / Linux / MSHV azure-mshv-scus-1 b7696e7b
37413122102 (1) #397 OpenVMM vmm-tests / Windows / WHP azure-windows-scus-1 b7696e7b

Through run 37413727664, these were the only vmm-tests jobs that ran on a *-scus-* runner. All three ran Rust 1.99.0 and failed with the same error. On 1.98.1 runners the same jobs built guest_test_uefi and passed, including at the same OpenVMM revision:

Run Head OpenVMM vmm-tests runners, all Rust 1.98.1 Result
37362885159 dev at bce2e919 b7696e7b azure-kvm-5, azure-mshv-7, azure-windows-5 Passed
37409832176 #394 at 33aa16ae 4cb5777f azure-kvm-4, azure-mshv-1, azure-windows-5 Passed
37413727664 dev at 02ba24ba 4cb5777f azure-kvm-1, azure-mshv-5, azure-windows-4 Passed

Latest incident

  • Run: workflow CI, run 37413122102, attempt 1.

  • Event: pull_request for collect-alpine-sources: preserve recipe line endings #397. The pull request is Ready, and the run's head, c50900aa14ae0f37cc9fcef311a1deb59abca8da, is still current. The tested merge ref merges that head into dev at 794f29b00ad2eebca5d2f7c8f1c578ac1b92e350.

  • OpenVMM source: b7696e7bac7344b9f415c64fbd6372d3959076c1.

  • Failed steps:

    • Run OpenVMM tests on MSHV in OpenVMM vmm-tests / Linux / MSHV.
    • Run OpenVMM tests on Windows in OpenVMM vmm-tests / Windows / WHP.

    Both failed in flowey's cargo build guest_test_uefi step before any test ran, so neither job uploaded test results.

  • Runners: both reported rustc 1.99.0 (b940084d7 2026-09-28) with LLVM 23.1.1.

    • azure-mshv-scus-1: machine esaurez-nvx-ci-backend-scus-rg-mshv-vmss000000, job labels self-hosted, linux, mshv, virtual-machine.
    • azure-windows-scus-1: machine vmss6grqr000001, job labels self-hosted, windows, whp, virtual-machine.
  • Impact: Required status check failed only because of these two jobs. Every other job that ran passed, including OpenVMM vmm-tests / Linux / KVM on azure-kvm-3.

  • Classification: a failure on the current head, caused by the runners' toolchain. collect-alpine-sources: preserve recipe line endings #397 changes only scripts/nvx_tools/collect_alpine_sources.py and its test.

$ rustup run stable-x86_64-unknown-linux-gnu cargo build --message-format=json-render-diagnostics -p guest_test_uefi --target x86_64-unknown-uefi --profile release --bin guest_test_uefi --target-dir /opt/nvx-runner/_work/nvx/nvx/openvmm/target/guest_test_uefi
error: linking with `rust-lld` failed: exit status: 1
  = note: rust-lld: error: undefined symbol: wcslen
          >>> referenced by /opt/nvx-runner/_work/nvx/nvx/openvmm/guest_test_uefi/src/uefi/mod.rs:19
          >>> referenced by /opt/nvx-runner/_work/_temp/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/uefi-0.35.0/src/system.rs:28
          >>>               ...:(uefi::system::firmware_vendor)
error: could not compile `guest_test_uefi` (bin "guest_test_uefi") due to 1 previous error

guest_test_uefi/src/uefi/mod.rs:19 prints system::firmware_vendor(), which reads the firmware's null-terminated UCS-2 vendor string through uefi 0.35.0. The Windows job reports the same two references.

Where NVX chooses the toolchain

  • Runner setup: scripts/setup/setup-linux-runner.sh and scripts/setup/setup-linux-mshv.sh set RUST_TOOLCHAIN=stable, and scripts/setup/setup-windows-whp.ps1 sets $RustToolchain = "stable". They require only Rust 1.95.0 or newer, so each runner gets whatever stable was when it was provisioned.
  • Test environment: OpenVMMBuildConstants.RUST_TOOLCHAIN in scripts/nvx_tools/build_constants.py is "stable". _prepare_openvmm_test_environment in scripts/nvx_tools/ci.py exports it as RUSTUP_TOOLCHAIN for cargo xflowey vmm-tests-run.
  • OpenVMM's own pin: OpenVMM pins its pipelines to Rust 1.95.0 in flowey/flowey_lib_hvlite/src/cfg_rustup_version.rs, both at b7696e7b and upstream, and declares rust-version = "1.95". NVX runs flowey with the runner's toolchain instead, so NVX CI is the first place OpenVMM meets a new compiler.

Next steps

  • Pin the Rust release that NVX CI uses for OpenVMM to a known-good version, such as 1.98.1, in the runner setup scripts and in OpenVMMBuildConstants.RUST_TOOLCHAIN, then update the *-scus-* runners. Before it installs anything, _prepare_openvmm_test_environment runs rustup target list --installed --toolchain <name>, which fails for a toolchain that isn't installed. Either install the pinned toolchain first, or make the query handle a missing toolchain.
  • Make the Validate Linux toolchain and Validate Windows toolchain steps require the pinned rustc version. Toolchain drift then fails with a clear error rather than a link error inside flowey.
  • Follow Regression in Rust 1.99: missing symbol wcslen rust-lang/rust#163614. Before moving the pin to a fixed release, confirm that guest_test_uefi links on a *-scus-* runner.
  • Optionally, as a stopgap until the pin lands, apply the workaround suggested upstream for no_std builds such as guest_test_uefi: -C llvm-args=-disable-loop-idiom-wcslen. It would have to be scoped to the x86_64-unknown-uefi build and to toolchains whose LLVM accepts the option, so pinning is simpler.

Until this is fixed, a rerun passes only if its vmm-tests jobs land on 1.98.1 runners.

Related: #318, whose run 37398037527 hit this first, and #397, whose run 37413122102 hit it after the pull request was marked Ready.

Note

Runner naming update (2026-10-07): Names in linked historical jobs and dated tables remain exactly as GitHub reported them. Current sequential aliases and runner registrations are mapped by VM instance in #211's rename note. Use the current name when connecting to or scheduling a runner.

Activity

  1. ppenna commented on Oct 6, 2026

    @ppenna
    ContributorAuthor

    Local reproduction

    I reproduced the failure on Windows with the x86_64-pc-windows-msvc host, the same host triple as the WHP job. I built OpenVMM b7696e7bac7344b9f415c64fbd6372d3959076c1 with flowey's build command, minus --message-format:

    rustup run <toolchain> cargo build -p guest_test_uefi --target x86_64-unknown-uefi --profile release --bin guest_test_uefi --target-dir <dir>
    
    Toolchain Extra flags Result
    1.99.0 (b940084d7 2026-09-28, LLVM 23.1.1) none Fails: undefined symbol: wcslen
    1.98.1 (48a229cea 2026-09-01, LLVM 22.1.8) none Builds guest_test_uefi.efi
    1.99.0 -C llvm-args=-disable-loop-idiom-wcslen Builds guest_test_uefi.efi
    1.98.1 -C llvm-args=-disable-loop-idiom-wcslen Builds guest_test_uefi.efi

    Rust 1.99.0 fails with the same references as CI:

    error: linking with `rust-lld` failed: exit code: 1
      = note: rust-lld: error: undefined symbol: wcslen
              >>> referenced by ...\guest_test_uefi\src\uefi\mod.rs:19
              >>> referenced by ...\uefi-0.35.0\src\system.rs:28
    error: could not compile `guest_test_uefi` (bin "guest_test_uefi") due to 1 previous error
    

    I passed the workaround as CARGO_TARGET_X86_64_UNKNOWN_UEFI_RUSTFLAGS, which Cargo joins with OpenVMM's target.'cfg(all())'.rustflags. The verbose rustc invocation for guest_test_uefi showed -Cforce-frame-pointers=yes, -Csymbol-mangling-version=v0, and the workaround flag, so the flags that OpenVMM sets were kept.

    LLVM 22.1.8 accepts the option too. The workaround therefore only needs to be scoped to the x86_64-unknown-uefi build; it doesn't need to depend on whether the runner has 1.98.1 or 1.99.0.

  2. ppenna commented on Oct 7, 2026

    @ppenna
    ContributorAuthor

    Update: 10 more failures since the description, all on Rust 1.99.0 *-scus-* runners

    Since the incident table in the description, OpenVMM vmm-tests has failed 10 more times. Each failure ran on a *-scus-* runner that reports rustc 1.99.0 (b940084d7 2026-09-28), and each stopped in flowey's cargo build guest_test_uefi with rust-lld: error: undefined symbol: wcslen before any test ran. They span eight pull requests and seven OpenVMM revisions.

    Run (attempt) Pull request Failed job Runner OpenVMM Next attempt
    37413122102 (2) #397 Windows / WHP azure-windows-scus-0 b7696e7b Attempt 3 passed on azure-windows-2
    37418428001 (1) #401 Linux / MSHV azure-mshv-scus-z1-1 e3708600 Superseded by a newer push
    37448382849 (1) #404 Linux / MSHV azure-mshv-scus-1 07850d27 Attempt 2 passed on azure-mshv-1
    37518293095 (1) #411 Linux / MSHV azure-mshv-scus-1 aa9f6c4f Not rerun in attempts 2 to 4
    37518293095 (5) #411 Linux / MSHV azure-mshv-scus-1 aa9f6c4f Attempt 6 passed on azure-mshv-1
    37520457443 (1) #413 Linux / MSHV azure-mshv-scus-z1-1 6a3855e6 Superseded by a newer push
    37529971686 (1) #412 Linux / MSHV azure-mshv-scus-z1-1 3951c841 Attempt 2 passed on azure-mshv-5
    37367403124 (3) #362 Linux / MSHV azure-mshv-scus-1 b7696e7b Not rerun
    37642981509 (1) #384 Linux / MSHV azure-mshv-scus-1 3e1f5bd4 Attempt 2 landed on azure-mshv-scus-1 again
    37642981509 (2) #384 Linux / MSHV azure-mshv-scus-1 3e1f5bd4 Not rerun

    These figures cover vmm-tests jobs started from 2026-10-06 00:00Z to 2026-10-07 16:13Z, including the three incidents in the description:

    • All 13 vmm-tests jobs that ran on a *-scus-* runner failed this way: 11 MSHV jobs on azure-mshv-scus-1 and azure-mshv-scus-z1-1, and 2 WHP jobs on azure-windows-scus-0 and azure-windows-scus-1. All 13 logs report Rust 1.99.0.
    • All 83 vmm-tests jobs that completed on the other 14 runners passed, and 4 more were cancelled by newer pushes. One log from each of those runners (azure-kvm-1, -3, -4, -5, and -6; azure-mshv-1, -5, -6, and -7; azure-windows-1 to -5) reports rustc 1.98.1 (48a229cea 2026-09-01).
    • MSHV is the most exposed lane: 11 of its 37 vmm-tests jobs landed on a *-scus-* runner. No WHP vmm-tests job has run on an azure-windows-scus-* runner since 2026-10-06 06:56Z.

    Latest incident

    Status

  3. ppenna commented on Oct 7, 2026

    @ppenna
    ContributorAuthor

    Update: same-commit comparison on #362, and dev now fails

    #362: one commit, two outcomes

    Attempts 2 and 3 of run 37367403124 tested the same merge commit, 932d7ca (#362's head e15b613 merged into dev at bce2e919), with OpenVMM b7696e7b. Only the MSHV job's runner changed:

    Attempt MSHV vmm-tests job Runner Rust (LLVM) Result
    2 112011873827 azure-mshv-6 1.98.1 (22.1.8) Passed
    3 112884150509 azure-mshv-scus-1 1.99.0 (23.1.1) undefined symbol: wcslen in cargo build guest_test_uefi

    In attempt 3, vmm-tests on azure-kvm-6 and azure-windows-3 ran Rust 1.98.1 and passed. #362 changes only scripts/nvx_tools/azurelinux.py, common.py, and ubuntu.py. The MSHV job is the only failure behind attempt 3's Required status check, so the same commit went from green to red because of the runner's toolchain alone.

    dev now fails too

    • Run: run 37652888580, attempt 1, push to dev at 8e1b611 with OpenVMM 762bc1c7, the eighth affected revision.
    • Failure: OpenVMM vmm-tests / Linux / MSHV ran on azure-mshv-scus-z1-1 with Rust 1.99.0 (LLVM 23.1.1) and failed the same way. It was the run's only failed job. vmm-tests passed on azure-kvm-4 and azure-windows-5.
    • Impact: this is the first dev push among the incidents here; every earlier one was a pull-request run. Persist performance baseline and Publish development release require openvmm-vmm-tests to succeed or be skipped, so this push persisted no performance history and published no development release.

    How flowey picks the compiler

    • OpenVMM's vmm_tests_run.rs configures flowey with ignore_rust_version: true (b7696e7b, unchanged at 762bc1c7). install_rust therefore ignores OpenVMM's 1.95.0 pin. Its detect active toolchain step takes the first word of rustup show active-toolchain (install_rust.rs), and run_cargo_build runs each build as rustup run <toolchain> cargo build ....
    • rustup show active-toolchain gives RUSTUP_TOOLCHAIN precedence over a rust-toolchain.toml file. _prepare_openvmm_test_environment exports RUSTUP_TOOLCHAIN=stable, which is why the failed log shows rustup run stable-x86_64-unknown-linux-gnu. A toolchain file alone therefore wouldn't change the compiler that vmm-tests uses.

    Pinning pitfalls in the current code

    • Job-local fallback: when the runner's stable lacks one of the vmm-tests targets and RUNNER_TEMP is set, _prepare_openvmm_test_environment installs stable into $RUNNER_TEMP/openvmm-rustup. Today that resolves to 1.99.0, so this path would fail the same way even on a 1.98.1 runner. Run 37367403124's failed job didn't take it: its log shows no toolchain or target installation. Pinning only the setup scripts wouldn't cover this path; changing OpenVMMBuildConstants.RUST_TOOLCHAIN, as the first next step proposes, would.
    • Toolchain name: the three setup scripts install only the toolchain named stable. A runner provisioned by them has no 1.98.1-<host> toolchain, even when its stable is 1.98.1. A 1.98.1 pin therefore hits the missing-toolchain case from the first next step on every self-hosted runner until they're reprovisioned, not only on the *-scus-* ones.
  4. jay23606 commented on Oct 16, 2026

    @jay23606
    Contributor

    Fixed by #429, which pins Rust 1.95.0 in rust-toolchain.toml, OpenVMMBuildConstants.RUST_TOOLCHAIN, and the setup tool manifest. Since it merged, OpenVMM vmm-tests pass on MSHV, KVM and WHP in dev runs 38926212043 and 38942383387.

  5. jay23606 commented on Oct 16, 2026

    @jay23606
    Contributor

    Fixed by #429, which pins Rust 1.95.0 in rust-toolchain.toml, OpenVMMBuildConstants.RUST_TOOLCHAIN, and the setup tool manifest. Since it merged, OpenVMM vmm-tests pass on MSHV, KVM and WHP in dev runs 38926212043 and 38942383387.

  6. ppenna commented on Oct 28, 2026

    @ppenna
    ContributorAuthor

    Closing: #429 fixed this, as the comments above say, and the remaining steps are done or no longer apply.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    confirmedIssue affects multiple people.

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions