Skip to content

ci: route hosted jobs to isolated Azure runners - #318

Closed
Enrique Saurez (esaurez) wants to merge 14 commits into
devfrom
copilot/ci-azure-utility-20261002
Closed

Enrique Saurez (esaurez) wants to merge 14 commits into
devfrom
copilot/ci-azure-utility-20261002

Conversation

@esaurez

@esaurez Enrique Saurez (esaurez) commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Move the remaining GitHub-hosted jobs in dev CI onto four dedicated Azure utility runners: Linux PR/fork, Linux trusted push, Windows PR/fork, and Windows trusted push. Keep the existing backend benchmark scale sets unchanged.
  • Route the new Linux/Windows crate checks to their matching PR or trusted runner and build the debug kernel on the Linux PR runner with rootless Docker, after the shared artifacts build succeeds. The single Linux PR runner serializes these builds; normal artifact consumers need not wait for the debug-only lane. Install additional Rust toolchains in job-private storage, and require the Windows PR runner to block instance metadata before running unreviewed code.
  • Preserve the existing backend runner labels. Add an opt-in validated Windows utility label override, while keeping the default Windows/WHP configuration unchanged.

Infrastructure

Two Linux and two Windows Regular Standard_D8s_v5 utility scale sets have been provisioned in subscription 7d07dbdd-8117-46e9-8f58-721da3f32fd4 using AzTemplates branch copilot/nvx-ci-utility-20261002. Each has one instance; the Windows VMSS use Windows Server 2025, Trusted Launch, 128-GiB OS and 512-GiB managed data disks, both approved SSH keys, and the reference Windows VMSS's exact data-disk initializer. Their PR and trusted administrator passwords are separate secrets in a dedicated vault. Each Windows managed identity can read only the existing runner-bootstrap secret. The existing Linux utility scale sets and Windows/WHP benchmark scale set were not reimaged or resized.

Bootstrap pins the setup script by immutable commit and checksum, survives prerequisite reboots, and registers role-specific labels. Both Windows utility runners are registered with separate PR/fork and trusted-push labels. The Windows PR crate check passed, and the trusted runner passed its bootstrap check-only validation.

CI stabilization

The latest dev already includes the stronger adversarial cleanup synchronization fixes; this rebase retains the still-needed typing guard without duplicating those fixes. No VMM runtime behavior changed.

The full PR workflow at 29ed346 passed on run 37409021969, attempt 2. The first attempt exceeded three MSHV latency thresholds on azure-mshv-scus-1; fresh measurements on azure-mshv-1 passed without benchmark, threshold, or baseline changes. Shared artifacts finished before the debug build started on the same PR utility runner.

Trust boundary

Fork PRs temporarily use persistent PR-only runners during the hosted-runner outage. Rootless Docker and runner metadata isolation do not provide ephemeral-runner isolation. In particular, a fork PR could leave state on the Linux PR runner that affects a later dev artifact build and release. This is the previously deferred review decision; do not merge until it is resolved. Revert fork jobs to hosted capacity when it recovers.

Copilot AI balanced review requested due to automatic review settings October 2, 2026 22:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 1 High severity · 3 Medium severity

Open (4)
What changed in this PR

Routes previously GitHub-hosted CI jobs onto dedicated Azure self-hosted Linux runners, separating PR/fork workloads from push-only jobs that use write credentials, and enforcing rootless Docker/per-job client config for Docker-based jobs.

Changes:

  • Update ci.yml job routing to nvx-utility-pr vs nvx-utility-trusted runners and add a rootless Docker enforcement step for relevant jobs.
  • Extend Linux runner setup to allow validated label overrides via --runner-labels.
  • Add/adjust composite actions and tests to validate runner label policy, routing, and rootless Docker usage.
File Description
scripts/​test_nvx_tools.py Adds assertions for runner label override behavior, workflow routing, and rootless Docker configuration.
scripts/​setup/​setup-linux-runner.sh Introduces --runner-labels override with validation and expected-label behavior changes.
scripts/​setup/​README.md Documents new runner label override and separates Docker-capable utility runner guidance.
.github/​workflows/​ci.yml Routes jobs to self-hosted Azure runners and adds rootless Docker requirement steps.
.github/​actions/​require-rootless-docker/​action.yml New composite action to force rootless Docker daemon + per-job DOCKER_CONFIG.
.github/​actions/​check-quality/​action.yml Adds self-hosted Python setup path using a per-job venv.
.github/​actionlint.yaml Adds new allowed self-hosted runner labels for actionlint.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread scripts/test_nvx_tools.py Resolved
Comment thread scripts/test_nvx_tools.py Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread scripts/test_nvx_tools.py Resolved
Copilot AI balanced review requested due to automatic review settings October 2, 2026 23:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Comment thread .github/workflows/ci.yml Resolved
Comment thread scripts/setup/README.md Resolved
Copilot AI balanced review requested due to automatic review settings October 3, 2026 00:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot AI balanced review requested due to automatic review settings October 3, 2026 00:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread .github/workflows/ci.yml
)
}}
runs-on: ubuntu-latest
runs-on: [self-hosted, linux, x64, nvx-utility-pr]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed: reusing a persistent runner for fork PR and dev push artifact builds can carry untrusted state into release inputs. This thread remains open while we decide whether to route push builds to the fork-free trusted VM or use an ephemeral/dedicated builder. No merge until this boundary is resolved.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot AI balanced review requested due to automatic review settings October 6, 2026 00:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot AI balanced review requested due to automatic review settings October 6, 2026 01:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use distinct labels for PR and credentialed jobs, and require rootless Docker for artifact builds.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Enrique Saurez (esaurez) and others added 12 commits October 5, 2026 19:40
Point artifact and quality jobs at the per-user systemd Docker daemon after moving it off a system-level service.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep Buildx configuration in the runner job temp directory because the service account has no writable home.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep hosted Python 3.10 selection, but use a fresh Python 3.12 virtualenv and per-job Docker state on the self-hosted utility runner.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Apply the repository-pinned Ruff formatter to the CI runner test before rerunning quality.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Scope Python to the quality action, clean verified per-job Docker state, honor the rootless runtime directory, and validate venv support and runner labels.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wait briefly for asynchronous process reaping and canary acceptance while preserving strict failure assertions; narrow the runner label test type guard.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Validate the PowerShell parser prerequisite before quality checks and clarify that utility jobs only have rootless Docker access.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@esaurez
Enrique Saurez (esaurez) force-pushed the copilot/ci-azure-utility-20261002 branch from 07a9737 to 12a36e5 Compare October 6, 2026 02:45
Copilot AI balanced review requested due to automatic review settings October 6, 2026 02:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread .github/workflows/ci.yml Resolved
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 6, 2026 03:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ppenna

Copy link
Copy Markdown
Contributor

Closing as not planned. The dedicated Azure utility runners are being removed from NVX CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants