Repository navigation
ci: route hosted jobs to isolated Azure runners - #318
Enrique Saurez (esaurez) wants to merge 14 commits into
Conversation
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
Review effort: Lite
Findings: 1
Open (4)
python3 -m venvis not guaranteed to work on Debian/Ubuntu unless thepython3-venvpackage is… · New The action creates a per-jobDOCKER_CONFIGdirectory but never cleans it up. On persistent… · New This assertion enforces that no job in the workflow ever usesubuntu-latest, which can cause… · New This test extractsvalidate_runner_labels()by splitting on exact string/whitespace, which is… · New
What changed in this PR
Routes previously GitHub-hosted CI jobs onto dedicated Azure self-hosted Linux runners, separating PR/fork workloads from push-only jobs that use write credentials, and enforcing rootless Docker/per-job client config for Docker-based jobs.
Changes:
- Update
ci.ymljob routing tonvx-utility-prvsnvx-utility-trustedrunners and add a rootless Docker enforcement step for relevant jobs. - Extend Linux runner setup to allow validated label overrides via
--runner-labels. - Add/adjust composite actions and tests to validate runner label policy, routing, and rootless Docker usage.
| File | Description |
|---|---|
| scripts/test_nvx_tools.py | Adds assertions for runner label override behavior, workflow routing, and rootless Docker configuration. |
| scripts/setup/setup-linux-runner.sh | Introduces --runner-labels override with validation and expected-label behavior changes. |
| scripts/setup/README.md | Documents new runner label override and separates Docker-capable utility runner guidance. |
| .github/workflows/ci.yml | Routes jobs to self-hosted Azure runners and adds rootless Docker requirement steps. |
| .github/actions/require-rootless-docker/action.yml | New composite action to force rootless Docker daemon + per-job DOCKER_CONFIG. |
| .github/actions/check-quality/action.yml | Adds self-hosted Python setup path using a per-job venv. |
| .github/actionlint.yaml | Adds new allowed self-hosted runner labels for actionlint. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
Review effort: Lite
Findings: 1
Open (7)
python3 -m venvis not guaranteed to work on Debian/Ubuntu unless thepython3-venvpackage is… Writing the temp venv’sbindirectory to$GITHUB_PATHmakes that PATH change persist for the… · New The Docker socket location is hard-coded to/run/user/$(id -u)/docker.sock, but rootless Docker… · New This test extracts the shell function body using exact string delimiters that depend on formatting… · New This test extractsvalidate_runner_labels()by splitting on exact string/whitespace, which is… This assertion enforces that no job in the workflow ever usesubuntu-latest, which can cause… The action creates a per-jobDOCKER_CONFIGdirectory but never cleans it up. On persistent…
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The self-hosted quality runner bootstrap does not provide the PowerShell executable required by its quality action.
Review effort: Balanced
Findings: 2
Open (4)
Provision and validate PowerShell on Linux quality runners · Newpython3 -m venvis not guaranteed to work on Debian/Ubuntu unless thepython3-venvpackage is… The action creates a per-jobDOCKER_CONFIGdirectory but never cleans it up. On persistent… Clarify Docker access limitation for rootless utility runners · New
Resolved since last review (5)
This test extracts the shell function body using exact string delimiters that depend on formatting… The Docker socket location is hard-coded to/run/user/$(id -u)/docker.sock, but rootless Docker… Writing the temp venv’sbindirectory to$GITHUB_PATHmakes that PATH change persist for the… This test extractsvalidate_runner_labels()by splitting on exact string/whitespace, which is… This assertion enforces that no job in the workflow ever usesubuntu-latest, which can cause…
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The setup documentation contradicts the newly introduced rootless Docker access model.
Review effort: Balanced
Findings: 2
Open (4)
Provision and validate PowerShell on Linux quality runnerspython3 -m venvis not guaranteed to work on Debian/Ubuntu unless thepython3-venvpackage is… The action creates a per-jobDOCKER_CONFIGdirectory but never cleans it up. On persistent… Clarify Docker access limitation for rootless utility runners
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Fork PR and release-bound push artifacts share persistent runner state, weakening the intended trust boundary.
Review effort: Balanced
Findings: 3
Open (5)
Shared runner lets fork PR state contaminate trusted release builds · New Provision and validate PowerShell on Linux quality runnerspython3 -m venvis not guaranteed to work on Debian/Ubuntu unless thepython3-venvpackage is… The action creates a per-jobDOCKER_CONFIGdirectory but never cleans it up. On persistent… Clarify Docker access limitation for rootless utility runners
| ) | ||
| }} | ||
| runs-on: ubuntu-latest | ||
| runs-on: [self-hosted, linux, x64, nvx-utility-pr] |
There was a problem hiding this comment.
Confirmed: reusing a persistent runner for fork PR and dev push artifact builds can carry untrusted state into release inputs. This thread remains open while we decide whether to route push builds to the fork-free trusted VM or use an ephemeral/dedicated builder. No merge until this boundary is resolved.
329ba3e to
1c04aec
Compare
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The open fork-to-push artifact isolation issue still permits untrusted persistent-runner state to affect release inputs.
Review effort: Balanced
Findings: 3
Open (5)
Shared runner lets fork PR state contaminate trusted release builds Provision and validate PowerShell on Linux quality runnerspython3 -m venvis not guaranteed to work on Debian/Ubuntu unless thepython3-venvpackage is… The action creates a per-jobDOCKER_CONFIGdirectory but never cleans it up. On persistent… Clarify Docker access limitation for rootless utility runners
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The acknowledged persistent-runner boundary still allows fork workloads to influence later release artifact builds.
Review effort: Balanced
Findings: 3
Open (5)
Shared runner lets fork PR state contaminate trusted release builds Provision and validate PowerShell on Linux quality runnerspython3 -m venvis not guaranteed to work on Debian/Ubuntu unless thepython3-venvpackage is… The action creates a per-jobDOCKER_CONFIGdirectory but never cleans it up. On persistent… Clarify Docker access limitation for rootless utility runners
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The acknowledged persistent Linux PR runner can carry fork-controlled state into later development artifact builds and releases.
Review effort: Balanced
Findings: 3
Open (5)
Shared runner lets fork PR state contaminate trusted release builds Provision and validate PowerShell on Linux quality runnerspython3 -m venvis not guaranteed to work on Debian/Ubuntu unless thepython3-venvpackage is… The action creates a per-jobDOCKER_CONFIGdirectory but never cleans it up. On persistent… Clarify Docker access limitation for rootless utility runners
Use distinct labels for PR and credentialed jobs, and require rootless Docker for artifact builds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Point artifact and quality jobs at the per-user systemd Docker daemon after moving it off a system-level service. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep Buildx configuration in the runner job temp directory because the service account has no writable home. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep hosted Python 3.10 selection, but use a fresh Python 3.12 virtualenv and per-job Docker state on the self-hosted utility runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Apply the repository-pinned Ruff formatter to the CI runner test before rerunning quality. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Scope Python to the quality action, clean verified per-job Docker state, honor the rootless runtime directory, and validate venv support and runner labels. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wait briefly for asynchronous process reaping and canary acceptance while preserving strict failure assertions; narrow the runner label test type guard. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Validate the PowerShell parser prerequisite before quality checks and clarify that utility jobs only have rootless Docker access. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
07a9737 to
12a36e5
Compare
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Persistent fork-to-release artifact contamination remains unresolved, and single-instance routing serializes supposedly independent artifact and debug builds.
Review effort: Balanced
Findings: 3
Open (6)
Shared runner lets fork PR state contaminate trusted release builds Provision and validate PowerShell on Linux quality runnerspython3 -m venvis not guaranteed to work on Debian/Ubuntu unless thepython3-venvpackage is… Debug and artifacts jobs contend for the sole runner · New The action creates a per-jobDOCKER_CONFIGdirectory but never cleans it up. On persistent… Clarify Docker access limitation for rootless utility runners
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The acknowledged persistent fork-to-push artifact isolation boundary remains unresolved and can affect release inputs.
Review effort: Balanced
Findings: 3
Open (5)
Shared runner lets fork PR state contaminate trusted release builds Provision and validate PowerShell on Linux quality runnerspython3 -m venvis not guaranteed to work on Debian/Ubuntu unless thepython3-venvpackage is… The action creates a per-jobDOCKER_CONFIGdirectory but never cleans it up. On persistent… Clarify Docker access limitation for rootless utility runners
Resolved since last review (1)
|
Closing as not planned. The dedicated Azure utility runners are being removed from NVX CI. |



Summary
devCI onto four dedicated Azure utility runners: Linux PR/fork, Linux trusted push, Windows PR/fork, and Windows trusted push. Keep the existing backend benchmark scale sets unchanged.Infrastructure
Two Linux and two Windows Regular Standard_D8s_v5 utility scale sets have been provisioned in subscription
7d07dbdd-8117-46e9-8f58-721da3f32fd4using AzTemplates branchcopilot/nvx-ci-utility-20261002. Each has one instance; the Windows VMSS use Windows Server 2025, Trusted Launch, 128-GiB OS and 512-GiB managed data disks, both approved SSH keys, and the reference Windows VMSS's exact data-disk initializer. Their PR and trusted administrator passwords are separate secrets in a dedicated vault. Each Windows managed identity can read only the existing runner-bootstrap secret. The existing Linux utility scale sets and Windows/WHP benchmark scale set were not reimaged or resized.Bootstrap pins the setup script by immutable commit and checksum, survives prerequisite reboots, and registers role-specific labels. Both Windows utility runners are registered with separate PR/fork and trusted-push labels. The Windows PR crate check passed, and the trusted runner passed its bootstrap check-only validation.
CI stabilization
The latest
devalready includes the stronger adversarial cleanup synchronization fixes; this rebase retains the still-needed typing guard without duplicating those fixes. No VMM runtime behavior changed.The full PR workflow at
29ed346passed on run 37409021969, attempt 2. The first attempt exceeded three MSHV latency thresholds onazure-mshv-scus-1; fresh measurements onazure-mshv-1passed without benchmark, threshold, or baseline changes. Shared artifacts finished before the debug build started on the same PR utility runner.Trust boundary
Fork PRs temporarily use persistent PR-only runners during the hosted-runner outage. Rootless Docker and runner metadata isolation do not provide ephemeral-runner isolation. In particular, a fork PR could leave state on the Linux PR runner that affects a later
devartifact build and release. This is the previously deferred review decision; do not merge until it is resolved. Revert fork jobs to hosted capacity when it recovers.