Skip to content

bootstrap: Make low priority the default - #165707

Open
kotauskas wants to merge 2 commits into
rust-lang:mainfrom
kotauskas:low-prio-by-default
Open

kotauskas wants to merge 2 commits into
rust-lang:mainfrom
kotauskas:low-prio-by-default

Conversation

@kotauskas

@kotauskas kotauskas commented Nov 9, 2026 •

Copy link
Copy Markdown

View all comments

The low-priority option is tucked away so far in bootstrap.example.toml that it's nearly impossible to discover by anyone who doesn't think they're doing anything niche (see #165702). However, it is beneficial to have it enabled on nearly every system, since a build is a CPU-saturating batch job that has no business preempting foreground applications, which predominantly have nice 0. Consider the following four scenarios ensuing from the change from low-priority = "false" to low-priority = "true":

  • There are no processes with non-negligible CPU usage between nice 0 and 10, and bootstrap builds do not get preempted by anything. A system on which this is the case will be unaffected by this change. Examples include CI runners and other build environments with proper build queuing.
  • There are foreground processes between nice 0 and 10 that are relatively light on CPU compared to a bootstrap build and were previously being heavily preempted by bootstrap builds. With this change, a system on which this is the case will see a significant improvement to the latency of interactive programs, enabling the user to comfortably do other things while the build is running without affecting the throughput of the build itself (as the programs are, as established above, light on CPU).
  • There are CPU-intensive but not CPU-saturating foreground processes (games, office and productivity software, Jupyter notebooks, etc.) between nice 0 and 10 that constantly compete with bootstrap builds for CPU time. With this change, a system on which this is the case will see CPU time allocation be biased toward the foreground CPU work at the expense of the build, which is most likely what the user wants, since they wouldn't run CPU-intensive things in the foreground if they didn't want them to have priority over background batch jobs whose completion they are passively waiting for.
  • There are CPU-saturating batch jobs between nice 0 and 10 that constantly compete with bootstrap builds for CPU time, interfering with each other and performing worse than if a batch job queue was in operation and one job ran after the other. With this change, a system on which this is the case will see the bootstrap build receive less CPU time, but the overall sum of batch jobs will complete in approximately the same amount of time regardless, and may even complete sooner due to reduced competition for CPU cache and OS page cache. In any case, this is a misconfigured and a priori suboptimally performing setup, and it will not see good performance unless some form of job queuing is implemented. Admins may implement a low-priority = "false" override as a hacky stopgap if it is desirable for the bootstrap build to engage in disruptive competition for resources and win against some other batch job running in parallel.

Some may wonder why agonizing over this is necessary to begin with if this is such a clear-cut case that operating systems are supposed to deal with automatically. The answer is that the operating systems of today are still not very good at making this sort of scheduling decision in that their capacity to implement such logic automatically is limited and non-robust in unintuitive ways.

For example, the automatic prioritization of foreground programs for scheduling purposes in Windows, known as quantum boosting, can only confer priority boosts to the threads of the process that owns a foreground window, which means that if the graphical application displayed within that window has an IPC dependency on a different process (as is the case with the likes of Jupyter notebooks and even regular web browsers), that dependency will not receive the quantum boost or any other kind of priority increase.

On Linux, meanwhile, no such heuristics exist to begin with (due to graphical applications not holding any special status in the scheduler), and the only mechanism to prevent CPU-intensive and CPU-saturating batch jobs from preempting foreground applications is the emergent heuristic of completely fair scheduling, in which tasks with less total CPU time accrued will have higher priority. This is even more flawed than quantum boosting on Windows, as long-running graphical applications will easily get preempted for long periods of time by the constantly spawning rustc processes of a build that have no history of high CPU usage.

In summary: there are commonly occurring setups on which this change is beneficial, commonly occurring setups on which this change does nothing, and extremely niche and typically broken use cases where this change does something that cannot be conclusively deemed positive or negative.

Large language models were not used in the production of this PR.

@rustbot

rustbot commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

This PR modifies bootstrap.example.toml.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Nov 9, 2026
@rustbot

rustbot commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the pull request, and welcome! The Rust Project has assigned @clubby789 (or someone else) to review your changes, you should hear from them (or someone else) within the next two weeks.

Please see the contribution instructions and our LLM policy for more information.

Why was this reviewer chosen?

The reviewer was selected based on:

  • Owners of files modified in this PR: bootstrap
  • bootstrap expanded to 6 candidates
  • Random selection from Mark-Simulacrum, clubby789, jieyouxu

@kotauskas

Copy link
Copy Markdown
Author

Change tracker entry probably not necessary, as elaborated above.

@kotauskas

Copy link
Copy Markdown
Author

On second thought, maybe it's worth telling people to get rid of things like alias x="nice ./x" from their setups if they didn't know about this option. I've added a change tracker entry.

@workingjubilee

Copy link
Copy Markdown
Member

I have never once heard of anyone using such an alias.

@bjorn3

bjorn3 commented Nov 9, 2026

Copy link
Copy Markdown
Member

Would it make sense to disable this on the dist profile? Otherwise on systems building multiple packages at the same time we might unnecessarily get slowed down.

@kotauskas

Copy link
Copy Markdown
Author

@bjorn3 Refer to case 4:

With this change, a system on which this is the case will see the bootstrap build receive less CPU time, but the overall sum of batch jobs will complete in approximately the same amount of time regardless, and may even complete sooner due to reduced competition for CPU cache and OS page cache.

"We" might get unnecessarily "slowed down", but that is not a bad thing, unless you are presuming that those running the (as mentioned previously, inefficient and misguided) cross-package-parallel build are measuring the wall time (as opposed to the CPU time) of the Rust build and drawing unsavory conclusions from that. You can't save people with horrible benchmarks from themselves.

@bjorn3

bjorn3 commented Nov 9, 2026

Copy link
Copy Markdown
Member

Increasing wall time means that it takes longer between updating the package definition for rustc and the update actually being available to end users or to build other packages, even if cpu time is identical. And given that rustc already takes a while to build (and is a dependency of other projects that are slow to build, like firefox), making it even longer to build seems bad.

@workingjubilee

Copy link
Copy Markdown
Member

I agree that building packages without handling your own priorities if you care about ordering seems like a "you" problem, but I also agree that there's no reason to have rustc nice its own build when we would, logically, mostly be competing against "peers" of rustc. A dist build should probably avoid trying to take part in prioritization on the assumption that the larger system has handled that.

@kotauskas

Copy link
Copy Markdown
Author

Let's say we have two jobs A and R, where R is the Rust build, being started at the same time, and a job B, which waits for R to finish. Consider the asymptotic starvation behavior: A runs while R gets exactly zero CPU time; R then gets all of the CPU time for itself and completes faster than if it was competing for time (but the wall time will make it look slower, because the timer starts when A and R start, not when A finishes and R begins receiving CPU time). Subsequently, B will get the CPU all to itself, which will only result in CPU sleep compared to equal scheduling if B cannot parallelize. Note that CPU sleeping during execution of B is also possible in the equal scheduling case if A and R complete sooner.

All of this is to say that situations in which rustc being deprioritized could significantly delay the build are very exotic. It would be like trying to rely on pipelining facilitated by build planning when the planning algorithm is "if we can start running a thing, do it".

Although, at the same time, the kinds of workflows that run dist builds often involve nicing the whole build at a higher level (like a renice +10 $$ in a makepkg.conf), so maybe doing deprioritization for dist builds would be unnecessary anyway.

I think I'll special-case dist builds when I get home. I would much prefer if there was some protocol for non-interactive batch job managers to tell their children that they shouldn't auto-prioritize, maybe like a NO_AUTO_PRIO environment variable.

@workingjubilee

Copy link
Copy Markdown
Member

Although, at the same time, the kinds of workflows that run dist builds often involve nicing the whole build at a higher level (like a renice +10 $$ in a makepkg.conf), so maybe doing deprioritization for dist builds would be unnecessary anyway.

Yes, exactly. That "should" be happening, so fiddling with our priority in the dist case seems like it just "gets in the way", even if

I think I'll special-case dist builds when I get home. I would much prefer if there was some protocol for non-interactive batch job managers to tell their children that they shouldn't auto-prioritize, maybe like a NO_AUTO_PRIO environment variable.

in a more perfect world we would have this, yes.

@clubby789

Copy link
Copy Markdown
Contributor

@bors r+

@rust-bors

rust-bors Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

📌 Commit d66db2b has been approved by clubby789

It is now in the queue for this repository.

@rust-bors rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 19, 2026
rust-bors Bot pushed a commit that referenced this pull request Nov 19, 2026
bootstrap: Make low priority the default

The `low-priority` option is tucked away so far in `bootstrap.example.toml` that it's nearly impossible to discover by anyone who doesn't think they're doing anything niche (see #165702). However, it is beneficial to have it enabled on nearly every system, since a build is a CPU-saturating batch job that has no business preempting foreground applications, which predominantly have nice 0. Consider the following three scenarios ensuing from the change from `low-priority = "false"` to `low-priority = "true"`:

- There are no processes with non-negligible CPU usage between nice 0 and 10, and bootstrap builds do not get preempted by anything. A system on which this is the case will be unaffected by this change. Examples include CI runners and other build environments with proper build queuing.
- There are foreground processes between nice 0 and 10 that are relatively light on CPU compared to a bootstrap build and were previously being heavily preempted by bootstrap builds. With this change, a system on which this is the case will see a significant improvement to the latency of interactive programs, enabling the user to comfortably do other things while the build is running without affecting the throughput of the build itself (as the programs are, as established above, light on CPU).
- There are CPU-intensive but not CPU-saturating foreground processes (games, office and productivity software, Jupyter notebooks, etc.) between nice 0 and 10 that constantly compete with bootstrap builds for CPU time. With this change, a system on which this is the case will see CPU time allocation be biased toward the foreground CPU work at the expense of the build, which is most likely what the user wants, since they wouldn't run CPU-intensive things in the foreground if they didn't want them to have priority over background batch jobs whose completion they are passively waiting for.
- There are CPU-saturating batch jobs between nice 0 and 10 that constantly compete with bootstrap builds for CPU time, interfering with each other and performing worse than if a batch job queue was in operation and one job ran after the other. With this change, a system on which this is the case will see the bootstrap build receive less CPU time, but the overall sum of batch jobs will complete in approximately the same amount of time regardless, and may even complete sooner due to reduced competition for CPU cache and OS page cache. In any case, this is a misconfigured and a priori suboptimally performing setup, and it will not see good performance unless some form of job queuing is implemented. Admins may implement a `low-priority = "false"` override as a hacky stopgap if it is desirable for the bootstrap build to engage in disruptive competition for resources and win against some other batch job running in parallel.

Some may wonder why agonizing over this is necessary to begin with if this is such a clear-cut case that operating systems are supposed to deal with automatically. The answer is that the operating systems of today are still not very good at making this sort of scheduling decision in that their capacity to implement such logic automatically is limited and non-robust in unintuitive ways.

For example, the automatic prioritization of foreground programs for scheduling purposes in Windows, known as [quantum boosting][quantboost], can only confer priority boosts to the threads of the process that owns a foreground window, which means that if the graphical application displayed within that window has an IPC dependency on a different process (as is the case with the likes of Jupyter notebooks and even regular web browsers), that dependency will not receive the quantum boost or any other kind of priority increase.

[quantboost]: https://web.archive.org/web/20250326185947/https://www.microsoftpressstore.com/articles/article.aspx?p=2233328&seqNum=7

On Linux, meanwhile, no such heuristics exist to begin with (due to graphical applications not holding any special status in the scheduler), and the only mechanism to prevent CPU-intensive and CPU-saturating batch jobs from preempting foreground applications is the emergent heuristic of completely fair scheduling, in which tasks with less total CPU time accrued will have higher priority. This is even more flawed than quantum boosting on Windows, as long-running graphical applications will easily get preempted for long periods of time by the constantly spawning rustc processes of a build that have no history of high CPU usage.

In summary: there are commonly occurring setups on which this change is beneficial, commonly occurring setups on which this change does nothing, and extremely niche and typically broken use cases where this change does something that cannot be conclusively deemed positive or negative.
@rust-bors

This comment has been minimized.

@rust-bors rust-bors Bot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Nov 19, 2026
@rust-bors

rust-bors Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

💔 Test for 41fd2d0 failed: CI. Failed job:

@rust-log-analyzer

This comment has been minimized.

Comment thread src/bootstrap/src/utils/job.rs Resolved
Comment thread src/bootstrap/src/utils/job.rs Resolved
Comment thread src/bootstrap/src/utils/job.rs Resolved
Comment thread src/bootstrap/src/utils/job.rs Resolved
Comment thread bootstrap.example.toml Resolved
Comment thread src/bootstrap/src/utils/job.rs Resolved
Comment thread src/bootstrap/src/utils/job.rs Resolved
@rustbot

rustbot commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

This PR modifies src/bootstrap/defaults.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

Comment thread src/bootstrap/src/core/config/config.rs Resolved
@clubby789

Copy link
Copy Markdown
Contributor

Can you squash the changes so the commit history is something like

  1. Change build.low-priority to true by default
  2. Respect existing nice value

@kotauskas

Copy link
Copy Markdown
Author

…Hopefully, that did what I was trying to do.

@rustbot ready

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Nov 19, 2026
Comment thread src/bootstrap/src/utils/job.rs Resolved
@clubby789

Copy link
Copy Markdown
Contributor

Structure is clearer now, and the change to respecting an existing nice value seems reasonable.
@bors try jobs=apple,bsd
r=me if the try job passes and with the nit applied.

@rust-bors

This comment has been minimized.

rust-bors Bot pushed a commit that referenced this pull request Nov 19, 2026
bootstrap: Make low priority the default


try-job: *apple*
try-job: *bsd*
@rust-bors

rust-bors Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

☀️ Try build successful (CI)
Build commit: 97b6bb6 (97b6bb670065741a3fc5697b1b3f4e10eaf66431)
Base parent: c884ecf (c884ecfbabb04b04fa5054dafc67ec258ecde745)

@clubby789

Copy link
Copy Markdown
Contributor

@bors r+

@rust-bors

rust-bors Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

📌 Commit 0c2fc3b has been approved by clubby789

It is now in the queue for this repository.

@rust-bors rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 19, 2026
jhpratt added a commit to jhpratt/rust that referenced this pull request Nov 20, 2026
…lubby789

bootstrap: Make low priority the default

The `low-priority` option is tucked away so far in `bootstrap.example.toml` that it's nearly impossible to discover by anyone who doesn't think they're doing anything niche (see rust-lang#165702). However, it is beneficial to have it enabled on nearly every system, since a build is a CPU-saturating batch job that has no business preempting foreground applications, which predominantly have nice 0. Consider the following four scenarios ensuing from the change from `low-priority = "false"` to `low-priority = "true"`:

- There are no processes with non-negligible CPU usage between nice 0 and 10, and bootstrap builds do not get preempted by anything. A system on which this is the case will be unaffected by this change. Examples include CI runners and other build environments with proper build queuing.
- There are foreground processes between nice 0 and 10 that are relatively light on CPU compared to a bootstrap build and were previously being heavily preempted by bootstrap builds. With this change, a system on which this is the case will see a significant improvement to the latency of interactive programs, enabling the user to comfortably do other things while the build is running without affecting the throughput of the build itself (as the programs are, as established above, light on CPU).
- There are CPU-intensive but not CPU-saturating foreground processes (games, office and productivity software, Jupyter notebooks, etc.) between nice 0 and 10 that constantly compete with bootstrap builds for CPU time. With this change, a system on which this is the case will see CPU time allocation be biased toward the foreground CPU work at the expense of the build, which is most likely what the user wants, since they wouldn't run CPU-intensive things in the foreground if they didn't want them to have priority over background batch jobs whose completion they are passively waiting for.
- There are CPU-saturating batch jobs between nice 0 and 10 that constantly compete with bootstrap builds for CPU time, interfering with each other and performing worse than if a batch job queue was in operation and one job ran after the other. With this change, a system on which this is the case will see the bootstrap build receive less CPU time, but the overall sum of batch jobs will complete in approximately the same amount of time regardless, and may even complete sooner due to reduced competition for CPU cache and OS page cache. In any case, this is a misconfigured and a priori suboptimally performing setup, and it will not see good performance unless some form of job queuing is implemented. Admins may implement a `low-priority = "false"` override as a hacky stopgap if it is desirable for the bootstrap build to engage in disruptive competition for resources and win against some other batch job running in parallel.

Some may wonder why agonizing over this is necessary to begin with if this is such a clear-cut case that operating systems are supposed to deal with automatically. The answer is that the operating systems of today are still not very good at making this sort of scheduling decision in that their capacity to implement such logic automatically is limited and non-robust in unintuitive ways.

For example, the automatic prioritization of foreground programs for scheduling purposes in Windows, known as [quantum boosting][quantboost], can only confer priority boosts to the threads of the process that owns a foreground window, which means that if the graphical application displayed within that window has an IPC dependency on a different process (as is the case with the likes of Jupyter notebooks and even regular web browsers), that dependency will not receive the quantum boost or any other kind of priority increase.

[quantboost]: https://web.archive.org/web/20250326185947/https://www.microsoftpressstore.com/articles/article.aspx?p=2233328&seqNum=7

On Linux, meanwhile, no such heuristics exist to begin with (due to graphical applications not holding any special status in the scheduler), and the only mechanism to prevent CPU-intensive and CPU-saturating batch jobs from preempting foreground applications is the emergent heuristic of completely fair scheduling, in which tasks with less total CPU time accrued will have higher priority. This is even more flawed than quantum boosting on Windows, as long-running graphical applications will easily get preempted for long periods of time by the constantly spawning rustc processes of a build that have no history of high CPU usage.

In summary: there are commonly occurring setups on which this change is beneficial, commonly occurring setups on which this change does nothing, and extremely niche and typically broken use cases where this change does something that cannot be conclusively deemed positive or negative.
rust-bors Bot pushed a commit that referenced this pull request Nov 20, 2026
Rollup of 21 pull requests

Successful merges:

 - #165707 (bootstrap: Make low priority the default)
 - #165716 (Refactor split_block in check_pointers to avoid Terminator: None)
 - #165934 (Avoid using Terminator:None in simplify_cfg)
 - #166050 (Use attribute parser for lang attributes in `LateResolutionVisitor`)
 - #166069 (bootstrap: fix keep-stage-std)
 - #166117 (Remove ebadf handling from stdio)
 - #166119 (Avoid an ICE on re-exports of private imports without a DefId)
 - #165094 (reflection: doc attributes from adts, fields, and variants)
 - #165216 (Clearer diagnostics when `&FnOnce` was provided when `FnOnce` was expected)
 - #165694 (view-types: make sure view types are !Sized)
 - #165802 (Add CI workflow to publish backtrace after a push to main)
 - #165883 (Report a recursion limit error instead of delaying a bug in RPITIT collector)
 - #165906 (fix ICE when a forced keyword follows a parameter pattern)
 - #166064 (Add const generic transmute min support)
 - #166136 (Move ub_checks() inside the const_eval_select branch.)
 - #166158 (bump some tools' `toml` dependency)
 - #166166 (Don't recover if not allowed in `parse_seq_to_before_tokens`)
 - #166176 (compiletest: Move all code out of the crate root)
 - #166185 (simd-test: Fix cfg syntax in the test macro)
 - #166194 (std: Unstably constify `std::process::abort`)
 - #166210 (Avoid terminator: None in insert_direct_enum_check)

Failed merges:

 - #166150 (Reflection: refactor Type and TypeKind)
 - #166160 (refactor CTFE-specific intrinsics to different file)
@jhpratt

jhpratt commented Nov 20, 2026

Copy link
Copy Markdown
Member

@bors r- #166228 (comment)

I presume you'll need to gate the errno dependency to the appropriate cfg, as the logs show that it's unused.

@rust-bors rust-bors Bot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Nov 20, 2026
@rust-bors

rust-bors Bot commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

This pull request was unapproved.

This PR was contained in a rollup (#166228), which was unapproved.

View changes since this unapproval

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants