Skip to content

perf(utils): use std::fs::remove_dir_all() - #5204

Merged
rami3l merged 5 commits into
rust-lang:mainfrom
FalkWoldmann:std-remove-dir-all-unix
Nov 9, 2026
Merged

rami3l merged 5 commits into
rust-lang:mainfrom
FalkWoldmann:std-remove-dir-all-unix

Conversation

@FalkWoldmann

@FalkWoldmann FalkWoldmann commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

On macOS and Linux, the remove_dir_all crate opens every entry it deletes to find out whether it's a directory. Endpoint security software scans files on open, so that becomes one scan per file. On my Mac with suhc a endpoint software, rustup toolchain uninstall of a nightly with the docs took 11 minutes. This does not happen when using the std fs::remove_dir_all :)

remove_dir_all 1.0 std
rustup toolchain uninstall, macOS with Defender, 72,597 entries 661 s 24.5 s
deleting a doc tree directly, Linux on btrfs without a scanner, 67,636 entries 1.65 s 0.41 s

I couldn't test windows and AFAIK the remove_dir_all does some speedup optimizations here. Would be interesting if we could switch windows to the std version as well

@djc djc left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you do a little git blame stuff to try and figure out why rustup started using the remove_dir_all crate? Presumably this predated the inclusion of this function in std?

I don't think your manual testing should be a prerequisite to using this on Windows -- we should just start using it everywhere (does that mean we don't need the raw helper anymore?) and see if Windows CI passes.

View changes since this review

@FalkWoldmann
FalkWoldmann force-pushed the std-remove-dir-all-unix branch from ee0d990 to ed0718e Compare November 9, 2026 10:39
@FalkWoldmann FalkWoldmann changed the title perf(utils): use std::fs::remove_dir_all() outside Windows perf(utils): use std::fs::remove_dir_all() Nov 9, 2026
@FalkWoldmann

Copy link
Copy Markdown
Contributor Author

Can you do a little git blame stuff to try and figure out why rustup started using the remove_dir_all crate? Presumably this predated the inclusion of this function in std?

I don't think your manual testing should be a prerequisite to using this on Windows -- we should just start using it everywhere (does that mean we don't need the raw helper anymore?) and see if Windows CI passes.

View changes since this review

Sure :) I adjusted the PR so it uses std everywhere.

Git Blame history:

@djc

djc commented Nov 9, 2026

Copy link
Copy Markdown
Contributor
  • Bump remove_dir_all #3244 (February 2023): the 0.8 security bump. From then on the crate used its own code on Unix too, which is where the per-file open comes from.

Can you look at why it started using its own code on Unix?

@FalkWoldmann

FalkWoldmann commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Bump remove_dir_all #3244 (February 2023): the 0.8 security bump. From then on the crate used its own code on Unix too, which is where the per-file open comes from.

Can you look at why it started using its own code on Unix?

tbh I couldn't find the reason for why they changed it. The commit message and changelog give no reason for dropping the std re-export. The code that needed the security fix was the two functions std doesn't have. My guess is that they wanted one handle-based implementation behind every entry point. For the record, the 0.8 changelog also says:

processes that might run with elevated privileges should figure out how to securely identify the directory they are going to delete ... Pragmatically, other processes should be fine with the path based entry points: this is the same interface std::fs::remove_dir_all offers

That supports the point that std's function is fine for rustup, which isn't a setuid binary. @rbtcollins made the same point in #3244

@ChrisDenton

Copy link
Copy Markdown
Member

Shouldn't we completely remove our use of the remove_dir_all crate in that case?

@rami3l rami3l left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a whole lot for all this investigation!

View changes since this review

@rami3l
rami3l added this pull request to the merge queue Nov 9, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 9, 2026
@rami3l
rami3l added this pull request to the merge queue Nov 9, 2026
Merged via the queue into rust-lang:main with commit 2ca996a Nov 9, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants