Repository navigation
Enable shared NVCC PCH caching for Modal submissions - #534
Merged
Merged
Conversation
Coverage reportClick to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||
msaroufim
marked this pull request as ready for review
November 15, 2026 23:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enable NVCC PCH by default for compatible
load_inlinesubmissions. Warm 17 profiles from five observed flag combinations for Torch/CUDA headers (excluding CUTLASS), then mount the shared cache read-only. Misses compile normally;KERNELBOT_NVCC_PCH=0disables it.CPU compile medians: two fresh 4-CPU/8-GiB workers per mode, same NVCC 13.4.99 and host PCH. Times include import, compilation, dependency capture, and artifact packing.
-O3-O3 --use_fast_math-O3-O3 --use_fast_math -arch=sm_100aAll 20 timed runs passed correctness and GPU artifact reuse. CUTLASS showed no clear benefit. Measurements predate the final catalog/metadata fixes; compiler, toolkit, and framework headers are unchanged.
CI passes. Overall, 33 instrumented cases passed, including five B200 cases. Final canary passed on T4/A100/H100; B200 confirmed CPU PCH consumption but timed out awaiting GPU capacity. L4 was compile/warm tested only. Excluded a C++17 profile that also fails on stock CUDA without PCH. Benchmark harnesses and raw data stay outside this patch.
Thanks to Michael Buch at NVIDIA for the compiler support and PCH guidance.