Repository navigation
Cache generated source code and hash in custom kernels - #4568
Merged
Merged
Conversation
CustomKernel now gets an optional precomputed source hash instead of hashing the full source per dispatch Found with a Time Profiler capture, M5 Ultra, Qwen 3.8 Flash-Next. 14% of host thread time was spent in this path Effect: graph build goes from 1.42->1.21ms per round. Tests included.
zcbenz
reviewed
Oct 1, 2026
zcbenz
reviewed
Oct 1, 2026
Member
|
Closing due to lack of activity. |
Contributor
Author
|
Sorry let me fix this PR. It is a solid perf boost. |
- CustomKernel::source_hash_ is a plain std::size_t. Callers that already have the hash pass it; the state()-order constructor used by import computes it from the source. - GeneratedSourceCache stores GeneratedSource by value and returns a reference. Entries are never erased, so the reference stays valid. - The new test only covers template args (int 1 vs bool true, same kernel name, different signature).
Contributor
Author
zcbenz
approved these changes
Oct 31, 2026
STRML
added a commit
to STRML/mlx-serve
that referenced
this pull request
Oct 31, 2026
…l-explore/mlx#4568 700520ae (Metal expm1 argument reduction accuracy) and 33927952 (sharded-to-all bias and integer split positions on quantized layers). Neither touches the custom-kernel cache, the SDPA dispatch, the quantized kernels or the CUDA backend: the carried sdpa_split patch applies unchanged and the CUDA patch stack replays in Linux and Windows order. AGENTS.md names the new pin and the carried mlx patch.
inureyes
added a commit
to lablup/mlxcel
that referenced
this pull request
Nov 6, 2026
… copy Upstream 4ca8d5d5a (ml-explore/mlx#4568) declares fast::template_arguments_hash in mlx/backend/common/custom_kernel.h, which mlx/fast.h includes. The fork keeps a private copy in mlx/backend/rocm/custom_kernel.cpp, so the unqualified call there matches both declarations and MSVC rejects it as ambiguous (C2668) when the ROCm backend and the common library are built together. Remove the fork copy; the upstream one also maps '-' to '_' in integer arguments so a negative template argument cannot produce an invalid kernel name. Recorded as LOCAL_FIXES item 73.
inureyes
added a commit
to lablup/mlxcel
that referenced
this pull request
Nov 6, 2026
…gnatures Upstream 4ca8d5d5a (ml-explore/mlx#4568) added a generated-source hash to fast::CustomKernel's constructor, and the fork's mlx/backend/rocm/custom_kernel.cpp still passed grid as the fourth argument. Pass std::hash<std::string>{}(kernel_source), as the CUDA backend does; the ROCm JIT module keeps keying on the source itself (LOCAL_FIXES item 33). Upstream 5ee1e8edf (ml-explore/mlx#4711) added add_out_to_temporary_on_allocation to reshape_gpu, which the shared gpu/primitives.cpp overlay now calls with four arguments. The fork's definition in mlx/backend/rocm/copy.hip takes the flag and registers the output as a temporary when it is set, matching CUDA. The first was a compile error on MSVC; the second would have been a link error on any ROCm build. Recorded as LOCAL_FIXES items 73 and 74.
inureyes
added a commit
to lablup/mlxcel
that referenced
this pull request
Nov 6, 2026
* update: advance MLX pin to 9f806b36 and refresh backend overlays Move the CMake pin from 7ca40033 (2026-10-26) to 9f806b36 (2026-11-05), a reproducible main snapshot 46 commits newer and 81 past stable v0.32.4, which the previous pin already postdated. Rebase the 14 drifted CPU, Metal and CUDA whole-file overlays with a three-way merge. Upstream #4803 now accumulates the CPU affine quantized matmul in float, so only fp_qmm and fp_qmm_t stay local. The new Metal affine qmv_mma (#4787) is placed behind the MLXCEL_QMV_WIDE off-switch so the speculative exactness retry still restores byte-equality between a verify block and the single-token chain. Metal device.cpp keeps the input-budget override across upstream's env rework, and quantized.metal re-appends mlxcel_affine_qmm_t_rows after the restructured instantiation macros. Two upstream API moves needed code changes: env::get_var is now config::get (#4647), used by the cxx bridge and the CUDA SDPA switches, and gpu_core_count() (#4781) needs IOKit on the macOS link line because upstream links it PRIVATE. Retarget the ROCm core files with the fork unchanged, resolve the fast.h and gpu/primitives.cpp conflicts, and regenerate CORE_RESIDUAL.diff. See docs/mlxcelverse/mlx-pin-2026-11-06.md. * fix(rocm): use upstream's template_arguments_hash instead of the fork copy Upstream 4ca8d5d5a (ml-explore/mlx#4568) declares fast::template_arguments_hash in mlx/backend/common/custom_kernel.h, which mlx/fast.h includes. The fork keeps a private copy in mlx/backend/rocm/custom_kernel.cpp, so the unqualified call there matches both declarations and MSVC rejects it as ambiguous (C2668) when the ROCm backend and the common library are built together. Remove the fork copy; the upstream one also maps '-' to '_' in integer arguments so a negative template argument cannot produce an invalid kernel name. Recorded as LOCAL_FIXES item 73. * fix(rocm): adapt the backend to upstream custom-kernel and reshape signatures Upstream 4ca8d5d5a (ml-explore/mlx#4568) added a generated-source hash to fast::CustomKernel's constructor, and the fork's mlx/backend/rocm/custom_kernel.cpp still passed grid as the fourth argument. Pass std::hash<std::string>{}(kernel_source), as the CUDA backend does; the ROCm JIT module keeps keying on the source itself (LOCAL_FIXES item 33). Upstream 5ee1e8edf (ml-explore/mlx#4711) added add_out_to_temporary_on_allocation to reshape_gpu, which the shared gpu/primitives.cpp overlay now calls with four arguments. The fork's definition in mlx/backend/rocm/copy.hip takes the flag and registers the output as a temporary when it is set, matching CUDA. The first was a compile error on MSVC; the second would have been a link error on any ROCm build. Recorded as LOCAL_FIXES items 73 and 74.
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CustomKernel now gets an optional precomputed source hash instead of hashing
the full source per dispatch
Found with a Time Profiler capture, M5 Ultra, Qwen 3.8 Flash-Next. 14%
of host thread time was spent in this path
Effect: graph build goes from 1.42->1.21ms per round.
Tests included.
Benchmark (M5 Ultra, N=10000 calls, best of 5 per run, main and PR interleaved; per call, build + eval in batches of 100)