Skip to content

Cache generated source code and hash in custom kernels - #4568

Merged
zcbenz merged 9 commits into
ml-explore:mainfrom
STRML:metal-kernel-source-cache
Oct 31, 2026
Merged

zcbenz merged 9 commits into
ml-explore:mainfrom
STRML:metal-kernel-source-cache

Conversation

@STRML

@STRML STRML commented Sep 26, 2026

Copy link
Copy Markdown
Contributor
  • ☑️ I understand it is strictly prohibited to use AI to write PR description
  • AI usage disclosure: was used to help identify the solution, not to write the commit

CustomKernel now gets an optional precomputed source hash instead of hashing
the full source per dispatch

Found with a Time Profiler capture, M5 Ultra, Qwen 3.8 Flash-Next. 14%
of host thread time was spent in this path

Effect: graph build goes from 1.42->1.21ms per round.

Tests included.

Benchmark (M5 Ultra, N=10000 calls, best of 5 per run, main and PR interleaved; per call, build + eval in batches of 100)

kernel body main (3 runs) PR (3 runs) change
300 lines, 19 KB 5.10 / 4.98 / 5.01 us 4.67 / 4.71 / 4.66 us ~ -7%
2000 lines, 130 KB 26.16 / 25.72 / 25.80 us 23.57 / 22.96 / 23.84 us ~ -9%
  • graph build alone: 300-line 1.05 -> 0.92 us/call; 2000-line ~2.4 -> 2.1 us/call
  • most of the gain is removing the per-dispatch hash; each CustomKernel still copies the cached source string

CustomKernel now gets an optional precomputed source hash instead of hashing
the full source per dispatch

Found with a Time Profiler capture, M5 Ultra, Qwen 3.8 Flash-Next. 14%
of host thread time was spent in this path

Effect: graph build goes from 1.42->1.21ms per round.

Tests included.
@STRML STRML changed the title Metal kernel source cache Perf: Metal kernel source hash cache Sep 26, 2026
Comment thread mlx/fast_primitives.h Resolved
Comment thread mlx/backend/common/metal_kernel.cpp Resolved
Comment thread python/tests/test_fast.py Resolved
@zcbenz zcbenz added the await response This pull request is waiting for response from the author. label Oct 2, 2026
@zcbenz

zcbenz commented Oct 29, 2026

Copy link
Copy Markdown
Member

Closing due to lack of activity.

@zcbenz zcbenz closed this Oct 29, 2026
@STRML

STRML commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Sorry let me fix this PR. It is a solid perf boost.

STRML added 2 commits October 29, 2026 07:21
- CustomKernel::source_hash_ is a plain std::size_t. Callers that already
  have the hash pass it; the state()-order constructor used by import
  computes it from the source.
- GeneratedSourceCache stores GeneratedSource by value and returns a
  reference. Entries are never erased, so the reference stays valid.
- The new test only covers template args (int 1 vs bool true, same
  kernel name, different signature).
@zcbenz zcbenz reopened this Oct 29, 2026
@zcbenz zcbenz removed the await response This pull request is waiting for response from the author. label Oct 29, 2026
@STRML

STRML commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

@zcbenz all three comments are addressed in a4cd4a7 on the branch (main merged in; test_fast.py and test_export_import.py pass locally on Metal). GitHub won't let me reopen a PR a maintainer closed - could you reopen it?

@zcbenz zcbenz changed the title Perf: Metal kernel source hash cache Cache generated source code and hash in custom kernels Oct 31, 2026
@zcbenz
zcbenz merged commit 4ca8d5d into ml-explore:main Oct 31, 2026
24 checks passed
STRML added a commit to STRML/mlx-serve that referenced this pull request Oct 31, 2026
…l-explore/mlx#4568

700520ae (Metal expm1 argument reduction accuracy) and 33927952 (sharded-to-all bias and integer split positions on quantized layers). Neither touches the custom-kernel cache, the SDPA dispatch, the quantized kernels or the CUDA backend: the carried sdpa_split patch applies unchanged and the CUDA patch stack replays in Linux and Windows order. AGENTS.md names the new pin and the carried mlx patch.
inureyes added a commit to lablup/mlxcel that referenced this pull request Nov 6, 2026
… copy

Upstream 4ca8d5d5a (ml-explore/mlx#4568) declares fast::template_arguments_hash in mlx/backend/common/custom_kernel.h, which mlx/fast.h includes. The fork keeps a private copy in mlx/backend/rocm/custom_kernel.cpp, so the unqualified call there matches both declarations and MSVC rejects it as ambiguous (C2668) when the ROCm backend and the common library are built together. Remove the fork copy; the upstream one also maps '-' to '_' in integer arguments so a negative template argument cannot produce an invalid kernel name.

Recorded as LOCAL_FIXES item 73.
inureyes added a commit to lablup/mlxcel that referenced this pull request Nov 6, 2026
…gnatures

Upstream 4ca8d5d5a (ml-explore/mlx#4568) added a generated-source hash to fast::CustomKernel's constructor, and the fork's mlx/backend/rocm/custom_kernel.cpp still passed grid as the fourth argument. Pass std::hash<std::string>{}(kernel_source), as the CUDA backend does; the ROCm JIT module keeps keying on the source itself (LOCAL_FIXES item 33).

Upstream 5ee1e8edf (ml-explore/mlx#4711) added add_out_to_temporary_on_allocation to reshape_gpu, which the shared gpu/primitives.cpp overlay now calls with four arguments. The fork's definition in mlx/backend/rocm/copy.hip takes the flag and registers the output as a temporary when it is set, matching CUDA.

The first was a compile error on MSVC; the second would have been a link error on any ROCm build. Recorded as LOCAL_FIXES items 73 and 74.
inureyes added a commit to lablup/mlxcel that referenced this pull request Nov 6, 2026
* update: advance MLX pin to 9f806b36 and refresh backend overlays

Move the CMake pin from 7ca40033 (2026-10-26) to 9f806b36 (2026-11-05), a reproducible main snapshot 46 commits newer and 81 past stable v0.32.4, which the previous pin already postdated.

Rebase the 14 drifted CPU, Metal and CUDA whole-file overlays with a three-way merge. Upstream #4803 now accumulates the CPU affine quantized matmul in float, so only fp_qmm and fp_qmm_t stay local. The new Metal affine qmv_mma (#4787) is placed behind the MLXCEL_QMV_WIDE off-switch so the speculative exactness retry still restores byte-equality between a verify block and the single-token chain. Metal device.cpp keeps the input-budget override across upstream's env rework, and quantized.metal re-appends mlxcel_affine_qmm_t_rows after the restructured instantiation macros.

Two upstream API moves needed code changes: env::get_var is now config::get (#4647), used by the cxx bridge and the CUDA SDPA switches, and gpu_core_count() (#4781) needs IOKit on the macOS link line because upstream links it PRIVATE.

Retarget the ROCm core files with the fork unchanged, resolve the fast.h and gpu/primitives.cpp conflicts, and regenerate CORE_RESIDUAL.diff.

See docs/mlxcelverse/mlx-pin-2026-11-06.md.

* fix(rocm): use upstream's template_arguments_hash instead of the fork copy

Upstream 4ca8d5d5a (ml-explore/mlx#4568) declares fast::template_arguments_hash in mlx/backend/common/custom_kernel.h, which mlx/fast.h includes. The fork keeps a private copy in mlx/backend/rocm/custom_kernel.cpp, so the unqualified call there matches both declarations and MSVC rejects it as ambiguous (C2668) when the ROCm backend and the common library are built together. Remove the fork copy; the upstream one also maps '-' to '_' in integer arguments so a negative template argument cannot produce an invalid kernel name.

Recorded as LOCAL_FIXES item 73.

* fix(rocm): adapt the backend to upstream custom-kernel and reshape signatures

Upstream 4ca8d5d5a (ml-explore/mlx#4568) added a generated-source hash to fast::CustomKernel's constructor, and the fork's mlx/backend/rocm/custom_kernel.cpp still passed grid as the fourth argument. Pass std::hash<std::string>{}(kernel_source), as the CUDA backend does; the ROCm JIT module keeps keying on the source itself (LOCAL_FIXES item 33).

Upstream 5ee1e8edf (ml-explore/mlx#4711) added add_out_to_temporary_on_allocation to reshape_gpu, which the shared gpu/primitives.cpp overlay now calls with four arguments. The fork's definition in mlx/backend/rocm/copy.hip takes the flag and registers the output as a temporary when it is set, matching CUDA.

The first was a compile error on MSVC; the second would have been a link error on any ROCm build. Recorded as LOCAL_FIXES items 73 and 74.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants