Repository navigation
Upstream correctness fixes since v0.32.4, and opt-in 64 x 64 affine qmm tiles on GPUs without NAX (numen-tech/gemma4-qat#1694, #1695) - #31
Conversation
(cherry picked from commit 5c582ab)
(cherry picked from commit afc03c5)
…ml-explore#4750) Co-authored-by: Cheng <git@zcbenz.com> (cherry picked from commit 8bbef75)
Co-authored-by: Rohith Kapelli <kapellirohith@gmail.com> (cherry picked from commit c5afbf3)
…plore#4775) Co-authored-by: Garry Ling <4526646+lgyStoic@users.noreply.github.com> (cherry picked from commit 32a183e)
(cherry picked from commit 93007e9)
(cherry picked from commit 4936667)
…ithout NAX (port of ml-explore#4826, numen-tech/gemma4-qat#1695) A measurement switch, off by default: unset, or any value other than "1", keeps today's 32 x 32 kernels bit for bit. With MLX_QMM_TILE64=1 (read on every call, so one process can A/B both routes), qmm() takes the BM 64 / BK 32 / BN 64 instantiation of affine_qmm_t when: - the GPU has no NAX (metal::is_nax_available() is false; qmm() returns into qmm_nax before this on a NAX GPU, so no NAX route changes); - the product is affine (not affine_sym, not the fp modes), transposed, non-batched (B == 1), without a global scale, 4- or 8-bit; - M >= 512 at 4 bits, M >= 2048 at 8 bits (upstream's thresholds); - upstream's fp4 tile rule (ml-explore#4709) holds: at least 160 64 x 64 output tiles (1024 on an Ultra, architecture suffix 'd'), and ceil(M / 32) even or M > 256 (M > 2048 from generation 15 and on an Ultra). The kernel body is the fork's affine_qmm_t, unchanged: it already takes BM/BK/BN template arguments, and the 32 x 32 code is upstream's at the same revision. BK stays 32. The kernel name gains `_bm_64_bn_64` before `_ib`, so a kernel trace tells the routes apart, and aligned_N is judged against the tile width (N % 64), because the aligned weight loader skips its N bound. The precompiled library carries the 72 new kernels (float, float16, bfloat16 x gs 32/64/128 x 4/8-bit x aligned or not x explicit or implied bias); the JIT builds them from the same template. Composition with the fork's routes: Apple6 never reaches qmm() (SIMT GEMM or the vector kernels); qmv, qmv_quad, qmv_wide and qmv_wide_rr act below the qmv batch limit; split-K keeps its choice, and at 512 rows hands every N >= 1024 to qmm(), which is the case this acts on; the ml-explore#1651 split-K fall-through is NAX-only. Threadgroup memory per 64 x 64 threadgroup: 10 KiB for fp16/bf16, 18 KiB for fp32 (5 / 9 KiB at 32 x 32). No version constant yet: one is set if the tiles ship.
test_quantized.py::test_qmm_tile64: 4-bit M 511-640 and 8-bit M 2047/2049, N 1280 / 1281 / 1312 (32- but not 64-aligned), gs 32/64/128, fp32/fp16/bf16, switch on and off, against the dequantized reference; implied bias equals the materialized one bit for bit; bit-equal routes below the thresholds. metal_gpu_family_tests.cpp: the kernel trace shows the _bm_64_bn_64 kernel with the switch on (GPU without NAX only), never with it off; the Apple6 MMA name list carries a 64 x 64 name.
…men-tech/gemma4-qat#1694) An 8-bit KV cache quantizes K and V on the GPU in groups of 64 along the head dim; ml-explore#4775 fixed groups that are all negative, which used to take 0 as their max. The test quantizes an fp16 [1, 2, 8, 256] K tensor in [-3, -2.9] on the CPU and the GPU and checks each group's dequantized error against half its own step, and the GPU against the CPU.
…threadgroups; an Ultra-sized shape; the K/V test skips without Metal (review of numen-tech/gemma4-qat#1694/ml-explore#1695)
…ace on every exit (review of numen-tech/gemma4-qat#1695)
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 81e2731f7f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // 64 x 64 tiles for large affine qmm on a GPU without NAX (the port of | ||
| // ml-explore/mlx#4826, numen-tech/gemma4-qat#1695): a transposed, non-batched | ||
| // 4-bit qmm of M >= 512 rows or 8-bit of M >= 2048 rows takes the BM 64 / BK 32 | ||
| // / BN 64 instantiation of affine_qmm_t (kernel names end `_bm_64_bn_64`) | ||
| // where the grid has enough tiles to fill the GPU: at least 160 64 x 64 tiles |
There was a problem hiding this comment.
Shorten the tile-selection commentary
The 13-line introduction repeats the thresholds and conditions implemented immediately below and adds upstream history and routing details. AGENTS.md requires concise code comments and prohibits redundant or excessive inline commentary. Reduce this block to a short explanation of the padding and occupancy rationale rather than restating the implementation.
AGENTS.md reference: AGENTS.md:L31-L32
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in f37bc70: the comment above use_qmm_tile64 is now three lines (what it is, the switch, where the occupancy rule comes from). The gate itself is unchanged.
| auto& d = metal::device(mlx::core::Device::gpu); | ||
| bool ultra = d.get_architecture().back() == 'd'; | ||
| CHECK_FALSE(has_tile64(names_with(false))); | ||
| CHECK(has_tile64(names_with(true)) == (!is_nax_available() && !ultra)); |
There was a problem hiding this comment.
Exclude Apple6 from the expected tile64 routes
On an A13, or when running Metal tests with MLX_FORCE_GPU_FAMILY=apple6, this assertion expects a tile64 kernel because NAX is unavailable and the architecture is not Ultra. However, QuantizedMatmul::eval_gpu routes this supported shape through qgemm_simt and returns before reaching qmm, regardless of MLX_QMM_TILE64. The new test therefore fails on a supported route even when dispatch is correct. Include !d.is_apple6() in the expected condition and verify that Apple6 never selects tile64.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in f37bc70: the expectation is now !is_nax_available() && !d.is_apple6() && !ultra. Checked on the M3 Max with MLX_FORCE_GPU_FAMILY=apple6 (the case passes: no _bm_64_bn_64 kernel in the trace) and without it (full C++ suite 330/330).
Review of #31: the tile-selection comment repeated the code and the PR; the kernel-trace test expected tiles on an Apple6 GPU (or under MLX_FORCE_GPU_FAMILY=apple6), which routes to qgemm_simt. Checked with MLX_FORCE_GPU_FAMILY=apple6 on the M3 Max.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f37bc70748
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # 64 x 64 affine qmm tiles on a GPU without NAX (ml-explore/mlx#4826, | ||
| # numen-tech/gemma4-qat#1695), behind MLX_QMM_TILE64=1 (read on every | ||
| # call). Both routes must match the dequantized reference, and the | ||
| # implied bias must equal the materialized one bit for bit. The shapes | ||
| # cover a tile-aligned N, an N that is 32- but not 64-aligned (the |
There was a problem hiding this comment.
Shorten the tile64 test commentary
The 12-line introduction repeats the assertions, shape cases, switch behavior, and upstream history that are already visible in the test. AGENTS.md requires concise comments and says to avoid redundant or excessive inline commentary. Reduce this block to a short explanation of the non-obvious boundary cases, such as why N must include a 32-aligned but not 64-aligned value.
AGENTS.md reference: AGENTS.md:L31-L32
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 12ce20c: the test_qmm_tile64 introduction is now two lines, on the one non-obvious case (N 1312 is 32- but not 64-aligned, so the aligned loader must not be picked for the 64-wide tile). The same commit trims the K/V test, the C++ trace test and the quantized.cpp template-argument comments. Comments only; test_quantized.py ran 82 passed, 1 skipped, and the C++ tile and Apple6 cases pass.
The Apple6 route (an A13, or MLX_FORCE_GPU_FAMILY=apple6) runs qgemm_simt and never reaches qmm(), as the fork's C++ test now expects (numen-tech/mlx#31 review). The check mirrors Device::is_apple6().
Review of #31: the test introductions repeated their assertions, shapes and history. Comments only.
Upstream correctness fixes since v0.32.4 (numen-tech/gemma4-qat#1694) and the 64 x 64 affine qmm tiles for GPUs without NAX, behind a switch that is off by default (numen-tech/gemma4-qat#1695, a port of ml-explore#4826). The base is
prism-v0.32.4-fixesat 22568bd.Cherry-picks (one commit each, upstream hash in the message)
5c582abc5)afc03c511)padfor outputs over 2^31 elements8bbef7591)c5afbf392)strided_scanreads and writes out of boundscumsumcalls (the samplers,Gemma4Unified) scan the contiguous last axis, which never takes the strided kerneldf46f5842)concatenatefor outputs over 2^31 elements32a183e12)w_maxstarted at 0, so the scale covered[min, 0], and 1-bit was wrongQuantizedKVCache.updateQuantized/maybeAffineQuantizeKVCachein mlx-swift-lm) quantizes live K/V on the GPU in groups of 64, and a group of K can be all negative. A test of that case is added below93007e9a8)49366677f)addmmepilogue precision:alpha * acc + beta * cis now done in the accumulator typeaddmmresults change, toward the fp32 resultLinearwith a bias (addMM), i.e. the Gemma 4 audio tower, the 12B unified model'sLinear(bias: true)layers, and the Qwen3-VL vision blocks in mlx-swift-lm. Gemma text has no biasesa159fb8f6)_qmm_dispatchwith the same fp32 accumulator (noted in the commit)quantizedMMin tests and oracle tools only. The sweep's CPU references dequantize to fp32 and usematmul, so they are not affectedDeclined: ml-explore#4568 (cached custom-kernel source and hash). It conflicts with the fork's rewritten custom-kernel path:
metal_kernel.cppbecamecustom_kernel.cppupstream, and the fork'sVariantCache(numen-tech/gemma4-qat#189: 158d0ce, 713bf19, bb8fa5f) already builds and hashes each variant once per process. The fork already has its benefit, and it is not a correctness fix.The 64 x 64 tiles (measurement form)
MLX_QMM_TILE64=1, read withgetenvon every qmm as the fork's other switches are. Unset means today's route, bit for bit. When it is set,qmm()runs a transposed, non-batched 4/8-bit affine qmm at BM = BN = 64 (BK 32) when all of these hold, as in upstream's ml-explore#4826 gate:ceil(M/32)is even, or M is past the padding bar (256 rows below generation 15, 2,048 from 15 on; an Ultra part always uses 2,048);ceil(N/64) * ceil(M/64)>= 160 (1,024 on an Ultra part).Kernel names gain
_bm_64_bn_64before_ib, so a kernel trace can tell the routes apart. The precompiled library instantiates 72 kernels (fp32/fp16/bf16 x g32/64/128 x 4/8-bit x aligned or not, each with an implied-bias twin). JIT builds get the same template parameters.How it composes with the other routes:
qmm()changes. A NAX GPU returns intoqmm_naxbefore the gate. Apple6 (A13) never reachesqmm(), becauseeval_gpuroutes it toqgemm_simtor the vector kernels.split_kcomes to 1 for N >= 1024 and it hands the call toqmm().g17p, NAX kept off by the app's policy) takes the tiles. So would a NAX GPU underMLX_DISABLE_NAX=1.alignedis now tested against the tile's N (N % bn), because the aligned loader skips its N bound.Threadgroup memory at 64 x 64: 10 KiB fp16/bf16, 18 KiB fp32. At 32 x 32 it is 5 / 9 KiB.
There is no version constant and no default change in this form. The shipping form, if the measurements support it, will be a second commit.
Tests
python/tests/test_quantized.py:test_qmm_tile64: shapes around both bars and the padding and aligned edges, at g32/64/128, fp32/fp16/bf16, 4 and 8 bits, explicit and implied bias. Each run is checked against the dequantized fp32 reference. Implied bias must equal the materialized bias bitwise, and below the bars both settings must give identical bits.test_affine_quantize_all_negative_kv_group_8bit: an fp16[1, 2, 8, 256]K tensor in[-3, -2.9], quantized in groups of 64 at 8 bits on the CPU and the GPU. Each group's dequantized error must be at most half its own step plus fp16 rounding, and the GPU must agree with the CPU.tests/metal_gpu_family_tests.cpp:MLX_QMM_TILE64and stops the trace on every exit;Run on the M3 Max: C++ 330/330 (after the review fix f37bc70 too, and the trace case also under
MLX_FORCE_GPU_FAMILY=apple6),test_quantized.py82 passed (1 skipped), conv/ops/blas/conv_transpose/load/apple6_route/fast 288 passed (19 skipped).Measurements (numen-tech/gemma4-qat#1695)
Full write-up:
swift/Tests/NumenCoreTests/Resources/measurements/qmm-tiles-64-1695.mdon gemma4-qat branchtiles-1695.Kernel: 64 x 64 batch time over 32 x 32, geomean over the tile cells. The tile and stock outputs are numerically identical in every cell.
Prefill end to end, ~1,500-token prompts:
Decode is unchanged, and greedy text is identical in every pair.
Unaffected routes: the M5 (NAX) and the A13 (Apple6) never take the tiles, and their outputs are bit-identical with the switch on.
Default: the results are mixed, so this PR keeps the switch off by default with no version constant.