Skip to content

Upstream correctness fixes since v0.32.4, and opt-in 64 x 64 affine qmm tiles on GPUs without NAX (numen-tech/gemma4-qat#1694, #1695) - #31

Merged
jeethu merged 16 commits into
prism-v0.32.4-fixesfrom
tiles-1695
Nov 16, 2026
Merged

jeethu merged 16 commits into
prism-v0.32.4-fixesfrom
tiles-1695

Conversation

@jeethu

@jeethu jeethu commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Upstream correctness fixes since v0.32.4 (numen-tech/gemma4-qat#1694) and the 64 x 64 affine qmm tiles for GPUs without NAX, behind a switch that is off by default (numen-tech/gemma4-qat#1695, a port of ml-explore#4826). The base is prism-v0.32.4-fixes at 22568bd.

Cherry-picks (one commit each, upstream hash in the message)

Upstream Commit What it fixes Fork overlap Numerics Reachable from gemma4-qat
ml-explore#4653 (5c582abc5) 07ed619 conv2d out-of-bounds reads for partial output-channel tiles none: the fork does not touch the conv kernels outputs in range are unchanged. The bug was out-of-bounds reads, not wrong values conv2d with an output-channel count that is not a tile multiple: the Gemma 4 audio subsample convs (by grep)
ml-explore#4744 (afc03c511) 41615bf GPU pad for outputs over 2^31 elements none unchanged below 2^31 no: no tensor of ours is that large. Taken to match upstream
ml-explore#4750 (8bbef7591) 6f31adb flipped bounds check in the Metal large-filter conv loader none only flipped convs change not by grep: we run no flipped convolution
ml-explore#4430 (c5afbf392) 6aebef5 strided_scan reads and writes out of bounds none the strided scan only not by grep: our cumsum calls (the samplers, Gemma4Unified) scan the contiguous last axis, which never takes the strided kernel
ml-explore#4758 (df46f5842) cd3b4d8 GPU concatenate for outputs over 2^31 elements none unchanged below 2^31 no. Taken to match upstream
ml-explore#4775 (32a183e12) a315e02 Metal/CUDA affine quantize of all-negative groups. w_max started at 0, so the scale covered [min, 0], and 1-bit was wrong none: the fork's quantize changes are in other kernels all-negative groups get their own range, so a finer step. All other groups are unchanged yes: 8-bit KV quantization (QuantizedKVCache.updateQuantized / maybeAffineQuantizeKVCache in mlx-swift-lm) quantizes live K/V on the GPU in groups of 64, and a group of K can be all negative. A test of that case is added below
ml-explore#4764 (93007e9a8) 9835ed5 gguf tensor to f16 size overflow none n/a no: we load no gguf. It applies cleanly, so it is taken
ml-explore#4769 (49366677f) db561a5 Metal addmm epilogue precision: alpha * acc + beta * c is now done in the accumulator type none fp16/bf16 addmm results change, toward the fp32 result yes: every Linear with a bias (addMM), i.e. the Gemma 4 audio tower, the 12B unified model's Linear(bias: true) layers, and the Qwen3-VL vision blocks in mlx-swift-lm. Gemma text has no biases
ml-explore#4803 (a159fb8f6) 46c7b5c the CPU affine qmm accumulates in fp32 conflict with the fork's implied-bias CPU branch, resolved by hand: the implied branch materializes the bias and now calls _qmm_dispatch with the same fp32 accumulator (noted in the commit) CPU-stream fp16/bf16 quantized matmuls change, toward the fp32 result CPU-stream quantizedMM in tests and oracle tools only. The sweep's CPU references dequantize to fp32 and use matmul, so they are not affected

Declined: ml-explore#4568 (cached custom-kernel source and hash). It conflicts with the fork's rewritten custom-kernel path: metal_kernel.cpp became custom_kernel.cpp upstream, and the fork's VariantCache (numen-tech/gemma4-qat#189: 158d0ce, 713bf19, bb8fa5f) already builds and hashes each variant once per process. The fork already has its benefit, and it is not a correctness fix.

The 64 x 64 tiles (measurement form)

MLX_QMM_TILE64=1, read with getenv on every qmm as the fork's other switches are. Unset means today's route, bit for bit. When it is set, qmm() runs a transposed, non-batched 4/8-bit affine qmm at BM = BN = 64 (BK 32) when all of these hold, as in upstream's ml-explore#4826 gate:

  • the GPU has no NAX;
  • M >= 512 (4-bit) or M >= 2048 (8-bit);
  • either ceil(M/32) is even, or M is past the padding bar (256 rows below generation 15, 2,048 from 15 on; an Ultra part always uses 2,048);
  • ceil(N/64) * ceil(M/64) >= 160 (1,024 on an Ultra part).

Kernel names gain _bm_64_bn_64 before _ib, so a kernel trace can tell the routes apart. The precompiled library instantiates 72 kernels (fp32/fp16/bf16 x g32/64/128 x 4/8-bit x aligned or not, each with an implied-bias twin). JIT builds get the same template parameters.

How it composes with the other routes:

  • Only qmm() changes. A NAX GPU returns into qmm_nax before the gate. Apple6 (A13) never reaches qmm(), because eval_gpu routes it to qgemm_simt or the vector kernels.
  • qmv, qmv_quad, qmv_wide and qmv_wide_rr act below the qmv batch limit, far under 512 rows.
  • Split-K is unchanged: at 512 rows its split_k comes to 1 for N >= 1024 and it hands the call to qmm().
  • The split-K fall-through of numen-tech/gemma4-qat#1651 is NAX only.
  • The A18 (g17p, NAX kept off by the app's policy) takes the tiles. So would a NAX GPU under MLX_DISABLE_NAX=1.
  • aligned is now tested against the tile's N (N % bn), because the aligned loader skips its N bound.

Threadgroup memory at 64 x 64: 10 KiB fp16/bf16, 18 KiB fp32. At 32 x 32 it is 5 / 9 KiB.

There is no version constant and no default change in this form. The shipping form, if the measurements support it, will be a second commit.

Tests

  • python/tests/test_quantized.py:
    • test_qmm_tile64: shapes around both bars and the padding and aligned edges, at g32/64/128, fp32/fp16/bf16, 4 and 8 bits, explicit and implied bias. Each run is checked against the dequantized fp32 reference. Implied bias must equal the materialized bias bitwise, and below the bars both settings must give identical bits.
    • test_affine_quantize_all_negative_kv_group_8bit: an fp16 [1, 2, 8, 256] K tensor in [-3, -2.9], quantized in groups of 64 at 8 bits on the CPU and the GPU. Each group's dequantized error must be at most half its own step plus fp16 rounding, and the GPU must agree with the CPU.
  • tests/metal_gpu_family_tests.cpp:
    • a kernel-trace case: the 64 x 64 kernel with the switch on, never with it off, never on NAX or Apple6, and not at this size on an Ultra part. It restores the caller's MLX_QMM_TILE64 and stops the trace on every exit;
    • the Apple6 MMA kernel list gains the new name.

Run on the M3 Max: C++ 330/330 (after the review fix f37bc70 too, and the trace case also under MLX_FORCE_GPU_FAMILY=apple6), test_quantized.py 82 passed (1 skipped), conv/ops/blas/conv_transpose/load/apple6_route/fast 288 passed (19 skipped).

Measurements (numen-tech/gemma4-qat#1695)

Full write-up: swift/Tests/NumenCoreTests/Resources/measurements/qmm-tiles-64-1695.md on gemma4-qat branch tiles-1695.

  • Kernel: 64 x 64 batch time over 32 x 32, geomean over the tile cells. The tile and stock outputs are numerically identical in every cell.

    GPU geomean
    A17 Pro 0.89
    M4 0.90
    A16 0.92
    M3 Max 0.95 (the 1312-column unaligned shape loses up to 1.44)
    A14 1.00 (fp16 at 544 rows 1.04, fp32 1.10)
  • Prefill end to end, ~1,500-token prompts:

    device model prefill change [95 % CI]
    15 Pro E2B +6.9 % [+5.9, +8.0]
    M4 iPad E4B +6.5 % [+5.9, +7.2]
    iPhone 15 E2B +4.1 % [+3.4, +4.9]
    12 Pro Max E2B +0.1 % [-0.2, +0.3]
    Mac E4B +9.5 %
    Mac MiniCPM5 1B -5.3 %

    Decode is unchanged, and greedy text is identical in every pair.

  • Unaffected routes: the M5 (NAX) and the A13 (Apple6) never take the tiles, and their outputs are bit-identical with the switch on.

  • Default: the results are mixed, so this PR keeps the switch off by default with no version constant.

navthings and others added 14 commits November 15, 2026 19:33
…ml-explore#4750)

Co-authored-by: Cheng <git@zcbenz.com>
(cherry picked from commit 8bbef75)
Co-authored-by: Rohith Kapelli <kapellirohith@gmail.com>
(cherry picked from commit c5afbf3)
…plore#4775)

Co-authored-by: Garry Ling <4526646+lgyStoic@users.noreply.github.com>
(cherry picked from commit 32a183e)
…re#4803)

(cherry picked from commit a159fb8)

Conflict resolved by hand: the fork's implied-bias branch of QuantizedMatmul::eval_cpu (gemma4-qat#190) takes the same fp32 row accumulator and passes it to _qmm_dispatch.
…ithout NAX (port of ml-explore#4826, numen-tech/gemma4-qat#1695)

A measurement switch, off by default: unset, or any value other than "1",
keeps today's 32 x 32 kernels bit for bit. With MLX_QMM_TILE64=1 (read on
every call, so one process can A/B both routes), qmm() takes the BM 64 /
BK 32 / BN 64 instantiation of affine_qmm_t when:

- the GPU has no NAX (metal::is_nax_available() is false; qmm() returns into
  qmm_nax before this on a NAX GPU, so no NAX route changes);
- the product is affine (not affine_sym, not the fp modes), transposed,
  non-batched (B == 1), without a global scale, 4- or 8-bit;
- M >= 512 at 4 bits, M >= 2048 at 8 bits (upstream's thresholds);
- upstream's fp4 tile rule (ml-explore#4709) holds: at least 160
  64 x 64 output tiles (1024 on an Ultra, architecture suffix 'd'), and
  ceil(M / 32) even or M > 256 (M > 2048 from generation 15 and on an Ultra).

The kernel body is the fork's affine_qmm_t, unchanged: it already takes
BM/BK/BN template arguments, and the 32 x 32 code is upstream's at the same
revision. BK stays 32. The kernel name gains `_bm_64_bn_64` before `_ib`, so a
kernel trace tells the routes apart, and aligned_N is judged against the tile
width (N % 64), because the aligned weight loader skips its N bound. The
precompiled library carries the 72 new kernels (float, float16, bfloat16 x gs
32/64/128 x 4/8-bit x aligned or not x explicit or implied bias); the JIT
builds them from the same template.

Composition with the fork's routes: Apple6 never reaches qmm() (SIMT GEMM or
the vector kernels); qmv, qmv_quad, qmv_wide and qmv_wide_rr act below the qmv
batch limit; split-K keeps its choice, and at 512 rows hands every N >= 1024
to qmm(), which is the case this acts on; the ml-explore#1651 split-K fall-through is
NAX-only. Threadgroup memory per 64 x 64 threadgroup: 10 KiB for fp16/bf16,
18 KiB for fp32 (5 / 9 KiB at 32 x 32).

No version constant yet: one is set if the tiles ship.
test_quantized.py::test_qmm_tile64: 4-bit M 511-640 and 8-bit M 2047/2049,
N 1280 / 1281 / 1312 (32- but not 64-aligned), gs 32/64/128, fp32/fp16/bf16,
switch on and off, against the dequantized reference; implied bias equals the
materialized one bit for bit; bit-equal routes below the thresholds.
metal_gpu_family_tests.cpp: the kernel trace shows the _bm_64_bn_64 kernel with
the switch on (GPU without NAX only), never with it off; the Apple6 MMA name
list carries a 64 x 64 name.
…men-tech/gemma4-qat#1694)

An 8-bit KV cache quantizes K and V on the GPU in groups of 64 along the head
dim; ml-explore#4775 fixed groups that are all negative, which used to take
0 as their max. The test quantizes an fp16 [1, 2, 8, 256] K tensor in
[-3, -2.9] on the CPU and the GPU and checks each group's dequantized error
against half its own step, and the GPU against the CPU.
…threadgroups; an Ultra-sized shape; the K/V test skips without Metal (review of numen-tech/gemma4-qat#1694/ml-explore#1695)
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T07:05:32.263968Z 12ce20c New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 81e2731f7f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread mlx/backend/metal/quantized.cpp Outdated
Comment on lines +170 to +174
// 64 x 64 tiles for large affine qmm on a GPU without NAX (the port of
// ml-explore/mlx#4826, numen-tech/gemma4-qat#1695): a transposed, non-batched
// 4-bit qmm of M >= 512 rows or 8-bit of M >= 2048 rows takes the BM 64 / BK 32
// / BN 64 instantiation of affine_qmm_t (kernel names end `_bm_64_bn_64`)
// where the grid has enough tiles to fill the GPU: at least 160 64 x 64 tiles

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Shorten the tile-selection commentary

The 13-line introduction repeats the thresholds and conditions implemented immediately below and adds upstream history and routing details. AGENTS.md requires concise code comments and prohibits redundant or excessive inline commentary. Reduce this block to a short explanation of the padding and occupancy rationale rather than restating the implementation.

AGENTS.md reference: AGENTS.md:L31-L32

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f37bc70: the comment above use_qmm_tile64 is now three lines (what it is, the switch, where the occupancy rule comes from). The gate itself is unchanged.

Comment thread tests/metal_gpu_family_tests.cpp Outdated
auto& d = metal::device(mlx::core::Device::gpu);
bool ultra = d.get_architecture().back() == 'd';
CHECK_FALSE(has_tile64(names_with(false)));
CHECK(has_tile64(names_with(true)) == (!is_nax_available() && !ultra));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Exclude Apple6 from the expected tile64 routes

On an A13, or when running Metal tests with MLX_FORCE_GPU_FAMILY=apple6, this assertion expects a tile64 kernel because NAX is unavailable and the architecture is not Ultra. However, QuantizedMatmul::eval_gpu routes this supported shape through qgemm_simt and returns before reaching qmm, regardless of MLX_QMM_TILE64. The new test therefore fails on a supported route even when dispatch is correct. Include !d.is_apple6() in the expected condition and verify that Apple6 never selects tile64.

Useful? React with 👍 / 👎.

@jeethu jeethu Nov 16, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f37bc70: the expectation is now !is_nax_available() && !d.is_apple6() && !ultra. Checked on the M3 Max with MLX_FORCE_GPU_FAMILY=apple6 (the case passes: no _bm_64_bn_64 kernel in the trace) and without it (full C++ suite 330/330).

Review of #31: the tile-selection comment repeated the
code and the PR; the kernel-trace test expected tiles on an Apple6 GPU
(or under MLX_FORCE_GPU_FAMILY=apple6), which routes to qgemm_simt.
Checked with MLX_FORCE_GPU_FAMILY=apple6 on the M3 Max.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f37bc70748

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread python/tests/test_quantized.py Outdated
Comment on lines +805 to +809
# 64 x 64 affine qmm tiles on a GPU without NAX (ml-explore/mlx#4826,
# numen-tech/gemma4-qat#1695), behind MLX_QMM_TILE64=1 (read on every
# call). Both routes must match the dequantized reference, and the
# implied bias must equal the materialized one bit for bit. The shapes
# cover a tile-aligned N, an N that is 32- but not 64-aligned (the

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Shorten the tile64 test commentary

The 12-line introduction repeats the assertions, shape cases, switch behavior, and upstream history that are already visible in the test. AGENTS.md requires concise comments and says to avoid redundant or excessive inline commentary. Reduce this block to a short explanation of the non-obvious boundary cases, such as why N must include a 32-aligned but not 64-aligned value.

AGENTS.md reference: AGENTS.md:L31-L32

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 12ce20c: the test_qmm_tile64 introduction is now two lines, on the one non-obvious case (N 1312 is 32- but not 64-aligned, so the aligned loader must not be picked for the 64-wide tile). The same commit trims the K/V test, the C++ trace test and the quantized.cpp template-argument comments. Comments only; test_quantized.py ran 82 passed, 1 skipped, and the C++ tile and Apple6 cases pass.

jeethu added a commit to numen-tech/mlx-swift that referenced this pull request Nov 16, 2026
The Apple6 route (an A13, or MLX_FORCE_GPU_FAMILY=apple6) runs
qgemm_simt and never reaches qmm(), as the fork's C++ test now expects
(numen-tech/mlx#31 review). The check mirrors Device::is_apple6().
Review of #31: the test introductions repeated their
assertions, shapes and history. Comments only.
@jeethu
jeethu merged commit fc7bf14 into prism-v0.32.4-fixes Nov 16, 2026
@jeethu
jeethu deleted the tiles-1695 branch November 16, 2026 07:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants