Skip to content

[Metal] Use 64x64 tiles for large affine QMM - #4826

Open
0xamlab wants to merge 2 commits into
ml-explore:mainfrom
0xamlab:metal-affine-qmm-64x64
Open

0xamlab wants to merge 2 commits into
ml-explore:mainfrom
0xamlab:metal-affine-qmm-64x64

Conversation

@0xamlab

@0xamlab 0xamlab commented Nov 3, 2026 •

Copy link
Copy Markdown

Fixes #4621. Large transposed affine QMM leaves performance on the table with the existing tile path. This adds 64x64 instantiations for 4-bit and 8-bit weights and selects them for non-NAX hardware with B == 1 when the M threshold and the existing padding and threadgroup-count gates pass: M >= 512 for 4-bit and M >= 2048 for 8-bit.

NAX, small-M, batched, non-transposed, and split-K dispatch are unchanged. The added 90-subtest regression passes across exact and partial tiles, group sizes 32/64/128, both bit widths, and float32/float16/bfloat16.

On an Apple M4 Max with bfloat16 activations, N = 12288, K = 4096, and group size 64, every measured shape has lower median latency on this branch. Main is c69303b00f5f61860436e832f23fa8c34b96f93a; each value is the median of three process-level medians, with 20 measured calls after three warmups and alternating main/PR order.

M Bits Main (ms) PR (ms) Reduction
512 4 5.894 4.465 24.2%
2048 4 28.483 17.618 38.1%
2048 8 19.422 17.519 9.8%
8192 4 164.584 155.337 5.6%
8192 8 160.275 133.895 16.5%

Run this three times on main and this branch, alternating order:

import statistics
import time
import mlx.core as mx

K, N, GROUP = 4096, 12288, 64
mx.random.seed(0)
w = (mx.random.normal((N, K)) * 0.02).astype(mx.bfloat16)
for bits in (4, 8):
    wq, scales, biases = mx.quantize(w, group_size=GROUP, bits=bits)
    for M in (512, 2048, 8192):
        x = mx.random.normal((1, M, K)).astype(mx.bfloat16)
        fn = lambda: mx.quantized_matmul(
            x, wq, scales, biases, transpose=True,
            group_size=GROUP, bits=bits,
        )
        for _ in range(3):
            mx.eval(fn())
        samples = []
        for _ in range(20):
            start = time.perf_counter()
            mx.eval(fn())
            samples.append((time.perf_counter() - start) * 1000)
        print(M, bits, f"{statistics.median(samples):.3f} ms")
  • ☑️ I understand it is strictly prohibited to use AI to write PR description
  • AI usage disclosure: Code and PR description prepared with AI assistance; the author reviewed and validated the changes and this description.

@nastya236

Copy link
Copy Markdown
Collaborator

Thanks for your contribution! Is there a specific model you’re optimising this kernel for?

@nastya236 nastya236 added the await response This pull request is waiting for response from the author. label Nov 17, 2026
@0xamlab

0xamlab commented Nov 17, 2026

Copy link
Copy Markdown
Author

Not a specific model. This targets the affine QMM kernel itself for large-M workloads on Apple Silicon. I measured five M4 Max configurations with 4/8-bit weights and BF16 activations, including M=2048 at 4-bit. I have not claimed an end-to-end model speedup; that would need a separate serving benchmark.

@zcbenz zcbenz added await verification This pull request is non-trivial and requires a human expert to verify its correctness. and removed await response This pull request is waiting for response from the author. labels Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

await verification This pull request is non-trivial and requires a human expert to verify its correctness.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

quantized_matmul takes 14- 41% longer than dequantize + bf16 matmul at prefill sizes (> 512 tokens)

3 participants