Skip to content

Carry ml-explore/mlx#4787 affine qmv_mma behind MLX_QMV_MMA=1, off by default (numen-tech/gemma4-qat#1701) - #32

Merged
jeethu merged 5 commits into
prism-v0.32.4-fixesfrom
verify-routes-1701
Nov 17, 2026
Merged

jeethu merged 5 commits into
prism-v0.32.4-fixesfrom
verify-routes-1701

Conversation

@jeethu

@jeethu jeethu commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

This PR carries ml-explore#4787's affine_qmv_mma (numen-tech/gemma4-qat#1701), off by default. With neither switch on, nothing changes: no other route, tile or default moves. The owner's call (2026-11-17) is to carry it for Gemma 4 E4B behind a switch and measure it end to end. numen-tech/gemma4-qat#1816 turns it on only for an E4B load that asks.

The switch. Neither switch is cached; the core reads them only for a call that passes every shape test, so decode at M = 1 reads neither.

  • metal::set_qmv_mma(bool) (metal.h) is a process flag. A host that switches the route per model load calls it rather than setenv/unsetenv, which can race another thread's dispatch that is reading the variable.
  • MLX_QMV_MMA=1 still turns the route on, for measurement harnesses and tests.
  • The route is on while either switch is on. The flag is read first, so a host that uses it never reads the environment.
  • metal::qmv_mma_device() reports whether this GPU can take the route at all.
  • kQmvMmaVersion = 2 (device.h). Version 1 was the variable alone.

The route. It sends an affine 4- or 8-bit transposed quantized matmul to affine_qmv_mma when all of these hold:

  • M is 4-16 (3-16 below gen 15) and K % 128 == 0.
  • N and K are >= 512, N x K >= 2^21, and N % 16 == 0.
  • The product has biases, and x, w and scales sit at aligned offsets.
  • The GPU has no NAX, is not Apple6 and is not an Ultra.

It runs ahead of the qmv batch limit. One addition beyond upstream: an implied-bias (_ib) twin for Gemma's 0-d bias factor. Not carried: the fp-mode qmv_mma, and qmv_nax (ml-explore#4712).

Commits.

  • 7cd5bf1e1 / 5bc665b8b: the spike and its test.
  • 446091ab5: R1 only, with the tile-8 switch dropped (Widen affine qmv_wide row tiles to 8 ml-explore/mlx#4728 lost 7-13 % at M = 8).
  • 7aa006437: the gate's shape, alignment and dtype tests now run before the environment read, using strcmp (it used to getenv and build a std::string on every quantized matmul).
  • 0ed8a2ca0: the process flag and qmv_mma_device(). The kernel-trace declarations move to metal.h, and no_metal.cpp defines them and the two new functions. Before this, a build without Metal referenced trace symbols it did not define (Codex P1). The test now takes eligibility from the core.

Test. test_qmv_mma_switch runs 2/4/8 bits x gs 32/64/128 x M 1-16, each with explicit and implied bias. It checks that:

  • With neither switch on, no product takes the route.
  • With the variable on, the output is within 2e-3 of fp32.
  • The route is taken exactly where the core's device check, the bit width and the row floor allow.
  • On the route, the implied factor is bitwise equal to the materialized bias.
  • The flag takes the route with the variable unset, and turning it off releases it.

What ran on 0ed8a2ca0 (M3 Max, applegpu_g15s):

  • pytest test_quantized.py: 83 passed, 1 skipped, 12654 subtests.
  • The bit check at E4B's text linear shapes gave output identical, line for line, to the earlier carry's log:
    • Where R1 is taken (M 4-16), 0.2-0.6 % of output elements differ from the fork's routes at M 4-8, and about 40 % at M 16, by at most 0.004.
    • M 1-3 and the sliding k/v shape are bit-identical.
  • A CMake build with MLX_BUILD_METAL=OFF links the Python module, and libmlx.a defines the four functions.
  • Codex review, two rounds:
    • Round 1: one P1 (the no-Metal link), two P2s (environment mutation racing a dispatch; the test assumed no NAX) and one P3 (mlx-swift had not picked up the reorder). All four are fixed in the commits above.
    • Round 2 (the incremental diff): no findings.

Not run here: end-to-end measurement, which is on numen-tech/gemma4-qat#1701 and ml-explore#1816.

Carried into mlx-swift by numen-tech/mlx-swift#33 (e42e82e). If this merges as a new commit, #33's submodule bumps to it before #33 merges.

…e#4728 qmv_wide tile-8 behind MLX_QMV_MMA / MLX_QMV_WIDE_TILE8, plus Python kernel-trace bindings

Both switches are off by default and read on every call. qmv_mma gets the
fork's implied-bias twin; the NAX half (qmv_nax) is not carried.
…itwise equal to ours; qmv_mma within 2e-3 of fp32, taken only where gated, implied bias equal to the materialized bias)
@jeethu jeethu changed the title Spike: upstream affine qmv_mma (#4787) and qmv_wide tile 8 (#4728) behind env switches (numen-tech/gemma4-qat#1701) Carry ml-explore/mlx#4787 affine qmv_mma behind MLX_QMV_MMA=1, off by default (numen-tech/gemma4-qat#1701) Nov 17, 2026
… read, so decode never reads MLX_QMV_MMA; strcmp, not std::string (numen-tech/gemma4-qat#1701)
…e environment; qmv_mma_device(); kQmvMmaVersion 2 (numen-tech/gemma4-qat#1701)

A host that switches the route per model load called setenv/unsetenv,
which can race another thread's dispatch reading MLX_QMV_MMA (Codex P2).
set_qmv_mma(bool) is an atomic flag the gate reads first; MLX_QMV_MMA=1
still turns it on and is read only by a call that passes every shape
test. qmv_mma_device() is the GPU half of the gate (no NAX, not Apple6,
not an Ultra) for bindings and tests.

The kernel-trace declarations move to metal.h, and no_metal.cpp defines
them and the two new functions, so a build without Metal links the
Python bindings (Codex P1). test_qmv_mma_switch takes eligibility from
the core instead of assuming no NAX (Codex P2) and covers the flag.
@jeethu
jeethu marked this pull request as ready for review November 17, 2026 13:55
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T13:57:29.929996Z 0ed8a2c Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@jeethu
jeethu merged commit 7a80bdd into prism-v0.32.4-fixes Nov 17, 2026
@jeethu
jeethu deleted the verify-routes-1701 branch November 17, 2026 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant