Repository navigation
Carry ml-explore/mlx#4787 affine qmv_mma behind MLX_QMV_MMA=1, off by default (numen-tech/gemma4-qat#1701) - #32
Merged
Conversation
…e#4728 qmv_wide tile-8 behind MLX_QMV_MMA / MLX_QMV_WIDE_TILE8, plus Python kernel-trace bindings Both switches are off by default and read on every call. qmv_mma gets the fork's implied-bias twin; the NAX half (qmv_nax) is not carried.
…itwise equal to ours; qmv_mma within 2e-3 of fp32, taken only where gated, implied bias equal to the materialized bias)
…surement switch; kQmvMmaVersion 1 (numen-tech/gemma4-qat#1701)
… read, so decode never reads MLX_QMV_MMA; strcmp, not std::string (numen-tech/gemma4-qat#1701)
…e environment; qmv_mma_device(); kQmvMmaVersion 2 (numen-tech/gemma4-qat#1701) A host that switches the route per model load called setenv/unsetenv, which can race another thread's dispatch reading MLX_QMV_MMA (Codex P2). set_qmv_mma(bool) is an atomic flag the gate reads first; MLX_QMV_MMA=1 still turns it on and is read only by a call that passes every shape test. qmv_mma_device() is the GPU half of the gate (no NAX, not Apple6, not an Ultra) for bindings and tests. The kernel-trace declarations move to metal.h, and no_metal.cpp defines them and the two new functions, so a build without Metal links the Python bindings (Codex P1). test_qmv_mma_switch takes eligibility from the core instead of assuming no NAX (Codex P2) and covers the flag.
jeethu
marked this pull request as ready for review
November 17, 2026 13:55
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR carries ml-explore#4787's
affine_qmv_mma(numen-tech/gemma4-qat#1701), off by default. With neither switch on, nothing changes: no other route, tile or default moves. The owner's call (2026-11-17) is to carry it for Gemma 4 E4B behind a switch and measure it end to end. numen-tech/gemma4-qat#1816 turns it on only for an E4B load that asks.The switch. Neither switch is cached; the core reads them only for a call that passes every shape test, so decode at M = 1 reads neither.
metal::set_qmv_mma(bool)(metal.h) is a process flag. A host that switches the route per model load calls it rather thansetenv/unsetenv, which can race another thread's dispatch that is reading the variable.MLX_QMV_MMA=1still turns the route on, for measurement harnesses and tests.metal::qmv_mma_device()reports whether this GPU can take the route at all.kQmvMmaVersion = 2(device.h). Version 1 was the variable alone.The route. It sends an affine 4- or 8-bit transposed quantized matmul to
affine_qmv_mmawhen all of these hold:x,wandscalessit at aligned offsets.It runs ahead of the qmv batch limit. One addition beyond upstream: an implied-bias (
_ib) twin for Gemma's 0-d bias factor. Not carried: the fp-modeqmv_mma, andqmv_nax(ml-explore#4712).Commits.
7cd5bf1e1/5bc665b8b: the spike and its test.446091ab5: R1 only, with the tile-8 switch dropped (Widen affine qmv_wide row tiles to 8 ml-explore/mlx#4728 lost 7-13 % at M = 8).7aa006437: the gate's shape, alignment and dtype tests now run before the environment read, usingstrcmp(it used togetenvand build astd::stringon every quantized matmul).0ed8a2ca0: the process flag andqmv_mma_device(). The kernel-trace declarations move to metal.h, andno_metal.cppdefines them and the two new functions. Before this, a build without Metal referenced trace symbols it did not define (Codex P1). The test now takes eligibility from the core.Test.
test_qmv_mma_switchruns 2/4/8 bits x gs 32/64/128 x M 1-16, each with explicit and implied bias. It checks that:What ran on
0ed8a2ca0(M3 Max,applegpu_g15s):pytest test_quantized.py: 83 passed, 1 skipped, 12654 subtests.MLX_BUILD_METAL=OFFlinks the Python module, andlibmlx.adefines the four functions.Not run here: end-to-end measurement, which is on numen-tech/gemma4-qat#1701 and ml-explore#1816.
Carried into mlx-swift by numen-tech/mlx-swift#33 (
e42e82e). If this merges as a new commit, #33's submodule bumps to it before #33 merges.