Repository navigation
metal: implied-bias gather kernels for one row per matrix (gather_qmv, gather_qmv_fast, gather_qmm_rhs + NAX), kGatherImpliedBiasVersion 1 (numen-tech/gemma4-qat#1833) - #33
Conversation
…, gather_qmv_fast, gather_qmm_rhs, gather_qmm_rhs_nax); kGatherImpliedBiasVersion 1 (numen-tech/gemma4-qat#1833) gather_qmm materialized every 0-d implied-bias factor. It now keeps the factor for a transposed product of one row per matrix on the Metal GPU with unpromoted scales at 1/2/4/8 bits, which is what mlx-lm's SwitchGLU presents for MoE experts. With M = 1 and transpose, GatherQMM::eval_gpu takes only gather_qmv(_fast) or, sorted with B >= 16 and B / E >= 4, gather_qmm_rhs (+NAX); those four kernels gain a trailing implied_bias template parameter and an "_ib" name. Everything else materializes as before: the matrix gather_qmm (+NAX), gather_qvm, the non-transposed sorted kernel, more than one row per matrix, promoted scales, 3/5/6-bit, the CPU and CUDA. eval_gpu throws if a 0-d bias reaches a route without a twin (a graph replayed with other shapes). GatherQMM::vjp gives a 0-d primal the gradients of the materialized graph: zero for the factor, f * d(bias) added to the scales. The rhs kernels do not go through adjust_matrix_offsets, so their bias pointer moves are skipped for an implied bias and the loader takes the flag. The JIT getters pass the trailing template argument on the implied branch only, so the template definition of every other kernel is unchanged. AOT lists: 1/2/4/8-bit "_ib" twins of the transposed kernels. Tests (python/tests/test_quantized.py): bit-pattern equality with explicit biases and the kernel trace on the 26B expert shapes at 4-bit g32 (decode, verify and prefill widths, sorted and unsorted, both sides of the rhs rule, ragged N, sliced weights, lhs indices), the other prebuilt formats in three dtypes, negative and signed-zero scales, subnormal fp16 scales; the materialized routes with no "_ib" kernel in the trace; gradients against the explicit scales * f graph. benchmarks/python/gather_qmm_ib_bench.py is the microbench. The NAX twin is compiled into the metallib here and not executed: this GPU (applegpu_g15s) has no NAX.
…d of throwing; the kept factor is a constant (stop_gradient); tests for a factor that rounds, the replay and forward mode (numen-tech/gemma4-qat#1833) Codex review of 2b4487f: - A shapeless compile traced with one row per matrix and replayed with more reached GatherQMM::eval_gpu with the 0-d factor and threw, where the graph ran before. eval_gpu now forms scales * f itself (the Multiply kernel the op's materialization uses) for a route without an "_ib" twin. - The op kept the raw factor, so a JVP with respect to it reached GatherQMM::jvp (not implemented) and a reverse pass propagated a zero cotangent into whatever produced it (sqrt at 0 gives 0/0). The kept factor now sits behind stop_gradient, as it does on the materialized path. - Tests: -5.3 and 0.7 as factors on the "_ib" routes in three dtypes (the product and T(f) both round, so a kernel that ignored the factor or formed it in another type fails); the shapeless replay with the kernel trace; JVP and a producer's gradient. - The bench saves and compares fp16 bit patterns.
…-tech/gemma4-qat#1833) gather_qmv_ is a prefix of gather_qmv_fast_, so the route check accepted either (Codex review of the Swift side).
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3c02a0c59f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…a zero-stride view of the scales' shape (numen-tech/gemma4-qat#1833) Codex P2 on #33. GatherQMM::eval_gpu's fallback (a shapeless replay that leaves the "_ib" routes) passed the 0-d factor to binary_op_gpu. With scales strided across matrices (scales[::2], a swapped expert grid, one expert broadcast over all), which ensure_row_contiguous_matrix keeps because each matrix is contiguous, the binary op takes its general path, and collapse_contiguous_dims reads every operand's strides at the scales' rank: past the end of the factor's empty strides. The factor is now broadcast (common broadcast(): a shared buffer with zero strides) to the scales' shape first, so the scalar path and the general path both get operands of one rank. On 3c02a0c the read did not crash or change a result here (Strides is a SmallVector, so it reads its own inline storage; the trace shows the general kernel g2_Multiply running on the rank-0 operand), which is why the tests compare bits and assert the kernels rather than expect a crash. Tests: the shapeless replay now runs on the CPU and the GPU over four layouts (contiguous, strided, permuted expert grid, broadcast) at M = 1, 3, 5 and 64, checks bit patterns against explicit biases, one gather kernel of the expected family, and the backend multiply only on the replays that left the "_ib" routes; the "_ib" routes themselves run the strided and broadcast layouts. No other backend op sees the 0-d factor: the CPU and CUDA never keep it (the op materializes at graph build, so a replay there runs the Multiply node), the "_ib" kernels read it in place, and ensure_row_contiguous_matrix and the sorted kernels' normalization return a 0-d array as is.
…the gather replay fallback broadcasts the factor to the scales' shape (numen-tech/gemma4-qat#1833) Codex P2 on numen-tech/mlx#33: a shapeless-compile replay that leaves the "_ib" routes multiplied strided scales by the 0-d factor through the binary op's general path, which read strides the factor does not have. Host code only (quantized.cpp): the generated kernel sources are unchanged.
…implied-bias branch; op gate without the redundant metal::is_available(); vjp uses the optional is_implied_bias; test helpers (traced, assert_same_bits, is_ib_kernel) and a shorter route check; bench gqmm helper (numen-tech/gemma4-qat#1833) Quality only: no kernel source changes, the non-implied JIT template definitions are the same strings, the tests draw the same random data in the same order and run the same subtests.
…lable() short-circuit (a rank-1 x on a non-Metal GPU stream threw the gather rank error before, not an out-of-range index); assert_same_bits runs assert_bitwise_equal unconditionally as the tests did; the replay test keeps endswith("_ib") (numen-tech/gemma4-qat#1833)
|
/simplify pass (end of run, epic numen-tech/gemma4-qat#1828). Two commits on top of
No kernel source changed: Gate (M3 Max,
Codex on the delta: 5 findings.
Skipped from the review: The NAX twin is still not executed. numen-tech/mlx-swift#34 carries this head; numen-tech/gemma4-qat#1851 re-pins, and its device sweep runs from the new pin. |
…the /simplify pass of numen-tech/mlx#33 (numen-tech/gemma4-qat#1833) Host code and tests only (jit_kernels.cpp keeps upstream's ternary and overrides the implied-bias branch; the same template definition strings, the trailing implied_bias argument only on that branch). The generated Metal sources are unchanged; cuda_jit_sources.h came out in another order (the sorted lines hash the same) and is left as committed.
Merge order: this PR first. numen-tech/mlx-swift#34 points
Source/Cmlx/mlxat this head (6a8d93ba0, after the end-of-run /simplify pass; see the comment below). If this merges as a new commit, the submodule moves to it andtools/update-mlx.shruns again before ml-explore#34 merges; numen-tech/gemma4-qat then re-pins.Implied-bias (0-d factor) twins of the gather kernels a one-row-per-expert MoE layer takes, for numen-tech/gemma4-qat#1833 (epic ml-explore#1828: the Gemma 4 26B's experts, 12.85 GB of its 15.3 GB pack, in the lattice format whose biases are
f * scale). Before this,gather_qmmmaterialized every 0-d factor: 1.43 GB of extra fp16 for the 26B.What changes
gather_qmm(ops.cpp) keeps the factor when all of these hold, and materializesscales * T(f)exactly as before otherwise:transposeis true;x.shape(-2) == 1(one row per matrix);implied_bias_kernels_exactholds on the promoted output type (scales unpromoted, 1/2/4/8 bits).With M = 1 and
transpose,GatherQMM::eval_gpureaches only these kernels (get_qmv_batch_limitnever returns less than 6), and each gains a trailingimplied_biastemplate parameter and an_ibname:affine_gather_qmv_fastimplied_biastoqmv_fast_impl; the host already sends zerob_stridesfor a 0-d biasaffine_gather_qmvqmv_implaffine_gather_qmm_rhsimplied_bias; the two bias pointer moves are skipped (this kernel does not go throughadjust_matrix_offsets)affine_gather_qmm_rhs_naxThe scope is narrower than the issue's "every gather kernel": mlx-lm's
SwitchGLU(switch_layers.py:159-197) expands x to one row per (token, expert), so the matrixgather_qmm(+NAX) andgather_qvmare never reached from it. They keep materializing, and so do the non-transposed sorted kernel, M > 1, promoted scales, 3/5/6-bit, the CPU (GatherQMM::eval_cpuwould callensure_row_contiguous_last_dimson a 0-d array) and CUDA.Other pieces:
GatherQMM::eval_gpuformsscales * fitself (the Multiply kernel the op's materialization uses) if a 0-d bias reaches a route without a twin. Only a shapeless-compile replay with other shapes gets there.stop_gradient, as on the materialized path.GatherQMM::vjpgives a 0-d primal the materialized graph's gradients: zero for the factor,f * d(bias)added to the scales.get_gather_qmm_kernel,get_gather_qmm_nax_kernel) takeimplied_biasand pass the trailing template argument on that branch alone, so every other kernel's template definition is unchanged. AOT lists: 1/2/4/8-bit_ibtwins of the transposed kernels.kGatherImpliedBiasVersion = 1(device.h), for a binding to gate on.Tests (
python/tests/test_quantized.py)Every comparison is on bit patterns against the explicit bias
scales * T(f), and on the GPU the kernel trace proves the route: one gather kernel, with_ibin its name exactly where the factor is kept.test_implied_bias_gather_routeslhs_indices; 1/2/4/8-bit at g32/64/128 in fp32, fp16, bf16; negative and signed-zero scales; factors -5.3 and 0.7 (the product and T(f) both round); subnormal fp16 scalestest_implied_bias_gather_qmmgather_qvm, the non-transposed sorted kernel, promoted scales, 3- and 6-bit. Equal bits, and no_ibkernel in the tracetest_implied_bias_gather_gradvjpwrt x and scales equals the explicitscales * fgraph bit for bit; the factor's gradient and tangent are zero; a producer of the factor (sqrtat 0) gets no NaNtest_implied_bias_gather_shapeless_replayWhat ran (M3 Max,
applegpu_g15s, no NAX)0ed8a2ca0(base)test_quantized.py2b4487f78test_quantized.py2b4487f78python/tests-DMLX_METAL_JIT=ON, what mlx-swift builds)2b4487f78test_quantized.py2b4487f78python/tests91dbc1531test_quantized.py91dbc1531python/tests91dbc1531test_quantized.py3c02a0c59test_quantized.py, three runs8b073840c(head)test_quantized.py8b073840cpython/tests8b073840ctest_quantized.py8b073840cpython/tests3c02a0c59only tightens a test assertion (the plaingather_qmvroute must not name the fast kernel).8b073840cis the review fix below (the replay fallback with strided scales).The one JIT failure is not this change.
test_fp_gather_qmm_non_multiple_of_32(nvfp4, 50 rows per matrix, K = 1040, fp32, thenvfp4_gather_qmm_tmatrix kernel) returned NaN once. That kernel reads past its inputs on the partial K tile, on base too. Repro (poison.py, the test's own call after the allocator's cache is filled with 0xFF buffers of the sizes the call uses, so the bytes behind each input are NaN as a float and as an fp8 scale; 200 runs):0ed8a2ca0, AOT3c02a0c59, JITThe reference matmul never returned NaN. Whether a run fails depends on what the allocator hands back, which is why the test passes almost always. Not fixed here (fp modes are outside this change); it wants its own issue.
Not run: the NAX twin.
affine_gather_qmm_rhs_nax_ibis compiled into the metallib here (the AOT build instantiates it at both row tiles) and was never executed: this GPU has no NAX. It mirrorsqmm_t_nax_tgp_impl's use of the same loader withimplied_bias, which the shippedaffine_qmm_t_nax_ibkernels run. gemma4-qat's NAX sweep gainsgather.sortedibcells that assert_gather_qmm_rhs_nax_+_ibon a NAX device. That sweep has not run yet (the M5 iPad was held by another session all night); numen-tech/gemma4-qat#1851 has the commands.Microbench (
benchmarks/python/gather_qmm_ib_bench.py)26B expert shapes, E = 128, top-8, 4-bit g32 fp16, median of 7 x 200 evals per arm, under
heavy.sh --timing. Base0ed8a2ca0and this branch (the C++ tree of91dbc1531; the build predates that commit, so its version string reads2b4487f78), run in the order base, branch, branch, base; each figure is the mean of an arm's two runs._ib) µsgather_qmmandgather_qvm, are the same fp16 bit patterns on base and branch (--save/--check, 14 arrays).Codex adversarial review (of
2b4487f78)Fixed in
91dbc1531(and the test assertion in3c02a0c59):eval_gpunow materializes.stop_gradientboundary: a JVP wrt the factor reachedGatherQMM::jvp(not implemented), and a zero cotangent flowed into the factor's producer.Fixed in
8b073840c(the P2 on this PR's review thread):binary_op_gpu; with scales strided across matrices the general path read strides the factor does not have. The factor is now broadcast to the scales' shape first.test_implied_bias_gather_shapeless_replayruns four layouts (contiguous, strided, a permuted expert grid, broadcast) on the CPU and the GPU at M = 1, 3, 5 and 64.Not changed, with the reason:
QuantizedMatmul's implied bias since Update docs (#177) ml-explore/mlx#190 as well; exported graphs are not moved across cores here.qmv_fast_impl/qmv_implstill form (never read) offset pointers from the factor. That is the shared body the shippedaffine_qmv_ibkernels have run since Update docs (#177) ml-explore/mlx#190; this PR does not touch it.Pre-existing, found by the review, not part of this change (both reproduce with explicit biases):
gather_qmmwith x broadcast over the indices leaves rows unwritten.xof shape (1, 1, K) with 16 sortedrhs_indicesand E = 4 on base0ed8a2ca0: row 0 matches the reference (max error 0.004), rows 1 to 15 are off by 9 to 13.eval_gpupassesx.size() / K(1) as the row count togather_qmm_rhs, which broadcasts x to 16 rows and then schedules one. mlx-lm's_gather_sortalways passes physical rows, so the MoE path does not hit it. A one-line fix (pass B); left for the owner to decide, since it changes the explicit-bias path.nvfp4_gather_qmm_treads past its inputs on a partial K tile (K = 1040): reproduced on base, table above.affine_gather_qmm_rhs_naxreads past the scales and weights on a partial K tile (K % 64 != 0, e.g. K = 96 at g32):load_safebounds the rows, not the columns. Read from the source only, not reproduced (no NAX here). The 26B's K (2816, 704) are multiples of 64.poison.py