Skip to content

metal: implied-bias gather kernels for one row per matrix (gather_qmv, gather_qmv_fast, gather_qmm_rhs + NAX), kGatherImpliedBiasVersion 1 (numen-tech/gemma4-qat#1833) - #33

Merged
jeethu merged 6 commits into
prism-v0.32.4-fixesfrom
gather-ib-1833
Nov 18, 2026

Conversation

@jeethu

@jeethu jeethu commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Merge order: this PR first. numen-tech/mlx-swift#34 points Source/Cmlx/mlx at this head (6a8d93ba0, after the end-of-run /simplify pass; see the comment below). If this merges as a new commit, the submodule moves to it and tools/update-mlx.sh runs again before ml-explore#34 merges; numen-tech/gemma4-qat then re-pins.

Implied-bias (0-d factor) twins of the gather kernels a one-row-per-expert MoE layer takes, for numen-tech/gemma4-qat#1833 (epic ml-explore#1828: the Gemma 4 26B's experts, 12.85 GB of its 15.3 GB pack, in the lattice format whose biases are f * scale). Before this, gather_qmm materialized every 0-d factor: 1.43 GB of extra fp16 for the 26B.

What changes

gather_qmm (ops.cpp) keeps the factor when all of these hold, and materializes scales * T(f) exactly as before otherwise:

  • the stream is the Metal GPU;
  • transpose is true;
  • x.shape(-2) == 1 (one row per matrix);
  • implied_bias_kernels_exact holds on the promoted output type (scales unpromoted, 1/2/4/8 bits).

With M = 1 and transpose, GatherQMM::eval_gpu reaches only these kernels (get_qmv_batch_limit never returns less than 6), and each gains a trailing implied_bias template parameter and an _ib name:

Kernel When Change
affine_gather_qmv_fast N % 8 == 0 and K a multiple of the fast block (256 at 4/8-bit) passes implied_bias to qmv_fast_impl; the host already sends zero b_strides for a 0-d bias
affine_gather_qmv the other shapes the same, to qmv_impl
affine_gather_qmm_rhs sorted, B >= 16 and B / E >= 4 the loader takes implied_bias; the two bias pointer moves are skipped (this kernel does not go through adjust_matrix_offsets)
affine_gather_qmm_rhs_nax the same on a NAX GPU the same

The scope is narrower than the issue's "every gather kernel": mlx-lm's SwitchGLU (switch_layers.py:159-197) expands x to one row per (token, expert), so the matrix gather_qmm (+NAX) and gather_qvm are never reached from it. They keep materializing, and so do the non-transposed sorted kernel, M > 1, promoted scales, 3/5/6-bit, the CPU (GatherQMM::eval_cpu would call ensure_row_contiguous_last_dims on a 0-d array) and CUDA.

Other pieces:

  • GatherQMM::eval_gpu forms scales * f itself (the Multiply kernel the op's materialization uses) if a 0-d bias reaches a route without a twin. Only a shapeless-compile replay with other shapes gets there.
  • The kept factor sits behind stop_gradient, as on the materialized path. GatherQMM::vjp gives a 0-d primal the materialized graph's gradients: zero for the factor, f * d(bias) added to the scales.
  • JIT getters (get_gather_qmm_kernel, get_gather_qmm_nax_kernel) take implied_bias and pass the trailing template argument on that branch alone, so every other kernel's template definition is unchanged. AOT lists: 1/2/4/8-bit _ib twins of the transposed kernels.
  • kGatherImpliedBiasVersion = 1 (device.h), for a binding to gate on.

Tests (python/tests/test_quantized.py)

Every comparison is on bit patterns against the explicit bias scales * T(f), and on the GPU the kernel trace proves the route: one gather kernel, with _ib in its name exactly where the factor is kept.

Test Covers
test_implied_bias_gather_routes 26B expert shapes at 4-bit g32 fp16, (K, N) = (2816, 704), (704, 2816), (2816, 708): 8 to 1064 rows, unsorted and sorted, both sides of the rhs rule at E = 8 and E = 128, ragged row and column tiles; sliced weights; lhs_indices; 1/2/4/8-bit at g32/64/128 in fp32, fp16, bf16; negative and signed-zero scales; factors -5.3 and 0.7 (the product and T(f) both round); subnormal fp16 scales
test_implied_bias_gather_qmm the materialized routes, CPU and GPU: 3 rows per matrix, the matrix kernel (t and n), gather_qvm, the non-transposed sorted kernel, promoted scales, 3- and 6-bit. Equal bits, and no _ib kernel in the trace
test_implied_bias_gather_grad vjp wrt x and scales equals the explicit scales * f graph bit for bit; the factor's gradient and tangent are zero; a producer of the factor (sqrt at 0) gets no NaN
test_implied_bias_gather_shapeless_replay a shapeless compile traced at M = 1 and replayed at M = 3 and 64

What ran (M3 Max, applegpu_g15s, no NAX)

Build Commit Run Result
AOT (metallib) 0ed8a2ca0 (base) test_quantized.py 83 passed, 1 skipped, 12654 subtests
AOT 2b4487f78 test_quantized.py 85 passed, 1 skipped, 12964 subtests
AOT 2b4487f78 python/tests 988 passed, 57 skipped, 23508 subtests
JIT (-DMLX_METAL_JIT=ON, what mlx-swift builds) 2b4487f78 test_quantized.py 85 passed, 1 skipped, 12964 subtests
JIT 2b4487f78 python/tests 988 passed, 57 skipped, 23508 subtests
AOT 91dbc1531 test_quantized.py 86 passed, 1 skipped, 13004 subtests
AOT 91dbc1531 python/tests 989 passed, 57 skipped, 23548 subtests
JIT 91dbc1531 test_quantized.py 86 passed, 1 skipped, 13003 subtests; 1 subtest failed, a pre-existing out-of-bounds read in an fp-mode kernel (below)
JIT 3c02a0c59 test_quantized.py, three runs 86 passed, 1 skipped, 13004 subtests, each run
AOT 8b073840c (head) test_quantized.py 86 passed, 1 skipped, 13046 subtests
AOT 8b073840c python/tests 989 passed, 57 skipped, 23590 subtests
JIT 8b073840c test_quantized.py 86 passed, 1 skipped, 13046 subtests
JIT 8b073840c python/tests 989 passed, 57 skipped, 23590 subtests

3c02a0c59 only tightens a test assertion (the plain gather_qmv route must not name the fast kernel). 8b073840c is the review fix below (the replay fallback with strided scales).

The one JIT failure is not this change. test_fp_gather_qmm_non_multiple_of_32 (nvfp4, 50 rows per matrix, K = 1040, fp32, the nvfp4_gather_qmm_t matrix kernel) returned NaN once. That kernel reads past its inputs on the partial K tile, on base too. Repro (poison.py, the test's own call after the allocator's cache is filled with 0xFF buffers of the sizes the call uses, so the bytes behind each input are NaN as a float and as an fp8 scale; 200 runs):

Build Clean After the 0xFF fill
base 0ed8a2ca0, AOT 0 of 200 NaN 83 of 200 NaN
this branch, AOT 0 of 200 NaN 120 of 200 NaN
this branch 3c02a0c59, JIT 0 of 200 NaN 61 of 200 NaN

The reference matmul never returned NaN. Whether a run fails depends on what the allocator hands back, which is why the test passes almost always. Not fixed here (fp modes are outside this change); it wants its own issue.

Not run: the NAX twin. affine_gather_qmm_rhs_nax _ib is compiled into the metallib here (the AOT build instantiates it at both row tiles) and was never executed: this GPU has no NAX. It mirrors qmm_t_nax_tgp_impl's use of the same loader with implied_bias, which the shipped affine_qmm_t_nax _ib kernels run. gemma4-qat's NAX sweep gains gather.sortedib cells that assert _gather_qmm_rhs_nax_ + _ib on a NAX device. That sweep has not run yet (the M5 iPad was held by another session all night); numen-tech/gemma4-qat#1851 has the commands.

Microbench (benchmarks/python/gather_qmm_ib_bench.py)

26B expert shapes, E = 128, top-8, 4-bit g32 fp16, median of 7 x 200 evals per arm, under heavy.sh --timing. Base 0ed8a2ca0 and this branch (the C++ tree of 91dbc1531; the build predates that commit, so its version string reads 2b4487f78), run in the order base, branch, branch, base; each figure is the mean of an arm's two runs.

layer tokens rows route explicit, base µs explicit, branch µs branch / base factor, base (materialized) µs factor, branch (_ib) µs branch / base
gate/up 1 8 gather_qmv_fast 247 217 0.88 321 228 0.71
gate/up 3 24 gather_qmv_fast 326 317 0.97 419 290 0.69
gate/up 5 40 gather_qmv_fast 389 369 0.95 460 379 0.82
gate/up 8 64 gather_qmv_fast 397 364 0.92 459 353 0.77
gate/up 64 512 gather_qmm_rhs 1400 1281 0.91 1471 1309 0.89
gate/up 512 4096 gather_qmm_rhs 2968 2928 0.99 3035 2747 0.91
down 1 8 gather_qmv 256 233 0.91 339 214 0.63
down 3 24 gather_qmv 329 284 0.86 389 288 0.74
down 5 40 gather_qmv 409 369 0.90 462 338 0.73
down 8 64 gather_qmv 494 446 0.90 535 414 0.77
down 64 512 gather_qmm_rhs 1400 1261 0.90 1425 1244 0.87
down 512 4096 gather_qmm_rhs 2902 2780 0.96 2927 2664 0.91
  • Explicit biases: the branch is not slower on any cell (0.86 to 0.99 of base; with this run order I read the gap as noise, not a gain).
  • The explicit-bias outputs of all 12 cells, plus the matrix gather_qmm and gather_qvm, are the same fp16 bit patterns on base and branch (--save / --check, 14 arrays).
  • The factor: 9 to 37 % faster than the materialized bias it replaces (no per-call multiply over the experts' scales).

Codex adversarial review (of 2b4487f78)

Fixed in 91dbc1531 (and the test assertion in 3c02a0c59):

  • A shapeless replay with more rows per matrix threw where the graph ran before. eval_gpu now materializes.
  • Keeping the raw factor dropped the stop_gradient boundary: a JVP wrt the factor reached GatherQMM::jvp (not implemented), and a zero cotangent flowed into the factor's producer.
  • The 4-bit tests would have passed with a hard-coded -8: factors -5.3 and 0.7 added.
  • The bench compared widened floats: it now compares fp16 bit patterns.

Fixed in 8b073840c (the P2 on this PR's review thread):

  • The replay fallback passed the 0-d factor to binary_op_gpu; with scales strided across matrices the general path read strides the factor does not have. The factor is now broadcast to the scales' shape first. test_implied_bias_gather_shapeless_replay runs four layouts (contiguous, strided, a permuted expert grid, broadcast) on the CPU and the GPU at M = 1, 3, 5 and 64.

Not changed, with the reason:

  • An exported graph carries the 0-d input, and an older core or a CUDA consumer cannot run it. True of QuantizedMatmul's implied bias since Update docs (#177) ml-explore/mlx#190 as well; exported graphs are not moved across cores here.
  • qmv_fast_impl / qmv_impl still form (never read) offset pointers from the factor. That is the shared body the shipped affine_qmv _ib kernels have run since Update docs (#177) ml-explore/mlx#190; this PR does not touch it.

Pre-existing, found by the review, not part of this change (both reproduce with explicit biases):

  • Sorted gather_qmm with x broadcast over the indices leaves rows unwritten. x of shape (1, 1, K) with 16 sorted rhs_indices and E = 4 on base 0ed8a2ca0: row 0 matches the reference (max error 0.004), rows 1 to 15 are off by 9 to 13. eval_gpu passes x.size() / K (1) as the row count to gather_qmm_rhs, which broadcasts x to 16 rows and then schedules one. mlx-lm's _gather_sort always passes physical rows, so the MoE path does not hit it. A one-line fix (pass B); left for the owner to decide, since it changes the explicit-bias path.
  • nvfp4_gather_qmm_t reads past its inputs on a partial K tile (K = 1040): reproduced on base, table above.
  • affine_gather_qmm_rhs_nax reads past the scales and weights on a partial K tile (K % 64 != 0, e.g. K = 96 at g32): load_safe bounds the rows, not the columns. Read from the source only, not reproduced (no NAX here). The 26B's K (2816, 704) are multiples of 64.
poison.py
"""Does the nvfp4 matrix gather_qmm (K = 1040, 50 rows per matrix) read memory it does not own?

The failing subtest of test_fp_gather_qmm_non_multiple_of_32, repeated after
filling the allocator's cache with NaN buffers of the sizes the call uses, so
that any byte read past an input, or any output element left unwritten, shows
up as NaN. Prints the number of runs with a NaN or an error over tolerance.
"""
import sys
import mlx.core as mx

E, K, N, B, M = 4, 1040, 128, 8, 50
mode = "nvfp4"
key = mx.random.key(0)
k1, k2, k3 = mx.random.split(key, 3)
dtype = mx.float32


def poison(sizes):
    # 0xFF bytes: NaN as float32 or float16, and NaN as an fp8 (e4m3) scale.
    junk = [mx.full((n,), 255, mx.uint8) for n in sizes for _ in range(4)]
    mx.eval(junk)
    del junk


def once(with_poison):
    if with_poison:
        # byte counts of every buffer the call touches: the packed weights,
        # the fp8 scales, x, the output (page-rounded by the allocator)
        w_bytes, s_bytes = E * N * (K // 8) * 4, E * N * (K // 16)
        sizes = [w_bytes, s_bytes, s_bytes + 1, B * M * K * 4, B * M * N * 4, 16384, 49152]
        poison(sizes)
    w = mx.random.normal(shape=(E, N, K), key=k2).astype(dtype)
    w_q, scales = mx.quantize(w, mode=mode)
    w_hat = mx.dequantize(w_q, scales, mode=mode)
    mx.eval(w_q, scales, w_hat)
    x = (mx.random.normal(shape=(B, M, K), key=k1) / K**0.5).astype(dtype)
    # every expert, the last one included: its last row's tail is the read
    # that can leave the buffers
    rhs = mx.array([3, 0, 1, 2, 3, 3, 2, 1], mx.uint32)
    lhs = mx.arange(B, dtype=mx.uint32)
    y_q = mx.gather_qmm(x, w_q, scales, lhs_indices=lhs, rhs_indices=rhs, transpose=True, mode=mode)
    y_hat = x @ mx.swapaxes(mx.take(w_hat, rhs, axis=0), -1, -2)
    err = (y_q - y_hat).abs().max().item()
    return err, mx.isnan(y_q).any().item(), mx.isnan(y_hat).any().item()


runs = int(sys.argv[1]) if len(sys.argv) > 1 else 200
for label, flag in (("clean", False), ("poisoned", True)):
    bad = nan_q = nan_ref = 0
    worst = 0.0
    for _ in range(runs):
        err, a, b = once(flag)
        nan_q += a
        nan_ref += b
        if not (err < 2e-3):
            bad += 1
        elif err > worst:
            worst = err
    print(f"{mx.__version__} {label}: {bad}/{runs} over tolerance, NaN in gather_qmm {nan_q}, NaN in the reference matmul {nan_ref}, worst finite {worst:.2e}")

…, gather_qmv_fast, gather_qmm_rhs, gather_qmm_rhs_nax); kGatherImpliedBiasVersion 1 (numen-tech/gemma4-qat#1833)

gather_qmm materialized every 0-d implied-bias factor. It now keeps the
factor for a transposed product of one row per matrix on the Metal GPU with
unpromoted scales at 1/2/4/8 bits, which is what mlx-lm's SwitchGLU presents
for MoE experts. With M = 1 and transpose, GatherQMM::eval_gpu takes only
gather_qmv(_fast) or, sorted with B >= 16 and B / E >= 4, gather_qmm_rhs
(+NAX); those four kernels gain a trailing implied_bias template parameter
and an "_ib" name.

Everything else materializes as before: the matrix gather_qmm (+NAX),
gather_qvm, the non-transposed sorted kernel, more than one row per matrix,
promoted scales, 3/5/6-bit, the CPU and CUDA. eval_gpu throws if a 0-d bias
reaches a route without a twin (a graph replayed with other shapes).
GatherQMM::vjp gives a 0-d primal the gradients of the materialized graph:
zero for the factor, f * d(bias) added to the scales.

The rhs kernels do not go through adjust_matrix_offsets, so their bias
pointer moves are skipped for an implied bias and the loader takes the
flag. The JIT getters pass the trailing template argument on the implied
branch only, so the template definition of every other kernel is unchanged.
AOT lists: 1/2/4/8-bit "_ib" twins of the transposed kernels.

Tests (python/tests/test_quantized.py): bit-pattern equality with explicit
biases and the kernel trace on the 26B expert shapes at 4-bit g32 (decode,
verify and prefill widths, sorted and unsorted, both sides of the rhs
rule, ragged N, sliced weights, lhs indices), the other prebuilt formats in
three dtypes, negative and signed-zero scales, subnormal fp16 scales; the
materialized routes with no "_ib" kernel in the trace; gradients against
the explicit scales * f graph. benchmarks/python/gather_qmm_ib_bench.py is
the microbench. The NAX twin is compiled into the metallib here and not
executed: this GPU (applegpu_g15s) has no NAX.
…d of throwing; the kept factor is a constant (stop_gradient); tests for a factor that rounds, the replay and forward mode (numen-tech/gemma4-qat#1833)

Codex review of 2b4487f:
- A shapeless compile traced with one row per matrix and replayed with
  more reached GatherQMM::eval_gpu with the 0-d factor and threw, where
  the graph ran before. eval_gpu now forms scales * f itself (the Multiply
  kernel the op's materialization uses) for a route without an "_ib"
  twin.
- The op kept the raw factor, so a JVP with respect to it reached
  GatherQMM::jvp (not implemented) and a reverse pass propagated a zero
  cotangent into whatever produced it (sqrt at 0 gives 0/0). The kept
  factor now sits behind stop_gradient, as it does on the materialized
  path.
- Tests: -5.3 and 0.7 as factors on the "_ib" routes in three dtypes (the
  product and T(f) both round, so a kernel that ignored the factor or
  formed it in another type fails); the shapeless replay with the kernel
  trace; JVP and a producer's gradient.
- The bench saves and compares fp16 bit patterns.
…-tech/gemma4-qat#1833)

gather_qmv_ is a prefix of gather_qmv_fast_, so the route check accepted either (Codex review of the Swift side).
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-18T05:04:39.752784Z 6a8d93b New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3c02a0c59f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread mlx/backend/metal/quantized.cpp Resolved
…a zero-stride view of the scales' shape (numen-tech/gemma4-qat#1833)

Codex P2 on #33. GatherQMM::eval_gpu's fallback (a shapeless replay that
leaves the "_ib" routes) passed the 0-d factor to binary_op_gpu. With
scales strided across matrices (scales[::2], a swapped expert grid, one
expert broadcast over all), which ensure_row_contiguous_matrix keeps
because each matrix is contiguous, the binary op takes its general path,
and collapse_contiguous_dims reads every operand's strides at the scales'
rank: past the end of the factor's empty strides. The factor is now
broadcast (common broadcast(): a shared buffer with zero strides) to the
scales' shape first, so the scalar path and the general path both get
operands of one rank.

On 3c02a0c the read did not crash or change a result here (Strides is a
SmallVector, so it reads its own inline storage; the trace shows the
general kernel g2_Multiply running on the rank-0 operand), which is why
the tests compare bits and assert the kernels rather than expect a crash.

Tests: the shapeless replay now runs on the CPU and the GPU over four
layouts (contiguous, strided, permuted expert grid, broadcast) at M = 1,
3, 5 and 64, checks bit patterns against explicit biases, one gather
kernel of the expected family, and the backend multiply only on the
replays that left the "_ib" routes; the "_ib" routes themselves run the
strided and broadcast layouts.

No other backend op sees the 0-d factor: the CPU and CUDA never keep it
(the op materializes at graph build, so a replay there runs the Multiply
node), the "_ib" kernels read it in place, and
ensure_row_contiguous_matrix and the sorted kernels' normalization return
a 0-d array as is.
jeethu added a commit to numen-tech/mlx-swift that referenced this pull request Nov 18, 2026
…the gather replay fallback broadcasts the factor to the scales' shape (numen-tech/gemma4-qat#1833)

Codex P2 on numen-tech/mlx#33: a shapeless-compile replay that leaves the
"_ib" routes multiplied strided scales by the 0-d factor through the
binary op's general path, which read strides the factor does not have.
Host code only (quantized.cpp): the generated kernel sources are unchanged.
…implied-bias branch; op gate without the redundant metal::is_available(); vjp uses the optional is_implied_bias; test helpers (traced, assert_same_bits, is_ib_kernel) and a shorter route check; bench gqmm helper (numen-tech/gemma4-qat#1833)

Quality only: no kernel source changes, the non-implied JIT template
definitions are the same strings, the tests draw the same random data in
the same order and run the same subtests.
…lable() short-circuit (a rank-1 x on a non-Metal GPU stream threw the gather rank error before, not an out-of-range index); assert_same_bits runs assert_bitwise_equal unconditionally as the tests did; the replay test keeps endswith("_ib") (numen-tech/gemma4-qat#1833)
@jeethu

jeethu commented Nov 18, 2026

Copy link
Copy Markdown
Member Author

/simplify pass (end of run, epic numen-tech/gemma4-qat#1828). Two commits on top of 8b073840c, pushed as a fast-forward: head 6a8d93ba0.

Commit What
1833b935a jit_kernels.cpp: both gather getters keep upstream's two-way ternary unchanged, and an if (is_affine && implied_bias) overrides it with the trailing true. The diff against 0ed8a2ca0 for that file goes from 140 to 40 lines, and every template definition string is the same as before (Codex checked all 8 combinations of affine / implied / global scale). primitives.cpp: the std::optional overload of is_implied_bias. ops.cpp: the keep/materialize if/else becomes a ternary. Tests: the helpers traced, assert_same_bits and is_ib_kernel, and check() in test_implied_bias_gather_routes takes the weights tuple (about 100 lines shorter). The tests draw the same random numbers in the same order and run the same 168 route subtests. Bench: a gqmm helper, make_weights without its unused parameters, and the matrix/qvm tail as a loop.
6a8d93ba0 Codex on the delta: keeps metal::is_available() in the op gate (without it, a rank-1 x on a non-Metal GPU stream hit an out-of-range index before the gather rank error); assert_same_bits runs assert_bitwise_equal unconditionally, as the tests did; the replay test keeps endswith("_ib").

No kernel source changed: git diff 8b073840c 6a8d93ba0 -- mlx/backend/metal/kernels is empty, so the microbench numbers above still describe the kernels.

Gate (M3 Max, applegpu_g15s, under heavy.sh)

Build Run Before (8b073840c) After (6a8d93ba0)
AOT test_quantized.py 86 passed, 1 skipped, 13046 subtests 86 passed, 1 skipped, 13046 subtests
AOT python/tests 989 passed, 57 skipped, 23590 subtests 989 passed, 57 skipped, 23590 subtests
JIT test_quantized.py 86 passed, 1 skipped, 13046 subtests 86 passed, 1 skipped, 13046 subtests
JIT python/tests 989 passed, 57 skipped, 23590 subtests 989 passed, 57 skipped, 23590 subtests
AOT the strided-replay repro (M 1 / 3 / 64) equal bits, _ib only at M 1 the same
AOT gather_qmm_ib_bench.py --check base.npz (the base 0ed8a2ca0 save) n/a 14 explicit-bias arrays equal bit for bit

ruff 0.16.10: format clean; check reports only the two findings already on 8b073840c (import order, line 2037). clang-format: clean.

Codex on the delta: 5 findings.

  • Fixed in 6a8d93ba0: the CUDA error path; the replay test's endswith("_ib"); the empty-array case of assert_same_bits.
  • Not changed: make_weights lost its unused bits / gs / dtype parameters. Nothing calls it from outside the script.
  • Not changed: the timed loop makes one more Python call (gqmm), about 0.1 µs against cells of 200 to 3000 µs, and the same on both arms.

Skipped from the review: zeros_like(scales) in place of the vjp's ternary would also touch upstream's non-implied token, and its equivalence rests on validation a shapeless replay does not repeat.

The NAX twin is still not executed. numen-tech/mlx-swift#34 carries this head; numen-tech/gemma4-qat#1851 re-pins, and its device sweep runs from the new pin.

jeethu added a commit to numen-tech/mlx-swift that referenced this pull request Nov 18, 2026
…the /simplify pass of numen-tech/mlx#33 (numen-tech/gemma4-qat#1833)

Host code and tests only (jit_kernels.cpp keeps upstream's ternary and
overrides the implied-bias branch; the same template definition strings,
the trailing implied_bias argument only on that branch). The generated
Metal sources are unchanged; cuda_jit_sources.h came out in another order
(the sorted lines hash the same) and is left as committed.
@jeethu
jeethu merged commit 897320f into prism-v0.32.4-fixes Nov 18, 2026
@jeethu
jeethu deleted the gather-ib-1833 branch November 18, 2026 08:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant