Skip to content

qmv_fast: take affine 1/4-bit at half-block K (partial last block) - #8

Merged
jeethu merged 2 commits into
prism-0.32.2-fixesfrom
fork/192-qmv-fast-4bit
Oct 5, 2026
Merged

jeethu merged 2 commits into
prism-0.32.2-fixesfrom
fork/192-qmv-fast-4bit

Conversation

@jeethu

@jeethu jeethu commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

Part of numen-tech/gemma4-qat#192 (epic numen-tech/gemma4-qat#178).

What

qmv_fast now takes affine/affine_sym 4-bit with K % 256 == 0 and 1-bit with K % 512 == 0, which is half its block. Before, those shapes needed a whole block (K % 512 for 4-bit, K % 1024 for 1-bit) and otherwise fell back to the generic qmv.

The target is Gemma 4 12B (QAT q4_0, gs32). Its q/k/v, gate/up and tied-head matmuls have K = 3840 = 7.5 blocks, so at M = 1 they ran qmv. Lizzy-7B's down_proj (K = 11008) is in the same position.

  • qmv_fast_impl (kernels/quantized.h): the partial-last-block guard is now "this lane's whole slice is in the row", i.e. aligned_end + (simd_lid + 1) * values_per_thread <= in_vec_size. Lanes outside it read nothing: no weights, scales, biases or x. All lanes still join the simd_sum. When K % values_per_thread == 0 this matches the old lane-start test, so it is hardening and not a behaviour change.
  • qmv_fast_k_alignment(bits, mode) (quantized.cpp): returns block / 2 for affine 1-bit and 4-bit, and block otherwise. Used by qmv and gather_qmv. fp_qmv_fast_impl has no tail, so fp modes keep the whole block.
    • The comment records the real requirement, K % values_per_thread, which K % group_size already implies. block/2 is the cutoff measured for 1/4-bit; widening it is numen-tech/gemma4-qat#260.
  • Tests: a C++ doctest and a Python test cover the newly fast 4-bit and 1-bit shapes against the generic qmv and an fp32 reference.

Numerics

  • Shapes that already took qmv_fast are bit-identical: the loop body is unchanged and the tail never runs for them.
  • Newly admitted 4-bit shapes switch kernel: qmv_fast sums 16 values per lane vs 8 in qmv, so outputs differ at rounding level.
    • In the Codex review's deterministic reproducer (4-bit, K = 256, gs64), 550 of 768 fp32 outputs and 2 of 768 fp16 outputs differ.
    • The error against an fp32 reference is never worse than qmv's plus one output-dtype ulp. The gemma4-qat QmvFastTailKernelTests assert exactly that.
  • End to end on the 12B and Lizzy-7B, greedy output is identical.

Evidence

See the gemma4-qat PR for the gate results: tok/s A/B, greedy diff, dispatch budget, and the full Swift suite.

Merge order

  1. this PR;
  2. numen-tech/mlx-swift (bumps Source/Cmlx/mlx to the merged commit);
  3. numen-tech/gemma4-qat (re-pins Package.resolved).

jeethu added 2 commits October 4, 2026 15:48
The guarded partial-block tail of qmv_fast_impl handles half a block, so
the host gate takes affine 1/4-bit K that are half-block multiples
(Gemma 4 12B K = 3840, down_proj K = 11008). fp modes keep the full
block: fp_qmv_fast_impl has no tail.

A tail lane now loads only if all of its values are in the row.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-04T17:25:55.770332Z 7d565ec PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant