Skip to content

fix: split quantized fused MoE experts in gemma4_unified sanitize - #156

Merged
inureyes merged 4 commits into
mainfrom
fix/issue-155-quantized-moe-split
Jun 4, 2026
Merged

inureyes merged 4 commits into
mainfrom
fix/issue-155-quantized-moe-split

Conversation

@inureyes

@inureyes inureyes commented Jun 4, 2026 •

Copy link
Copy Markdown
Member

Summary

Generalize the gemma4_unified fused-MoE expert split in sanitize_gemma4_unified_weights so it handles the quantized triplet (.weight + .scales + .biases), not just the bare non-quantized .weight. Previously the match guards required the exact key …experts.gate_up_proj / …experts.down_proj, so for a quantized MoE checkpoint none of the .weight/.scales/.biases legs matched, the split never fired, the fused tensors fell through to prefix normalization, and switch_glu construction could not find its per-projection quantized parts.

What changed

  • src/loading/vlm_gemma_unified.rs
    • split_fused_moe_key matches a fused expert key (gate_up_proj / down_proj) in both the bare form and each quantized component leg, returning the experts. prefix and the component suffix ("" / .weight / .scales / .biases).
    • split_fused_gate_up_component splits each present component along the output (doubled-FFN) axis at the same half boundary into switch_glu.gate_proj.<c> (first half) and switch_glu.up_proj.<c> (second half); down_proj is renamed under switch_glu unchanged.
    • emit_component_suffix keeps the bare non-quantized path emitting …switch_glu.<proj>.weight (original naming) while quantized legs keep their own suffix.
    • Replaced the now-inaccurate "aspirational" comment with an accurate description of the implemented split and its orientation.
    • Both fused-MoE split branches now wrap their output keys in normalize_gemma4_unified_key so a model.-prefixed checkpoint's split expert tensors land under the normalized language_model.model.… namespace, consistent with every other sanitized tensor. The normalization is idempotent for already-normalized keys.
  • src/loading/vlm_gemma_unified_tests.rs
    • Added a synthetic quantized fused fixture and shape/key assertions (unified_sanitize_splits_quantized_moe_switch_glu).
    • Added a numerical dequantize-equivalence gate (unified_sanitize_quantized_split_dequant_equivalence).
    • Added guard test unified_sanitize_moe_split_normalizes_model_prefix covering the model.language_model.* → language_model.model.* rewrite through the split path.

Orientation (the layout-risk resolution)

  • Non-quantized .weight is stored [num_experts, in, 2*ffn] (doubled dim is the last axis) and keeps the existing transpose([0,2,1]) → split-axis-1 path.
  • Quantized legs are already stored in the post-transpose [num_experts, 2*ffn, in/…] layout that gather_qmm(transpose=true) consumes, with MLX affine grouping along the last (input/contract) axis. They are split on the output axis 1 directly, with no transpose — transposing packed ints or per-group scales/biases would corrupt them, while slicing independent output rows partitions weight/scales/biases at the identical half boundary with no group straddling. This mirrors the mistral4 (src/models/mistral4.rs) and qwen3_5_moe (src/models/qwen3_5.rs) fused splits, which likewise slice the output axis of the already-[E, out, in] tensor without transposing.

Validation caveat

No real quantized-MoE gemma4_unified checkpoint exists yet for end-to-end validation. Correctness is proven via the synthetic dequantize-equivalence round-trip (dequantize(split(W)) == split(dequantize(W)) along the output axis, asserted with max-abs-diff == 0.0 over a fixture with distinct per-output-row values so a wrong-axis or group-straddling split would fail) and by mirroring the mistral4 / qwen3_5_moe precedent. The dense reference model (gemma-4-12b-it-4bit) and the non-quantized MoE path are unaffected (the existing non-quantized split test stays green).

Test plan

  • cargo test --lib unified_sanitize — 6 passed (existing non-quant split + 2 new quantized tests including the dequant-equivalence gate + new normalize-prefix guard)
  • cargo check --lib --tests — clean
  • cargo clippy --lib --tests -- -D warnings — clean
  • cargo fmt — clean

Closes #155

inureyes added 3 commits June 4, 2026 16:57
Generalize the gemma4_unified fused-MoE expert split in sanitize_gemma4_unified_weights so it handles the quantized triplet (.weight + .scales + .biases), not just the bare non-quantized .weight. Previously the match guards required the exact key …experts.gate_up_proj / …experts.down_proj, so for a quantized MoE checkpoint none of the .weight/.scales/.biases legs matched and the split never fired — the fused tensors fell through to prefix normalization and switch_glu construction could not find its per-projection quantized parts.

split_fused_moe_key now matches both the bare key and each quantized component leg, returning the experts. prefix and the component suffix ("" / ".weight" / ".scales" / ".biases"). split_fused_gate_up_component splits each present component along the output (doubled-FFN) axis at the same half boundary into switch_glu.gate_proj.<c> and switch_glu.up_proj.<c>; down_proj is renamed under switch_glu unchanged.

Orientation: the bare non-quantized .weight is stored [E, in, 2*ffn] and keeps the existing transpose([0,2,1]) → split-axis-1 path. Quantized legs are already stored in the post-transpose [E, 2*ffn, in/…] layout that gather_qmm(transpose=true) consumes, with MLX affine grouping along the last (input/contract) axis, so they are split on the output axis 1 directly with no transpose — transposing packed ints or per-group scales/biases would corrupt them, while slicing independent output rows partitions weight/scales/biases at the identical half boundary with no group straddling. This mirrors the mistral4 / qwen3_5_moe fused splits, which also slice the output axis of the already-[E, out, in] tensor without transposing.
Route the emitted switch_glu suffix through emit_component_suffix so the bare non-quantized gate_up_proj / down_proj keys still emit …switch_glu.<proj>.weight (the original naming), while quantized legs keep their own .weight / .scales / .biases suffix. Collapse the nested experts. guard into a let-chain to satisfy clippy collapsible_if.
Add two tests to the gemma4_unified loader sanitize suite, mirroring the existing non-quantized split test.

unified_sanitize_splits_quantized_moe_switch_glu constructs a synthetic quantized fused triplet (gate_up_proj.{weight,scales,biases} in the [num_experts, 2*ffn, in/…] on-disk layout) plus a down_proj triplet, and asserts sanitize emits exactly switch_glu.{gate,up,down}_proj.{weight,scales,biases} with the correct half-split shapes, with the fused/bare keys removed.

unified_sanitize_quantized_split_dequant_equivalence is the numerical gate: it asserts dequantize(split(W)) == split(dequantize(W)) along the output axis (max abs diff == 0.0), proving the sanitize split partitions weight/scales/biases at the same half boundary with no group straddling — without needing a real quantized-MoE gemma4_unified checkpoint. The fixture uses distinct per-output-row values so a wrong-axis or group-straddling split would change the dequantized result, keeping the gate non-vacuous.
@inureyes inureyes added status:review Under review type:bug Bug fixes, error corrections, or issue resolutions priority:low Low priority area:models Model architectures, weights, loading, metadata labels Jun 4, 2026
@inureyes

inureyes commented Jun 4, 2026

Copy link
Copy Markdown
Member Author

Implementation Review Summary

Intent

Generalize the gemma4_unified fused-MoE expert sanitize split so it handles the quantized triplet (.weight/.scales/.biases), not just the bare non-quantized .weight, emitting the full switch_glu.{gate,up,down}_proj.{weight[,scales,biases]} set.

Findings Addressed

  • None required code changes. No CRITICAL/HIGH issues found.

Remaining Items (informational, not regressions — pre-existing behavior preserved by this PR)

  • (MEDIUM/latent, pre-existing) The MoE split branch continues before normalize_gemma4_unified_key, so it keeps whatever raw prefix the on-disk expert key carried. End-to-end correctness therefore assumes the raw fused-expert key already arrives in the final language_model.model.…experts.… form. A future real checkpoint that ships experts under model.language_model.… or bare language_model.… would land them under an unnormalized prefix and SwitchGeGLU::from_weights would hard-error Missing weight. This is identical to the original non-quant branch (not introduced by this PR) and is not exercised by any current checkpoint (the dense gemma-4-12b-it-4bit has no experts). Worth a follow-up note, but out of scope for fix: split quantized fused MoE experts (gate_up_proj/down_proj scales+biases) in gemma4_unified sanitize #155.
  • (LOW/contractual) split_fused_moe_key classifies any …experts.gate_up_proj.weight as a quantized leg (split on axis 1, no transpose). This is correct under the documented format — across mistral4/qwen3_5/gemma4_unified the non-quant fused tensor is always the bare key experts.gate_up_proj; a .weight suffix denotes the quantized packed-int leg. No format-impossible case exists, so no action needed.

Verification

  • All stated requirements implemented (quantized triplet split on output axis 1 without transpose; down_proj renamed not split; non-quant path keeps transpose+slice; aspirational comment replaced with an accurate one)
  • No placeholder/mock code remaining (real dequantize FFI used in the equivalence gate)
  • Integrated into project code flow (emits exactly the …switch_glu.{gate,up,down}_proj.{weight,scales,biases} keys that gemma4 SwitchGeGLU / SwitchLinear::Quantized + the compiled_switch_qgeglu_forward fused path consume via gather_qmm(transpose=true))
  • Project conventions followed (mirrors the mistral4 / qwen3_5_moe output-axis fused split; slice_axis refactor is behavior-preserving vs. the original slice)
  • Existing modules reused where applicable (slice_axis, transpose_axes, dequantize)
  • No unintended structural changes (sanitize signature, prefix-normalization, and the rest of the loader unchanged)
  • Tests pass (cargo test --lib unified_sanitize → 5 passed incl. shape/key + dequant-equivalence gate; cargo check --lib --tests clean; cargo clippy --lib --tests -- -D warnings clean)

The orientation decision is sound: SwitchLinear::Quantized::forward calls gather_qmm(transpose=true), which consumes [E, out, in_packed] with affine grouping on the last (input) axis, so the output-axis-1 split with no transpose is lossless with no group straddling — confirmed numerically by dequantize(split(W)) == split(dequantize(W)) (max-abs-diff 0.0 over distinct per-output-row values).

Wrap output keys from both fused-MoE split branches (gate_up_proj and
down_proj) in normalize_gemma4_unified_key so a model.-prefixed checkpoint's
split expert tensors land under the normalized language_model.model.… namespace,
consistent with every other sanitized tensor. The call is idempotent for already-normalized keys, so the dense reference and all existing tests are behavior-unchanged.

Add guard test unified_sanitize_moe_split_normalizes_model_prefix covering the
model.language_model.* → language_model.model.* rewrite through the split path.
@inureyes

inureyes commented Jun 4, 2026

Copy link
Copy Markdown
Member Author

PR Finalization Complete

Summary

  • Tests: 6/6 passing (unified_sanitize_drops_rotary_and_lm_head, unified_sanitize_drops_audio_when_absent, unified_sanitize_splits_moe_switch_glu, unified_sanitize_moe_split_normalizes_model_prefix, unified_sanitize_splits_quantized_moe_switch_glu, unified_sanitize_quantized_split_dequant_equivalence)
  • Documentation: PR body updated — test plan count corrected to 6, split branches' prefix-normalization behavior documented in the "What changed" section
  • Lint/Format: cargo fmt applied, cargo clippy --lib --tests -- -D warnings clean, cargo check --lib --tests clean

Commit 91318f2 pushed to fix/issue-155-quantized-moe-split. Not merged.

@inureyes inureyes added status:done Completed and removed status:review Under review labels Jun 4, 2026
@inureyes
inureyes merged commit 6c1861e into main Jun 4, 2026
5 checks passed
@inureyes
inureyes deleted the fix/issue-155-quantized-moe-split branch June 4, 2026 08:27
@inureyes inureyes added this to the 0.1 milestone Jun 21, 2026
inureyes added a commit that referenced this pull request Aug 7, 2026
## Summary

`loading::vlm::gemma_unified::tests::unified_sanitize_quantized_split_dequant_equivalence` was the only failing test on `main`. The root cause is a real defect, but not in the gemma4_unified sanitize split: `mlx::core::dequantize` silently returns wrong values whenever its `biases` input is not row-contiguous. The test expectation (exact equality, `assert_eq!(max_abs_diff(...), 0.0)`) was sound and is unchanged.

## Root cause

The sanitize split is correct. Slicing the fused quantized `gate_up_proj` on the output axis selects exactly the right `weight` / `scales` / `biases` bytes: dequantizing a row-contiguous copy of the sanitize output reproduces `split(dequantize(W))` bit for bit on both legs. Isolating the three inputs one at a time shows the result is wrong if and only if `biases` is strided, regardless of the layout of `weight` and `scales`.

`quantize_impl` on the MLX Metal backend (<https://github.com/ml-explore/mlx/blob/main/mlx/backend/metal/quantized.cpp>) binds `w` (buffer 0) and `scales` (buffer 1) on the shared compute encoder and only then calls `ensure_row_contiguous` on `biases`. For a strided `biases`, that call enqueues a copy kernel on the same encoder, and the copy rebinds buffers 0 and 1 to its own operands. The dequantize dispatch that follows reads `w` and `scales` from the copy's operands instead of its own, so the affine kernel computes `biases * (q + 1)` with `q` decoded from the raw bias bytes. That model matches the observed numbers exactly: the gate leg's minimum was `-3.344`, which is `min(biases) * 16`, and its maximum was `-0.1793`, which is `max(biases) * 1`.

This is an upstream ordering defect, not a contract on the caller. `QuantizedMatmul::eval_gpu` in the same file hoists every `ensure_row_contiguous*` above its first binding, and so did `fast::Quantize::eval_gpu` until ml-explore/mlx#3757 folded it into the shared `quantize_impl`.

## Regression window

The test has been red since `94a8608a` (#1046, 2026-08-06), which bumped the MLX pin from `b7c3dd6d` (2026-07-17) to `2c46b953` (2026-08-05). The upstream commit that reordered the bindings, `a79332de6` (ml-explore/mlx#3757), is an ancestor of `2c46b953` and is not an ancestor of `b7c3dd6d`; the pre-bump tree still has all three contiguity copies hoisted above the first `set_input_array`. The test passed from the day it was added in #156 until that pin bump, so it was red for two days, not silently red for months. The window was established by reading `fast::Quantize::eval_gpu` in both pinned MLX trees (both are present in the local FetchContent clone) rather than by rebuilding mlxcel at the old pin, which would have cost a cold MLX C++ build.

## What changed

- `src/lib/mlxcel-core/cpp/mlx_cxx_bridge.cpp`: the `dequantize` shim forces `biases` row-contiguous before calling `mlx::core::dequantize`. `biases` is the only input whose contiguity copy is enqueued after a binding, so it is the only one guarded, and `mlx::core::contiguous` shares the buffer instead of copying when the input is already row-contiguous, so the common path is unchanged. The comment records the mechanism and the upstream reference for the next MLX bump.
- `src/lib/mlxcel-core/src/ffi_tests.rs`: adds `dequantize_commutes_with_output_axis_slice_on_strided_inputs`, which pins the guard at the FFI boundary where the defect actually lives, covering both halves of the split.
- `src/loading/vlm_gemma_unified.rs`: doc-comment only. `split_fused_gate_up_component` keeps emitting strided views rather than materializing copies, because `gather_qmm` and `quantized_matmul` receive per-input batch strides explicitly and consume the views zero-copy; materializing would add a second copy of every fused expert tensor at load time for no correctness gain. The comment now says consumers must not assume row-contiguity there.

## Notes

- Both legs confirmed. The `up_proj` assertion that the earlier failure masked also passes; before the fix it was wrong by 4.0477 against an expected 0.
- `mistral4` and `qwen3_5_moe` split fused MoE experts the same way and therefore also hand strided quantized triplets downstream. They are not changed here: the guard is at the `dequantize` boundary, so it covers every caller.
- The other in-tree `mlx::core::dequantize` call sites (`mlx_cxx_nemotron.cpp`, the fused-MoE parity reference in `mlx_cxx_kernels.cpp`, and `quantized_embedding`) feed `biases` from `take` or `astype`, both of which produce row-contiguous output, so none of them were exposed.

## Test plan

- [x] `cargo test --profile test-fast --features metal,accelerate -p mlxcel --lib unified_sanitize` (6 passed, 0 failed; was 5 passed, 1 failed)
- [x] `cargo test --profile test-fast --features metal,accelerate -p mlxcel-core --lib dequantize_commutes_with_output_axis_slice_on_strided_inputs` (passed)
- [x] Negative control: the same test fails with a 3.71 max absolute difference when the guard is reverted, so it is not vacuous
- [x] `cargo test --profile test-fast --features metal,accelerate -p mlxcel --test granite4_vision_parity` (3 passed, real 4-bit checkpoint)
- [x] `cargo fmt --check`
- [x] `cargo clippy --profile test-fast --features metal,accelerate -p mlxcel-core --lib --tests -- -D warnings`
- [x] `make verify-test`

Closes #1074


## Review follow-ups

A second commit adds comments only, no behavior change: the three call sites that reach `mlx::core::dequantize` directly (`quantized_embedding` in the bridge, the nemotron embedding path, and the fused-MoE parity reference) now record inline that they are safe only because `biases` comes from `take` / `astype`; the shim's cost claim is corrected (`mlx::core::contiguous` always adds a `Contiguous` node, and the elision happens at eval and only for a standalone row-contiguous input); the shim records why the guard is unconditional rather than gated on `flags().row_contiguous` (flags are not meaningful before eval, so the obvious short-circuit would skip the copy exactly when it is needed); and the Rust declaration states the normalization where callers look.

## Not covered

- **`gather_qmm` on strided expert weights is not pinned by a test.** This PR deliberately keeps `split_fused_gate_up_component` emitting strided views, and that decision rests on `QuantizedMatmul::eval_gpu` and `GatherQMM::eval_gpu` hoisting every `ensure_row_contiguous_matrix` above their first binding and passing per-input batch strides explicitly. That was verified by reading the pinned MLX source, not by a test. A future MLX bump that regresses it the way ml-explore/mlx#3757 regressed `dequantize` would corrupt quantized MoE inference for gemma4_unified, mistral4, and qwen3_5_moe silently. A companion FFI test comparing `gather_qmm` on sliced expert weights against `gather_qmm` on materialized copies would close that gap and is worth a follow-up issue.
- **The MLX bump checklist was not updated.** `CLAUDE.md` carries the post-bump re-validation list and would be the right home for "re-check the `dequantize` biases guard, regression test `dequantize_commutes_with_output_axis_slice_on_strided_inputs`". It is gitignored in this repository, so the entry cannot land through this PR and is left to the maintainer.

## Observed while verifying

`make verify-test` was run twice on this branch. Both runs were green except for one real-model VLM parity test asserting `text-only logits must be finite`, and it was a **different** test each time (`granite4_vision_parity` in the first run, `hunyuan_vl_parity` in the second). Each passes in isolation, including its full test binary. That looks like load-dependent flakiness in the real-checkpoint parity tests when the whole workspace runs concurrently, not a regression from this change; worth its own issue.
@inureyes inureyes self-assigned this Aug 31, 2026
be-free-snowboy pushed a commit to freestylejs/snowbrain-mlxcel that referenced this pull request Oct 27, 2026
…blup#156)

* fix: split quantized fused MoE experts in gemma4_unified sanitize

Generalize the gemma4_unified fused-MoE expert split in sanitize_gemma4_unified_weights so it handles the quantized triplet (.weight + .scales + .biases), not just the bare non-quantized .weight. Previously the match guards required the exact key …experts.gate_up_proj / …experts.down_proj, so for a quantized MoE checkpoint none of the .weight/.scales/.biases legs matched and the split never fired — the fused tensors fell through to prefix normalization and switch_glu construction could not find its per-projection quantized parts.

split_fused_moe_key now matches both the bare key and each quantized component leg, returning the experts. prefix and the component suffix ("" / ".weight" / ".scales" / ".biases"). split_fused_gate_up_component splits each present component along the output (doubled-FFN) axis at the same half boundary into switch_glu.gate_proj.<c> and switch_glu.up_proj.<c>; down_proj is renamed under switch_glu unchanged.

Orientation: the bare non-quantized .weight is stored [E, in, 2*ffn] and keeps the existing transpose([0,2,1]) → split-axis-1 path. Quantized legs are already stored in the post-transpose [E, 2*ffn, in/…] layout that gather_qmm(transpose=true) consumes, with MLX affine grouping along the last (input/contract) axis, so they are split on the output axis 1 directly with no transpose — transposing packed ints or per-group scales/biases would corrupt them, while slicing independent output rows partitions weight/scales/biases at the identical half boundary with no group straddling. This mirrors the mistral4 / qwen3_5_moe fused splits, which also slice the output axis of the already-[E, out, in] tensor without transposing.

* fix: preserve .weight suffix for bare non-quantized MoE split

Route the emitted switch_glu suffix through emit_component_suffix so the bare non-quantized gate_up_proj / down_proj keys still emit …switch_glu.<proj>.weight (the original naming), while quantized legs keep their own .weight / .scales / .biases suffix. Collapse the nested experts. guard into a let-chain to satisfy clippy collapsible_if.

* test: add quantized fused-MoE split + dequant-equivalence gate

Add two tests to the gemma4_unified loader sanitize suite, mirroring the existing non-quantized split test.

unified_sanitize_splits_quantized_moe_switch_glu constructs a synthetic quantized fused triplet (gate_up_proj.{weight,scales,biases} in the [num_experts, 2*ffn, in/…] on-disk layout) plus a down_proj triplet, and asserts sanitize emits exactly switch_glu.{gate,up,down}_proj.{weight,scales,biases} with the correct half-split shapes, with the fused/bare keys removed.

unified_sanitize_quantized_split_dequant_equivalence is the numerical gate: it asserts dequantize(split(W)) == split(dequantize(W)) along the output axis (max abs diff == 0.0), proving the sanitize split partitions weight/scales/biases at the same half boundary with no group straddling — without needing a real quantized-MoE gemma4_unified checkpoint. The fixture uses distinct per-output-row values so a wrong-axis or group-straddling split would change the dequantized result, keeping the gate non-vacuous.

* fix: normalize prefix in gemma4_unified fused-MoE split keys

Wrap output keys from both fused-MoE split branches (gate_up_proj and
down_proj) in normalize_gemma4_unified_key so a model.-prefixed checkpoint's
split expert tensors land under the normalized language_model.model.… namespace,
consistent with every other sanitized tensor. The call is idempotent for already-normalized keys, so the dense reference and all existing tests are behavior-unchanged.

Add guard test unified_sanitize_moe_split_normalizes_model_prefix covering the
model.language_model.* → language_model.model.* rewrite through the split path.
be-free-snowboy pushed a commit to freestylejs/snowbrain-mlxcel that referenced this pull request Oct 27, 2026
## Summary

`loading::vlm::gemma_unified::tests::unified_sanitize_quantized_split_dequant_equivalence` was the only failing test on `main`. The root cause is a real defect, but not in the gemma4_unified sanitize split: `mlx::core::dequantize` silently returns wrong values whenever its `biases` input is not row-contiguous. The test expectation (exact equality, `assert_eq!(max_abs_diff(...), 0.0)`) was sound and is unchanged.

## Root cause

The sanitize split is correct. Slicing the fused quantized `gate_up_proj` on the output axis selects exactly the right `weight` / `scales` / `biases` bytes: dequantizing a row-contiguous copy of the sanitize output reproduces `split(dequantize(W))` bit for bit on both legs. Isolating the three inputs one at a time shows the result is wrong if and only if `biases` is strided, regardless of the layout of `weight` and `scales`.

`quantize_impl` on the MLX Metal backend (<https://github.com/ml-explore/mlx/blob/main/mlx/backend/metal/quantized.cpp>) binds `w` (buffer 0) and `scales` (buffer 1) on the shared compute encoder and only then calls `ensure_row_contiguous` on `biases`. For a strided `biases`, that call enqueues a copy kernel on the same encoder, and the copy rebinds buffers 0 and 1 to its own operands. The dequantize dispatch that follows reads `w` and `scales` from the copy's operands instead of its own, so the affine kernel computes `biases * (q + 1)` with `q` decoded from the raw bias bytes. That model matches the observed numbers exactly: the gate leg's minimum was `-3.344`, which is `min(biases) * 16`, and its maximum was `-0.1793`, which is `max(biases) * 1`.

This is an upstream ordering defect, not a contract on the caller. `QuantizedMatmul::eval_gpu` in the same file hoists every `ensure_row_contiguous*` above its first binding, and so did `fast::Quantize::eval_gpu` until ml-explore/mlx#3757 folded it into the shared `quantize_impl`.

## Regression window

The test has been red since `94a8608a` (lablup#1046, 2026-08-06), which bumped the MLX pin from `b7c3dd6d` (2026-07-17) to `2c46b953` (2026-08-05). The upstream commit that reordered the bindings, `a79332de6` (ml-explore/mlx#3757), is an ancestor of `2c46b953` and is not an ancestor of `b7c3dd6d`; the pre-bump tree still has all three contiguity copies hoisted above the first `set_input_array`. The test passed from the day it was added in lablup#156 until that pin bump, so it was red for two days, not silently red for months. The window was established by reading `fast::Quantize::eval_gpu` in both pinned MLX trees (both are present in the local FetchContent clone) rather than by rebuilding mlxcel at the old pin, which would have cost a cold MLX C++ build.

## What changed

- `src/lib/mlxcel-core/cpp/mlx_cxx_bridge.cpp`: the `dequantize` shim forces `biases` row-contiguous before calling `mlx::core::dequantize`. `biases` is the only input whose contiguity copy is enqueued after a binding, so it is the only one guarded, and `mlx::core::contiguous` shares the buffer instead of copying when the input is already row-contiguous, so the common path is unchanged. The comment records the mechanism and the upstream reference for the next MLX bump.
- `src/lib/mlxcel-core/src/ffi_tests.rs`: adds `dequantize_commutes_with_output_axis_slice_on_strided_inputs`, which pins the guard at the FFI boundary where the defect actually lives, covering both halves of the split.
- `src/loading/vlm_gemma_unified.rs`: doc-comment only. `split_fused_gate_up_component` keeps emitting strided views rather than materializing copies, because `gather_qmm` and `quantized_matmul` receive per-input batch strides explicitly and consume the views zero-copy; materializing would add a second copy of every fused expert tensor at load time for no correctness gain. The comment now says consumers must not assume row-contiguity there.

## Notes

- Both legs confirmed. The `up_proj` assertion that the earlier failure masked also passes; before the fix it was wrong by 4.0477 against an expected 0.
- `mistral4` and `qwen3_5_moe` split fused MoE experts the same way and therefore also hand strided quantized triplets downstream. They are not changed here: the guard is at the `dequantize` boundary, so it covers every caller.
- The other in-tree `mlx::core::dequantize` call sites (`mlx_cxx_nemotron.cpp`, the fused-MoE parity reference in `mlx_cxx_kernels.cpp`, and `quantized_embedding`) feed `biases` from `take` or `astype`, both of which produce row-contiguous output, so none of them were exposed.

## Test plan

- [x] `cargo test --profile test-fast --features metal,accelerate -p mlxcel --lib unified_sanitize` (6 passed, 0 failed; was 5 passed, 1 failed)
- [x] `cargo test --profile test-fast --features metal,accelerate -p mlxcel-core --lib dequantize_commutes_with_output_axis_slice_on_strided_inputs` (passed)
- [x] Negative control: the same test fails with a 3.71 max absolute difference when the guard is reverted, so it is not vacuous
- [x] `cargo test --profile test-fast --features metal,accelerate -p mlxcel --test granite4_vision_parity` (3 passed, real 4-bit checkpoint)
- [x] `cargo fmt --check`
- [x] `cargo clippy --profile test-fast --features metal,accelerate -p mlxcel-core --lib --tests -- -D warnings`
- [x] `make verify-test`

Closes lablup#1074


## Review follow-ups

A second commit adds comments only, no behavior change: the three call sites that reach `mlx::core::dequantize` directly (`quantized_embedding` in the bridge, the nemotron embedding path, and the fused-MoE parity reference) now record inline that they are safe only because `biases` comes from `take` / `astype`; the shim's cost claim is corrected (`mlx::core::contiguous` always adds a `Contiguous` node, and the elision happens at eval and only for a standalone row-contiguous input); the shim records why the guard is unconditional rather than gated on `flags().row_contiguous` (flags are not meaningful before eval, so the obvious short-circuit would skip the copy exactly when it is needed); and the Rust declaration states the normalization where callers look.

## Not covered

- **`gather_qmm` on strided expert weights is not pinned by a test.** This PR deliberately keeps `split_fused_gate_up_component` emitting strided views, and that decision rests on `QuantizedMatmul::eval_gpu` and `GatherQMM::eval_gpu` hoisting every `ensure_row_contiguous_matrix` above their first binding and passing per-input batch strides explicitly. That was verified by reading the pinned MLX source, not by a test. A future MLX bump that regresses it the way ml-explore/mlx#3757 regressed `dequantize` would corrupt quantized MoE inference for gemma4_unified, mistral4, and qwen3_5_moe silently. A companion FFI test comparing `gather_qmm` on sliced expert weights against `gather_qmm` on materialized copies would close that gap and is worth a follow-up issue.
- **The MLX bump checklist was not updated.** `CLAUDE.md` carries the post-bump re-validation list and would be the right home for "re-check the `dequantize` biases guard, regression test `dequantize_commutes_with_output_axis_slice_on_strided_inputs`". It is gitignored in this repository, so the entry cannot land through this PR and is left to the maintainer.

## Observed while verifying

`make verify-test` was run twice on this branch. Both runs were green except for one real-model VLM parity test asserting `text-only logits must be finite`, and it was a **different** test each time (`granite4_vision_parity` in the first run, `hunyuan_vl_parity` in the second). Each passes in isolation, including its full test binary. That looks like load-dependent flakiness in the real-checkpoint parity tests when the whole workspace runs concurrently, not a regression from this change; worth its own issue.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:models Model architectures, weights, loading, metadata priority:low Low priority status:done Completed type:bug Bug fixes, error corrections, or issue resolutions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: split quantized fused MoE experts (gate_up_proj/down_proj scales+biases) in gemma4_unified sanitize

1 participant