Repository navigation
perf(moe): attribute M = 1 expert gather gap and wire Inkling/V4 MoE fast paths (#1859) - #2310
Merged
Merged
Conversation
…r indices (#1859) - Wire `InklingSparseMoe::forward` at single-token decode (`flat.shape()[0] == 1`) to `SwitchGLU::forward_fused_kernel` when `gate_scale` and `out_scale` are absent (`inkling-small-mlx-4bit`), collapsing each sparse MoE layer's routed expert path from 6 GPU kernel launches and 15 graph nodes (`3 gather_qmv` + `compiled_swiglu_activation` + `moe_weighted_sum`) to 2 custom kernel launches and 5 graph nodes while preserving the scaled gather path for `inkling-small-nvfp4`. - Update `LimitedSwitchGlu::forward` in `deepseek_v4_moe.rs` to use `expand_dims_multi(x, &[-3, -2])`, `prepare_gather_indices`, and `SwitchLinear::forward_indexed`, eliminating 6 redundant index-synthesis graph nodes per MoE layer (`258` nodes/token across 43 layers when `MoEGate` returns `INT32` indices). - Refresh `Used by:` doc comments on `fused_moe_max_dff_from`, `SwitchLinear::forward`, `SwitchLinear::forward_indexed`, `SwitchGLU::forward_fused_kernel`, and `prepare_gather_indices` in `switch_layers.rs`. - Add unit tests covering `InklingSparseMoe` fused routing vs. per-expert scale fallback and `LimitedSwitchGlu` bit-exactness and `AsType(uint32)` node reduction.
Member
Author
Implementation Review SummaryIntent
Findings Addressed
Remaining Items
Verification
|
Member
Author
PR Finalization CompleteSummary
All checks passing. Ready for merge. |
inureyes
added a commit
that referenced
this pull request
Oct 20, 2026
The 2026-09-08 rows measured the pre-#2310 decode path, recorded two VLM rows on a dirty tree, and labeled the baseline with a stale 0.6.16 from a detached mlx-vlm pin. Swap all ten for a fresh 2026-10-20 four-way run at 058d7f7 on current main (MLX 243bdf09): mlxcel text and VLM rows plus mlx-vlm --text-via-vlm and VLM baselines for the same three checkpoints, baselines now recording mlx-vlm 0.7.9 from upstream main. The decode gap to the mlx-vlm baseline persists on M3 Ultra after #2310 wired the fused MoE kernel: mlxcel decodes at 23.7% (deepseek-v4-flash-4bit), 15.9% (inkling-small-mlx-4bit) and 18.5% (inkling-small-nvfp4) of the baseline on the text protocol, and 15.8% / 18.5% on the VLM protocol, each within ~3% of the September numbers on both sides. A same-build control with MLXCEL_FUSED_MOE=0 decodes at 7.80 tok/s against 7.78 with the fused kernel enabled (metal_m3ultra_2026-10-20_nofused_single_inkling-small-mlx-4bit.csv), so the kernel is not the lever on this host: the gap must come from elsewhere in the batch-1 expert path. The nvfp4 control arm stays on the gather path and moved by the same amount as the fused-path model, so the rerun is internally consistent. Refs #1858, #1859.
inureyes
added a commit
that referenced
this pull request
Oct 20, 2026
) * bench: add M3 Ultra baselines for the three large MoE checkpoints bench_mlxlm.py hardcoded its memory budget at 85% of 128 GB, so on the 512 GB M3 Ultra it skipped deepseek-v4-flash-4bit and both inkling-small checkpoints as SKIP:oom_estimate even though bench_decode.sh had just measured them. The budget now follows the host's own memory (hw.memsize on macOS, MemTotal on Linux), mirroring bench_decode.sh, and falls back to 128 GB when memory cannot be read. PYLM_BENCH_MAX_GB and the ROCm carve-out override still apply. deepseek_v4 and inkling exist only in mlx-vlm, so the plain text path recorded FAIL:warmup for them. The new --text-via-vlm flag runs the same text protocol (synthesized 512-token prompt, no chat template, no image) but loads through mlx-vlm, records mlx-vlm as baseline_version, and writes to a separate pylm_*_textvlm_* file. The ten CSVs are the 2026-09-08 M3 Ultra measurements: mlxcel text and VLM rows plus the mlx-vlm text and VLM baselines for the same three checkpoints. Validated with py_compile, --help, and a module load that resolves the budget to 435 GB on this host. * refactor(bench): reuse _host_memory_bytes for the pylm memory budget The memory-budget fix added `_detect_memory_bytes()`, which shells out to `sysctl -n hw.memsize` on Darwin and parses `/proc/meminfo` on Linux. The file already had `_host_memory_bytes()`, a single portable `os.sysconf` call used by the ROCm hardware tagging, and both return the same value on the 512 GB M3 Ultra. The budget now calls `_host_memory_bytes()` and the duplicate is gone. The helper moves above the module-level budget computation, since the budget runs at import time and the helper used to be defined later in the file. Behavior is unchanged: 435.2 GiB on the 512 GB host, `PYLM_BENCH_MAX_GB` still overrides (65 GiB checked), and the ROCm path still resolves the helper. * bench: replace September CSVs with a post-#2310 rerun The 2026-09-08 rows measured the pre-#2310 decode path, recorded two VLM rows on a dirty tree, and labeled the baseline with a stale 0.6.16 from a detached mlx-vlm pin. Swap all ten for a fresh 2026-10-20 four-way run at 058d7f7 on current main (MLX 243bdf09): mlxcel text and VLM rows plus mlx-vlm --text-via-vlm and VLM baselines for the same three checkpoints, baselines now recording mlx-vlm 0.7.9 from upstream main. The decode gap to the mlx-vlm baseline persists on M3 Ultra after #2310 wired the fused MoE kernel: mlxcel decodes at 23.7% (deepseek-v4-flash-4bit), 15.9% (inkling-small-mlx-4bit) and 18.5% (inkling-small-nvfp4) of the baseline on the text protocol, and 15.8% / 18.5% on the VLM protocol, each within ~3% of the September numbers on both sides. A same-build control with MLXCEL_FUSED_MOE=0 decodes at 7.80 tok/s against 7.78 with the fused kernel enabled (metal_m3ultra_2026-10-20_nofused_single_inkling-small-mlx-4bit.csv), so the kernel is not the lever on this host: the gap must come from elsewhere in the batch-1 expert path. The nvfp4 control arm stays on the gather path and moved by the same amount as the fused-path model, so the rerun is internally consistent. Refs #1858, #1859.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #1859.
Attributes the single-token (
n_tokens = 1) decode throughput gap betweenmlxcelandmlx-lm/mlx-vlmondeepseek-v4-flash-4bit,inkling-small-mlx-4bit, andinkling-small-nvfp4, and lands the two concrete MoE fast paths identified by the attribution.Step 2 Attribution Record (
docs/benchmark_results/moe-m1-gather-attribution-2026-10-14.md)9072e2adprior to any edits tosrc/models/switch_layers.rs,src/models/deepseek_v4_moe.rs, orsrc/models/inkling/mlp.rs.GatherQMM::eval_gpu(overlays/metal/mlx/backend/metal/quantized.cpp:2060-2088): ChecksM < get_qmv_batch_limit(K, N, d). Sinceget_qmv_batch_limit >= 6on all Apple GPUs (12..33on Max/Ultra/M5) andM = 1,GatherQMMunconditionally returns throughgather_qmv(affine_gather_qmv_fast_bfloat16_gs_64_b_4orfp_gather_qmv_fast_bfloat16_nvfp4). Commit8ad62d4b(#1772,bm = 32insidegather_qmm) is unreachable atM = 1.GatherQMM::eval_gpu(overlays/cuda/mlx/backend/cuda/quantized/quantized.cpp:526-547): Checksif (can_use_qmm_sm80) { if (can_use_qmv && (M * B < 8)) call_qmv(); else call_qmm_sm80(); }. Unliketop_k = 8models (M * B = 8 >= 8),deepseek-v4-flash-4bit,inkling-small-mlx-4bit, andinkling-small-nvfp4all usetop_k = 6, soM * B = 1 * 6 = 6 < 8is true at single-token decode. Furthermore,supports_qmm_sm80is false fornvfp4(inkling-small-nvfp4). All three models therefore already dispatchM = 1, B = 6expert projections tomlx::core::cu::gather_qmv_kernel, neverqmm_sm80(tile_m = 16).GatherQMM::eval_gpu(overlays/rocm/mlx/backend/rocm/quantized/qmm.hip:6506-6726): DispatchesM = 1, B = 6togather_qmv_wide_kernel(affinebf16) orgather_qmv_kernel(nvfp4).forward_fused_kernelinventory: 17 model families wired toSwitchGLU::forward_fused_kernel+ 2 family-local (qwen3_next,gemma4) = 19 families on096152c6(growing to 20 withinklingin this PR).deepseek-v4-flash-4bitdecode gap breakdown: Over 85% of the gap is non-MoE—specifically the 2x per-layer (86x/token) unfused 20-iteration float32HyperConnectionSinkhorn loops (src/models/deepseek_v4_hyper.rs:38-42, 115-200, ~290 graph nodes/layer = ~12,470 nodes/token vs. upstream's fused_hc_sinkhorn_collapse_kernel), plus MLA KV compression and Indexer. Within the MoE block,LimitedSwitchGlu::forwardmissed#1713/ PR#2137, emitting 6 redundant index-synthesis graph nodes per layer (258nodes/token across 43 layers).inkling-small-mlx-4bitandinkling-small-nvfp4decode gap breakdown:InklingSparseMoe::forward(src/models/inkling/mlp.rs:217-223) never calledSwitchGLU::forward_fused_kernel, leavinginkling-small-mlx-4bit(affine4-bit gs64,gate_scale = None, out_scale = None) on the 6-launch / 15-nodegather_qmv+moe_weighted_sumpath instead of the 2-launch / 5-node fused kernel path. Non-MoE contributors are the 4x per-layer f32 short convolutions, unfused banded relative-position bias, 24-oproute_weights, and on GB10 CUDA forinkling-small-nvfp4,MLX_MAX_MB_PER_BUFFER = 25(data_size()element count) forcing a CUDA graph commit pernvfp4gather_qmm.Step 3 Implementation (
f7733f8b)src/models/inkling/mlp.rs: WireInklingSparseMoe::forwardat single-token decode (flat.shape()[0] == 1 && self.gate_scale.is_none() && self.out_scale.is_none() && fused_moe_enabled()) toSwitchGLU::forward_fused_kernel, saving 4 GPU kernel launches and 10 graph nodes per sparse MoE layer oninkling-small-mlx-4bitwhile keepinginkling-small-nvfp4(gate_scale = Some, out_scale = Some) onforward_with_expert_scales.src/models/deepseek_v4_moe.rs: UpdateLimitedSwitchGlu::forwardto useexpand_dims_multi(x, &[-3, -2]),prepare_gather_indices, andSwitchLinear::forward_indexed, eliminating 6 redundant index-synthesis graph nodes per layer (258nodes/token across 43 layers).src/models/switch_layers.rs: UpdateUsed by:doc comments onfused_moe_max_dff_from,SwitchLinear::forward,SwitchLinear::forward_indexed,SwitchGLU::forward_fused_kernel, andprepare_gather_indices.Verification
cargo fmt --all -- --checkMLXCEL_DEVICE=cpu cargo test --profile test-fast --lib models::deepseek_v4::tests::limited_switch_glu -- --test-threads=1(1 passed)MLXCEL_DEVICE=cpu cargo test --profile test-fast --lib models::inkling::mlp::tests -- --test-threads=1(3 passed)MLXCEL_DEVICE=cpu cargo test --profile test-fast --lib models::inkling::tests -- --test-threads=1(passed)MLXCEL_DEVICE=cpu cargo test --profile test-fast --lib models::switch_layers::gather_index_tests -- --test-threads=1(5 passed)cargo clippy --profile test-fast -p mlxcel --lib --tests -- -D warnings(0 warnings)