Skip to content

perf(moe): attribute M = 1 expert gather gap and wire Inkling/V4 MoE fast paths (#1859) - #2310

Merged
inureyes merged 3 commits into
mainfrom
perf/issue-1859-moe-decode-attribution
Oct 14, 2026
Merged

inureyes merged 3 commits into
mainfrom
perf/issue-1859-moe-decode-attribution

Conversation

@inureyes

Copy link
Copy Markdown
Member

Summary

Closes #1859.

Attributes the single-token (n_tokens = 1) decode throughput gap between mlxcel and mlx-lm / mlx-vlm on deepseek-v4-flash-4bit, inkling-small-mlx-4bit, and inkling-small-nvfp4, and lands the two concrete MoE fast paths identified by the attribution.

Step 2 Attribution Record (docs/benchmark_results/moe-m1-gather-attribution-2026-10-14.md)

  • Committed first in 9072e2ad prior to any edits to src/models/switch_layers.rs, src/models/deepseek_v4_moe.rs, or src/models/inkling/mlp.rs.
  • Metal GatherQMM::eval_gpu (overlays/metal/mlx/backend/metal/quantized.cpp:2060-2088): Checks M < get_qmv_batch_limit(K, N, d). Since get_qmv_batch_limit >= 6 on all Apple GPUs (12..33 on Max/Ultra/M5) and M = 1, GatherQMM unconditionally returns through gather_qmv (affine_gather_qmv_fast_bfloat16_gs_64_b_4 or fp_gather_qmv_fast_bfloat16_nvfp4). Commit 8ad62d4b (#1772, bm = 32 inside gather_qmm) is unreachable at M = 1.
  • CUDA GatherQMM::eval_gpu (overlays/cuda/mlx/backend/cuda/quantized/quantized.cpp:526-547): Checks if (can_use_qmm_sm80) { if (can_use_qmv && (M * B < 8)) call_qmv(); else call_qmm_sm80(); }. Unlike top_k = 8 models (M * B = 8 >= 8), deepseek-v4-flash-4bit, inkling-small-mlx-4bit, and inkling-small-nvfp4 all use top_k = 6, so M * B = 1 * 6 = 6 < 8 is true at single-token decode. Furthermore, supports_qmm_sm80 is false for nvfp4 (inkling-small-nvfp4). All three models therefore already dispatch M = 1, B = 6 expert projections to mlx::core::cu::gather_qmv_kernel, never qmm_sm80 (tile_m = 16).
  • ROCm GatherQMM::eval_gpu (overlays/rocm/mlx/backend/rocm/quantized/qmm.hip:6506-6726): Dispatches M = 1, B = 6 to gather_qmv_wide_kernel (affine bf16) or gather_qmv_kernel (nvfp4).
  • Reconciled forward_fused_kernel inventory: 17 model families wired to SwitchGLU::forward_fused_kernel + 2 family-local (qwen3_next, gemma4) = 19 families on 096152c6 (growing to 20 with inkling in this PR).
  • deepseek-v4-flash-4bit decode gap breakdown: Over 85% of the gap is non-MoE—specifically the 2x per-layer (86x/token) unfused 20-iteration float32 HyperConnection Sinkhorn loops (src/models/deepseek_v4_hyper.rs:38-42, 115-200, ~290 graph nodes/layer = ~12,470 nodes/token vs. upstream's fused _hc_sinkhorn_collapse_kernel), plus MLA KV compression and Indexer. Within the MoE block, LimitedSwitchGlu::forward missed #1713 / PR #2137, emitting 6 redundant index-synthesis graph nodes per layer (258 nodes/token across 43 layers).
  • inkling-small-mlx-4bit and inkling-small-nvfp4 decode gap breakdown: InklingSparseMoe::forward (src/models/inkling/mlp.rs:217-223) never called SwitchGLU::forward_fused_kernel, leaving inkling-small-mlx-4bit (affine 4-bit gs64, gate_scale = None, out_scale = None) on the 6-launch / 15-node gather_qmv + moe_weighted_sum path instead of the 2-launch / 5-node fused kernel path. Non-MoE contributors are the 4x per-layer f32 short convolutions, unfused banded relative-position bias, 24-op route_weights, and on GB10 CUDA for inkling-small-nvfp4, MLX_MAX_MB_PER_BUFFER = 25 (data_size() element count) forcing a CUDA graph commit per nvfp4 gather_qmm.

Step 3 Implementation (f7733f8b)

  • src/models/inkling/mlp.rs: Wire InklingSparseMoe::forward at single-token decode (flat.shape()[0] == 1 && self.gate_scale.is_none() && self.out_scale.is_none() && fused_moe_enabled()) to SwitchGLU::forward_fused_kernel, saving 4 GPU kernel launches and 10 graph nodes per sparse MoE layer on inkling-small-mlx-4bit while keeping inkling-small-nvfp4 (gate_scale = Some, out_scale = Some) on forward_with_expert_scales.
  • src/models/deepseek_v4_moe.rs: Update LimitedSwitchGlu::forward to use expand_dims_multi(x, &[-3, -2]), prepare_gather_indices, and SwitchLinear::forward_indexed, eliminating 6 redundant index-synthesis graph nodes per layer (258 nodes/token across 43 layers).
  • src/models/switch_layers.rs: Update Used by: doc comments on fused_moe_max_dff_from, SwitchLinear::forward, SwitchLinear::forward_indexed, SwitchGLU::forward_fused_kernel, and prepare_gather_indices.

Verification

  • cargo fmt --all -- --check
  • MLXCEL_DEVICE=cpu cargo test --profile test-fast --lib models::deepseek_v4::tests::limited_switch_glu -- --test-threads=1 (1 passed)
  • MLXCEL_DEVICE=cpu cargo test --profile test-fast --lib models::inkling::mlp::tests -- --test-threads=1 (3 passed)
  • MLXCEL_DEVICE=cpu cargo test --profile test-fast --lib models::inkling::tests -- --test-threads=1 (passed)
  • MLXCEL_DEVICE=cpu cargo test --profile test-fast --lib models::switch_layers::gather_index_tests -- --test-threads=1 (5 passed)
  • cargo clippy --profile test-fast -p mlxcel --lib --tests -- -D warnings (0 warnings)

…r indices (#1859)

- Wire `InklingSparseMoe::forward` at single-token decode (`flat.shape()[0] == 1`) to `SwitchGLU::forward_fused_kernel` when `gate_scale` and `out_scale` are absent (`inkling-small-mlx-4bit`), collapsing each sparse MoE layer's routed expert path from 6 GPU kernel launches and 15 graph nodes (`3 gather_qmv` + `compiled_swiglu_activation` + `moe_weighted_sum`) to 2 custom kernel launches and 5 graph nodes while preserving the scaled gather path for `inkling-small-nvfp4`.
- Update `LimitedSwitchGlu::forward` in `deepseek_v4_moe.rs` to use `expand_dims_multi(x, &[-3, -2])`, `prepare_gather_indices`, and `SwitchLinear::forward_indexed`, eliminating 6 redundant index-synthesis graph nodes per MoE layer (`258` nodes/token across 43 layers when `MoEGate` returns `INT32` indices).
- Refresh `Used by:` doc comments on `fused_moe_max_dff_from`, `SwitchLinear::forward`, `SwitchLinear::forward_indexed`, `SwitchGLU::forward_fused_kernel`, and `prepare_gather_indices` in `switch_layers.rs`.
- Add unit tests covering `InklingSparseMoe` fused routing vs. per-expert scale fallback and `LimitedSwitchGlu` bit-exactness and `AsType(uint32)` node reduction.
@inureyes inureyes added status:review Under review type:performance Performance improvements priority:medium Medium priority labels Oct 14, 2026
@inureyes

Copy link
Copy Markdown
Member Author

Implementation Review Summary

Intent

Attribute single-token MoE decode throughput gap across Metal, CUDA, and ROCm, land fused decode fast path for affine Inkling checkpoints, and share gathered index synthesis for DeepSeek-V4.

Findings Addressed

  • Step 2 attribution record committed before code changes (9072e2ad) with comprehensive kernel dispatch traces across Metal, CUDA, and ROCm (CRITICAL)
  • InklingSparseMoe::forward single-token affine decode wired to SwitchGLU::forward_fused_kernel while preserving per-expert scale fallback for NVFP4 checkpoints (HIGH)
  • LimitedSwitchGlu::forward modernized to expand_dims_multi + prepare_gather_indices + forward_indexed, eliminating 6 redundant index-synthesis graph nodes per layer (HIGH)
  • Used by: doc comments refreshed across switch_layers.rs for fused_moe_max_dff_from, SwitchLinear::forward, SwitchLinear::forward_indexed, SwitchGLU::forward_fused_kernel, and prepare_gather_indices (LOW)
  • Unit test suites added in deepseek_v4_tests.rs and inkling/mlp.rs pinning bit-exactness, AST node reduction, and routing guards (HIGH)

Remaining Items

  • None

Verification

  • All stated requirements implemented
  • No placeholder/mock code remaining
  • Integrated into project code flow
  • Project conventions followed
  • Existing modules reused where applicable
  • No unintended structural changes
  • Tests pass

@inureyes

Copy link
Copy Markdown
Member Author

PR Finalization Complete

Summary

  • Tests: Verified unit tests in src/models/deepseek_v4_tests.rs (limited_switch_glu_shares_gather_indices_and_is_bit_identical_to_per_call_form, 1 passed), src/models/inkling/mlp.rs (3 passed: concatenated_shared_experts_equal_explicit_expert_sum, inkling_sparse_moe_routes_one_token_affine_through_fused_kernel, inkling_sparse_moe_with_expert_scales_stays_on_scaled_gather_path), src/models/switch_layers.rs (gather_index_tests, 5 passed), and src/models/inkling/tests (17 passed).
  • Documentation: Verified comprehensive Step 2 attribution record committed in 9072e2ad (docs/benchmark_results/moe-m1-gather-attribution-2026-10-14.md) documenting exact kernel dispatch across Metal, CUDA, and ROCm, graph node counts, and rationale for not changing M = 1 tile sizes. Verified Used by: doc comments updated across src/models/switch_layers.rs (fused_moe_max_dff_from, SwitchLinear::forward, SwitchLinear::forward_indexed, SwitchGLU::forward_fused_kernel, prepare_gather_indices).
  • Lint/Format: Verified cargo fmt --all -- --check clean and cargo clippy --profile test-fast -p mlxcel --lib --tests -- -D warnings clean (0 warnings).

All checks passing. Ready for merge.

@inureyes inureyes added status:done Completed and removed status:review Under review labels Oct 14, 2026
@inureyes
inureyes merged commit 5a50847 into main Oct 14, 2026
11 checks passed
@inureyes
inureyes deleted the perf/issue-1859-moe-decode-attribution branch October 14, 2026 03:24
inureyes added a commit that referenced this pull request Oct 20, 2026
The 2026-09-08 rows measured the pre-#2310 decode path, recorded two VLM rows on a dirty tree, and labeled the baseline with a stale 0.6.16 from a detached mlx-vlm pin. Swap all ten for a fresh 2026-10-20 four-way run at 058d7f7 on current main (MLX 243bdf09): mlxcel text and VLM rows plus mlx-vlm --text-via-vlm and VLM baselines for the same three checkpoints, baselines now recording mlx-vlm 0.7.9 from upstream main.

The decode gap to the mlx-vlm baseline persists on M3 Ultra after #2310 wired the fused MoE kernel: mlxcel decodes at 23.7% (deepseek-v4-flash-4bit), 15.9% (inkling-small-mlx-4bit) and 18.5% (inkling-small-nvfp4) of the baseline on the text protocol, and 15.8% / 18.5% on the VLM protocol, each within ~3% of the September numbers on both sides.

A same-build control with MLXCEL_FUSED_MOE=0 decodes at 7.80 tok/s against 7.78 with the fused kernel enabled (metal_m3ultra_2026-10-20_nofused_single_inkling-small-mlx-4bit.csv), so the kernel is not the lever on this host: the gap must come from elsewhere in the batch-1 expert path. The nvfp4 control arm stays on the gather path and moved by the same amount as the fused-path model, so the rerun is internally consistent.

Refs #1858, #1859.
inureyes added a commit that referenced this pull request Oct 20, 2026
)

* bench: add M3 Ultra baselines for the three large MoE checkpoints

bench_mlxlm.py hardcoded its memory budget at 85% of 128 GB, so on the 512 GB M3 Ultra it skipped deepseek-v4-flash-4bit and both inkling-small checkpoints as SKIP:oom_estimate even though bench_decode.sh had just measured them. The budget now follows the host's own memory (hw.memsize on macOS, MemTotal on Linux), mirroring bench_decode.sh, and falls back to 128 GB when memory cannot be read. PYLM_BENCH_MAX_GB and the ROCm carve-out override still apply.

deepseek_v4 and inkling exist only in mlx-vlm, so the plain text path recorded FAIL:warmup for them. The new --text-via-vlm flag runs the same text protocol (synthesized 512-token prompt, no chat template, no image) but loads through mlx-vlm, records mlx-vlm as baseline_version, and writes to a separate pylm_*_textvlm_* file.

The ten CSVs are the 2026-09-08 M3 Ultra measurements: mlxcel text and VLM rows plus the mlx-vlm text and VLM baselines for the same three checkpoints.

Validated with py_compile, --help, and a module load that resolves the budget to 435 GB on this host.

* refactor(bench): reuse _host_memory_bytes for the pylm memory budget

The memory-budget fix added `_detect_memory_bytes()`, which shells out to `sysctl -n hw.memsize` on Darwin and parses `/proc/meminfo` on Linux. The file already had `_host_memory_bytes()`, a single portable `os.sysconf` call used by the ROCm hardware tagging, and both return the same value on the 512 GB M3 Ultra.

The budget now calls `_host_memory_bytes()` and the duplicate is gone. The helper moves above the module-level budget computation, since the budget runs at import time and the helper used to be defined later in the file. Behavior is unchanged: 435.2 GiB on the 512 GB host, `PYLM_BENCH_MAX_GB` still overrides (65 GiB checked), and the ROCm path still resolves the helper.

* bench: replace September CSVs with a post-#2310 rerun

The 2026-09-08 rows measured the pre-#2310 decode path, recorded two VLM rows on a dirty tree, and labeled the baseline with a stale 0.6.16 from a detached mlx-vlm pin. Swap all ten for a fresh 2026-10-20 four-way run at 058d7f7 on current main (MLX 243bdf09): mlxcel text and VLM rows plus mlx-vlm --text-via-vlm and VLM baselines for the same three checkpoints, baselines now recording mlx-vlm 0.7.9 from upstream main.

The decode gap to the mlx-vlm baseline persists on M3 Ultra after #2310 wired the fused MoE kernel: mlxcel decodes at 23.7% (deepseek-v4-flash-4bit), 15.9% (inkling-small-mlx-4bit) and 18.5% (inkling-small-nvfp4) of the baseline on the text protocol, and 15.8% / 18.5% on the VLM protocol, each within ~3% of the September numbers on both sides.

A same-build control with MLXCEL_FUSED_MOE=0 decodes at 7.80 tok/s against 7.78 with the fused kernel enabled (metal_m3ultra_2026-10-20_nofused_single_inkling-small-mlx-4bit.csv), so the kernel is not the lever on this host: the gap must come from elsewhere in the batch-1 expert path. The nvfp4 control arm stays on the gather path and moved by the same amount as the fused-path model, so the rerun is internally consistent.

Refs #1858, #1859.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority:medium Medium priority status:done Completed type:performance Performance improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(moe): attribute the MoE decode gap at the M = 1 expert gather

1 participant