Skip to content

feat(speculative): GLM-4.7-Flash (glm4_moe_lite) MTP drafter - #1753

Merged
inureyes merged 4 commits into
mainfrom
feature/issue-1326-glm4-moe-lite-mtp-drafter
Sep 10, 2026
Merged

inureyes merged 4 commits into
mainfrom
feature/issue-1326-glm4-moe-lite-mtp-drafter

Conversation

@inureyes

@inureyes inureyes commented Sep 10, 2026 •

Copy link
Copy Markdown
Member

GLM-4.7-Flash keeps its one next-token-prediction layer as model.layers.47.* and the community 4-bit conversions drop it, so the drafter has to be split out of the raw checkpoint into its own directory. This adds mlxcel split-mtp (a mlxcel-surgery op that writes a standalone glm4_moe_lite_mtp directory and refuses a checkpoint whose index names no nextn layer), the stateful Glm4MoeLiteMtpDraftModel built on the decoder's own TransformerBlock, and an MtpTarget adapter for Glm4MoeLiteModel with an absorbed-MLA verify forward, trim-based rollback on a model-owned per-sequence cache slot, and the block-vs-chain exactness probe behind the shared gate. The offline CLI, the server burst, the tick slice and the speculative bench admit the family; requests that adopted a prompt-cache prefix decline to classic because the MTP slot cannot see the pool cache.

Two decisions to keep when reviewing: the verify attention runs one materialized query row at a time, because an unmaterialized slice of the batched query put MLX on a different kernel and the rows drifted from the decode chain data-dependently; and the drafter is fed the post-final-norm hidden, which measured higher acceptance than the pre-norm residual (below, and in the drafter's module docs). The MoE is deliberately left batched: SwitchGLU switches to the gather-sort path at n_tokens * top_k >= 64, so the default block_size = 2 takes the same reduction as the decode chain while a wide --draft-block-size crosses the threshold and is left for the probe to decline. The drafter is built in the binary crate through models::drafter_loader::load_drafter because it reuses the glm4_moe_lite block; core refuses the model_type by name.

Changes during review

  • The KV cache mode now reaches the MTP slot. glm4_moe_lite is a pool-cached family, so the scheduler upgrades its caches inside CachePool after allocation and the model never needed a mode table; the MTP slot this PR adds is built by the model instead, and Glm4MoeLiteModel took the no-op set_kv_cache_layer_modes default. It is the first pool-cached family to acquire a model-owned slot. The model now holds a KvCacheLayerModes table, implements both trait methods, and every slot install plus both arms of the block-vs-chain probe build through make_configured_caches, so the probe measures the arithmetic the session runs. The offline path injects the same resolved table before dispatch, scoped to this family. No configuration reaches the mismatch today: resolve_kv_cache_mode_for_model (fix(core): symmetric Turbo4 on MLA-latent caches builds the K quantizer from the V head_dim and reads past the sign vectors, and the allowlist fallback the CLI announces does not exist #1350) already resolves every quantized mode to fp16 for the MLA-latent families on both the --kv-cache-mode and the --kv-bits route, so this closes the structural gap rather than a live regression, and holds the invariant if the family ever gains a calibrated quantized mode.
  • split-mtp --block-size is bounded at 16. The recorded value becomes the server's default verify width and this family materializes one query row per verify position, so an unchecked --block-size 20000 produced a directory that loaded fine and then built roughly a million graph nodes per round with the scheduler tick held.
  • The --force guard screens on whatever makes a directory a checkpoint, not on model.safetensors alone, so an --output pointed at a sharded checkpoint no longer silently replaces its config and tokenizer files while orphaning the shards; --force also clears a stale index that would otherwise send the loader to those shards.
  • decompose_kv_b_proj converts its geometry fields with i32::try_from and range-checks the head-dim product. as i32 could wrap past the shape check that exists to keep a malformed config from reaching reshape and aborting MLX across the bridge, and that helper is now on the target loader's path too.
  • Smaller: num_nextn_predict_layers + 1 is saturating in both places; a config missing both block_size and num_nextn_predict_layers is told which field is missing; prefill_from_target_hidden clears runtime state before the empty-prompt return so a recycled drafter cannot inherit the previous session's seed; peek_drafter_model_type gets its Used by: note.
  • Documented: automatic prefix caching and MTP do not combine on this family. An MTP-served request's target K/V lives on the model-owned slot rather than in CachePool, so it donates nothing and a later turn finds no prefix to adopt. Confirmed safe (no partial entry can reach the store on either the dense or the paged arm); the cost is the lost optimization, and docs/supported-models.md now says so rather than implying adoption normally works.

Rebased onto #1751 (LFM2 / LFM2.5 DSpark on the DFlash loop), which landed in the same files: the two block-size peeks key on different DrafterKinds and both stay, the drafter error enum carries both new variants, load_drafter in the burst resolves to the binary-crate wrapper this branch needs while sampler_is_greedy keeps coming from core, and the model-name match carries both families' arms.

Validation

Real checkpoints: target models/glm-4.7-flash-4bit (16 GB), drafter models/glm-4.7-flash-mtp-4bit produced by ./target/release/mlxcel split-mtp --model models/glm-4.7-flash-bf16 --output models/glm-4.7-flash-mtp-4bit --q-bits 4 (705 MB, block_size: 2). Apple M5 Max, release binary rebuilt from this tree after the rebase. Every CLI run: mlxcel generate -m models/glm-4.7-flash-4bit -p "Explain in two sentences why the sky is blue." -n 128 --temp 0 --show-reasoning with MLXCEL_PRINT_TOKEN_IDS=1, ids compared with cmp, --draft-model models/glm-4.7-flash-mtp-4bit on the MTP arms.

  • Four arms under MLXCEL_QMV_WIDE=0 (classic and MTP, MLXCEL_FUSED_MOE unset and =0) are id-identical over all 128 tokens, md5 f22d2a4b. MLXCEL_FUSED_MOE changes nothing in any arm because glm4_moe_lite never dispatches the fused single-token MoE kernel. Re-run three times: before the review fixes, after them, and after the rebase, with the same md5 each time.
  • Mean accepted length 1.753 (55 of 73 proposals), unchanged across all three runs. The pre-norm tap (MLXCEL_GLM_MTP_HIDDEN_TAP=pre) measured 1.716 (53 of 74); post-final-norm stays the default and both taps emit the same ids.
  • A classic run with the default wide qmv kernel differs from every other arm at one near-tie position; that arm also differs from the narrow classic run, so the divergence is the feat(speculative): let the MTP gate buy back exactness by dropping qmv_wide #1199 kernel-selection effect the exactness gate exists for (the gate logs MTP exactness probe failed under qmv_wide ... passed without it. Disabling qmv_wide for this process), not the MTP path.
  • Opt-in gate MLXCEL_TEST_GLM_MTP_TARGET=models/glm-4.7-flash-4bit cargo test ... real_checkpoint_verify_block_matches_decode_chain -- --nocapture: 128 positions, 0 byte mismatches, worst abs diff 0.0, 0 argmax flips.
  • Server on port 19326, both sides under MLXCEL_QMV_WIDE=0: mlxcel-server -m models/glm-4.7-flash-4bit --model-draft models/glm-4.7-flash-mtp-4bit versus the same server with no drafter, POST /completion with the chat-templated prompt, temperature: 0, n_predict: 128, return_tokens: true. Both stop at EOS after 36 ids and the two id streams are identical. Startup logs MTP exactness probe passed: verify block is byte-identical to the single-token chain block_size=2; the request served through the tick slice at 18 rounds, 16 accepted, emitted_per_verify 1.889, and /v1/internal/mtp-policy reports profiling with acceptance_rate: 0.889. Both servers stopped afterwards.
  • --kv-cache-mode int8 on this checkpoint: classic and MTP are id-identical over 64 tokens and the banner reports fp16 (requested int8; effective fp16; applied to 47 of 47 layers), which is fix(core): symmetric Turbo4 on MLA-latent caches builds the K quantizer from the V head_dim and reads past the sign vectors, and the allowlist fallback the CLI announces does not exist #1350's MLA-latent substitution and the reason the slot mismatch was not operator-visible.
  • Tests: cargo test --profile test-fast -p mlxcel-surgery (160), --lib models::glm4_moe_lite (33), -p mlxcel-core drafter:: (179) and mla:: (41), --bin mlxcel commands::split_mtp (2). cargo clippy --lib --tests --features metal,accelerate -- -D warnings at root and for both member crates, plus cargo clippy --bins, all clean; cargo fmt --all -- --check clean.

Not run: the full --workspace gate (machine shared with other agents; narrow scopes only). Batched B > 1 windows decline to classic by design and were not exercised.

Closes #1326

@inureyes inureyes added status:review Under review type:enhancement New features, capabilities, or significant additions priority:low Low priority area:models Model architectures, weights, loading, metadata area:docs User and developer documentation area:core mlxcel-core: MLX FFI, primitives, KV cache, layers area:inference Generation, sampling, decoding (incl. speculative, DRY) area:cli Command-line interface / CLI flags modelsize:medium 4-bit checkpoint 10-50GB; comfortable on a 128GB dev box modeltype:text Text-only language model arch:moe Sparse mixture-of-experts decoder labels Sep 10, 2026
@inureyes
inureyes force-pushed the feature/issue-1326-glm4-moe-lite-mtp-drafter branch from 7832755 to 3677957 Compare September 10, 2026 11:26
…t adapter

GLM-4.7-Flash stores its one next-token-prediction layer as `model.layers.47.*` and the community 4-bit conversions drop it, so the drafter must be extracted from the raw checkpoint into its own directory. This adds `mlxcel split-mtp` (a surgery op that writes a standalone `glm4_moe_lite_mtp` directory and refuses a checkpoint whose index names no nextn layer), the stateful `Glm4MoeLiteMtpDraftModel` built on the decoder's own `TransformerBlock`, and an `MtpTarget` adapter for `Glm4MoeLiteModel` with an absorbed-MLA verify forward, trim-based rollback on a model-owned per-sequence cache slot, and the block-vs-chain exactness probe behind the shared gate. The offline CLI, the server burst, the tick slice and the speculative bench admit the family; requests that adopted a prompt-cache prefix decline to classic because the MTP slot cannot see the pool cache.

Two decisions worth keeping: the verify attention runs one materialized query row at a time, because an unmaterialized slice of the batched query put MLX on a different kernel and the rows drifted from the decode chain data-dependently; and the drafter is fed the post-final-norm hidden, which measured 1.753 mean accepted length against 1.716 for the pre-norm residual (`MLXCEL_GLM_MTP_HIDDEN_TAP=pre`).

Validated on `models/glm-4.7-flash-4bit` with the 4-bit drafter split from `models/glm-4.7-flash-bf16`: 128 greedy tokens are id-identical between classic decode and the MTP pairing under the same `qmv` kernel selection (`MLXCEL_QMV_WIDE=0`, the state the gate pins on Apple GPU generation 15+), with and without `MLXCEL_FUSED_MOE=0` (the family never dispatches the fused kernel). A wide-kernel classic run differs at one near-tie position, the #1199 effect, not the MTP path. The opt-in gate replays the same 128 positions through the `M = 2` verify block with 0 byte mismatches, and `mlxcel-server --model-draft` on port 19326 serves the same 128 ids through the tick slice. Narrow suites, clippy `-D warnings` and fmt pass.

Closes #1326
The tick-slice drafter-return sites (`park_speculative_slice`, `finalize_speculative_slice`) carried no `LoadedModel::Glm4MoeLite` arm, so every parked and every finished GLM MTP slice session fell into the defensive `_ => None` arm and dropped the drafter handle instead of returning it to the worker slot, reloading the split-out drafter from disk once per request. The tree's own `every_mtp_dispatch_site_covers_every_capable_variant` guard was already failing on both sites; the narrow filters this branch was validated with do not reach `server::batch::speculative_burst_tests`.

`split_mtp_dir` now refuses an `--output` that canonicalizes to the source. A sharded raw checkpoint carries no `model.safetensors`, so the CLI's `--force` guard does not fire on it, and the run would replace the source `config.json` with the drafter's and truncate its tokenizer files: `std::fs::copy` on a path to itself returns `Ok(0)` after opening the destination with `O_TRUNC`, verified on this host.

`split_mtp` now refuses a next-token-prediction layer that arrives already quantized. `stack_experts` rejects unstacked packed experts by name, but a conversion whose experts were already stacked passed that check and reached the bf16 pass, which rewrote the packed uint32 payload as floats with no error.

Both refusals carry unit tests. Narrow suites, clippy `-D warnings` and `cargo fmt --check` pass.

Refs #1326
…lit-mtp

Review follow-ups on the GLM-4.7-Flash MTP drafter.

`glm4_moe_lite` is a pool-cached family, so the scheduler upgrades its caches inside `CachePool` after allocation (`apply_kv_cache_mode_to`) and the model never needed a mode table. The MTP slot this branch added is built by the model instead, from an unconditional `KVCache::new()`, and `Glm4MoeLiteModel` took the no-op `set_kv_cache_layer_modes` default, so nothing carried the resolved per-layer table to the caches an engaged MTP session actually runs on. It is the first pool-cached family to acquire a model-owned slot; Gemma 4 and Qwen 3.5 are model-owned throughout and both implement the hook. The model now holds a `KvCacheLayerModes` table, implements both trait methods, and every slot install and both arms of the block-vs-chain probe build through `make_configured_caches`, so the probe measures the arithmetic the session runs rather than an FP16 stand-in. The offline path injects the same resolved table before dispatch, since it builds no `GenerationConfig` and nothing else on it would reach the slot; that injection is scoped to this family because correcting the other MTP families' identical gap changes what their offline runs measure.

No configuration reaches the mismatch today: `resolve_kv_cache_mode_for_model` (#1350) resolves every quantized mode to fp16 for the MLA-latent families, `glm4_moe_lite` among them, on both the `--kv-cache-mode` and the `--kv-bits` route, so the pool caches an MTP request runs beside are fp16 as well. The fix closes the structural gap and holds the invariant if this family ever gains a calibrated quantized mode.

`split-mtp` gains two refusals. `--block-size` is bounded at 16: the recorded value becomes the server's default verify width, this family materializes one query row per verify position, and an unchecked `--block-size 20000` produced a directory that loaded fine and then built roughly a million graph nodes per round with the scheduler tick held. The `--force` guard now screens on whatever makes a directory a checkpoint rather than on `model.safetensors` alone, so an `--output` pointed at a sharded checkpoint no longer silently replaces its config and tokenizer files while orphaning the shards, and `--force` clears a stale index that would otherwise send the loader to those shards.

Smaller: `decompose_kv_b_proj` converts its four geometry fields with `i32::try_from` and range-checks the head-dim product, because `as i32` could wrap past the shape check that exists to keep a malformed config from reaching `reshape` and aborting MLX across the bridge, and that helper is now on the target loader's path too. `num_nextn_predict_layers + 1` is saturating in both places, and a config missing both `block_size` and `num_nextn_predict_layers` is told which field is missing instead of being blamed for a `block_size` nobody wrote. `prefill_from_target_hidden` clears runtime state before the empty-prompt return so a recycled drafter cannot inherit the previous session's seed. `peek_drafter_model_type` gets the `Used by:` note. The hooks module doc records why the MoE stays batched while the attention does not, and `docs/supported-models.md` states that automatic prefix caching and MTP do not combine on this family.

Rebased onto #1751 (LFM2 / LFM2.5 DSpark on the DFlash loop), which landed in the same files: the two block-size peeks key on different `DrafterKind`s and both stay, the drafter error enum carries both new variants, `load_drafter` in the burst resolves to the binary-crate wrapper this branch needs while `sampler_is_greedy` keeps coming from core, and the model-name match carries both families' arms.

Re-validated on `models/glm-4.7-flash-4bit` with the 4-bit drafter: 128 greedy tokens are id-identical across classic and MTP, fused MoE on and off (md5 `f22d2a4b`), mean accepted length 1.753 over 73 rounds, unchanged from before these commits. Narrow suites (surgery 160, root `models::glm4_moe_lite` 33, `mlxcel-core` `drafter::` 179 and `mla::` 41, bin `commands::split_mtp` 2), clippy `-D warnings` on lib, tests and bins for all three crates, and `cargo fmt --check` pass.

Refs #1326
@inureyes
inureyes force-pushed the feature/issue-1326-glm4-moe-lite-mtp-drafter branch from f31b405 to 6ddcefe Compare September 10, 2026 13:09
@inureyes inureyes added status:done Completed and removed status:review Under review labels Sep 10, 2026
Bilingual report for PR #1753 (issue #1326), committed into the branch ahead of the merge because `TECHNICAL_REPORTS/.keep-reports` is present.

Covers the `split-mtp` surgery op and every refusal it grew, the stateful drafter and the post-final-norm hidden tap the acceptance measurement settled on, the two width-sensitive pieces of the verify forward decided in opposite directions, the review finding that the model-owned MTP slot never saw the resolved KV cache mode table, the documented prefix-caching and MTP non-combination, the rebase onto #1751, and the validation the branch actually ran alongside what it does not establish.

Refs #1326
@inureyes
inureyes merged commit e3ca8af into main Sep 10, 2026
13 checks passed
@inureyes
inureyes deleted the feature/issue-1326-glm4-moe-lite-mtp-drafter branch September 10, 2026 13:39
inureyes added a commit that referenced this pull request Sep 10, 2026
Records the split-mtp guard fixes: the refusal text, the --force contract that refuses beside any foreign safetensors file, and the producer-side bits and group-size validators, with the real-run result that the drafter output is byte-identical to the pre-PR binary's.

Updates the PR #1753 report's resolved follow-up bullets to the final guard scope (any foreign *.safetensors, group size validated too), and corrects twelve doc comments that cited the PR #1778 review as "issue #1778".
inureyes added a commit that referenced this pull request Sep 10, 2026
…#1778)

## Summary

The PR #1753 technical report listed three defects in `split-mtp` among its follow-ups. This PR fixes all three. Review of the first version found five more gaps in the same guards, and those are fixed here as well.

- **Refusal text.** The `MAX_BLOCK_SIZE` refusal rendered with two runs of 14 literal spaces. It is now one sentence with single spaces.
- **`--force` never deletes weight files.** `prepare_output_dir(dir, force)` refuses and names the file when the output directory holds any `*.safetensors` other than the drafter's own `model.safetensors`. The loader's `glob_safetensors` reads every `*.safetensors` in a directory, so a stale `model-0000N-of-...` or `consolidated.safetensors` would otherwise be loaded as part of the drafter. A stale `model.safetensors.index.json` is still cleared when no foreign file remains. An unreadable directory fails closed.
- **Quantization parameters are checked before any tensor work.** `SUPPORTED_AFFINE_BITS` (2, 3, 4, 5, 6, 8) and `SUPPORTED_AFFINE_GROUP_SIZES` (32, 64, 128) now live in `mlxcel-core::layers`, next to `validate_affine_quantization_bits` and `validate_affine_quantization_group_size`. Before this PR, `--q-bits 1` or `--q-group-size 16` did the tensor work and then aborted the process inside MLX's quantize. Gemma 4's private copy of the bit set now reads the shared constant. `validate_quantization_params` still accepts `1..=32` at load time (#929).
- **Checks run before mutation.** `preflight_checks` (bits, group size, output equals source) runs before `prepare_output_dir`. A bad flag or `-o` pointing at the source can no longer touch the directory first.

## Validation

- Workspace gate on 059ea43: exit 0, 123 binaries, 11,038 passed, 0 failed. Clippy, fmt, `verify-versions`, `verify-kernel-dtype-keys` and `verify-llama-compat` are clean. CI is green.
- Every new or strengthened test was run with its fix reverted and failed by name. With only the load-time `1..=32` check left, `--q-bits 1` aborts the test binary (`std::invalid_argument: [quantize] The requested number of bits 1 is not supported`, SIGABRT).
- Real run with a release binary at 059ea43 against `glm-4.7-flash-bf16`:
  - `--q-bits 4` writes the drafter: 54 tensors, 0.72 GB, `glm4_moe_lite_mtp`, 4-bit affine, group size 64. `model.safetensors` and `config.json` are byte-identical to the output of the pre-PR binary (the #1772 merge).
  - `--force` over a planted `model-00001-of-00003.safetensors` exits 1 and names that file. Every file's name, size, mtime and inode is unchanged.
  - `--q-bits 7` exits 1 within a second with `quantization bits (7) must be one of 2, 3, 4, 5, 6, 8`, and no `drafter2` is created.

Closes #1763.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arch:moe Sparse mixture-of-experts decoder area:cli Command-line interface / CLI flags area:core mlxcel-core: MLX FFI, primitives, KV cache, layers area:docs User and developer documentation area:inference Generation, sampling, decoding (incl. speculative, DRY) area:models Model architectures, weights, loading, metadata modelsize:medium 4-bit checkpoint 10-50GB; comfortable on a 128GB dev box modeltype:text Text-only language model priority:low Low priority status:done Completed type:enhancement New features, capabilities, or significant additions

Projects

None yet

1 participant