Repository navigation
feat(speculative): GLM-4.7-Flash (glm4_moe_lite) MTP drafter - #1753
Merged
Merged
Conversation
inureyes
force-pushed
the
feature/issue-1326-glm4-moe-lite-mtp-drafter
branch
from
September 10, 2026 11:26
7832755 to
3677957
Compare
…t adapter GLM-4.7-Flash stores its one next-token-prediction layer as `model.layers.47.*` and the community 4-bit conversions drop it, so the drafter must be extracted from the raw checkpoint into its own directory. This adds `mlxcel split-mtp` (a surgery op that writes a standalone `glm4_moe_lite_mtp` directory and refuses a checkpoint whose index names no nextn layer), the stateful `Glm4MoeLiteMtpDraftModel` built on the decoder's own `TransformerBlock`, and an `MtpTarget` adapter for `Glm4MoeLiteModel` with an absorbed-MLA verify forward, trim-based rollback on a model-owned per-sequence cache slot, and the block-vs-chain exactness probe behind the shared gate. The offline CLI, the server burst, the tick slice and the speculative bench admit the family; requests that adopted a prompt-cache prefix decline to classic because the MTP slot cannot see the pool cache. Two decisions worth keeping: the verify attention runs one materialized query row at a time, because an unmaterialized slice of the batched query put MLX on a different kernel and the rows drifted from the decode chain data-dependently; and the drafter is fed the post-final-norm hidden, which measured 1.753 mean accepted length against 1.716 for the pre-norm residual (`MLXCEL_GLM_MTP_HIDDEN_TAP=pre`). Validated on `models/glm-4.7-flash-4bit` with the 4-bit drafter split from `models/glm-4.7-flash-bf16`: 128 greedy tokens are id-identical between classic decode and the MTP pairing under the same `qmv` kernel selection (`MLXCEL_QMV_WIDE=0`, the state the gate pins on Apple GPU generation 15+), with and without `MLXCEL_FUSED_MOE=0` (the family never dispatches the fused kernel). A wide-kernel classic run differs at one near-tie position, the #1199 effect, not the MTP path. The opt-in gate replays the same 128 positions through the `M = 2` verify block with 0 byte mismatches, and `mlxcel-server --model-draft` on port 19326 serves the same 128 ids through the tick slice. Narrow suites, clippy `-D warnings` and fmt pass. Closes #1326
The tick-slice drafter-return sites (`park_speculative_slice`, `finalize_speculative_slice`) carried no `LoadedModel::Glm4MoeLite` arm, so every parked and every finished GLM MTP slice session fell into the defensive `_ => None` arm and dropped the drafter handle instead of returning it to the worker slot, reloading the split-out drafter from disk once per request. The tree's own `every_mtp_dispatch_site_covers_every_capable_variant` guard was already failing on both sites; the narrow filters this branch was validated with do not reach `server::batch::speculative_burst_tests`. `split_mtp_dir` now refuses an `--output` that canonicalizes to the source. A sharded raw checkpoint carries no `model.safetensors`, so the CLI's `--force` guard does not fire on it, and the run would replace the source `config.json` with the drafter's and truncate its tokenizer files: `std::fs::copy` on a path to itself returns `Ok(0)` after opening the destination with `O_TRUNC`, verified on this host. `split_mtp` now refuses a next-token-prediction layer that arrives already quantized. `stack_experts` rejects unstacked packed experts by name, but a conversion whose experts were already stacked passed that check and reached the bf16 pass, which rewrote the packed uint32 payload as floats with no error. Both refusals carry unit tests. Narrow suites, clippy `-D warnings` and `cargo fmt --check` pass. Refs #1326
…lit-mtp Review follow-ups on the GLM-4.7-Flash MTP drafter. `glm4_moe_lite` is a pool-cached family, so the scheduler upgrades its caches inside `CachePool` after allocation (`apply_kv_cache_mode_to`) and the model never needed a mode table. The MTP slot this branch added is built by the model instead, from an unconditional `KVCache::new()`, and `Glm4MoeLiteModel` took the no-op `set_kv_cache_layer_modes` default, so nothing carried the resolved per-layer table to the caches an engaged MTP session actually runs on. It is the first pool-cached family to acquire a model-owned slot; Gemma 4 and Qwen 3.5 are model-owned throughout and both implement the hook. The model now holds a `KvCacheLayerModes` table, implements both trait methods, and every slot install and both arms of the block-vs-chain probe build through `make_configured_caches`, so the probe measures the arithmetic the session runs rather than an FP16 stand-in. The offline path injects the same resolved table before dispatch, since it builds no `GenerationConfig` and nothing else on it would reach the slot; that injection is scoped to this family because correcting the other MTP families' identical gap changes what their offline runs measure. No configuration reaches the mismatch today: `resolve_kv_cache_mode_for_model` (#1350) resolves every quantized mode to fp16 for the MLA-latent families, `glm4_moe_lite` among them, on both the `--kv-cache-mode` and the `--kv-bits` route, so the pool caches an MTP request runs beside are fp16 as well. The fix closes the structural gap and holds the invariant if this family ever gains a calibrated quantized mode. `split-mtp` gains two refusals. `--block-size` is bounded at 16: the recorded value becomes the server's default verify width, this family materializes one query row per verify position, and an unchecked `--block-size 20000` produced a directory that loaded fine and then built roughly a million graph nodes per round with the scheduler tick held. The `--force` guard now screens on whatever makes a directory a checkpoint rather than on `model.safetensors` alone, so an `--output` pointed at a sharded checkpoint no longer silently replaces its config and tokenizer files while orphaning the shards, and `--force` clears a stale index that would otherwise send the loader to those shards. Smaller: `decompose_kv_b_proj` converts its four geometry fields with `i32::try_from` and range-checks the head-dim product, because `as i32` could wrap past the shape check that exists to keep a malformed config from reaching `reshape` and aborting MLX across the bridge, and that helper is now on the target loader's path too. `num_nextn_predict_layers + 1` is saturating in both places, and a config missing both `block_size` and `num_nextn_predict_layers` is told which field is missing instead of being blamed for a `block_size` nobody wrote. `prefill_from_target_hidden` clears runtime state before the empty-prompt return so a recycled drafter cannot inherit the previous session's seed. `peek_drafter_model_type` gets the `Used by:` note. The hooks module doc records why the MoE stays batched while the attention does not, and `docs/supported-models.md` states that automatic prefix caching and MTP do not combine on this family. Rebased onto #1751 (LFM2 / LFM2.5 DSpark on the DFlash loop), which landed in the same files: the two block-size peeks key on different `DrafterKind`s and both stay, the drafter error enum carries both new variants, `load_drafter` in the burst resolves to the binary-crate wrapper this branch needs while `sampler_is_greedy` keeps coming from core, and the model-name match carries both families' arms. Re-validated on `models/glm-4.7-flash-4bit` with the 4-bit drafter: 128 greedy tokens are id-identical across classic and MTP, fused MoE on and off (md5 `f22d2a4b`), mean accepted length 1.753 over 73 rounds, unchanged from before these commits. Narrow suites (surgery 160, root `models::glm4_moe_lite` 33, `mlxcel-core` `drafter::` 179 and `mla::` 41, bin `commands::split_mtp` 2), clippy `-D warnings` on lib, tests and bins for all three crates, and `cargo fmt --check` pass. Refs #1326
inureyes
force-pushed
the
feature/issue-1326-glm4-moe-lite-mtp-drafter
branch
from
September 10, 2026 13:09
f31b405 to
6ddcefe
Compare
Bilingual report for PR #1753 (issue #1326), committed into the branch ahead of the merge because `TECHNICAL_REPORTS/.keep-reports` is present. Covers the `split-mtp` surgery op and every refusal it grew, the stateful drafter and the post-final-norm hidden tap the acceptance measurement settled on, the two width-sensitive pieces of the verify forward decided in opposite directions, the review finding that the model-owned MTP slot never saw the resolved KV cache mode table, the documented prefix-caching and MTP non-combination, the rebase onto #1751, and the validation the branch actually ran alongside what it does not establish. Refs #1326
This was referenced Sep 10, 2026
inureyes
added a commit
that referenced
this pull request
Sep 10, 2026
Records the split-mtp guard fixes: the refusal text, the --force contract that refuses beside any foreign safetensors file, and the producer-side bits and group-size validators, with the real-run result that the drafter output is byte-identical to the pre-PR binary's. Updates the PR #1753 report's resolved follow-up bullets to the final guard scope (any foreign *.safetensors, group size validated too), and corrects twelve doc comments that cited the PR #1778 review as "issue #1778".
inureyes
added a commit
that referenced
this pull request
Sep 10, 2026
…#1778) ## Summary The PR #1753 technical report listed three defects in `split-mtp` among its follow-ups. This PR fixes all three. Review of the first version found five more gaps in the same guards, and those are fixed here as well. - **Refusal text.** The `MAX_BLOCK_SIZE` refusal rendered with two runs of 14 literal spaces. It is now one sentence with single spaces. - **`--force` never deletes weight files.** `prepare_output_dir(dir, force)` refuses and names the file when the output directory holds any `*.safetensors` other than the drafter's own `model.safetensors`. The loader's `glob_safetensors` reads every `*.safetensors` in a directory, so a stale `model-0000N-of-...` or `consolidated.safetensors` would otherwise be loaded as part of the drafter. A stale `model.safetensors.index.json` is still cleared when no foreign file remains. An unreadable directory fails closed. - **Quantization parameters are checked before any tensor work.** `SUPPORTED_AFFINE_BITS` (2, 3, 4, 5, 6, 8) and `SUPPORTED_AFFINE_GROUP_SIZES` (32, 64, 128) now live in `mlxcel-core::layers`, next to `validate_affine_quantization_bits` and `validate_affine_quantization_group_size`. Before this PR, `--q-bits 1` or `--q-group-size 16` did the tensor work and then aborted the process inside MLX's quantize. Gemma 4's private copy of the bit set now reads the shared constant. `validate_quantization_params` still accepts `1..=32` at load time (#929). - **Checks run before mutation.** `preflight_checks` (bits, group size, output equals source) runs before `prepare_output_dir`. A bad flag or `-o` pointing at the source can no longer touch the directory first. ## Validation - Workspace gate on 059ea43: exit 0, 123 binaries, 11,038 passed, 0 failed. Clippy, fmt, `verify-versions`, `verify-kernel-dtype-keys` and `verify-llama-compat` are clean. CI is green. - Every new or strengthened test was run with its fix reverted and failed by name. With only the load-time `1..=32` check left, `--q-bits 1` aborts the test binary (`std::invalid_argument: [quantize] The requested number of bits 1 is not supported`, SIGABRT). - Real run with a release binary at 059ea43 against `glm-4.7-flash-bf16`: - `--q-bits 4` writes the drafter: 54 tensors, 0.72 GB, `glm4_moe_lite_mtp`, 4-bit affine, group size 64. `model.safetensors` and `config.json` are byte-identical to the output of the pre-PR binary (the #1772 merge). - `--force` over a planted `model-00001-of-00003.safetensors` exits 1 and names that file. Every file's name, size, mtime and inode is unchanged. - `--q-bits 7` exits 1 within a second with `quantization bits (7) must be one of 2, 3, 4, 5, 6, 8`, and no `drafter2` is created. Closes #1763.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GLM-4.7-Flash keeps its one next-token-prediction layer as
model.layers.47.*and the community 4-bit conversions drop it, so the drafter has to be split out of the raw checkpoint into its own directory. This addsmlxcel split-mtp(amlxcel-surgeryop that writes a standaloneglm4_moe_lite_mtpdirectory and refuses a checkpoint whose index names no nextn layer), the statefulGlm4MoeLiteMtpDraftModelbuilt on the decoder's ownTransformerBlock, and anMtpTargetadapter forGlm4MoeLiteModelwith an absorbed-MLA verify forward, trim-based rollback on a model-owned per-sequence cache slot, and the block-vs-chain exactness probe behind the shared gate. The offline CLI, the server burst, the tick slice and the speculative bench admit the family; requests that adopted a prompt-cache prefix decline to classic because the MTP slot cannot see the pool cache.Two decisions to keep when reviewing: the verify attention runs one materialized query row at a time, because an unmaterialized slice of the batched query put MLX on a different kernel and the rows drifted from the decode chain data-dependently; and the drafter is fed the post-final-norm hidden, which measured higher acceptance than the pre-norm residual (below, and in the drafter's module docs). The MoE is deliberately left batched:
SwitchGLUswitches to the gather-sort path atn_tokens * top_k >= 64, so the defaultblock_size = 2takes the same reduction as the decode chain while a wide--draft-block-sizecrosses the threshold and is left for the probe to decline. The drafter is built in the binary crate throughmodels::drafter_loader::load_drafterbecause it reuses theglm4_moe_liteblock; core refuses the model_type by name.Changes during review
glm4_moe_liteis a pool-cached family, so the scheduler upgrades its caches insideCachePoolafter allocation and the model never needed a mode table; the MTP slot this PR adds is built by the model instead, andGlm4MoeLiteModeltook the no-opset_kv_cache_layer_modesdefault. It is the first pool-cached family to acquire a model-owned slot. The model now holds aKvCacheLayerModestable, implements both trait methods, and every slot install plus both arms of the block-vs-chain probe build throughmake_configured_caches, so the probe measures the arithmetic the session runs. The offline path injects the same resolved table before dispatch, scoped to this family. No configuration reaches the mismatch today:resolve_kv_cache_mode_for_model(fix(core): symmetric Turbo4 on MLA-latent caches builds the K quantizer from the V head_dim and reads past the sign vectors, and the allowlist fallback the CLI announces does not exist #1350) already resolves every quantized mode to fp16 for the MLA-latent families on both the--kv-cache-modeand the--kv-bitsroute, so this closes the structural gap rather than a live regression, and holds the invariant if the family ever gains a calibrated quantized mode.split-mtp --block-sizeis bounded at 16. The recorded value becomes the server's default verify width and this family materializes one query row per verify position, so an unchecked--block-size 20000produced a directory that loaded fine and then built roughly a million graph nodes per round with the scheduler tick held.--forceguard screens on whatever makes a directory a checkpoint, not onmodel.safetensorsalone, so an--outputpointed at a sharded checkpoint no longer silently replaces its config and tokenizer files while orphaning the shards;--forcealso clears a stale index that would otherwise send the loader to those shards.decompose_kv_b_projconverts its geometry fields withi32::try_fromand range-checks the head-dim product.as i32could wrap past the shape check that exists to keep a malformed config from reachingreshapeand aborting MLX across the bridge, and that helper is now on the target loader's path too.num_nextn_predict_layers + 1is saturating in both places; a config missing bothblock_sizeandnum_nextn_predict_layersis told which field is missing;prefill_from_target_hiddenclears runtime state before the empty-prompt return so a recycled drafter cannot inherit the previous session's seed;peek_drafter_model_typegets itsUsed by:note.CachePool, so it donates nothing and a later turn finds no prefix to adopt. Confirmed safe (no partial entry can reach the store on either the dense or the paged arm); the cost is the lost optimization, anddocs/supported-models.mdnow says so rather than implying adoption normally works.Rebased onto #1751 (LFM2 / LFM2.5 DSpark on the DFlash loop), which landed in the same files: the two block-size peeks key on different
DrafterKinds and both stay, the drafter error enum carries both new variants,load_drafterin the burst resolves to the binary-crate wrapper this branch needs whilesampler_is_greedykeeps coming from core, and the model-name match carries both families' arms.Validation
Real checkpoints: target
models/glm-4.7-flash-4bit(16 GB), draftermodels/glm-4.7-flash-mtp-4bitproduced by./target/release/mlxcel split-mtp --model models/glm-4.7-flash-bf16 --output models/glm-4.7-flash-mtp-4bit --q-bits 4(705 MB,block_size: 2). Apple M5 Max, release binary rebuilt from this tree after the rebase. Every CLI run:mlxcel generate -m models/glm-4.7-flash-4bit -p "Explain in two sentences why the sky is blue." -n 128 --temp 0 --show-reasoningwithMLXCEL_PRINT_TOKEN_IDS=1, ids compared withcmp,--draft-model models/glm-4.7-flash-mtp-4biton the MTP arms.MLXCEL_QMV_WIDE=0(classic and MTP,MLXCEL_FUSED_MOEunset and=0) are id-identical over all 128 tokens, md5f22d2a4b.MLXCEL_FUSED_MOEchanges nothing in any arm becauseglm4_moe_litenever dispatches the fused single-token MoE kernel. Re-run three times: before the review fixes, after them, and after the rebase, with the same md5 each time.MLXCEL_GLM_MTP_HIDDEN_TAP=pre) measured 1.716 (53 of 74); post-final-norm stays the default and both taps emit the same ids.qmvkernel differs from every other arm at one near-tie position; that arm also differs from the narrow classic run, so the divergence is the feat(speculative): let the MTP gate buy back exactness by dropping qmv_wide #1199 kernel-selection effect the exactness gate exists for (the gate logsMTP exactness probe failed under qmv_wide ... passed without it. Disabling qmv_wide for this process), not the MTP path.MLXCEL_TEST_GLM_MTP_TARGET=models/glm-4.7-flash-4bit cargo test ... real_checkpoint_verify_block_matches_decode_chain -- --nocapture: 128 positions, 0 byte mismatches, worst abs diff 0.0, 0 argmax flips.MLXCEL_QMV_WIDE=0:mlxcel-server -m models/glm-4.7-flash-4bit --model-draft models/glm-4.7-flash-mtp-4bitversus the same server with no drafter,POST /completionwith the chat-templated prompt,temperature: 0,n_predict: 128,return_tokens: true. Both stop at EOS after 36 ids and the two id streams are identical. Startup logsMTP exactness probe passed: verify block is byte-identical to the single-token chain block_size=2; the request served through the tick slice at 18 rounds, 16 accepted,emitted_per_verify1.889, and/v1/internal/mtp-policyreportsprofilingwithacceptance_rate: 0.889. Both servers stopped afterwards.--kv-cache-mode int8on this checkpoint: classic and MTP are id-identical over 64 tokens and the banner reportsfp16 (requested int8; effective fp16; applied to 47 of 47 layers), which is fix(core): symmetric Turbo4 on MLA-latent caches builds the K quantizer from the V head_dim and reads past the sign vectors, and the allowlist fallback the CLI announces does not exist #1350's MLA-latent substitution and the reason the slot mismatch was not operator-visible.cargo test --profile test-fast -p mlxcel-surgery(160),--lib models::glm4_moe_lite(33),-p mlxcel-core drafter::(179) andmla::(41),--bin mlxcel commands::split_mtp(2).cargo clippy --lib --tests --features metal,accelerate -- -D warningsat root and for both member crates, pluscargo clippy --bins, all clean;cargo fmt --all -- --checkclean.Not run: the full
--workspacegate (machine shared with other agents; narrow scopes only). Batched B > 1 windows decline to classic by design and were not exercised.Closes #1326