Repository navigation
Studio: launch llama-server with --backend-sampling on CUDA when it does not speculate - #14409
Open
danielhanchen wants to merge 7 commits into
Open
danielhanchen wants to merge 7 commits into
danielhanchen wants to merge 7 commits into
Conversation
llama.cpp can run the per-slot sampler chain inside the decode graph (--backend-sampling). Without it every decode step copies each slot's full-vocabulary logits to the host and walks them on the CPU. Colab L4, -c 8192 -np 4 --kv-unified, Studio's own sampler chains: - Qwen3.5-0.8B (presence 1.5): 20.3 -> 7.0 ms per token at 4 streams, 8.1 -> 4.9 ms at 1 stream, flood throughput 154 -> 341 tok/s - Qwen3-0.6B: 7.3 -> 6.1 ms at 4 streams, 4.4 -> 4.1 ms at 1 stream - cost: ~20-25 ms more TTFT per request (llama.cpp re-reserves its scheduler for each new sampler) With identical logits and seed it draws the same tokens as the CPU chain. The flag is per request in llama-server and steps aside on its own for grammar / tools, a reasoning budget and pre-sampling logprobs. Studio adds it only when the binary advertises it (new probe key supports_backend_sampling, fails closed), a GPU holds the output layer, the backend is not Vulkan or macOS, and no speculative decoding runs (penalties cannot run on the GPU when a step verifies several tokens per slot, and there it measured no gain). A user's own flag and UNSLOTH_LLAMA_BACKEND_SAMPLING=0 win.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
llama.cpp's GPU min_p keeps logit - (max_logit + log p) > 0, so at min_p 1 (or within float rounding of it) it drops even the top token and draws from NaN probabilities, where the CPU chain keeps the argmax. Requests whose min_p is past 0.999 now carry backend_sampling: false, only when the spawned llama-server enables the GPU sampler (or the client asked for it per request), so older binaries never see the field. Applied at _open_stream, the GGUF TTS POST, _build_passthrough_payload (OpenAI, Responses and both Anthropic passthroughs) and the raw /v1/completions proxy. A launch whose default --min-p / LLAMA_ARG_MIN_P is past 0.999 does not get --backend-sampling at all. [skip ci]
llama.cpp's backend sampler aborts the whole server on a top_k past its candidates (ggml_top_k asserts, and every backend sampler is probed on a 128K-candidate graph), so a /v1/completions body with top_k 1000000, top_k 140000 on a 151936-token vocabulary, or a custom sampler order with temperature 0 ahead of top_k is a remote crash. It also samples another distribution than the CPU chain for min_keep > 1, dynatemp, duplicate logit_bias tokens, and a fixed seed without top_p (top_k's GPU order is not stable, so the seed stops reproducing). guard_gpu_min_p becomes guard_gpu_sampler, one check over every field the GPU part of the chain reads, on the same call sites (_open_stream, the GGUF TTS POST, the passthrough builder, raw /v1/completions). top_k is held to the GGUF's token count and the 128K probe (256 when the vocabulary is unknown); an omitted field takes the launch's own flag or LLAMA_ARG_* default, a launch default outside the envelope keeps the flag off, and an unparseable value is outside.
…ff the GPU prefix Launch flags the GPU sampler envelope parses are now matched through _flag_name, as common/arg.cpp folds a long option's underscores: --sampler_seq tk enabled backend sampling and recorded no custom order, so temperature 0 with top_k 20 aborted llama-server. Every --spec-type is appended (a later none no longer hides an earlier type), and the argv value wins over the env twin. XTC, DRY, typical-p, top-n-sigma and mirostat have no backend implementation, so an active one split the chain into a GPU prefix and a CPU suffix that saw the candidates top-p / min-p had masked to -inf; XTC at threshold 0 then kept an excluded token. A request with any of them active, from its own fields or a launch default, now samples on the CPU, and a launch default that activates one drops --backend-sampling. A NaN sampling value is outside the envelope.
…in /props
llama.cpp also takes sampling defaults, and backend-sampling itself, from a
config.ini (/etc/llama.cpp, ${XDG_CONFIG_HOME:-~/.config}/llama.cpp, %PROGRAMDATA% /
%APPDATA% on Windows) that no argv or env shows, so [default] sampler-seq = tk
plus a temperature 0, top_k 20 request still aborted the server.
Once the server is up, and before it is published, read
default_generation_settings.params from /props (the same params every request
copies its defaults from) and guard omitted fields with those; the argv reading
only fills logit_bias, which /props leaves out, and stays the pre-launch gate.
A GPU-sampling server whose /props is unreadable or lacks a field, or one with
no readable /props and a config.ini present, pins every request to the CPU.
Every spawn re-reads it, so a respawn or reload refreshes the defaults.
…ampler /props leaves logit_bias out, and llama.cpp appends the biases of every config section it applies, which the GPU sampler does not sum.
danielhanchen
added a commit
that referenced
this pull request
Oct 21, 2026
…mpile packing test off leaked env (#14425) ## Problem Two tests on main fail in CI for reasons in the tests, not the product. They are red on unrelated PRs, for example #14409 and #14365. **1. `tests/python/test_v100_fullft_precision.py` (10 tests, `Repo tests (CPU, studio)`)** ``` tests/python/test_v100_fullft_precision.py:80: in _decide exec(CODE, {"torch": torch, "os": os}, {"args": args, "model": model}) E NameError: name 'args' is not defined ``` The test runs the `mixed_precision` template from `unsloth/models/rl.py` with separate globals and locals dicts. #14394 added `_unsloth_precision_requested = tuple(getattr(args, k, False) for k in (...))` to that template. In the generated trainer this runs inside a function, where the generator sees `args`, so the product is fine. A generator inside `exec` with a separate locals dict cannot see those locals, so the test raises `NameError`. **2. `tests/test_legacy_block_compile_install.py::test_packed_calls_keep_the_unsloth_layer_forward` (`Repo tests (CPU, rest)`)** ``` tests/test_legacy_block_compile_install.py:105: in test_packed_calls_keep_the_unsloth_layer_forward E assert [] ``` The test (from #14398) expects an unpacked call to go through the compiled block. `_block_compile._wrap` keeps every call on the stock forward when `UNSLOTH_COMPILE_DISABLE` is `1`, `UNSLOTH_BLOCK_COMPILE` is `0`, or `UNSLOTH_FORCE_FLOAT32` is `1`. Eight test modules run `os.environ.setdefault("UNSLOTH_COMPILE_DISABLE", "1")` at import time, so the result depends on which modules a worker collected. The test passes alone and fails with any of those variables set. ## Fix Test only: - `test_v100_fullft_precision.py`: `exec` the template in one namespace, as in the trainer's function scope. Results are still read from `args` and the environment. - `test_legacy_block_compile_install.py`: the packing test clears the three variables with `monkeypatch.delenv` before wrapping. ## Verification - `test_v100_fullft_precision.py`: on main 10 failed (the `NameError` above); with this change 10 passed. - `test_legacy_block_compile_install.py` on main: - with any one of `UNSLOTH_COMPILE_DISABLE=1`, `UNSLOTH_BLOCK_COMPILE=0` or `UNSLOTH_FORCE_FLOAT32=1` set: 1 failed (the packing test), 5 passed; - with this change: 6 passed in each of those three cases and with none set.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Launch llama-server with
--backend-sampling(llama.cpp's GPU sampler) on CUDA GPU launches that do not speculate. With it the per-slot sampler chain runs inside the decode graph, so a decode step no longer copies every slot's full-vocabulary logits to the host and walks them on the CPU.Why
A Colab L4 profile of a 4-slot server (
-c 8192 -np 4 --kv-unified) showed the GPU idle for about 2.1 ms of a 7.4 ms decode step, about 1.3 ms of it CPU sampling. On the same L4, with Studio's own sampler chains, 4 slots, interleaved rounds:--backend-samplingA/A noise on ITL was 0.1 ms (Qwen3) and 0.9 ms (Qwen3.5).
Correctness
When the flag is added
All of: the binary's
--helpadvertises it (fails closed, so older builds launch exactly as before); a GPU is detected; the build has the CUDA backend (ROCm, SYCL and Vulkan are not measured yet); not macOS; the model is not placed on the CPU; no speculative decoding (penalties cannot run on the GPU when a step verifies several tokens, and there was no gain beside drafts); the user has not passed the flag orLLAMA_ARG_BACKEND_SAMPLING;UNSLOTH_LLAMA_BACKEND_SAMPLING=0is not set.Studio's Auto mode adds
--spec-defaultfor a plain GGUF, so today this engages when speculation is off or stands down.Testing
studio/backend/tests/test_backend_sampling_launch.py: 47 tests (capability probe, every gate, speculation detection, end-to-end launch argv). On main the file does not import; 8 behaviour tests fail against main's launch.test_capability_detection.py).