Skip to content

Handle reasoning budget - #20297

Merged
pwilkin merged 13 commits into
ggml-org:masterfrom
pwilkin:reasoning-budget
Mar 11, 2026
Merged

pwilkin merged 13 commits into
ggml-org:masterfrom
pwilkin:reasoning-budget

Conversation

@pwilkin

@pwilkin pwilkin commented Mar 9, 2026 •

Copy link
Copy Markdown
Member

Adds proper handling for --reasoning-budget.

Currently, --reasoning-budget is just a stub that handles one case: 0, and the only think that does is set enable_thinking to false.

This PR adds the following flags:

  • --reasoning on (short -rea on) - enable reasoning via kwargs on model
  • --reasoning off (short -rea off) - disable reasoning via kwargs on model
  • --reasoning-budget-message - a message to be appended before the reasoning close marker to inform the model the reasoning was terminated due to budget constraints, i.e. " ... reasoning budget exceeded" or "... okay, now let's answer."

Also, --reasoning-budget now adds an extra grammar with a mechanism called delayed launch. Basically what this does is, when the opening trigger for the grammar is triggered, tokens get counted down and we also watch for the disarm trigger. If the disarm trigger doesn't get triggered before the countdown, the grammar gets launched.

This allows setting a real token budget limit on reasoning for models. This also allows disabling thinking for models that normally do not allow that by setting the budget to 0, which is now a different behavior than --disable-reasoning. Note: while possible, this isn't recommended since if a model was trained to only work with reasoning, it might exhibit aberrant behavior (for example, try to open an extra reasoning section).

Supersedes #17750

@pwilkin
pwilkin requested review from ggerganov and ngxson as code owners March 9, 2026 15:07
@pwilkin

pwilkin commented Mar 9, 2026

Copy link
Copy Markdown
Member Author

AI disclosure: I used Claude Opus in making most of the changes, auditing and modifying the critical code myself.

@pwilkin

pwilkin commented Mar 9, 2026

Copy link
Copy Markdown
Member Author

Oh, I didn't mention it in the note but this of course entails support for multiple grammars for one server task since the tool grammar is still there.

@pwilkin

pwilkin commented Mar 9, 2026

Copy link
Copy Markdown
Member Author

Some interesting observations from early tests (on Qwen3.5 9B Q8_0):

  • full model humaneval is around 93%
  • non-reasoning (-dre) is around 88%
  • using reasoning_budget 1000 and 400 is actually pretty similar and improves to about 89%
  • however this relies on having a --reasoning-budget-message (I used " ... reasoning budget exceeded, need to answer."). Without that, performance drops to a terrible 79%.

@ggerganov ggerganov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll probably need to understand this deeper, but on first look this seems very heavy logic. How important is this functionality?

Specifically the changes in common_sampler seem disproportionately large compared to what this brings to the existing logic. Look for ways to simplify.

@pwilkin

pwilkin commented Mar 9, 2026

Copy link
Copy Markdown
Member Author

I'll probably need to understand this deeper, but on first look this seems very heavy logic. How important is this functionality?

A lot of people have been requesting this, especially with the Qwen3.5 models that are seen as too verbose with their reasoning.

The changes in the sampler code are basically to the grammar sampler, since the idea is (a) to support more than one grammar simultaneously and (b) to support delayed grammar application (with token counting). Maybe this can be simplified by instead inserting another grammar sampler? Not sure how viable that would be.

@aldehir

aldehir commented Mar 9, 2026 •

Copy link
Copy Markdown
Contributor

In my opinion, we need to think long term.

The grammar sampler is incredibly inefficient. We had to revert a change @ggerganov wanted to make that shifts the grammar to the start of the chain to support backend sampling.

Merging this will increase the reliance on the grammar sampler and make it more challenging to optimize in the future.

I'm of the opinion that a dedicated, simple, reasoning sampler that lives in common would be enough and can be used at the start of the chain--so long as it aligns with the grammar used (if any).

@ggerganov

Copy link
Copy Markdown
Member

Yes, framing this as a reasoning sampler should definitely be explored.

@pwilkin
pwilkin force-pushed the reasoning-budget branch from 1df4d24 to b201c80 Compare March 9, 2026 21:15
@pwilkin

pwilkin commented Mar 9, 2026 •

Copy link
Copy Markdown
Member Author

@ggerganov @aldehir aight, reverted all the grammar changes and instead reimplemented it as a clean new reasoning parser.

I tested on cli, there is no noticeable overhead on generation (152 t/s both with and without the sampler).

Comment thread common/arg.cpp Resolved
Comment thread common/arg.cpp Resolved
Comment thread tools/server/server-context.cpp Resolved
@aldehir

aldehir commented Mar 9, 2026 •

Copy link
Copy Markdown
Contributor

Unless @ggerganov thinks otherwise, I would put it under common until it reaches maturity before exposing it in the public API. I imagine there will be quite a bit of churn with all the models to support.

Other notes:

  • Is arm_immediately needed? Why not define an initial state instead?
  • Need to add a soft/hard cap to handle partial utf8 sequences. When I tested this with a grammar approach, I would often see incomplete utf8 codepoints. Instead, we could enforce a soft cap then continue until we hit a clean boundary or hit the hard cap.
  • Would like to see some vision for incorporating other reasoning budget strategies. For example, Nemotron Nano 2 (which they claim to also support in their 3-series).
3.4. Budget Control Evaluation
Nemotron Nano V2 allows users to specify how many thinking tokens the model may generate before
producing the final answer. The final answer is the portion of text typically shown to end users.
This feature is implemented by counting tokens after the model begins generating the <think>
token. Once the budget is reached, the inference setup attempts to insert a closing </think> tag.
Rather than inserting it immediately, we let the model finish its current sentence and place the
tag at the next newline. In extreme cases where no newline appears, the system enforces closure
within 500 tokens past the budget: if no newline occurs by the (budget+ 500)th token, the </think>
tag is forcibly inserted.

https://arxiv.org/abs/2508.14444

Overall, I think this is a cleaner approach. It isolates the complexity rather than polluting the already complex grammar sampling logic.

@aldehir aldehir left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need some tests around the apply/accept logic. I had some in my example, but feel free to improvise.

Comment thread include/llama.h Resolved
@CISC

CISC commented Mar 9, 2026

Copy link
Copy Markdown
Member

@pwilkin

pwilkin commented Mar 9, 2026

Copy link
Copy Markdown
Member Author

Funny, wonder what happened here: https://github.com/ggml-org/llama.cpp/actions/runs/22877806623/job/66373589713?pr=20297

GitHub merge running on Windows? :D

@pwilkin

pwilkin commented Mar 10, 2026

Copy link
Copy Markdown
Member Author

Aight I got rid of the explosive terminology and fixed the newlines in the process :)

@pwilkin

pwilkin commented Mar 10, 2026

Copy link
Copy Markdown
Member Author

Okay, UTF-8 and tests are done, think this one's ready.

@github-actions github-actions Bot added the testing Everything test related label Mar 10, 2026
@pwilkin
pwilkin requested review from CISC, aldehir and ggerganov March 10, 2026 14:11
my-other-github-account pushed a commit to my-other-github-account/llama.cpp that referenced this pull request May 15, 2026
* v1

* Finished!

* Handlie cli

* Reasoning sampler

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Less explosive terminology :)

* Add utf-8 case and tests

* common : migrate reasoning budget sampler to common

* cont : clean up

* cont : expose state and allow passing as initial state

* cont : remove unused imports

* cont : update state machine doc string

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
my-other-github-account pushed a commit to my-other-github-account/llama.cpp that referenced this pull request May 15, 2026
my-other-github-account pushed a commit to my-other-github-account/llama.cpp that referenced this pull request May 15, 2026
This change refactors the reasoning_budget_message parameter from the
common params into the sampling parameters specifically. It also removes
the reasoning_budget common parameter and standardizes on the existing
reasoning_budget_tokens parameter in the sampling configuration.

Issue: ggml-org#20429
Original PR: ggml-org#20297
baramofme pushed a commit to baramofme/llama-cpp-turboquant that referenced this pull request May 23, 2026
This change refactors the reasoning_budget_message parameter from the
common params into the sampling parameters specifically. It also removes
the reasoning_budget common parameter and standardizes on the existing
reasoning_budget_tokens parameter in the sampling configuration.

Issue: ggml-org#20429
Original PR: ggml-org#20297
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 5, 2026
* v1

* Finished!

* Handlie cli

* Reasoning sampler

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Less explosive terminology :)

* Add utf-8 case and tests

* common : migrate reasoning budget sampler to common

* cont : clean up

* cont : expose state and allow passing as initial state

* cont : remove unused imports

* cont : update state machine doc string

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 5, 2026
MrLordCat referenced this pull request in MrLordCat/llama.cpp-rdna-lab Jul 16, 2026
* v1

* Finished!

* Handlie cli

* Reasoning sampler

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Less explosive terminology :)

* Add utf-8 case and tests

* common : migrate reasoning budget sampler to common

* cont : clean up

* cont : expose state and allow passing as initial state

* cont : remove unused imports

* cont : update state machine doc string

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
MrLordCat referenced this pull request in MrLordCat/llama.cpp-rdna-lab Jul 16, 2026
MrLordCat pushed a commit to MrLordCat/llama.cpp-rdna-lab that referenced this pull request Jul 16, 2026
This change refactors the reasoning_budget_message parameter from the
common params into the sampling parameters specifically. It also removes
the reasoning_budget common parameter and standardizes on the existing
reasoning_budget_tokens parameter in the sampling configuration.

Issue: ggml-org/llama.cpp#20429
Original PR: ggml-org/llama.cpp#20297
The-Monk pushed a commit to The-Monk/llama.cpp that referenced this pull request Aug 5, 2026
This change refactors the reasoning_budget_message parameter from the
common params into the sampling parameters specifically. It also removes
the reasoning_budget common parameter and standardizes on the existing
reasoning_budget_tokens parameter in the sampling configuration.

Issue: ggml-org/llama.cpp#20429
Original PR: ggml-org/llama.cpp#20297
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Aug 16, 2026
Gemma-4 26B-A4B hat ghost thought channels — das Modell emittiert
{thought}-Tokens in Endlosschleife trotz enable_thinking=False.
--reasoning off (PR ggml-org#20297) ist der zuverlässige Fix, chat_template_kwargs
wurde durch Autoparser-Bug ggml-org#20338 zeitweise ignoriert.

AGENTS.md ergänzt mit vollständiger Thinking-Doku.
zommiommy pushed a commit to zommiommy/llama.cpp that referenced this pull request Aug 18, 2026
* v1

* Finished!

* Handlie cli

* Reasoning sampler

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Less explosive terminology :)

* Add utf-8 case and tests

* common : migrate reasoning budget sampler to common

* cont : clean up

* cont : expose state and allow passing as initial state

* cont : remove unused imports

* cont : update state machine doc string

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
zommiommy pushed a commit to zommiommy/llama.cpp that referenced this pull request Aug 18, 2026
zommiommy pushed a commit to zommiommy/llama.cpp that referenced this pull request Aug 18, 2026
This change refactors the reasoning_budget_message parameter from the
common params into the sampling parameters specifically. It also removes
the reasoning_budget common parameter and standardizes on the existing
reasoning_budget_tokens parameter in the sampling configuration.

Issue: ggml-org#20429
Original PR: ggml-org#20297
Vect0rM added a commit to AtomicBot-ai/atomic-chat-core that referenced this pull request Nov 3, 2026
…#68)

* fix(hermes): patch only the top-level model block of config.yaml

The writer rewrote the first `default:`, `provider:` and `base_url:` lines
anywhere in ~/.hermes/config.yaml. Hermes repeats those keys under
auxiliary, delegation, cron, moa, tts, memory and others, so on any config
where one of them comes before `model:` (or where `model:` is a scalar or
missing) the patch landed in another block, sometimes breaking the YAML,
and left the chat route untouched. Hermes then ran on its built-in
defaults (empty model, provider auto), fell through to OpenRouter and got
"HTTP 400: No models provided" before any request reached Atomic Chat.

Edit only direct children of the top-level `model:` key: replace
`default`, `provider` and `base_url` in place (dropping a nested value a
scalar replaces), add the missing ones at the block's indent, turn a
scalar `model: <id>` into a block, and append a model section when there
is none. Also never add a second top-level `providers:` or
`custom_providers:` key (Hermes' ruamel reader rejects duplicate keys and
then runs on defaults), and recognise those keys with a trailing comment.

Golden fixture outputs are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(server): turn llama.cpp thinking off for Anthropic requests that ask for none

The Anthropic API thinks only when a request sets `thinking` to enabled or
adaptive. llama.cpp's /v1/messages maps only `enabled` and otherwise leaves
the chat template at the server default, which --jinja turns on for every
template that can think. Claude Code's session-title request (no `thinking`,
max_tokens 32000) made Qwen3.5 4B think for 20k tokens, and on a one-slot
server the user's own turn waited behind it (QA B08).

For a /messages request to a llama.cpp engine (llamacpp, llamacpp-upstream,
atomic-prism) that has no `thinking` or `thinking: {type: "disabled"}` and
does not set the switch itself, the core now adds
`chat_template_kwargs: {enable_thinking: false}`, which llama.cpp's Anthropic
conversion passes to the template. The chat-completions fallback gets the
same kwargs. Requests that ask for thinking pass byte for byte; other
engines are untouched. The proxy-http contract records the divergence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(server): pin the byte-exact relay of llama.cpp streams to usage-asking clients

B11 (Kilo Code showed no reply while the inspector recorded a completed
stream): Kilo (AI SDK openai-compatible with includeUsage) and OpenClaude
both send stream_options.include_usage themselves, so the proxy neither
rewrites their request nor strips anything from the reply. Pin that with a
llama.cpp b12019-shaped thinking-model reply written in small pieces cut
mid-line and between an event's two newlines: the client gets the upstream
bytes unchanged, watched or not, and a client that did not ask loses only
the usage trailer the proxy asked for, with the finish chunk and [DONE]
intact across split reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(server): keep max_tokens on a llama.cpp /messages fallback

A /messages request that llama.cpp's template rejects (Claude Code sends
mid-conversation system messages, which Qwen3.5's template refuses) is
retried as chat completions. The translation dropped max_tokens, as the
Rust proxy did, so on a server started with n_predict -1 the reply ran
until the context was full. llama.cpp's own /v1/messages honours the cap;
its fallback now keeps it too. Other engines keep the translation as it
was. The proxy-http contract records the divergence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(server): merge mid-conversation system messages before llama.cpp sees them

Claude Code sends its reminders as role "system" messages between turns
to a model it does not know. llama.cpp's /v1/messages passes them to the
chat template where they stand, and a strict template (Qwen3.5: "System
message must be at the beginning.") answers 500, so every later turn went
through the chat-completions fallback. That fallback merges the system
messages into the leading one but loses the reply's token usage (Claude
Code reported input_tokens 0 and never learned how full its context was)
and the turn's thinking blocks.

For a /messages request to a llama.cpp engine, the core now merges those
messages into the top-level system the same way before forwarding, so the
engine answers natively. Other engines and conversations without system
messages are sent as they came.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(integrations): hand the served context window to the agent writers

Agents told nothing about the window assume 128K-272K and never compact
before a 16K-64K local server refuses. ConfigureInput gains an optional
contextLength, and `atomic-chat launch` reads the window the session was
loaded with from its /props (falling back to the asked size without fit).
The writers use it in follow-up commits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(integrations): Zed's real config dir and the served window for Zed and ZCode

- zedConfigDir(): %APPDATA%\Zed on Windows, $XDG_CONFIG_HOME/zed on
  Linux, ~/.config/zed on macOS, as Zed resolves it (parity with the
  app's zed_config_dir).
- Zed: max_tokens = served window, max_output_tokens = min(8192, ctx/4),
  images: false (Zed's own default); 32768 when the window is unknown,
  since Zed requires a value.
- ZCode: contextWindow = served window, omitted when unknown so ZCode's
  default stands; output cap min(8192, ctx/4).
- Both keep the user's default model they replace in a
  <config>.atomic-chat-backup.json sidecar for the app's Disconnect.
- Golden zed fixtures: images false, and the sidecar where the seed had
  another provider as the default model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(integrations): find Zed and ZCode Preview off PATH in `launch`

Mirrors the app's zed_cli_candidates / zed_app_candidates and the ZCode
Preview build of zcode_app_candidates_for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(integrations): tell Kilo, OpenCode, MiMo and pi the served window and keep what Connect replaces

The OpenCode-family writers now describe the model the way these agents
need for a local server:

- `limit: { context, output: min(8192, ctx/4) }` when the window is known
  (omitted when not), so OpenCode and Kilo compact instead of running with
  `limit.context = 0`, and MiMo stops assuming a 1M window;
- `reasoning: true` and `interleaved: { field: "reasoning_content" }`, so
  a thinking model's answer is shown as thoughts plus text and its
  reasoning is sent back within a tool loop (B11);
- provider options `timeout: false, headerTimeout: false`, verified in
  the Kilo, OpenCode 1.15 and MiMo schemas: a long prompt on a slow
  machine no longer hits the 5-minute first-byte timeout.

pi gets `name`, `contextWindow`, `maxTokens` and a per-model
`compaction.modelOverrides["atomic/<id>"]` (reserve and recent history a
quarter of the window), the schema documented in pi's settings reference.

OpenCode and MiMo are read as JSON5 like Kilo (comments and trailing
commas are dropped on write); an existing `opencode.jsonc` /
`mimocode.jsonc` is written instead of the `.json` it overrides, and our
leftovers in that `.json` are removed. KILO_CONFIG_DIR, OPENCODE_CONFIG,
OPENCODE_CONFIG_DIR, MIMOCODE_HOME, PI_CODING_AGENT_DIR and
XDG_CONFIG_HOME are honoured. MiMo's `$schema` is the one MiMo serves.

Before `model`, `provider.atomic`, pi's default provider/model or its
`providers.atomic` are overwritten, a value that is not ours is kept in
`<file>.atomic-chat-backup.json` for the app's Disconnect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(integrations): run Goose with its own environment, start Kilo on our model

Goose no longer gets OPENAI_API_KEY, OPENAI_HOST and GOOSE_* written
user-wide (shell rc block, `setx`), which pointed every OpenAI SDK tool on
the machine at the local server. `launch goose` sets them on the child
only: GOOSE_PROVIDER, GOOSE_MODEL, OPENAI_BASE_URL with the server's API
prefix (instead of the bare host plus a hard-coded OPENAI_BASE_PATH),
OPENAI_API_KEY, and GOOSE_CONTEXT_LIMIT / GOOSE_MAX_TOKENS when the
window is known (names checked against goose 1.57). An ambient
OPENAI_HOST / OPENAI_BASE_PATH is cleared, since Goose prefers it to
OPENAI_BASE_URL. Configuring Goose now only removes what earlier
releases wrote: our marked block in .zshenv/.bashrc/.bash_profile (our
export lines only) and, on Windows, the user variables recognisably set
by `setx`.

Kilo's saved per-agent pick beats `model` in its config, so `launch kilo`
passes `--model atomic/<id>` (the TUI's own flag).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(contract): agent-config fixtures for the new Kilo, OpenCode, MiMo, pi and Goose output

Regenerated by the app's Rust writers (`opencode_family.rs`
`dump_family_fixtures`) and replayed byte for byte here: the model entry
with thinking, no timeouts and the window, pi's window and compaction
override, the backup files Connect leaves for Disconnect, the `.jsonc`
target with its `.json` cleaned, MiMo's corrected `$schema`, and Goose
writing nothing but removing what earlier releases wrote. Cases may carry
`context_length` (index note). CHECKSUM recomputed over this tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(models): count sliding-window KV layer by layer, with a window per slot

The KV estimate averaged a sliding-window model over every layer,
(full + window) / 2. llama.cpp b12019 keeps the whole window only in the
global layers and pad256(n_swa x sequences + n_ubatch) cells in each
sliding one (src/llama-kv-cache-iswa.cpp:73), so Gemma 3 was counted at
two and a half times its real cache at 32K and small Macs were narrowed
far more than needed.

The estimate now reads the per-layer sliding_window_pattern (inlined by
the GGUF reader at any length, like head_count_kv), a period from the
GGUF, or the architecture's built-in pattern (gemma2/3/3n, gpt-oss,
cohere2, exaone4, afmoe, smallthinker); Gemma 4's own SWA head sizes and
shared KV layers count too. An architecture with a window but no known
pattern is counted in full, the safe side. A new layout argument gives
the sequences sharing a unified cache and the micro-batch size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(llamacpp): agent-friendly defaults: shared KV slots, answer and thinking caps

Integrations audit 2026-11-03 (S3, S4, S5, X6, P3), llama.cpp b12019:

- Slots: -kvu is always sent, so --parallel N shares one KV pool of the
  whole window (src/llama-context.cpp:389-390) and costs no window memory
  beyond each sliding layer's per-slot window. parallel defaults to 2, so
  a title or classifier request no longer waits behind an agent turn.
  The fit margin and the small-Mac ceiling no longer multiply the KV
  cache by the slots (load-plan.ts); the ceiling is the pool itself.
- Continuous batching is sent both ways (--cont-batching /
  --no-cont-batching): llama.cpp has it on unless told otherwise, so the
  setting could never turn it off. Its default is now on, as llama.cpp's.
- n_predict 0 is automatic: 16384 tokens, never above the window core
  knows. Clients without max_tokens (Codex via Responses, Kilo, OpenCode)
  no longer generate until the window is full. -1 stays "no limit".
- New reasoning_budget setting (default 8192) sent as --reasoning-budget
  with --reasoning-budget-message from b8287, the first build that takes
  a token count (ggml-org/llama.cpp#20297); requests' own budgets win.
- --cache-ram 1024/2048 MiB on machines with at most 8/16 GiB of RAM
  (from b6723, ggml-org/llama.cpp#16391) instead of llama.cpp's 8 GiB.
- DEFAULT_CTX_LEN is 32768: a fit-off load without a window gets 32K (or
  the trained context) instead of the whole trained context, and MLX's
  default window is 32K; on a Mac below 16 GiB both are narrowed toward
  the previous 16K by the KV estimate, never below it.
- isContextOverflowFinishLength takes the engine's own answer cap into
  account: a length stop at --n-predict is no longer a context overflow.

The args fixtures from the Rust builder are compared without -kvu and
--no-cont-batching, recorded as deliberate divergences; the engine
contract gains the new flags and build gates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(managed): automatic context length and request count for vLLM and SGLang

Integrations audit 2026-11-03, S2: vLLM and SGLang defaulted to 8192
tokens and one request, so every coding agent's 13-27K-token prompt was
refused with context_length_exceeded and side requests queued behind the
agent's turn.

context_length and max_num_seqs / max_running_requests now default to 0,
automatic: the load picks the widest pair the engine's own memory rule
passes on the card it re-probes right before the container — 64K with
two requests, 32K with two, 32K alone, 16K, then the old 8K alone, at
which the memory check refuses exactly as before. The window never
exceeds the checkpoint's own (max_position_embeddings and the other keys
vLLM derives it from), leaves room for the output cap, and with a KV
size fixed by hand holds one full sequence of it. Both engines keep one
paged pool sized by their memory share, so the window only bounds a
request. The chosen pair reaches the container through the launch plan
and the session limits (/v1/models context) through planContextLength.
Values set by hand stay as they are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(settings): move stored values off the defaults core no longer ships

settings.json stores every provider value, defaults included, so a new
schema default reached fresh installs only. On open, the store now moves
a stored value that still equals an old default to the new one, once,
and records the migration under state.default_migrations; a value the
user sets afterwards, the old default included, stays. A moved value
bumps the revision so the app mirrors it.

Migrations (integrations audit 2026-11-03): llama.cpp parallel 1 -> 2,
cont_batching false -> true (llama.cpp always batched continuously),
n_predict -1 -> 0 (automatic cap); vLLM/SGLang context_length 8192 -> 0
and request count 1 -> 0 (automatic).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(integrations): keep Claude Code and OpenClaude writers in step with the app

The `launch` writers for Claude Code and OpenClaude now match the app's
claude_config.rs: each keeps a `<file>.atomic-chat-backup.json` sidecar
recording what every key held before, so the app's Disconnect can undo a
connection the CLI made; a reconnect undoes the previous one first, and
values an earlier release wrote are taken out rather than recorded.

Claude Code: empty CLAUDE_CODE_USE_BEDROCK/VERTEX/FOUNDRY, the terminal
title off, and with a known window CLAUDE_CODE_MAX_CONTEXT_TOKENS plus
CLAUDE_CODE_MAX_OUTPUT_TOKENS (min(8192, ctx/4)); $CLAUDE_CONFIG_DIR is
honoured.

OpenClaude: the profile carries the server's key again (OC-2: the
keyless profile got a 401 from a keyed server), maxContextLength and a
per-model output cap with a known window, empty values for the shell
variables that make OpenClaude skip its active profile, and the startup
profile its OPENAI_API_KEY. Our profile is matched by its id only, and
$OPENCLAUDE_CONFIG_DIR and the legacy .config.json are honoured.

Golden fixtures for both agents regenerated; the in-place replacement
case became openclaude__users_own_atomic_chat_profile_kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(server): keep mid-conversation system messages in place for llama.cpp

6f87e30 moved every system message inside an Anthropic conversation into
the top-level system. That rewrites the start of the prompt whenever Claude
Code sends a reminder, and llama.cpp then processes the whole conversation
again (13-27k tokens on hybrid and sliding-window models, whose prompt
cache cannot reuse anything past the change).

For a llama.cpp engine, only system messages before the first turn now
join `system`. A later one stays where it is, as user text wrapped in
<system-reminder>, merged into a neighbouring user turn so the roles still
alternate: before the text of the next user message, or at the end of its
last tool_result when it answers tool calls (llama.cpp puts a user
message's text before its tool results, so a text block there would land
between the call and its result). With no user message after it, it goes
to the end of the one before, or into a user message of its own. A
reminder sent with clear_at "next_user_message" is dropped once a user
message follows it. The chat-completions fallback gets the same messages.

The Responses shim (Codex to llama.cpp) placed developer and system items
the same hoisting way; it now keeps later ones in place too, never between
a tool call and its output. The shim contract case that recorded the Rust
hoisting is corrected as a known divergence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(server): advertise the served window and answer overflows as prompt too long

Agents did not know the window the local engine really serves, so none
of them compacted in time (audit X1). GET /v1/models now gives each local
model whose window is known `context_length` and `meta.n_ctx` (the field
llama-server's own list uses), and Muse Code's catalogue advertises the
real window instead of its 32k guess.

The window comes from the session: a declared session (vLLM, SGLang,
TensorRT-LLM) states it; a llama.cpp session is asked once per loaded
generation through /props (default_generation_settings.n_ctx, answered
from metadata without a slot). The core wires SessionWindows into the
server as `contextWindow`; a session target now carries its generation.
Other engines advertise none rather than a guess.

On /v1/messages, a llama.cpp context overflow (exceed_context_size_error
with n_prompt_tokens/n_ctx, or its message) on the native call or on the
chat-completions fallback is answered as the Anthropic API's 400
invalid_request_error "prompt is too long: N tokens > M maximum", the
wording Claude Code compacts on; the fallback is no longer tried for it.
OpenAI routes keep llama.cpp's own error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(server): keep each request's context window and cached prompt tokens

The Activity inspector showed the window of whatever session served the
model when the row was opened, not the one the request ran with (QA B04).
The trace now records the window of the session a request goes to when it
is routed (the session's declared window or the one SessionWindows read)
and sends it as `ctx_len` in the finish fields, so the log keeps it after
a reload or an unload.

The finish fields also carry `cached_tokens`, the part of the prompt the
engine reused, from llama.cpp's timings.cache_n, OpenAI's
prompt_tokens_details.cached_tokens and Anthropic's
cache_read_input_tokens. Prompt tokens now count the whole prompt:
Anthropic's input_tokens leaves out cache reads and writes, and llama.cpp's
timings.prompt_n only what it processed. A whole Anthropic reply is read
for usage and stop reason too. The Responses shim carries the cached part
into input_tokens_details instead of a hard-coded 0.

The inspector-telemetry contract records the new field as a known
divergence (null in every recorded case).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(server): give Codex's MCP namespace tools to llama.cpp

Codex sends the tools of each MCP server as a Responses namespace
({type: "namespace", name, tools: [...]}) and runs a call to one only
with that namespace. The Responses shim kept only top-level `function`
tools, so a local model never saw a Codex MCP tool (audit S7).

The shim now offers every function in a namespace as a chat function
named <ns>__<name> (the plain name in Codex's default `functions`
namespace; no second delimiter after a namespace that already ends in
one), described by its own description or else the namespace's. A
function_call item replayed from history and a namespaced tool_choice
use the same name. In the reply, whole or streamed, a call to such a name
is given back its `name` and `namespace`, so Codex routes it to the MCP
server. Freeform tools inside a namespace are dropped like top-level ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(server): cover the system-reminder wrapper

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(integrations): tell Codex the served window in its profile

With the session's window known, the Atomic Chat profile carries
model_context_window and model_auto_compact_token_limit (three quarters
of it), so Codex compacts before a 16K-64K local server refuses instead
of assuming 272K. Unknown, neither line is written. Byte-identical to
the desktop writer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(integrations): Cline, Droid, Copilot and dsh get the window and a backup

Ports the desktop writers (app wip/fam-codex), byte for byte:

- A known window is written with a reply cap of a quarter of it (8192
  at most), and nothing is invented when it is unknown: Cline's
  contextWindow/maxTokens on the entry `cline auth` saved, Droid's
  maxContextLimit/maxOutputTokens (instead of a fixed 16384), Copilot's
  maxPromptTokens/maxOutputTokens/maxContextWindowTokens, dsh's route.
- Each writer records what it overwrote in a
  <config>.atomic-chat-backup.json sidecar, which the desktop's
  Disconnect restores from.
- Droid names its entry (id, index) and selects it through
  sessionDefaultSettings.model; the legacy top-level selector is removed.
- Copilot writes an atomic-chat provider and model into
  .copilot/providers.json instead of persisting COPILOT_* in the shell
  rc, removes exactly the rc block an earlier release wrote, and
  `launch` selects the model with COPILOT_MODEL=atomic-chat/<model>.
- dsh 0.2 imports settings.yaml once into the profile, so the route and
  the default model go into profiles/web/cordis.patch.yml, re-serializing
  only our rows; a settings.yaml still present is kept in step, never
  created, and settings.yaml.imported is never touched.

The cline/copilot/droid/dsh agent-config fixtures change accordingly
(index note); the desktop writers produce the same bytes for every
copilot, droid and dsh case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(cli): start npm-shim agents through cmd.exe on Windows

`atomic-chat launch` spawned the detected launcher without a shell. On
Windows an npm-installed agent is a `.cmd` shim, which Node refuses to
spawn directly (EINVAL) and does not find by its bare name (ENOENT), so
every npm agent failed to start. Shims and bare names now run through
`cmd.exe /d /s /c "<line>"` with each token quoted for cmd; executables
and other platforms spawn as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(integrations): Hermes, OpenClaw and Atomic Agent writers record what they replace

Hermes: one route for the app and the CLI, bare `custom` with model.base_url,
model.api_key and model.context_length, which needs no custom_providers entry
(an inline list cannot take one) and carries the server key. The served
window is written as served, 65536 only when unknown. HERMES_HOME is honoured
on every platform (~, $NAME, ${NAME}, %NAME% expanded) and the sticky profile
from `hermes profile use` is the one written. No skeleton is seeded.

OpenClaw: the model entry carries contextWindow and maxTokens when the window
is known; the request budget moves to the provider (timeoutSeconds 600) and
the agents.defaults.timeoutSeconds: 240 older releases seeded is dropped;
gateway.auth.mode "none" is seeded only when no token or password is set.

Atomic Agent: llm.fallback.appendLocal false when there is no fallback block,
so a failure of ours is not retried on local-llama :8080; userModels carries
the served window.

Each writer records the keys it overwrote in <config>.atomic-chat-backup.json
(json-key-backup.ts), the same sidecar the app writes, so the app's Disconnect
restores a connection made by `atomic-chat launch`. The golden fixtures of the
three agents are regenerated and replayed byte for byte against the app's
Rust writers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(integrations): OpenHands, Poolside and Muse get their environment at launch only

Their writers persist nothing any more: the variables go on the child
`launch` starts (and inline in the app's terminal). Connect removes the
marked shell-rc block an older release wrote, from any rc file it may have
used, and nothing else; the old prefix sweep that deleted the user's own
`export META_*` / `POOLSIDE_*` / `LLM_*` lines is gone with the writer.

Poolside keeps `/v1` in POOLSIDE_STANDALONE_BASE_URL as its docs give it and
is told the served window (POOLSIDE_STANDALONE_CONTEXT_LENGTH) through
childEnv, which `launch` now passes the session's window to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(contract): recompute the app fixture checksum

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(integrations): cover the per-key backup and the legacy shell block

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: record the agent-serving defaults and the window the server advertises

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: keep context-growth tests independent of host RAM, /props and path separators

- sessions: mock node:os totalmem so the 32K default is not narrowed on
  small CI runners (7 GiB macOS, Linux arm64).
- public-server: load the overflowing model at an explicit 4K window
  instead of relying on the old 8K default.
- api-events: the external engine stub answers the /props window read and
  counts only chat requests.
- launch: ZCode's window is the one /v1/models advertises.
- json-backup: `~/` joins with the platform separator on Windows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: bring the new agent writers and overflow mapping up to their coverage floors

The macOS unit job runs test:coverage, which failed its per-file floors
on the code this branch added. Tests now cover:

- goose: every legacy local-host range, the Windows reg/PowerShell path
  and its IO_ERROR, the placeholder vs server key.
- hermes: CRLF input, %YAML directives, wrapped scalars, the providers
  block, the Disconnect backup reader, ~ and %NAME% homes, .env proxy
  bypass.
- opencode-style, openclaude, pi: stale $schema, unreadable backups, the
  sibling .json beside a .jsonc, earlier startup profiles.
- errors: anthropicPromptTooLong from fields and both message wordings,
  MLX KV-cache wordings, clientMaxTokens.
- mlx model-files, fit-margin, catalog: memory facts, shard renames, the
  256-token grid, Zed under LOCALAPPDATA.

goose.ts drops a `?? ''` fallback on a regex group that always matches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(e2e): give a cold Windows daemon 45 s to print its ready line

On windows-11-arm the first daemon of each e2e file spends 10-15 s in
the PowerShell hardware probe before it is ready (main included); on a
slow runner that crossed the 20 s wait and failed the start-up tests.
Other platforms keep 20 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

examples server testing Everything test related

Projects

None yet

Development

Successfully merging this pull request may close these issues.