Retired for now (2026-10-22). No further development is planned on ninfer-ext for the time being. The repository, its artifacts and its docs stay as they are, and the tree still builds and serves everything documented below; it is simply not being worked on.
The chart above is where this project stands on the workload it was built for. On one RTX 5090, Qwen3.8-Flash-Next NVFP4 at the full 262,144-token context, two engines of the same lineage measured faster at every depth in both greedy and sampled decoding. Prefill: 8.1k-9.2k tok/s for basalt and 6.8k-8.0k for capy-flash against 4.2k-6.3k here. Decode, greedy: 176-265 and 138-198 tok/s against 73-106; sampling lowers all three (146-242, 130-178, 57-99) and changes no ordering. The difference is the expert strategy: ninfer-ext copies every expert-cache miss over PCIe and computes it on the GPU, while the other two compute misses on the CPU cores, which on this host wins by a wide margin.
Conditions: one RTX 5090 at a 475 W power limit (driver 615.71, CUDA 13.3), 262,144-token context, 256 output tokens per request, MTP speculation on, INT8 KV, one request at a time, a cold unique prompt per point, fastest tuned configuration of each engine. Measured with an external cross-engine harness, not
ninfer_bench, so these numbers are not comparable with the tool-based tables further down.
Both alternatives below belong to the same NInfer lineage, both are actively developed, and both measured faster here on this model, this box, this test and comparable tuning effort. If you came for Qwen3.8-Flash-Next at long context, use one of them:
- basalt - the pick for one 32 GB card at long context.
Fastest of the three on both axes at every depth measured (prefill 8.1k-9.2k tok/s, decode
176-265 tok/s greedy), and the most complete packaging: one
.basaltfile carries the weights, tokenizer, chat template, MTP drafter, expert profile and vision projector, so there is no sidecar to keep in step. The adaptive expert tier that follows routing,basalt-probederiving suggested flags from the machine's own measured rates, a parity test per kernel and a PPL/KLD quality gate against Q8_0 are the kind of work that never shows up in a throughput table, and they are why it is the one worth building on. - capy-flash - the pick for two 16 GB consumer cards. It tensor-parallelises this 125B model over PCIe without peer-to-peer, partitions the expert cache between the cards and computes the misses on the CPU, then gets the agent work right: idle conversations move to RAM and hand their VRAM back to the expert cache, a continued conversation keeps the experts its next answer needs, and the prefix cache spans turns. Its quality reporting is the most careful of the three - the same prompts, paired, against NVIDIA's NVFP4 checkpoint served by vLLM on a GB300 (GSM8K 0.9750 vs 0.9810, MMLU-Pro 0.8413 vs 0.8452) - and its installer checks the machine and verifies every download before writing a byte.
Thanks to both maintainers for carrying this work forward.
ninfer-ext is an extended fork of Neroued/ninfer, a from-scratch C++/CUDA inference engine for maximum single-GPU performance on one NVIDIA GeForce RTX 5090.
It adds Qwen3.8-Flash-Next (a ~180B MoE that stock NInfer cannot load at all), speculative decoding beyond the upstream default, EXL3 quantization, constrained decoding and tool calls, deeper serving work for long-running agents, and Gemma 4 text and vision through the same Engine.
It is not faster than stock everywhere: versus stock NInfer reports both sides, including the two cases where stock is ahead.
Measured on one RTX 5090; the test system and conditions are
below. Every model runs through the same ninfer-serve, CLI and Engine.
The numbers come from different builds, and each section says which: the master charts, the MTP
comparison and the Qwen3.6-27B quotes are from the current tree, while the serving and single-request
tables are from fork commits 19f38b77 and f8106aa9 (same Engine; only benchmark sizing changed) and
the Flash-Next rows from fc305acd and 7fbabfbc (CUDA 13.3 and 13.4). Two later commits — a
tensor-core A16 kernel for NVFP4 MLP projections and the adaptive-MTP batch draft length — are not
reflected in those two tables; versus stock carries the same caveat.
One figure per model, drawn by tools/bench/plot_master.py from the runs
in profiles/bench/all_models/. Each has four panels: prefill and decode against context (a line per
concurrency) and the same two against concurrency (a line per context). Seven models have one;
Qwen3.6-27B is quoted under serving throughput instead, because its campaign
was run separately and is not in that root. The decode panels carry the same-session MTP run as a
dashed line labelled with its mode, so plain and speculative decode sit in one chart.
Prefill is the per-request rate at c1 and the point aggregate at c2–c8 (prompt tokens over the whole makespan, so it includes each request's decode); decode is the steady rate over complete intervals whose decode batch equals the concurrency. Panels are sparse on purpose, because a concurrency can only reach the contexts its KV window fits — 262,144 tokens at one lane, 32,768 at eight on this card.
Gemma 4 26B-A4B QAT — decode at one lane 330.5 → 320.0 → 291.0 → 212.5 tok/s across 300 / 8k /
31k / 123k tokens of context, 674.9 at eight lanes; mean NLL 5.039971 (PPL 154.466) over
ninfer-ppl-1m-v1 full.
Gemma 4 31B IT M1 — decode 53–155 tok/s.
Qwen3.6-35B-A3B — prefill 5,310–16,723 tok/s, decode 257–1,376.
Qwen3.6-35B-A3B nvfp4 — prefill 5,838–23,343 tok/s, decode 253–1,338.
Qwen3.8-27B groupwise-int — prefill 2,255–3,264 tok/s, decode 80–422.
Qwen3.8-27B nvfp4 — prefill 4,034–9,795 tok/s, decode 76–545.
Qwen3.8-Flash-Next nvfp4 — prefill 3,319–5,232 tok/s, and the only model here that runs the
260k-token fixture; decode 69.7 → 252.5 tok/s plain from one lane to eight, 93.4 → 265.5 with MTP. Its
numbers are host-bound: the routed experts live in pinned host RAM and stream over PCIe, so the
host and the link matter as much as the GPU.
MTP against plain decode, same session, both modes in one invocation
The runs the dashed lines in the decode panels come from, in detail: decode at a ~300-token context, gain at two / four / eight lanes. Speculation pays where decode is weight-bound and stops paying once the engine already has a full batch: the dense 27B models gain about 2x at every concurrency, while on Gemma 4 26B-A4B at four lanes MTP is slower than plain decode (606 → 409 tok/s) because the draft passes cost more than the tokens they commit.
| Model | c2 | c4 | c8 | acceptance |
|---|---|---|---|---|
| Gemma 4 26B-A4B QAT | 1.39x | 0.67x | 1.03x | 0.82–0.85 |
| Qwen3.6-35B-A3B | 1.46x | 1.12x | 0.98x | 0.72–0.75 |
Qwen3.6-35B-A3B nvfp4 |
1.43x | 1.42x | 1.57x | 0.71–0.74 |
Qwen3.8-27B groupwise-int |
2.21x | 2.03x | 1.78x | 0.81–0.82 |
Qwen3.8-27B nvfp4 |
2.10x | 1.98x | 1.87x | 0.59–0.63 |
Gemma 4 runs MTP on the full proposal head only (1–7 draft tokens) and refuses the optimized one, so
--lm-head-draft must not be passed for it.
Aggregate decode tok/s from ninfer-serve: C concurrent requests, mean of two runs. Every row writes
512-token essays except Qwen3.8-Flash-Next, whose rows are 1,024-token reasoning generations —
the protocol the master charts use. Its essay measurement was taken on an older
build and is superseded; docs/flash-next.md keeps it with its commit.
| Artifact | Mode | C=1 | C=2 | C=4 | C=8 |
|---|---|---|---|---|---|
Qwen3.8-27B nvfp4 |
plain | 78.6 | 149.0 | 254.3 | 492.3 |
--spec mtp |
127.2 | 253.1 | 423.0 | 730.4 | |
Qwen3.8-27B groupwise-int |
plain | 81.8 | 146.0 | 260.1 | 415.9 |
--spec mtp |
127.7 | 219.3 | 340.3 | 453.2 | |
Qwen3.6-35B-A3B groupwise-int |
plain | 386.6 | 619.8 | 972.3 | 1,324.4 |
--spec mtp |
491.9 | 663.5 | 850.9 | 1,001.0 | |
--spec dflash --draft-tokens 7 |
355.4 | 452.1 | 507.7 | 898.9 | |
Qwen3.8-Flash-Next nvfp4 |
plain | 69.7 | 116.1 | 169.2 | 252.5 |
--spec mtp --draft-tokens 3 |
93.4 | 231.3 | 194.7 | 265.5 |
The 35B-A3B rows above write 512-token essays, which favours plain decode on that artifact: it has plain in bold at the top concurrency. The load decides, and two other protocols in this README measure it. On the 1,024-token reasoning generations of the master charts, 35B-A3B gains 1.46x at two lanes and 1.12x at four while tying at eight. On the 4,096-token AIME load of versus stock, MTP K=3 beats plain at C=4 and DFlash7 is fastest at C=8. Recommended settings answers it per model.
Qwen3.6-27B was measured in its own campaign rather than this one — a 293-token prompt, an
8,192-token generation per request, INT8 group-64 KV and MTP=3 — so its numbers are quoted, not
tabulated above: 185.8 / 247.0 / 309.5 / 535.0 tok/s aggregate decode at C=1/2/4/8, with MTP=3 and
73–79% acceptance on long reasoning. With MTP off it decodes 77.6 / 70.7 / 64.5 / 54.8 tok/s at
7.7k / 64.5k / 130k / 260k tokens of context, at 3,218 / 2,656 / 2,185 / 1,615 tok/s prefill. The
groupwise-int card carries the full tables — MTP
acceptance by scenario, the AIME runs, TTFT and the evaluation scores — and the
nvfp4 card its own set.
Flash-Next's rows are the reasoning load, and they are the current build. Its decode gains from MTP at every concurrency — 1.34x at one lane, 1.99x at two, 1.15x at four, 1.05x at eight — and its prefill reaches 4,861 / 5,232 / 4,468 / 3,319 tok/s for 7.7k / 64.5k / 130k / 260k-token prompts, the last of which no other model here can run. Its numbers are host-bound: the routed experts live in pinned host RAM and stream over PCIe, so the host and the link matter as much as the GPU. (detail)
| Artifact | Prefill 4k | Prefill 16k | tg128 |
tg128 adaptive MTP |
tg128 MTP K=3 |
|---|---|---|---|---|---|
Qwen3.8-Flash-Next nvfp4 |
3,411 | 3,350 | 125.2 | 130.2 (52%) | 126.9 (45%) |
Qwen3.8-27B nvfp4 |
9,208 | 8,067 | 76.8 | 114.1 (42%) | 117.4 (38%) |
Qwen3.8-27B groupwise-int |
3,027 | 2,872 | 83.1 | 111.0 (41%) | 102.0 (29%) |
Qwen3.6-35B-A3B groupwise-int |
18,171 | 16,463 | 393.3 | 594.9 (72%) | 545.4 (56%) |
All values are tok/s; percentages are MTP draft acceptance. DFlash on Qwen3.6-35B-A3B reaches
385.1 tok/s at tg128 with 7 drafts (20% accepted) and 219.5 with 15 (7% accepted), against 393.3
without speculation.
| Component | Details |
|---|---|
| GPU | NVIDIA GeForce RTX 5090, 32 GB, PCIe Gen5 x16 |
| NVIDIA driver | 610.57.04 |
| CUDA toolkit | 13.4 (nvcc V13.4.92) |
| CPU | Intel Core Ultra 9 285K (24 cores, up to 5.7 GHz) |
| RAM | 247 GiB usable |
| Model storage | WD_BLACK SN850P 8 TB NVMe, ext4 |
| OS | Ubuntu 26.10 (development branch), Linux 7.3.0-5-generic |
Flash-Next keeps its routed experts in pinned host RAM and streams them over PCIe, so its numbers also depend on host memory and the PCIe link, not only on the GPU. Model load time depends on the NVMe drive.
- Tool.
ninfer_benchthrough the public Engine: BF16 KV, CUDA Graphs, 3 measured repetitions after 1 warm-up, and the default prefill chunk. - Commands.
-p 512,4096,16384 -n 128 -pg 2048,128without speculation. With speculation:-n 128 -pg 2048,128, plus--spec mtp,--spec mtp --draft-tokens 3 --fixed-draftor--spec dflash --draft-tokens 7|15. The Flash-Next FP8 figures add--kv-dtype fp8. tg128decodes from a one-token seed, so its speed and MTP acceptance depend on the generated text. After the 2,048-token corpus prompt, drafts are accepted 94–100% of the time, which flatters MTP, so those numbers are omitted.- Serve tables.
ninfer-serve --max-context 4096 --max-concurrency 8 --kv-capacity auto, BF16 KV,temperature=0with thinking off: one warm-up wave, then one measured wave of C concurrent 512-token essays on eight fixed topics. - Master charts.
run_bench_sweep.pydrives each artifact throughcorpus-makespan(prefill against context) anddecode-saturation(steady decode, at the fixture's context and at padded contexts); see serving benchmarks. - Qwen3.6-27B. Measured in its own campaign — a 293-token prompt, an 8,192-token generation per
request, INT8 group-64 KV, CUDA Graphs and MTP=3 — and recorded in the
groupwise-intandnvfp4model cards, with the numbers quoted under serving throughput. It has no master chart: those are drawn from the runs inprofiles/bench/all_models/, and the sweep skips a model whose artifact is absent from the benchmark root. - Raw reports. The chart, MTP and Qwen3.6-27B numbers resolve to runs under
profiles/bench/(all_models/,mtp_decode/) or to the model cards; the Flash-Next single-request reports are inprofiles/bench/readme_20260926b/. The serving table's own campaign is not on this disk, so those numbers are inherited from the measurement whose commits are named above rather than re-derivable here. Upstream's published methodology is in the performance index.
- 64-bit Linux and an RTX 5090 (
sm_120a). No other GPU is supported. - A CUDA toolkit supporting
sm_120a. CUDA 13.3 is validated. - CMake 3.28 or newer, a C++23 host compiler, Ninja and
pkg-config. - FFmpeg development libraries (
libavformat,libavcodec,libavutil,libswscale) andlibcurl >= 7.85. - For Qwen3.8-Flash-Next only: about 65 GiB of host RAM and 127 GB of disk (details).
git clone https://github.com/giveen/ninfer-ext.git
cd ninfer-ext
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -jThis builds build/apps/ninfer-serve (HTTP server), build/apps/ninfer (one-shot CLI) and
build/apps/ninfer-perplexity. There is no install target, so run from the build tree.
cmake --preset dev also builds the tests and benchmarks
(build configuration).
Upstream's official v3 artifacts work unchanged:
| Model | Weights | Artifact | Download |
|---|---|---|---|
| Qwen3.8-27B | nvfp4 |
qwen3_8_27b_nvfp4.ninfer |
neroued/Qwen3.8-27B-nvfp4-NInfer |
| Qwen3.8-27B | groupwise-int |
qwen3_8_27b.ninfer |
neroued/Qwen3.8-27B-NInfer |
| Qwen3.8-27B | exl3 4.0 bpw |
qwen3_8_27b_exl3_4bpw.ninfer |
jabbatheduck/ninfer-ext-models |
| Qwen3.8-27B | exl3 3.5 bpw |
qwen3_8_27b_exl3_3p5bpw.ninfer |
jabbatheduck/ninfer-ext-models |
| Qwen3.6-27B | nvfp4 |
qwen3_6_27b_nvfp4.ninfer |
neroued/Qwen3.6-27B-nvfp4-NInfer |
| Qwen3.6-27B | groupwise-int |
qwen3_6_27b.ninfer |
neroued/Qwen3.6-27B-NInfer |
| Qwen3.6-35B-A3B | groupwise-int |
qwen3_6_35b_a3b.ninfer |
neroued/Qwen3.6-35B-A3B-NInfer |
| Qwen3.8-Flash-Next | nvfp4 |
qwen3.8-flash-next/qwen3_8_flash_next_nvfp4.ninfer (+ 3 .part files) |
jabbatheduck/ninfer-ext-models |
| Qwen3.8-Flash-Next | exl3 4.0 bpw |
qwen3.8-flash-next-exl3-4bpw/…exl3_4bpw.ninfer (+ 2 .part files) |
jabbatheduck/ninfer-ext-models |
| Qwen3.8-Flash-Next | exl3 3.5 bpw |
qwen3.8-flash-next-exl3-3p5bpw/…exl3_3p5bpw.ninfer (+ 2 .part files) |
jabbatheduck/ninfer-ext-models |
| Gemma 4 31B | M1 (Q6/Q5 g64) |
gemma4-31b-it-m1/gemma4_31b_it_m1.ninfer |
jabbatheduck/ninfer-ext-models |
| Gemma 4 26B-A4B | QAT int4 experts + Q8 | gemma4-26b-a4b-qat/gemma4_26b_a4b_qat.ninfer |
jabbatheduck/ninfer-ext-models |
hf download neroued/Qwen3.8-27B-nvfp4-NInfer qwen3_8_27b_nvfp4.ninfer --local-dir models
hf download jabbatheduck/ninfer-ext-models qwen3_8_27b_exl3_4bpw.ninfer --local-dir models
hf download jabbatheduck/ninfer-ext-models --include "gemma4-26b-a4b-qat/*" --local-dir modelsThe exl3, Flash-Next and Gemma 4 rows are this fork's own artifacts, published on Hugging Face; they
download and run unchanged. Qwen3.8-27B with DFlash2 weights has to be converted yourself
(instructions). Converted artifacts embed
this fork's chat template (conversion guide); an existing v2 download can
be upgraded locally.
./build/apps/ninfer-serve models/qwen3_8_27b_nvfp4.ninfer \
--max-context 32768 --kv-capacity auto --max-concurrency 2 \
--spec mtp --draft-tokens 5 --fixed-draft --lm-head-draftcurl http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model": "qwen3.8-27b",
"messages": [{"role": "user", "content": "Reply with one short sentence."}],
"max_tokens": 64}'The server speaks OpenAI Chat Completions and Responses, and Anthropic Messages, including streaming
and tools. --model-id changes the model name it reports; --host 0.0.0.0 exposes it beyond
localhost, and --api-key requires a key. GET /metrics serves Prometheus metrics named after
vLLM's (metrics). For a one-shot answer without a server:
./build/apps/ninfer models/qwen3_8_27b_nvfp4.ninfer \
--prompt "Explain prefill and decode." --max-context 32768 --spec mtp --lm-head-draftThe CLI guide, the HTTP serving guide and the
examples cover every option; each binary's --help lists the current flags.
Docker. docker build --tag ninfer:local ., then pass the ninfer-serve command with
--host 0.0.0.0 to docker run --gpus all, with the model directory mounted.
Each of these has its own page; this README keeps only the numbers.
| Feature | Page |
|---|---|
| Qwen3.8-Flash-Next on one 32 GB GPU | docs/flash-next.md |
| Speculative decoding: MTP, DFlash, DFlash2, EAGLE3, prompt lookup | docs/speculative-decoding.md |
| EXL3 quantization at 4.0 and 3.5 bpw | docs/quantization.md |
| KV streaming, context cache and serving extras | docs/kv-and-context.md |
| Gemma 4 31B and 26B-A4B (text, vision, MTP) | docs/gemma4.md |
| Constrained decoding and constrained tool calls | docs/maintainer/constrained-decoding.md |
| Versus stock NInfer | docs/vs-stock-ninfer.md |
| Engineering: conversion, C++23, merged upstream PRs | docs/engineering.md |
Capabilities: text with thinking and non-thinking modes; image, multi-image, video and mixed
multimodal input (--vision); chunked prefill and CUDA Graph decode of one to eight concurrent
requests; speculative decoding as listed above; BF16, INT8, FP8, NVFP4 and K8V4 KV storage; private
and shared exact-prefix reuse; KV streaming beyond GPU memory; offline perplexity scoring; OpenAI
Responses and Chat Completions plus Anthropic Messages, with streaming, tools, token counting and
usage; Prometheus /metrics.
Limits: one RTX 5090, one resident model, and a startup-fixed one to eight active requests;
bounded FIFO admission, with no preemption, priority, multi-GPU or distributed serving. --max-context
is the per-request limit and --kv-capacity sizes the shared KV pool (auto sizes it from the memory
left after weights); with --kv-stream, contexts beyond the pool spill to Host KV. Tool calls are
parsed and returned to the client; NInfer does not execute them. Per-model limits — Gemma 4's KV
storages, its MTP proposal head, and the context/concurrency trade — are in
Gemma 4, and the rest in each feature page.
- Documentation index
- CLI · HTTP serving · Weight conversion
- Performance records · Perplexity evaluation
- Benchmark harness
- Contributing
NInfer is designed and developed by Neroued. The architecture, the artifact format and the official artifacts are theirs. Evaluation scores for the official artifacts are in their model cards. If NInfer is useful to you, consider supporting upstream on Ko-fi.
Both projects are licensed under the Apache License 2.0.
The official artifacts derive from Qwen/Qwen3.6-27B,
Qwen/Qwen3.8-27B,
Qwen/Qwen3.6-35B-A3B, the fixed packed weights of
rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm
and the fixed mixed FP8/NVFP4 weights of
unsloth/Qwen3.8-27B-NVFP4. Flash-Next artifacts are
converted from nvidia/Qwen3.8-Flash-Next-NVFP4,
and the Gemma 4 ones from google/gemma-4-31B-it and google/gemma-4-26B-A4B-it under the Gemma Terms
of Use. Vendored dependencies keep their own licenses under third_party/.













