Skip to content
This repository was archived by the owner on Nov 2, 2026. It is now read-only.
giveenPublic archive

About

ninfer-ext: Forked for speed.

Resources

Contributing

Stars

26 stars

Watchers

0 watching

Forks

Repository files navigation

ninfer-ext

Qwen3.8-Flash-Next NVFP4 at a 262,144-token context: prefill and decode against depth for basalt, capy-flash and ninfer-ext, greedy and sampled

Retired for now (2026-10-22). No further development is planned on ninfer-ext for the time being. The repository, its artifacts and its docs stay as they are, and the tree still builds and serves everything documented below; it is simply not being worked on.

The chart above is where this project stands on the workload it was built for. On one RTX 5090, Qwen3.8-Flash-Next NVFP4 at the full 262,144-token context, two engines of the same lineage measured faster at every depth in both greedy and sampled decoding. Prefill: 8.1k-9.2k tok/s for basalt and 6.8k-8.0k for capy-flash against 4.2k-6.3k here. Decode, greedy: 176-265 and 138-198 tok/s against 73-106; sampling lowers all three (146-242, 130-178, 57-99) and changes no ordering. The difference is the expert strategy: ninfer-ext copies every expert-cache miss over PCIe and computes it on the GPU, while the other two compute misses on the CPU cores, which on this host wins by a wide margin.

Conditions: one RTX 5090 at a 475 W power limit (driver 615.71, CUDA 13.3), 262,144-token context, 256 output tokens per request, MTP speculation on, INT8 KV, one request at a time, a cold unique prompt per point, fastest tuned configuration of each engine. Measured with an external cross-engine harness, not ninfer_bench, so these numbers are not comparable with the tool-based tables further down.

Recommended instead

Both alternatives below belong to the same NInfer lineage, both are actively developed, and both measured faster here on this model, this box, this test and comparable tuning effort. If you came for Qwen3.8-Flash-Next at long context, use one of them:

  • basalt - the pick for one 32 GB card at long context. Fastest of the three on both axes at every depth measured (prefill 8.1k-9.2k tok/s, decode 176-265 tok/s greedy), and the most complete packaging: one .basalt file carries the weights, tokenizer, chat template, MTP drafter, expert profile and vision projector, so there is no sidecar to keep in step. The adaptive expert tier that follows routing, basalt-probe deriving suggested flags from the machine's own measured rates, a parity test per kernel and a PPL/KLD quality gate against Q8_0 are the kind of work that never shows up in a throughput table, and they are why it is the one worth building on.
  • capy-flash - the pick for two 16 GB consumer cards. It tensor-parallelises this 125B model over PCIe without peer-to-peer, partitions the expert cache between the cards and computes the misses on the CPU, then gets the agent work right: idle conversations move to RAM and hand their VRAM back to the expert cache, a continued conversation keeps the experts its next answer needs, and the prefix cache spans turns. Its quality reporting is the most careful of the three - the same prompts, paired, against NVIDIA's NVFP4 checkpoint served by vLLM on a GB300 (GSM8K 0.9750 vs 0.9810, MMLU-Pro 0.8413 vs 0.8452) - and its installer checks the machine and verifies every download before writing a byte.

Thanks to both maintainers for carrying this work forward.

ninfer-ext

ninfer-ext is an extended fork of Neroued/ninfer, a from-scratch C++/CUDA inference engine for maximum single-GPU performance on one NVIDIA GeForce RTX 5090.

It adds Qwen3.8-Flash-Next (a ~180B MoE that stock NInfer cannot load at all), speculative decoding beyond the upstream default, EXL3 quantization, constrained decoding and tool calls, deeper serving work for long-running agents, and Gemma 4 text and vision through the same Engine.

It is not faster than stock everywhere: versus stock NInfer reports both sides, including the two cases where stock is ahead.

Benchmarks

Measured on one RTX 5090; the test system and conditions are below. Every model runs through the same ninfer-serve, CLI and Engine.

The numbers come from different builds, and each section says which: the master charts, the MTP comparison and the Qwen3.6-27B quotes are from the current tree, while the serving and single-request tables are from fork commits 19f38b77 and f8106aa9 (same Engine; only benchmark sizing changed) and the Flash-Next rows from fc305acd and 7fbabfbc (CUDA 13.3 and 13.4). Two later commits — a tensor-core A16 kernel for NVFP4 MLP projections and the adaptive-MTP batch draft length — are not reflected in those two tables; versus stock carries the same caveat.

Master charts

One figure per model, drawn by tools/bench/plot_master.py from the runs in profiles/bench/all_models/. Each has four panels: prefill and decode against context (a line per concurrency) and the same two against concurrency (a line per context). Seven models have one; Qwen3.6-27B is quoted under serving throughput instead, because its campaign was run separately and is not in that root. The decode panels carry the same-session MTP run as a dashed line labelled with its mode, so plain and speculative decode sit in one chart.

Prefill is the per-request rate at c1 and the point aggregate at c2–c8 (prompt tokens over the whole makespan, so it includes each request's decode); decode is the steady rate over complete intervals whose decode batch equals the concurrency. Panels are sparse on purpose, because a concurrency can only reach the contexts its KV window fits — 262,144 tokens at one lane, 32,768 at eight on this card.

Gemma 4 26B-A4B QAT — decode at one lane 330.5 → 320.0 → 291.0 → 212.5 tok/s across 300 / 8k / 31k / 123k tokens of context, 674.9 at eight lanes; mean NLL 5.039971 (PPL 154.466) over ninfer-ppl-1m-v1 full.

Gemma 4 26B-A4B prefill and decode against context and concurrency

Gemma 4 31B IT M1 — decode 53–155 tok/s.

Gemma 4 31B prefill and decode against context and concurrency

Qwen3.6-35B-A3B — prefill 5,310–16,723 tok/s, decode 257–1,376.

Qwen3.6-35B-A3B prefill and decode against context and concurrency

Qwen3.6-35B-A3B nvfp4 — prefill 5,838–23,343 tok/s, decode 253–1,338.

Qwen3.6-35B-A3B nvfp4 prefill and decode against context and concurrency

Qwen3.8-27B groupwise-int — prefill 2,255–3,264 tok/s, decode 80–422.

Qwen3.8-27B prefill and decode against context and concurrency

Qwen3.8-27B nvfp4 — prefill 4,034–9,795 tok/s, decode 76–545.

Qwen3.8-27B nvfp4 prefill and decode against context and concurrency

Qwen3.8-Flash-Next nvfp4 — prefill 3,319–5,232 tok/s, and the only model here that runs the 260k-token fixture; decode 69.7 → 252.5 tok/s plain from one lane to eight, 93.4 → 265.5 with MTP. Its numbers are host-bound: the routed experts live in pinned host RAM and stream over PCIe, so the host and the link matter as much as the GPU.

Qwen3.8-Flash-Next nvfp4 prefill and decode against context and concurrency

MTP against plain decode, same session, both modes in one invocation

The runs the dashed lines in the decode panels come from, in detail: decode at a ~300-token context, gain at two / four / eight lanes. Speculation pays where decode is weight-bound and stops paying once the engine already has a full batch: the dense 27B models gain about 2x at every concurrency, while on Gemma 4 26B-A4B at four lanes MTP is slower than plain decode (606 → 409 tok/s) because the draft passes cost more than the tokens they commit.

Model c2 c4 c8 acceptance
Gemma 4 26B-A4B QAT 1.39x 0.67x 1.03x 0.82–0.85
Qwen3.6-35B-A3B 1.46x 1.12x 0.98x 0.72–0.75
Qwen3.6-35B-A3B nvfp4 1.43x 1.42x 1.57x 0.71–0.74
Qwen3.8-27B groupwise-int 2.21x 2.03x 1.78x 0.81–0.82
Qwen3.8-27B nvfp4 2.10x 1.98x 1.87x 0.59–0.63

Gemma 4 runs MTP on the full proposal head only (1–7 draft tokens) and refuses the optimized one, so --lm-head-draft must not be passed for it.

Gemma 4 26B-A4B decode with and without MTP

Qwen3.6-35B-A3B decode with and without MTP

Qwen3.6-35B-A3B nvfp4 decode with and without MTP

Qwen3.8-27B decode with and without MTP

Qwen3.8-27B nvfp4 decode with and without MTP

Serving throughput

Aggregate decode tok/s from ninfer-serve: C concurrent requests, mean of two runs. Every row writes 512-token essays except Qwen3.8-Flash-Next, whose rows are 1,024-token reasoning generations — the protocol the master charts use. Its essay measurement was taken on an older build and is superseded; docs/flash-next.md keeps it with its commit.

Artifact Mode C=1 C=2 C=4 C=8
Qwen3.8-27B nvfp4 plain 78.6 149.0 254.3 492.3
--spec mtp 127.2 253.1 423.0 730.4
Qwen3.8-27B groupwise-int plain 81.8 146.0 260.1 415.9
--spec mtp 127.7 219.3 340.3 453.2
Qwen3.6-35B-A3B groupwise-int plain 386.6 619.8 972.3 1,324.4
--spec mtp 491.9 663.5 850.9 1,001.0
--spec dflash --draft-tokens 7 355.4 452.1 507.7 898.9
Qwen3.8-Flash-Next nvfp4 plain 69.7 116.1 169.2 252.5
--spec mtp --draft-tokens 3 93.4 231.3 194.7 265.5

The 35B-A3B rows above write 512-token essays, which favours plain decode on that artifact: it has plain in bold at the top concurrency. The load decides, and two other protocols in this README measure it. On the 1,024-token reasoning generations of the master charts, 35B-A3B gains 1.46x at two lanes and 1.12x at four while tying at eight. On the 4,096-token AIME load of versus stock, MTP K=3 beats plain at C=4 and DFlash7 is fastest at C=8. Recommended settings answers it per model.

Qwen3.6-27B was measured in its own campaign rather than this one — a 293-token prompt, an 8,192-token generation per request, INT8 group-64 KV and MTP=3 — so its numbers are quoted, not tabulated above: 185.8 / 247.0 / 309.5 / 535.0 tok/s aggregate decode at C=1/2/4/8, with MTP=3 and 73–79% acceptance on long reasoning. With MTP off it decodes 77.6 / 70.7 / 64.5 / 54.8 tok/s at 7.7k / 64.5k / 130k / 260k tokens of context, at 3,218 / 2,656 / 2,185 / 1,615 tok/s prefill. The groupwise-int card carries the full tables — MTP acceptance by scenario, the AIME runs, TTFT and the evaluation scores — and the nvfp4 card its own set.

Flash-Next's rows are the reasoning load, and they are the current build. Its decode gains from MTP at every concurrency — 1.34x at one lane, 1.99x at two, 1.15x at four, 1.05x at eight — and its prefill reaches 4,861 / 5,232 / 4,468 / 3,319 tok/s for 7.7k / 64.5k / 130k / 260k-token prompts, the last of which no other model here can run. Its numbers are host-bound: the routed experts live in pinned host RAM and stream over PCIe, so the host and the link matter as much as the GPU. (detail)

Single-request benchmark

Artifact Prefill 4k Prefill 16k tg128 tg128 adaptive MTP tg128 MTP K=3
Qwen3.8-Flash-Next nvfp4 3,411 3,350 125.2 130.2 (52%) 126.9 (45%)
Qwen3.8-27B nvfp4 9,208 8,067 76.8 114.1 (42%) 117.4 (38%)
Qwen3.8-27B groupwise-int 3,027 2,872 83.1 111.0 (41%) 102.0 (29%)
Qwen3.6-35B-A3B groupwise-int 18,171 16,463 393.3 594.9 (72%) 545.4 (56%)

All values are tok/s; percentages are MTP draft acceptance. DFlash on Qwen3.6-35B-A3B reaches 385.1 tok/s at tg128 with 7 drafts (20% accepted) and 219.5 with 15 (7% accepted), against 393.3 without speculation.

Test system

Component Details
GPU NVIDIA GeForce RTX 5090, 32 GB, PCIe Gen5 x16
NVIDIA driver 610.57.04
CUDA toolkit 13.4 (nvcc V13.4.92)
CPU Intel Core Ultra 9 285K (24 cores, up to 5.7 GHz)
RAM 247 GiB usable
Model storage WD_BLACK SN850P 8 TB NVMe, ext4
OS Ubuntu 26.10 (development branch), Linux 7.3.0-5-generic

Flash-Next keeps its routed experts in pinned host RAM and streams them over PCIe, so its numbers also depend on host memory and the PCIe link, not only on the GPU. Model load time depends on the NVMe drive.

Benchmark conditions

  • Tool. ninfer_bench through the public Engine: BF16 KV, CUDA Graphs, 3 measured repetitions after 1 warm-up, and the default prefill chunk.
  • Commands. -p 512,4096,16384 -n 128 -pg 2048,128 without speculation. With speculation: -n 128 -pg 2048,128, plus --spec mtp, --spec mtp --draft-tokens 3 --fixed-draft or --spec dflash --draft-tokens 7|15. The Flash-Next FP8 figures add --kv-dtype fp8.
  • tg128 decodes from a one-token seed, so its speed and MTP acceptance depend on the generated text. After the 2,048-token corpus prompt, drafts are accepted 94–100% of the time, which flatters MTP, so those numbers are omitted.
  • Serve tables. ninfer-serve --max-context 4096 --max-concurrency 8 --kv-capacity auto, BF16 KV, temperature=0 with thinking off: one warm-up wave, then one measured wave of C concurrent 512-token essays on eight fixed topics.
  • Master charts. run_bench_sweep.py drives each artifact through corpus-makespan (prefill against context) and decode-saturation (steady decode, at the fixture's context and at padded contexts); see serving benchmarks.
  • Qwen3.6-27B. Measured in its own campaign — a 293-token prompt, an 8,192-token generation per request, INT8 group-64 KV, CUDA Graphs and MTP=3 — and recorded in the groupwise-int and nvfp4 model cards, with the numbers quoted under serving throughput. It has no master chart: those are drawn from the runs in profiles/bench/all_models/, and the sweep skips a model whose artifact is absent from the benchmark root.
  • Raw reports. The chart, MTP and Qwen3.6-27B numbers resolve to runs under profiles/bench/ (all_models/, mtp_decode/) or to the model cards; the Flash-Next single-request reports are in profiles/bench/readme_20260926b/. The serving table's own campaign is not on this disk, so those numbers are inherited from the measurement whose commits are named above rather than re-derivable here. Upstream's published methodology is in the performance index.

Quick start

1. Requirements

  • 64-bit Linux and an RTX 5090 (sm_120a). No other GPU is supported.
  • A CUDA toolkit supporting sm_120a. CUDA 13.3 is validated.
  • CMake 3.28 or newer, a C++23 host compiler, Ninja and pkg-config.
  • FFmpeg development libraries (libavformat, libavcodec, libavutil, libswscale) and libcurl >= 7.85.
  • For Qwen3.8-Flash-Next only: about 65 GiB of host RAM and 127 GB of disk (details).

2. Build

git clone https://github.com/giveen/ninfer-ext.git
cd ninfer-ext
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

This builds build/apps/ninfer-serve (HTTP server), build/apps/ninfer (one-shot CLI) and build/apps/ninfer-perplexity. There is no install target, so run from the build tree. cmake --preset dev also builds the tests and benchmarks (build configuration).

3. Get a model

Upstream's official v3 artifacts work unchanged:

Model Weights Artifact Download
Qwen3.8-27B nvfp4 qwen3_8_27b_nvfp4.ninfer neroued/Qwen3.8-27B-nvfp4-NInfer
Qwen3.8-27B groupwise-int qwen3_8_27b.ninfer neroued/Qwen3.8-27B-NInfer
Qwen3.8-27B exl3 4.0 bpw qwen3_8_27b_exl3_4bpw.ninfer jabbatheduck/ninfer-ext-models
Qwen3.8-27B exl3 3.5 bpw qwen3_8_27b_exl3_3p5bpw.ninfer jabbatheduck/ninfer-ext-models
Qwen3.6-27B nvfp4 qwen3_6_27b_nvfp4.ninfer neroued/Qwen3.6-27B-nvfp4-NInfer
Qwen3.6-27B groupwise-int qwen3_6_27b.ninfer neroued/Qwen3.6-27B-NInfer
Qwen3.6-35B-A3B groupwise-int qwen3_6_35b_a3b.ninfer neroued/Qwen3.6-35B-A3B-NInfer
Qwen3.8-Flash-Next nvfp4 qwen3.8-flash-next/qwen3_8_flash_next_nvfp4.ninfer (+ 3 .part files) jabbatheduck/ninfer-ext-models
Qwen3.8-Flash-Next exl3 4.0 bpw qwen3.8-flash-next-exl3-4bpw/…exl3_4bpw.ninfer (+ 2 .part files) jabbatheduck/ninfer-ext-models
Qwen3.8-Flash-Next exl3 3.5 bpw qwen3.8-flash-next-exl3-3p5bpw/…exl3_3p5bpw.ninfer (+ 2 .part files) jabbatheduck/ninfer-ext-models
Gemma 4 31B M1 (Q6/Q5 g64) gemma4-31b-it-m1/gemma4_31b_it_m1.ninfer jabbatheduck/ninfer-ext-models
Gemma 4 26B-A4B QAT int4 experts + Q8 gemma4-26b-a4b-qat/gemma4_26b_a4b_qat.ninfer jabbatheduck/ninfer-ext-models
hf download neroued/Qwen3.8-27B-nvfp4-NInfer qwen3_8_27b_nvfp4.ninfer --local-dir models
hf download jabbatheduck/ninfer-ext-models qwen3_8_27b_exl3_4bpw.ninfer --local-dir models
hf download jabbatheduck/ninfer-ext-models --include "gemma4-26b-a4b-qat/*" --local-dir models

The exl3, Flash-Next and Gemma 4 rows are this fork's own artifacts, published on Hugging Face; they download and run unchanged. Qwen3.8-27B with DFlash2 weights has to be converted yourself (instructions). Converted artifacts embed this fork's chat template (conversion guide); an existing v2 download can be upgraded locally.

4. Serve and send a request

./build/apps/ninfer-serve models/qwen3_8_27b_nvfp4.ninfer \
  --max-context 32768 --kv-capacity auto --max-concurrency 2 \
  --spec mtp --draft-tokens 5 --fixed-draft --lm-head-draft
curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "qwen3.8-27b",
       "messages": [{"role": "user", "content": "Reply with one short sentence."}],
       "max_tokens": 64}'

The server speaks OpenAI Chat Completions and Responses, and Anthropic Messages, including streaming and tools. --model-id changes the model name it reports; --host 0.0.0.0 exposes it beyond localhost, and --api-key requires a key. GET /metrics serves Prometheus metrics named after vLLM's (metrics). For a one-shot answer without a server:

./build/apps/ninfer models/qwen3_8_27b_nvfp4.ninfer \
  --prompt "Explain prefill and decode." --max-context 32768 --spec mtp --lm-head-draft

The CLI guide, the HTTP serving guide and the examples cover every option; each binary's --help lists the current flags.

Docker. docker build --tag ninfer:local ., then pass the ninfer-serve command with --host 0.0.0.0 to docker run --gpus all, with the model directory mounted.

Features

Each of these has its own page; this README keeps only the numbers.

Feature Page
Qwen3.8-Flash-Next on one 32 GB GPU docs/flash-next.md
Speculative decoding: MTP, DFlash, DFlash2, EAGLE3, prompt lookup docs/speculative-decoding.md
EXL3 quantization at 4.0 and 3.5 bpw docs/quantization.md
KV streaming, context cache and serving extras docs/kv-and-context.md
Gemma 4 31B and 26B-A4B (text, vision, MTP) docs/gemma4.md
Constrained decoding and constrained tool calls docs/maintainer/constrained-decoding.md
Versus stock NInfer docs/vs-stock-ninfer.md
Engineering: conversion, C++23, merged upstream PRs docs/engineering.md

Capabilities and limits

Capabilities: text with thinking and non-thinking modes; image, multi-image, video and mixed multimodal input (--vision); chunked prefill and CUDA Graph decode of one to eight concurrent requests; speculative decoding as listed above; BF16, INT8, FP8, NVFP4 and K8V4 KV storage; private and shared exact-prefix reuse; KV streaming beyond GPU memory; offline perplexity scoring; OpenAI Responses and Chat Completions plus Anthropic Messages, with streaming, tools, token counting and usage; Prometheus /metrics.

Limits: one RTX 5090, one resident model, and a startup-fixed one to eight active requests; bounded FIFO admission, with no preemption, priority, multi-GPU or distributed serving. --max-context is the per-request limit and --kv-capacity sizes the shared KV pool (auto sizes it from the memory left after weights); with --kv-stream, contexts beyond the pool spill to Host KV. Tool calls are parsed and returned to the client; NInfer does not execute them. Per-model limits — Gemma 4's KV storages, its MTP proposal head, and the context/concurrency trade — are in Gemma 4, and the rest in each feature page.

Documentation

Credits and license

NInfer is designed and developed by Neroued. The architecture, the artifact format and the official artifacts are theirs. Evaluation scores for the official artifacts are in their model cards. If NInfer is useful to you, consider supporting upstream on Ko-fi.

Both projects are licensed under the Apache License 2.0.

The official artifacts derive from Qwen/Qwen3.6-27B, Qwen/Qwen3.8-27B, Qwen/Qwen3.6-35B-A3B, the fixed packed weights of rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm and the fixed mixed FP8/NVFP4 weights of unsloth/Qwen3.8-27B-NVFP4. Flash-Next artifacts are converted from nvidia/Qwen3.8-Flash-Next-NVFP4, and the Gemma 4 ones from google/gemma-4-31B-it and google/gemma-4-26B-A4B-it under the Gemma Terms of Use. Vendored dependencies keep their own licenses under third_party/.

About

ninfer-ext: Forked for speed.

Resources

Contributing

Stars

26 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages