Skip to content
KikoCisPublic

About

An architecture-agnostic LLM inference + quantization engine in Rust (experimental).

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

5 Commits

Folders and files

Repository files navigation

coco — an architecture-agnostic LLM engine in Rust

coco

An architecture-agnostic LLM inference + quantization engine, in Rust. Built on candle + Metal.

License Rust GPU status

Website How it works Blog

⚠️ Experimental — work in progress. coco is an early-stage research engine under active development. It builds and the core pieces are validated against oracles, but its APIs, on-disk formats, and coverage are unstable and it is not production-ready. Expect rough edges, breaking changes, and gaps. Use it to learn from and hack on, not to ship.

One engine, not one-file-per-architecture. Every model reduces to a config row plus a single pluggable per-layer seam — so supporting a new architecture is an afternoon, not a rewrite.

Status: experimental, early but real. It builds clean and the headline pieces are validated against reference oracles. It is honest about what it can't do yet (see Limitations). A research engine to learn from — not a drop-in llama.cpp replacement.


What works today

  • Four architectures, all wired end-to-end through one generic DenseDecoder:
    • Qwen2, Qwen3 (QK-norm), Qwen3-MoE (per-layer dense/MoE routing), DeepSeek V2/V3 (MLA latent attention + shared-expert MoE + YaRN RoPE).
  • One pluggable seam — the SequenceMixer trait. Three mixers implement it and dispatch per layer: GqaAttention (full GQA + flash-decode + persistent KV cache), MlaAttention (DeepSeek latent), GatedDeltaNet (linear recurrent).
  • turboquant — a pure-Rust, data-free 3-bit quantizer (FWHT incoherence rotation + Lloyd-Max codebook + per-block fp16 radius). No Python, no calibration set, no external quant libs.
    • Measured on Qwen2.5-1.5B: whole model 2944 → 934 MB (3.15×), projections 5.12×, ~3.125 bpw, relFro 0.184.
    • Weights stay GPU-resident and compressed, decompressed on-the-fly inside fused Metal kernels — never materialized back to fp16.
    • Output is byte-for-byte identical to the fp16 baseline on the checked cases.
  • Fused Metal kernel suite (qlinear.rs): on-GPU rotation, quantized matvec that reads compressed codes directly (register-blocked variants), fused RMSNorm, and split-K flash-decode with in-kernel KV append.

Validated (reproducible)

  • GatedDeltaNet forward matches its reference oracle at max abs err 1.825e-7 (mean 3.5e-8), threshold 1e-4 — cargo run --release --bin gdn_ref, fixtures in tests/fixtures/gdn_oracle/.
  • Unit tests cover FWHT self-inverse, rotated-fast-path == dense matmul, and 3-bit roundtrip (cargo test --release, all green).
  • turboquant output is byte-for-byte identical to the fp16 baseline on the checked cases.
  • cargo build --release → 0 errors.

Limitations (honest)

  • Qwen3.5 is recognized but does not run yet. Its config is parsed and the GatedDeltaNet mixer is validated in isolation, but the per-layer linear/full-attention dispatch isn't wired into DecoderLayer yet — build_model returns a diagnostic error on purpose rather than building a half-working model.
  • MLA runs the candle fallback (no fused kernel yet); its parity-vs-reference is not re-verified in this release (needs an external checkpoint).
  • Speed wins are short-context only. Decode is bandwidth-bound, so reading fewer weight bytes wins at ctx 0–128; the advantage inverts by ~2K–8K because prefill is still O(seq²).
  • The integrated binary ships the 4.x-bpw nibble format by default; the true-3-bit (934 MB) path is self-checked but not yet CLI-default.
  • No checkpoint persistence / GGUF interop — models are quantized at load.
  • Test coverage is thin (a handful of unit tests). A real suite — target ≥1200 cases covering integer/float overflow, numerical edge cases, and security/robustness (malformed inputs, adversarial shapes/dims) — is planned but not yet written. Do not assume the engine is hardened.

Quick start

cargo build --release
# run generation (point --model at a HF model dir with config.json, tokenizer.json, *.safetensors)
cargo run --release --bin coco -- --model ./model --prompt "The capital of France is"
# validate the GatedDeltaNet port against its oracle
cargo run --release --bin gdn_ref
cargo test --release

Layout

file what
src/config.rs arch registry + config parsing (ARCH_REGISTRY, resolve_arch, build_model dispatch)
src/model.rs DenseDecoder, DecoderLayer, the SequenceMixer trait + its 3 impls, MoeFfn
src/turboquant.rs the data-free quantizer (rotation + codebook + radius)
src/qlinear.rs fused Metal kernels (quantized matvec, flash-decode, rmsnorm)
src/eval.rs / src/merge.rs eval harness + evolutionary merge
tests/fixtures/ golden oracles for validation

The story

The design decisions here are written up, for humans, on the Bitácora — with interactive demos. The coco series:

  1. Every new AI model breaks the tooling. I built an engine that shrugs it off. — the SequenceMixer seam
  2. How I ported a new kind of "attention" to Rust without missing a single decimal — oracle-first validation
  3. Shrinking an AI model 3× with no data and no quality loss — turboquant
  4. Why a 3 GB model runs faster than a 14 GB one (and when it stops) — the bandwidth thesis
  5. The GPU trick: multiplying straight from compressed data, without unpacking it — fused Metal kernels
  6. I optimized the GPU and it came out 20× slower. The lesson. — occupancy over traffic-saved
  7. When the honest thing is for your program to say "not yet" — the diagnostic bail

License

Apache-2.0. See LICENSE.

About

An architecture-agnostic LLM inference + quantization engine in Rust (experimental).

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages