Skip to content

[ROCm] Add a WMMA kernel for the fused 4-bit GEMM on RDNA3, RDNA3.5 and RDNA4 - #2145

Open
sstamenk wants to merge 3 commits into
bitsandbytes-foundation:mainfrom
sstamenk:feature/rocm-wmma-gemm-4bit-pr
Open

sstamenk wants to merge 3 commits into
bitsandbytes-foundation:mainfrom
sstamenk:feature/rocm-wmma-gemm-4bit-pr

Conversation

@sstamenk

@sstamenk sstamenk commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

On Radeon GPUs, the fused 4-bit GEMM (gemm_4bit) is a SIMT kernel that only wins up to M = 8 rows. Above that, ROCm falls back to dequantizing the whole weight matrix and calling F.linear. That is slow for small and medium batches: the dequantized weights are written to and read back from DRAM, and the SIMT kernel's cost grows linearly with M.

This PR adds a WMMA (matrix-core) kernel for the fused 4-bit GEMM on gfx11 (RDNA3/RDNA3.5) and gfx12 (RDNA4), and dispatches to it from the existing ROCm path. The fused path now stays faster than dequant + F.linear up to M = 384-512 rows instead of 8. This mirrors the CUDA SIMT → MMA → dequant + BLAS split.

speedup vs main

Kernel (csrc/gemm_4bit_wmma.cu)

  • gfx11 (v_wmma_f32_16x16x16_{f16,bf16}): a wave32 WMMA runs as two independent 8x16x16 half-wave products. Each half-wave takes its own 32-element K slice and the halves are summed with permlanex16, giving 8 real rows per WMMA without padding M to 16.
  • gfx12 (*_w32_gfx12 WMMAs): one full 16x16x16 product; each half-wave supplies its own K slice, summed inside the WMMA.
  • 4-bit weights are decoded through a 256-entry LDS table (byte → pair of NF4/FP4 centroids). absmax is applied in fp32 after the WMMAs. On gfx11 fp16 large tiles, the B operand is pre-scaled with packed half math instead. Nested (double-quant) statistics are supported.
  • A is double-buffered in LDS. K is split across warps with a deterministic LDS reduction (no atomics). A per-block K-chunk rotation avoids DRAM channel camping, and blocks walk row tiles first so row tiles of the same output columns share B in L2.
  • fp32 uses a 3-way split of each operand into bf16 (6 WMMAs per tile) and stays fp32-accurate. Against a float64 product of the same dequantized weights, the mean absolute error over the mean absolute output is at most ~9e-7 on gfx1100 and ~1.5e-6 on gfx1201 (blocksize 64), where the test allows 2e-6. Inf and NaN propagate as in fp32 math.
  • Input range: absmax scales the fp32 sum of A × centroid over a quantization group, so |A| must stay below ~FLT_MAX / blocksize (~5e36 at blocksize 64; fp32/bf16 inputs only). The fp32 split keeps full accuracy for |A| ≥ ~1e-33. Both are far outside real activations.

Dispatch

Requires K % 64 == 0 and blocksize >= 32; anything else keeps the current path.

  • SIMT vs WMMA (wmma_preferred(), called from csrc/gemm_4bit.cu): bf16/fp16 use WMMA from M = 1 on gfx11/gfx115x and from M = 2 on gfx120x. fp32 picks SIMT or WMMA from a small per-target table by M and N.

  • Tile selection: per-target rules for gfx110x/gfx1103, gfx115x and gfx120x as tables (per M bucket, up to three leaves on N, K or K/N, then a default tile), fitted from tile sweeps over 40 LLM linear-layer shapes at pinned GPU clocks. The launcher instantiates exactly the tiles the tables list.

  • Where the rules live: the SIMT-vs-WMMA rules, the tile tables and the tuning-target constants are in csrc/gemm_4bit_wmma_rules.cuh. The second commit only moves them there from nested if/else in the launchers; the third applies review fixes (Inf handling in fp32, generic-target builds, documented input ranges). Host programs comparing old and new code give identical choices on all 14.0e9 (target, dtype, M, N, K) tile picks and 21.2e6 (target, dtype, M, N) SIMT/WMMA picks checked, and the build has the same 470 WMMA kernels.

  • Fused vs dequant + F.linear (_gemm_4bit_use_custom_rocm), as the largest M that stays on the fused path:

    target bf16 / fp16 fp32
    gfx110x (W7900, 7900 XTX) 384 128
    gfx1103 (Phoenix) 384 384
    gfx115x (Strix Halo) 512 512
    gfx120x (R9700) 384 512

    Measured on gfx1100, gfx1103, gfx1151 and gfx1201. gfx1101/gfx1102 (RX 7800 XT/7700 XT/7600), gfx1150/gfx1152/gfx1153 (Strix Point, Krackan) and gfx1200 (RX 9060 XT) share the WMMA instructions of their family and use the rules of its measured chip (gfx1100, gfx1151, gfx1201) without having been measured. They have fewer CUs than those chips, so the tiles and M limits may be off for them, but the WMMA path should still be well ahead of SIMT and dequant + F.linear for M from about 9 to 128. Builds for gfx11-generic / gfx12-generic get the same kernels.

    The M limits were calibrated with blocksize 64 (the default) and weights streamed from DRAM. With blocksize 32 and weights already in cache, gfx110x at M ≈ 384 can be up to ~20% slower than dequant + F.linear.

CUDA, CDNA (gfx9) and CPU paths are unchanged.

Performance

Speedup of the default dispatch over main (SIMT for M ≤ 8, dequant + F.linear above). Each cell is the total time over 40 LLM linear-layer shapes (Llama 3 8B/70B, Qwen, Mistral, Gemma, ...). Setup: NF4, blocksize 64, weights streamed from DRAM (no L2 reuse between iterations), pinned clocks (amd-smi set -l HIGH or power_dpm_force_performance_level=high).

GPU dtype M=1 M=8 M=16 M=32 M=64 M=128 M=256
Radeon Pro W7900 (gfx1100) fp16 1.07x 2.87x 5.23x 3.67x 2.38x 1.62x 1.16x
bf16 1.06x 2.80x 5.76x 3.99x 2.54x 1.63x 1.16x
fp32 1.00x 1.63x 4.35x 2.54x 1.40x 1.05x (dequant)
Radeon AI PRO R9700 (gfx1201) fp16 1.00x 2.72x 5.86x 4.67x 2.97x 1.96x 1.28x
bf16 1.00x 2.66x 5.85x 4.63x 2.99x 1.96x 1.28x
fp32 1.00x 1.54x 6.04x 3.85x 2.34x 1.74x 1.45x
Radeon 8060S (gfx1151) fp16 1.11x 2.58x 6.50x 5.54x 3.92x 2.98x 2.20x
bf16 1.09x 3.00x 7.06x 5.87x 4.08x 2.83x 1.99x
fp32 1.00x 1.59x 6.03x 3.48x 2.12x 1.76x 1.35x
Radeon 760M (gfx1103) fp16 1.01x 2.50x 8.63x 7.47x 4.89x 2.78x 1.70x
bf16 1.02x 2.34x 8.59x 7.40x 4.79x 2.71x 1.64x
fp32 1.00x 2.05x 9.20x 5.33x 3.86x 3.49x 3.31x

At M = 1-2 the gain is small and some shapes lose. On gfx11, up to ~10-15% of shapes are slower than SIMT at M = 1, by up to 16%. On gfx1201 at M = 2, 6-7 of the 40 shapes are slower, by up to 0.6x; WMMA starts at M = 2 there because it wins on total time. From M = 3-4 on, bf16/fp16 are faster on every shape. Within the M limits, the default dispatch's total time is within ~2% of a per-shape oracle over SIMT, WMMA and dequant + F.linear at most M (largest gap 6.3%, gfx1151 fp32 at M = 3).

Kernels vs the bandwidth limit

Median over the 40 shapes, with each kernel forced: latency (top) and achieved bandwidth (bottom) against the time to read the GEMM's bytes once at peak DRAM bandwidth (dashed).

Radeon Pro W7900 (gfx1100)
W7900

Radeon AI PRO R9700 (gfx1201)
R9700

Radeon 8060S (gfx1151)
8060S

Radeon 760M (gfx1103)
760M

Tests

  • New tests: odd shapes for the 4-bit GEMM; test_matmul_4bit_fp32_precision (fp32 against a float64 reference); test_matmul_4bit_fp32_special_values (Inf/NaN propagation on the SIMT and WMMA paths).
  • pytest tests/test_functional.py tests/test_linear4bit.py -k "(4bit or Linear4bit or gemv) and not fsdp" on the head commit (test_fsdp_state_dict_save_4bit is skipped because it fails on main too in this venv: its torchrun subprocess cannot import bitsandbytes):
    • gfx1100 (RX 7900 XTX): 2422 passed, 23 failed
    • gfx1201 (R9700): 2422 passed, 23 failed
    • gfx1151 (Radeon 8060S): 2423 passed, 22 failed
    • gfx1100 and gfx1201 with a gfx11-generic;gfx12-generic build: 2422 passed, 23 failed (the WMMA kernels run)
    • gfx1103 (Radeon 760M): 2445 passed, 0 failed
    • Every failure is an fp32 test_gemv_4bit case (see below).
  • Edge cases checked by two independent reviews: a sweep reaching every rule-table bucket and leaf (6,480 runs: bf16/fp16/fp32, blocksize 32/64/256, K and K+64, nested and plain absmax, NF4/FP4, bias) matched a float64 reference on gfx1100 and gfx1201.
  • Known failures, unrelated to this PR: 23 fp32 test_gemv_4bit cases fail on gfx1100 and gfx1201, identically on main (22 on gfx1151, where main was not compared). The test compares gemv_4bit to dequant + F.linear, and on these GPUs the BLAS fp32 result is the less accurate side. Against float64, F.linear (hipBLASLt on gfx11, rocBLAS/hipBLASLt on gfx1201) has ~4.8e-8 mean error and gemv_4bit has ~1.0e-8. With TORCH_BLAS_PREFER_HIPBLASLT=0, all 64 fp32 cases pass on gfx1100. See also Fix fp32 4-bit GEMV test tolerance for ROCm/gfx1200 #2113.
  • pre-commit run --all-files passes.
  • Build cost: the WMMA object is 2.7 MB for two targets (gfx1100;gfx1201) and takes ~18 s to compile, against 0.41 MB and ~4 s for the SIMT object.

Full plots and data

The benchmark harness, raw measurements and all plots are on a separate branch that is not part of this PR: benchmark/rocm-wmma-gemm-4bit (this PR plus two commits).

  • Plots: for each GPU, SIMT vs WMMA vs dequant + BLAS against the bandwidth limit, as a median over the shapes and for every one of the 40 shapes in each dtype.
  • Harness and data: bench.py, the tile sweeps, the rule and cutoff fitting scripts, and the CSVs the rules and plots come from.
  • Tuning overrides patch: the tile sweeps need to force a kernel or a tile, which this PR does not support. The patch adds a -DBNB_WMMA_DEV=ON build option with environment overrides for that (BNB_GEMM4BIT_KERNEL, BNB_WMMA_CFG, BNB_WMMA_ROW_TILES_FIRST, BNB_WMMA_PRESCALE). Apply it with git apply on top of this PR to re-tune the rules or calibrate another GPU.

🤖 Generated with Claude Code

@sstamenk
sstamenk marked this pull request as draft November 16, 2026 15:46
@sstamenk
sstamenk force-pushed the feature/rocm-wmma-gemm-4bit-pr branch 2 times, most recently from a8ca462 to 9d6b3f7 Compare November 16, 2026 16:28
…nd RDNA4

Adds csrc/gemm_4bit_wmma.cu, a WMMA-based fused 4-bit (NF4/FP4) GEMM for
Radeon GPUs, and dispatches to it from the existing ROCm SIMT/dequant path.
It extends the fused path's advantage over dequant+F.linear from a few rows to
several hundred, mirroring the CUDA SIMT > MMA > dequant+BLAS dispatch.

Kernel
- gfx11 (RDNA3/RDNA3.5) v_wmma_f32_16x16x16_{f16,bf16}: a wave32 WMMA runs as
  two independent 8x16x16 half-wave products, so each half-wave takes its own
  32-K slice (summed with permlanex16); 8 real rows per WMMA.
- gfx12 (RDNA4) _w32_gfx12 WMMAs: one full 16x16x16 product; each half-wave
  supplies its own K slice, summed inside the WMMA.
- Weights decoded through a 256-entry LDS byte -> centroid-pair table; absmax
  applied in fp32 after the WMMAs (gfx11 fp16 large tiles pre-scale the B
  operand with packed half math instead; nested statistics keep the fp32 path).
  A is double-buffered in LDS, split-K across warps with a deterministic LDS
  reduction, per-block K-chunk rotation against DRAM channel camping, and
  row-tiles-first block order so row tiles of the same columns share B in L2.
- fp32 is emulated with 3-way split bf16 operands (6 WMMAs per tile) and stays
  fp32-accurate (~3-5e-6 relative error vs a float64 reference).

Dispatch (K % 64 == 0, blocksize >= 32)
- Tile rules per target (gfx110x, gfx1103, gfx1151, gfx1201), fitted from tile
  sweeps over 40 LLM linear-layer shapes at a pinned GPU performance level.
- SIMT vs WMMA in csrc/gemm_4bit.cu: bf16/fp16 WMMA from M = 1 (gfx11/gfx115x)
  or M = 2 (gfx1201); fp32 picks SIMT or WMMA by M and N.
- Fused vs dequant+F.linear in _gemm_4bit_use_custom_rocm: gfx110x bf16/fp16
  through M = 384 and fp32 through 128; gfx1103 through 384 for all dtypes;
  gfx1151 through 512; gfx1201 bf16/fp16 through 384 and fp32 through 512.
  gfx1101/gfx1102, gfx1150/gfx1152/gfx1153 and gfx1200 use the rules of their
  family's measured chip (gfx1100, gfx1151, gfx1201): same WMMA instructions,
  not measured.

Performance (speedup of the total time over 40 LLM linear-layer shapes vs the
previous dispatch, SIMT for M <= 8 and dequant+F.linear above; NF4 blocksize 64,
weights streamed from DRAM, pinned clocks; bf16/fp16 at M = 8 / 16 / 128 / 256):
- Radeon Pro W7900 (gfx1100):      2.8-2.9x / 5.2-5.8x / 1.6x / 1.2x
- Radeon AI PRO R9700 (gfx1201):   2.7x / 5.9x / 2.0x / 1.3x
- Radeon 8060S (gfx1151):          2.6-3.0x / 6.5-7.1x / 2.8-3.0x / 2.0-2.2x
- Radeon 760M (gfx1103):           2.3-2.5x / 8.6x / 2.7-2.8x / 1.6-1.7x
fp32 gains 1.5-2.1x at M = 8 and 4.4-9.2x at M = 16. At M = 1-2 the gain
is small and some shapes lose (up to 16% at M = 1 on gfx11; on gfx1201 at
M = 2, 6-7 of 40 shapes by up to 0.6x).
Within the caps the default dispatch's total time is within ~2% of a
per-shape oracle over SIMT, WMMA and dequant+F.linear at most M (largest gap
6.3%, gfx1151 fp32 M = 3).

Tests: extra odd shapes for the 4-bit GEMM and an fp32 precision test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sstamenk
sstamenk force-pushed the feature/rocm-wmma-gemm-4bit-pr branch from 9d6b3f7 to d24b85d Compare November 16, 2026 16:33
…es.cuh

The tile selection was ~180 lines of nested if/else per target and dtype in
launch_gemm_4bit_wmma_arch, with conditions written several ways, and the SIMT vs
WMMA cutoffs were inline in gemm_4bit(). Both are now data in
csrc/gemm_4bit_wmma_rules.cuh, next to the tuning-target constants:
- wmma_preferred(target, fp32, M, N): SIMT or WMMA (bf16/fp16 rule plus the
  per-target fp32 cutoff tables).
- Tile rule tables per target and dtype: a list of M buckets, each with up to
  three leaves (N < x, K < x or K/N < a/b) and a default tile, in the shape the
  tile sweeps are fitted in. wmma_pick() does the lookup; the launcher collects
  each table's distinct tiles at compile time and dispatches to their
  instantiations, so exactly the same kernels are built as before.

No behaviour change. Host programs comparing the previous code with the new
lookups agree on all 14.0e9 (target, dtype, M, N, K) tile choices (M = 1-1024,
N and K in multiples of 64 up to 65536 plus every threshold +-1) and all 21.2e6
(target, dtype, M, N) SIMT/WMMA choices (N in multiples of 32 plus every cutoff
+-1); the build contains the same 470 WMMA kernel symbols.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sstamenk
sstamenk force-pushed the feature/rocm-wmma-gemm-4bit-pr branch from 68332d3 to 002dd31 Compare November 16, 2026 17:35
…gets, review fixes

- fp32: an Inf activation became NaN. split_bf16 kept Inf/NaN in the leading part,
  which is also multiplied by B's lower parts, and those are zero for every
  centroid exact in bf16 (Inf * 0). Non-finite values now go to the last part,
  which only meets B's leading part, so Inf/NaN propagate as in fp32 math. Finite
  results are unchanged. New test: test_matmul_4bit_fp32_special_values (fails on
  the WMMA path before this change).
- Builds for gfx11-generic / gfx12-generic emitted no WMMA kernel bodies but the
  dispatch still selected WMMA by device name, so the first launch aborted on the
  missing symbol. The device pass now treats __gfx11_generic__ / __gfx12_generic__
  like their family (one BNB_WMMA_DEVICE_ARCH instead of BNB_WMMA_RDNA3/4, which
  also drops most new -Wexpansion-to-defined warnings).
- Document the input ranges of the WMMA path: absmax is applied to the fp32 sum of
  A x centroid over a quantization group, so |A| must stay below ~FLT_MAX / blocksize
  (fp32/bf16 inputs only); the three-part split-bf16 keeps fp32 accuracy for
  |A| >= ~1e-33.
- Python: check K % 64 once; note that the M limits were calibrated with blocksize
  64 and weights streamed from DRAM.
- Named tuning targets instead of literal 11/12; fix stale test comments (the fp32
  precision bound and its metric, the gfx12 wide-N tile).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sstamenk
sstamenk marked this pull request as ready for review November 17, 2026 14:21

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant