Skip to content

ggml-cuda: provide static workspace for cuBLAS handles - #26574

Merged
ggerganov merged 5 commits into
ggml-org:masterfrom
heislera763:cublas-workspace
Aug 20, 2026
Merged

ggerganov merged 5 commits into
ggml-org:masterfrom
heislera763:cublas-workspace

Conversation

@heislera763

Copy link
Copy Markdown
Contributor

Overview

This change assigns a static workspace buffer per-device for cublas, allocating the nvidia recommended workspace size according to the device arch.

You will also notice that cublas handle usage has been slightly adjusted so that we automatically set the workspace after every SetStream, because according to nvidia docs: "The cublasSetStream() function unconditionally resets the cuBLAS library workspace back to the default workspace pool" (I'm not sure why nvidia set it up to behave this way, but that's how it works)

This is an attempt to address the issues found in #26119 and probably also #26554, which would have both been affected by my change in #25749. There seems to be a correlation between these crashes and Volta/Turing + graphs, but reading through the docs I'm unsure why it wouldn't effect other archs, maybe the dynamic workspace allocation works differently on these older archs?

Changes are primarily based on information from: https://docs.nvidia.com/cuda/cublas/index.html#cublassetworkspace

Additional information

Anecdotally, I haven't had issues running this branch for a few days, and the user from #26119 says that this patch fixed the crash they were seeing.

Also, I saw no apparent performance changes

master:

llama-bench -m qwen3.6-27b-f16.gguf -ngl 999 -sm layer,tensor
ggml_cuda_init: found 8 CUDA devices (Total VRAM: 259953 MiB):
  Device 0: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
  Device 1: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
  Device 2: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
  Device 3: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
  Device 4: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
  Device 5: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
  Device 6: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
  Device 7: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
| model                          |       size |     params | backend    | ngl |     sm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | --------------: | -------------------: |
| qwen35 27B F16                 |  50.89 GiB |    27.32 B | CUDA       | 999 |  layer |           pp512 |      1085.01 ± 50.09 |
| qwen35 27B F16                 |  50.89 GiB |    27.32 B | CUDA       | 999 |  layer |           tg128 |         15.04 ± 0.01 |
| qwen35 27B F16                 |  50.89 GiB |    27.32 B | CUDA       | 999 | tensor |           pp512 |      1226.95 ± 80.99 |
| qwen35 27B F16                 |  50.89 GiB |    27.32 B | CUDA       | 999 | tensor |           tg128 |         43.93 ± 1.43 |

build: 7bd8282c3 (10267)

This PR:

llama-bench -m qwen3.6-27b-f16.gguf -ngl 999 -sm layer,tensor
ggml_cuda_init: found 8 CUDA devices (Total VRAM: 259953 MiB):
  Device 0: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
  Device 1: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
  Device 2: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
  Device 3: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
  Device 4: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
  Device 5: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
  Device 6: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
  Device 7: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
| model                          |       size |     params | backend    | ngl |     sm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | --------------: | -------------------: |
| qwen35 27B F16                 |  50.89 GiB |    27.32 B | CUDA       | 999 |  layer |           pp512 |      1089.95 ± 38.71 |
| qwen35 27B F16                 |  50.89 GiB |    27.32 B | CUDA       | 999 |  layer |           tg128 |         15.05 ± 0.01 |
| qwen35 27B F16                 |  50.89 GiB |    27.32 B | CUDA       | 999 | tensor |           pp512 |      1237.18 ± 58.83 |
| qwen35 27B F16                 |  50.89 GiB |    27.32 B | CUDA       | 999 | tensor |           tg128 |         43.81 ± 1.45 |

build: 6b40024a3 (10268)

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: Yes, I used Gemini 3.6 flash via antigravity cli extensively for this PR. I'm not deeply familiar with cuda/cuBLAS and how llama.cpp uses them, but I have the read sections of the nvidia docs on this topic and I think this change is the correct approach

@heislera763
heislera763 requested a review from a team as a code owner August 4, 2026 13:07
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Aug 4, 2026
@Peuqui

Peuqui commented Aug 4, 2026

Copy link
Copy Markdown

Testing this on the #26554 setup (2× RTX 8000 Turing + 3× V100 Volta, DeepSeek-V4-Flash-0731 + DSpark sidecar, 5-GPU layer split).

Heads-up for other testers: the current branch base (~b10267 master) has a deepseek4 load regression unrelated to this PR — GGML_ASSERT(ggml_nelements(a) == (ne0*ne1*ne2*ne3)) failed at ggml.c:3614, deterministic on every model load. b10257 loads the same GGUF fine, so it came in somewhere in b10258..b10267. I cherry-picked this PR's commit onto b10257 to test the workspace fix in isolation — loads and infers fine there. Now running it under production load (spec-draft-n-max 5, CUDA graphs enabled, the exact config that produced 4 crashes) and will report back after a few long sessions.

@heislera763

Copy link
Copy Markdown
Contributor Author

@Peuqui Thanks for the info about that dsv4 issue, looks like I made this PR right before the fix got in. I went ahead and merged in master so that won't be an issue for anyone else

@blakemartz

Copy link
Copy Markdown

Tested this PR as-submitted (head 810d90cb) — validation from another hardware class: Linux + Ada (2× RTX 4090), text-only, --spec-type draft-dspark. Same crash class as #26119, same model/spec combo as #26554.

Setup: Ampere Altra aarch64 + 2× RTX 4090, CUDA 12.8 (V12.8.93), driver 580.126.18. DeepSeek-V4-Flash-0731 MXFP4, experts on CPU via -ot, -c 262144 -b 4096 -ub 4096 -fa 1, single slot, DSpark sidecar with --spec-draft-n-max 3 -ngld 99 -otd 'exps=CPU' -devd CUDA1. CUDA graphs on (defaults). No mmproj/vision.

First hit the crash in production on b91 (596a5795) after ~7h50m of agentic traffic. Then found a ~5-minute deterministic trigger: a greedy 5-prompt sequence ending in two consecutive high-acceptance file-edit re-outputs (docstring-insert at ~93% draft acceptance, then a symbol-rename at ~99%), which reuses a warm slot at high LCP similarity (f_sim ≈ 0.99). On unpatched 596a5795: 2/2 crashes, both on the final rename prompt, identical signature:

CUDA error: an unsupported value or parameter was passed to the function
  current device: 1, in function ggml_cuda_mul_mat_cublas_impl at ggml/src/ggml-cuda/ggml-cuda.cu:1541
  cublasSgemm_v2(ctx.cublas_handle(), CUBLAS_OP_T, CUBLAS_OP_N, ne01, ne11, ne10, ...)

(device 1 is the drafter's device; gdb backtrace runs update_slots → llama_decode → ggml_backend_cuda_graph_compute → ggml_cuda_mul_mat_cublas; full logs preserved.)

On this PR branch (810d90cb), same config, same harness, fresh server load per run: the identical sequence ran clean 0/2, decode at parity with baseline-master + this patch cherry-picked (17.3–17.7 t/s spec-on mean across runs; spec-off unchanged, 12.78 vs 13.06 t/s on the respective bases — within same-day noise on this rig), per-prompt draft acceptance bit-identical. Stability-positive, perf-neutral here.

@BLSharda

BLSharda commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

There are few other places in repo where cublasSetStream is used ssm-scan.cu, out-prod.cu and solve_tri.cu. Do you want to patch them to use ctx.cublas_handle(stream)?

@Peuqui

Peuqui commented Aug 5, 2026

Copy link
Copy Markdown

Confirming the fix from the #26554 setup (2× RTX 8000 Turing + 3× V100 Volta, DeepSeek-V4-Flash-0731 UD-Q4_K_XL + DSpark sidecar, 5-GPU layer split, --spec-draft-n-max 5, CUDA graphs enabled — the exact config that produced 4 crashes in ~2 days): running this PR's commit cherry-picked onto b10257 under production load for ~24h now, including several long agentic coding sessions (27k-token prompts, 30–45 min continuous generation) — zero crashes. Performance unchanged. Together with @blakemartz's deterministic repro this now covers Volta, Turing and Ada. Thanks for the quick fix!

@ORippler ORippler left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for investigating this!

For my understanding: For these older GPUs, do we run cublas also during decode? Or is this a case where we have cudaGraphs enabled for prefill (unintentionally).

Current proposal does not work with GGML_CUDA_GRAPH_OPT=1

Comment thread ggml/src/ggml-cuda/common.cuh Resolved
@ORippler

ORippler commented Aug 10, 2026 •

Copy link
Copy Markdown
Collaborator

Also, what software stack are affected folks on (CTK, driver, cuBLAS)?

This is an attempt to address the issues found in #26119 and probably also #26554, which would have both been affected by my change in #25749. There seems to be a correlation between these crashes and Volta/Turing + graphs, but reading through the docs I'm unsure why it wouldn't effect other archs, maybe the dynamic workspace allocation works differently on these older archs?

cuBLAS will behave differently across SW/HW combinations, hence me asking

@Peuqui

Peuqui commented Aug 10, 2026

Copy link
Copy Markdown

Software stack: Driver 580.173.02 (CUDA 13.0 capable), but the build uses CUDA Toolkit 12.0 (nvcc 12.0.140) and links cuBLAS 12.0.2.224 (system CUDA-12.0 lib). GPUs: 2× Quadro RTX 8000 (Turing, sm_75) + 3× Tesla V100 (Volta, sm_70), built with CMAKE_CUDA_ARCHITECTURES=70;75.

cuBLAS during decode: Yes — the abort is cublasSgemm_v2 inside server_context_impl::decode → update_slots (see the stack frames in the crash reports on #26554), so cuBLAS runs during decode, not just prefill. CUDA graphs were at their default (enabled); I did not set GGML_CUDA_GRAPH_OPT=1. Speculative decoding (DSpark / MTP) was active, so the decode path processes verify-batches of varying shape.

I haven't tested GGML_CUDA_GRAPH_OPT=1 yet — can try it if useful. (The FP32 Sgemm in the decode path is itself unusual — possibly the DSpark markov head / sparse-attention indexer producing a degenerate shape that the static-workspace patch happens to make safe.)

@heislera763

Copy link
Copy Markdown
Contributor Author

@ORippler I think this is only during decode.

Also, I've changed the handle and workspace pointers to also be per-stream, rather than just per-device

@Peuqui

Peuqui commented Aug 11, 2026

Copy link
Copy Markdown

Following up on my earlier note — I said I hadn't tested GGML_CUDA_GRAPH_OPT=1 yet and could try it if useful. I've now tested it, on the current head of this PR including the per-stream change.

Tested revision: e2c3b56a4 ("account for concurrent streams when using GGML_CUDA_GRAPH_OPT") on top of 0458b380e.

Stack:

Driver 580.173.02 (CUDA 13.0 capable)
Built with CUDA Toolkit 12.0 (nvcc 12.0.140), links cuBLAS 12.0.2.224
CMAKE_CUDA_ARCHITECTURES=70;75
GPUs: 2x Quadro RTX 8000 (Turing, sm_75) + 3x Tesla V100-PCIE-32GB (Volta, sm_70)
Model: DeepSeek-V4-Flash 284B-A13B UD-Q4_K_XL, -ngl 99 -c 193536, layer split
       over all 5 GPUs, flash-attn on, speculative decoding (DSpark draft)

I ran a 2x2 matrix — -np 1 and -np 4, each with GGML_CUDA_GRAPH_OPT on and off — 12 requests per cell, 48 requests total. Each round fires 4 requests simultaneously, three decode-heavy (4000 tokens each) plus one prefill-heavy, so with -np 4 several slots decode concurrently. No crashes, no CUDA error, no ggml_abort, no GGML_ASSERT in any cell.

One caveat about that first matrix: all requests used the same prompt text, so llama.cpp matched the prefix cache (f_sim_best = 1.000) and skipped the prefill — 4 prompt tokens instead of ~28k. That makes it a decode-only test with essentially constant tensor shapes, which is weak coverage for a cuBLAS issue. So I repeated the -np 1 cells with a unique prompt per request (8.5k-11.7k prompt tokens each, varying in length and content, with no cache hits in the logs). Both cells came out clean as well: 12/12 requests, 125,385 prompt tokens processed per cell, no crashes. That brings the total to 72 requests across all configurations, none of which reproduced the abort.

On throughput, GGML_CUDA_GRAPH_OPT made no measurable difference here. Steady-state aggregate (excluding the first round, which carries the ~176 s model load):

                              same prompt   unique prompts
-np 4, graph-opt on            37.8 tok/s        -
-np 4, graph-opt off           39.6 tok/s        -
-np 1, graph-opt on            27.0 tok/s   24.4 tok/s
-np 1, graph-opt off           26.4 tok/s   24.9 tok/s

In the unique-prompt run the two cells track each other closely round by round
(18.9 / 24.0 / 24.8 with the option on, 18.8 / 24.8 / 25.0 with it off), which
is the cleaner comparison of the two since prompts and seeds were identical
across cells and the prefill was actually paid.

The two configurations disagree on the direction, and the round-to-round spread within a single cell (4-7%) is as large as the difference between cells, so I read this as no effect rather than a small win either way. Two rounds per cell is not enough to claim more than that.

For context on the original crash: it hit cublasSgemm_v2 inside server_context_impl::decode -> update_slots with graphs at their default setting, i.e. without GGML_CUDA_GRAPH_OPT. Since moving to this PR's build the crashes are gone, and they have stayed gone under all of the above.

@ORippler ORippler left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't be fulfilling any of the conditions here that would lead to failure mode with cudaGraphs (we neither launch from device side, nor child graphs, nor are we in a CiG context, nor do we output to host *). I unfortunately don't have time to assist in root-causing this further.

However, the changes here seem to fix the reported issues. Please adjust the other call sites to cublasSetStream as well to fetch the persistent handle, as suggested in #26574 (comment)

@heislera763

Copy link
Copy Markdown
Contributor Author

@ORippler Agreed on the root cause not being clear, I recall reading about a number of scenarios where this can go wrong, but it didn't seem like any actually applied in this case. I've been working on this using gemini and it seems to think it might be window WDDM mode related, but I haven't been able to nail down a reason any more specific than "WDDM doesn't play well with CUDA"

Regardless, I've removed the usage of cublasSetStream so that we do everything through the handle, as suggested.

While I was doing that I realized that solve_tri was the only place where we were passing anything to the handle, but I don't think this was really doing anything as far as I can tell. So, I've removed the function overloading to simplify things to just use the parameterless cublas_handle(), I think the logic reads a bit cleaner this way and I can't think of case where you would need to specify either the device or stream explicitly. I will revert that if you think I'm going to far with things and/or maybe it is useful in some scenarios.

@Peuqui

Peuqui commented Aug 13, 2026

Copy link
Copy Markdown

One datapoint on the WDDM theory: my crashes from #26554 happened on native Linux (Ubuntu 24.04, kernel 7.0.0-28-generic, driver 580.173.02 — no WSL, no display attached to any of the NVIDIA cards, pure compute). So WDDM can't be the whole story — either there are two distinct failure modes, or the root cause is platform-independent.

GPUs are free right now — I'll run my production-load confirmation on the current head (063e8c0d2, i.e. including the cublasSetStream removal and the parameterless cublas_handle()) on the 2× RTX 8000 + 3× V100 setup and report back.

@Peuqui

Peuqui commented Aug 13, 2026

Copy link
Copy Markdown

Confirmation on the current head done. Rebuilt 063e8c0d2 (build 10419) on the same stack as my earlier reports (native Ubuntu 24.04, driver 580.173.02, CUDA Toolkit 12.0, 2× Quadro RTX 8000 sm_75 + 3× Tesla V100 sm_70, DeepSeek-V4-Flash UD-Q4_K_XL + DSpark draft, 5-GPU layer split).

Re-ran the unique-prompt series (4 concurrent clients against -np 1, 3 rounds, full ~8–9k-token prefill per request, 4000 generated tokens each):

  • GGML_CUDA_GRAPH_OPT=1: 12/12 OK
  • default (no GRAPH_OPT): 12/12 OK

24/24 clean, zero crash markers in the server logs. Per-request throughput is on par with the previously tested revision e2c3b56a4 (differences within run-to-run noise), so the cublasSetStream removal and the parameterless cublas_handle() cost nothing measurable on Volta+Turing either.

@ORippler

Copy link
Copy Markdown
Collaborator

@Peuqui how long does it take for you to repro the issue? If it's not too long, I may send debugging patches your way so we can try to root-cause this more/fully. If you don't have the capacity, we can also go ahead with the workaround.

Has anybody faced this for single-GPU? Or does it only occur for multi-GPU setups

@Peuqui

Peuqui commented Aug 13, 2026

Copy link
Copy Markdown

@ORippler Repro data point: per my report in #26554 the abort hit reliably after ~2,000–2,600 decoded tokens of a single long reasoning-heavy response on the unfixed build (one crash logged at exactly 2,518 decoded) — so a repro attempt there is a single long-generation request, i.e. a few minutes, not hours. On this PR's head I've since accumulated a real burn-in: several 30–60 min continuous agentic generations plus 24 synthetic 4k-token runs — zero aborts.

Single-GPU: can't say — my setup is a 5-GPU layer split (2× Turing + 3× Volta), I've never run this workload single-GPU.

From my side the workaround is solid and I'd be fine with merging it as is. If deterministic iteration is needed for root-causing, @blakemartz's 2/2 repro is probably the more efficient target than my box — but if patches land here anyway, I can run the occasional soak test.

@ORippler

ORippler commented Aug 14, 2026 •

Copy link
Copy Markdown
Collaborator

While I was doing that I realized that solve_tri was the only place where we were passing anything to the handle, but I don't think this was really doing anything as far as I can tell. So, I've removed the function overloading to simplify things to just use the parameterless cublas_handle(), I think the logic reads a bit cleaner this way and I can't think of case where you would need to specify either the device or stream explicitly.

@pwilkin was there a reason you explicitly queried device in #17793 for solve_tri? I would tend to agree with @heislera763's assessment.

@ORippler

Copy link
Copy Markdown
Collaborator

@blakemartz mind running https://github.com/ORippler/llama.cpp/tree/osimons/cublas-workspace-rootcause to root-cause further based on your stable repro?

Invoke like so

  GGML_CUDA_CUBLAS_DEBUG=1 \
  GGML_CUDA_CUBLAS_WORKSPACE_SIZE=default \
  ./server ...

and

  GGML_CUDA_CUBLAS_DEBUG=1 \
  GGML_CUDA_CUBLAS_WORKSPACE_SIZE=0 \
  ./server ...

And post logs?

@ORippler ORippler left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed with @blakemartz that this is due to cublas not reclaiming per-cudaGraph allocations made to the default workspace in cublas < 12.4. This PR introduces cublasSetWorkspace() as the documented workaround for older cublas versions.

@pwilkin please chime in on the need for explicit device query #26574 (comment)

Comment on lines -714 to +719
}
if (cublas_handles[i] != nullptr) {
CUBLAS_CHECK(cublasDestroy(cublas_handles[i]));
if (cublas_handles[i][j] != nullptr) {
CUBLAS_CHECK(cublasDestroy(cublas_handles[i][j]));
}
if (cublas_workspaces[i][j] != nullptr) {
CUDA_CHECK(cudaFree(cublas_workspaces[i][j]));
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

teardown should happen cublas_handle -> workspace -> stream, not stream -> handle -> workspace. Morever, the current device should be set. But this can happen in a follow-up PR, as existing logic already did the teardown in the incorrect way

Comment thread ggml/src/ggml-cuda/common.cuh Resolved
@ORippler ORippler added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Aug 19, 2026
@ggerganov

Copy link
Copy Markdown
Member

@ORippler Good to merge or waiting for @pwilkin to chime in?

@ORippler

Copy link
Copy Markdown
Collaborator

@ORippler Good to merge or waiting for @pwilkin to chime in?

Good to merge imo

@ggerganov
ggerganov merged commit d9b6be0 into ggml-org:master Aug 20, 2026
23 of 24 checks passed
@ggerganov

Copy link
Copy Markdown
Member

Do we expect this to have an effect on modern RTX hardware, or is it just related to issues with V100?

@pwilkin

pwilkin commented Aug 20, 2026

Copy link
Copy Markdown
Member

Sorry, missed this one - yeah, per-stream should be fine.

@Peuqui

Peuqui commented Aug 20, 2026

Copy link
Copy Markdown

Data point from the reporter side of #26554, on the V100 question specifically.

My box is a mixed 5-GPU layer split, pinned per process via CUDA_VISIBLE_DEVICES with the two Turing cards first:

  • CUDA0, CUDA1 = Quadro RTX 8000 (Turing, sm_75)
  • CUDA2, CUDA3, CUDA4 = Tesla V100-PCIE-32GB (Volta, sm_70)

Every abort I logged reported current device: 0 — so a Quadro RTX 8000, not a V100. The three V100s in the same process never aborted, including CUDA4, which is where the DSpark drafter was pinned (--device-draft CUDA4).

Caveat so this isn't over-read: device 0 is simply where the failing FP32 GEMM landed given my --tensor-split 11,12,8,7,5 — the first 11 layers sit on that card. This does not show that Turing is more susceptible than Volta, and it does not show that the V100s would have survived a different layer distribution. All I can claim is that in this particular Turing+Volta mix, the card that actually aborted was the Turing one.

The stronger data point against a V100-only theory is @blakemartz's: 2x RTX 4090 (Ada), no Volta in the machine at all, deterministic repro 2/2 before the patch and 0/2 after.

So Volta, Turing and Ada are all covered by reports now. Nobody has reported from Hopper or Blackwell in either direction, so I wouldn't extrapolate there.

@ORippler

Copy link
Copy Markdown
Collaborator

Do we expect this to have an effect on modern RTX hardware, or is it just related to issues with V100?

This fix is for older CUBLAS versions (i.e. CUBLAS < 12.4), mainly used by older HW/CTK setups. Other issue like #27368 and #27102 need further/independent investigation and fixes

therealkenc pushed a commit to therealkenc/llama.cpp that referenced this pull request Aug 24, 2026
* provide static workspace for cuBLAS handles

* account for concurrent streams when using GGML_CUDA_GRAPH_OPT

* drop cublas_handle overloads and remove direct cublasSetStream calls

* Update ggml/src/ggml-cuda/common.cuh

---------

Co-authored-by: Oliver Simons <osimons@nvidia.com>
srossitto79 added a commit to srossitto79/llama.cpp that referenced this pull request Aug 25, 2026
Conflicts resolved:

- common/CMakeLists.txt: keep both the fork's jsonl.cpp/.h and upstream's
  new json.cpp/.h (common_json wrapper, ggml-org#27511).

- ggml/src/ggml-cuda/out-prod.cu: take upstream's removal of the redundant
  cublasSetStream (ggml-org#26574 binds the cuBLAS handle to its stream at creation).
  The fork's conditional lda for dequantized (quantized) src0 is unchanged;
  upstream's unconditional lda decl is dropped as it would redeclare it.

- ggml/src/ggml-metal/ggml-metal.metal: upstream split the monolithic
  source into kernels/*.metal (ggml-org#26561). The fork's kernels were ported into
  the new layout:
    * quantize_q3_K / quantize_q4_K / quantize_mxfp4 -> kernels/quantize.h
    * kernel_cpy_* q3_K/q4_K/mxfp4 instantiations    -> kernels/quantize.metal
    * adamw gclip, out_prod, out_prod_id, get_rows_back, repeat_back,
      cross_entropy_loss{,_back}                     -> kernels/misc.metal
    * rms_norm_back                                  -> kernels/norm.metal
    * soft_max_back                                  -> kernels/softmax.metal
  No CMake or enum changes are needed: kernel->library routing is built from
  each compiled library's functionNames.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
thecodacus pushed a commit to thecodacus/llama.cpp that referenced this pull request Sep 7, 2026
* provide static workspace for cuBLAS handles

* account for concurrent streams when using GGML_CUDA_GRAPH_OPT

* drop cublas_handle overloads and remove direct cublasSetStream calls

* Update ggml/src/ggml-cuda/common.cuh

---------

Co-authored-by: Oliver Simons <osimons@nvidia.com>
zbrad pushed a commit to zbrad/llama.cpp that referenced this pull request Sep 10, 2026
* provide static workspace for cuBLAS handles

* account for concurrent streams when using GGML_CUDA_GRAPH_OPT

* drop cublas_handle overloads and remove direct cublasSetStream calls

* Update ggml/src/ggml-cuda/common.cuh

---------

Co-authored-by: Oliver Simons <osimons@nvidia.com>
korcan-h added a commit to korcan-h/qvac-fabric-llm.cpp that referenced this pull request Oct 23, 2026
out-prod.cu called cublasSetStream() on the shared cuBLAS handle, which
resets the handle's workspace to cuBLAS's default pool. Upstream removed
this call in PR ggml-org#26574 together with the static workspace; the OUT_PROD
dequantization patch re-added the line by accident from rebases.

After the first OUT_PROD (training backward), every cuBLAS call then
allocated with cudaMallocAsync, so a CUDA graph captured later contained
graph memory allocations.

On the qvac windows CI GPUs those fail, and finetuning aborted with
"MUL_MAT failed: CUDA error: out of memory".

The handle is already bound to ctx.stream() with a fixed 4 MiB workspace
on Ada, so the call is not needed.

Signed-off-by: Korcan Hussein <korcan.hussein@collabora.com>
korcan-h added a commit to tetherto/qvac-fabric-llm.cpp that referenced this pull request Oct 23, 2026
out-prod.cu called cublasSetStream() on the shared cuBLAS handle, which
resets the handle's workspace to cuBLAS's default pool. Upstream removed
this call in PR ggml-org#26574 together with the static workspace; the OUT_PROD
dequantization patch re-added the line by accident from rebases.

After the first OUT_PROD (training backward), every cuBLAS call then
allocated with cudaMallocAsync, so a CUDA graph captured later contained
graph memory allocations.

On the qvac windows CI GPUs those fail, and finetuning aborted with
"MUL_MAT failed: CUDA error: out of memory".

The handle is already bound to ctx.stream() with a fixed 4 MiB workspace
on Ada, so the call is not needed.

Signed-off-by: Korcan Hussein <korcan.hussein@collabora.com>
ikawrakow pushed a commit to ikawrakow/ik_llama.cpp that referenced this pull request Oct 30, 2026
cublasSetStream() resets a cuBLAS handle to the default workspace pool, and cuBLAS calls captured into a CUDA graph may then add memory nodes (cudaMallocAsync/cudaFreeAsync). The decode graph is re-captured every 256 KV positions, and each re-capture left ~80 MiB behind on the main GPU.

Bind a per-(device, stream) workspace after every cublasSetStream(), as mainline does since ggml-org/llama.cpp#26574.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CUDA Related to the CUDA backend ggml changes relating to the ggml tensor library for machine learning merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Eval bug: CUDA "unsupported value" in cublasSgemm (mul_mat_cublas) after ~2500 decoded tokens with DSpark speculative decoding (deepseek4, multi-GPU)

7 participants