Repository navigation
ggml-cuda: provide static workspace for cuBLAS handles - #26574
Conversation
|
Testing this on the #26554 setup (2× RTX 8000 Turing + 3× V100 Volta, DeepSeek-V4-Flash-0731 + DSpark sidecar, 5-GPU layer split). Heads-up for other testers: the current branch base (~b10267 master) has a deepseek4 load regression unrelated to this PR — |
|
@Peuqui Thanks for the info about that dsv4 issue, looks like I made this PR right before the fix got in. I went ahead and merged in master so that won't be an issue for anyone else |
|
Tested this PR as-submitted (head Setup: Ampere Altra aarch64 + 2× RTX 4090, CUDA 12.8 (V12.8.93), driver 580.126.18. DeepSeek-V4-Flash-0731 MXFP4, experts on CPU via First hit the crash in production on b91 ( (device 1 is the drafter's device; gdb backtrace runs On this PR branch ( |
|
There are few other places in repo where |
|
Confirming the fix from the #26554 setup (2× RTX 8000 Turing + 3× V100 Volta, DeepSeek-V4-Flash-0731 UD-Q4_K_XL + DSpark sidecar, 5-GPU layer split, |
ORippler
left a comment
There was a problem hiding this comment.
Thanks for investigating this!
For my understanding: For these older GPUs, do we run cublas also during decode? Or is this a case where we have cudaGraphs enabled for prefill (unintentionally).
Current proposal does not work with GGML_CUDA_GRAPH_OPT=1
|
Also, what software stack are affected folks on (CTK, driver, cuBLAS)?
cuBLAS will behave differently across SW/HW combinations, hence me asking |
|
Software stack: Driver 580.173.02 (CUDA 13.0 capable), but the build uses CUDA Toolkit 12.0 (nvcc 12.0.140) and links cuBLAS 12.0.2.224 (system CUDA-12.0 lib). GPUs: 2× Quadro RTX 8000 (Turing, sm_75) + 3× Tesla V100 (Volta, sm_70), built with cuBLAS during decode: Yes — the abort is I haven't tested |
1f005bf to
e2c3b56
Compare
|
@ORippler I think this is only during decode. Also, I've changed the handle and workspace pointers to also be per-stream, rather than just per-device |
|
Following up on my earlier note — I said I hadn't tested Tested revision: Stack: I ran a 2x2 matrix — One caveat about that first matrix: all requests used the same prompt text, so llama.cpp matched the prefix cache ( On throughput, In the unique-prompt run the two cells track each other closely round by round The two configurations disagree on the direction, and the round-to-round spread within a single cell (4-7%) is as large as the difference between cells, so I read this as no effect rather than a small win either way. Two rounds per cell is not enough to claim more than that. For context on the original crash: it hit |
ORippler
left a comment
There was a problem hiding this comment.
We shouldn't be fulfilling any of the conditions here that would lead to failure mode with cudaGraphs (we neither launch from device side, nor child graphs, nor are we in a CiG context, nor do we output to host *). I unfortunately don't have time to assist in root-causing this further.
However, the changes here seem to fix the reported issues. Please adjust the other call sites to cublasSetStream as well to fetch the persistent handle, as suggested in #26574 (comment)
|
@ORippler Agreed on the root cause not being clear, I recall reading about a number of scenarios where this can go wrong, but it didn't seem like any actually applied in this case. I've been working on this using gemini and it seems to think it might be window WDDM mode related, but I haven't been able to nail down a reason any more specific than "WDDM doesn't play well with CUDA" Regardless, I've removed the usage of cublasSetStream so that we do everything through the handle, as suggested. While I was doing that I realized that solve_tri was the only place where we were passing anything to the handle, but I don't think this was really doing anything as far as I can tell. So, I've removed the function overloading to simplify things to just use the parameterless cublas_handle(), I think the logic reads a bit cleaner this way and I can't think of case where you would need to specify either the device or stream explicitly. I will revert that if you think I'm going to far with things and/or maybe it is useful in some scenarios. |
|
One datapoint on the WDDM theory: my crashes from #26554 happened on native Linux (Ubuntu 24.04, kernel 7.0.0-28-generic, driver 580.173.02 — no WSL, no display attached to any of the NVIDIA cards, pure compute). So WDDM can't be the whole story — either there are two distinct failure modes, or the root cause is platform-independent. GPUs are free right now — I'll run my production-load confirmation on the current head ( |
|
Confirmation on the current head done. Rebuilt Re-ran the unique-prompt series (4 concurrent clients against
24/24 clean, zero crash markers in the server logs. Per-request throughput is on par with the previously tested revision |
|
@Peuqui how long does it take for you to repro the issue? If it's not too long, I may send debugging patches your way so we can try to root-cause this more/fully. If you don't have the capacity, we can also go ahead with the workaround. Has anybody faced this for single-GPU? Or does it only occur for multi-GPU setups |
|
@ORippler Repro data point: per my report in #26554 the abort hit reliably after ~2,000–2,600 decoded tokens of a single long reasoning-heavy response on the unfixed build (one crash logged at exactly 2,518 decoded) — so a repro attempt there is a single long-generation request, i.e. a few minutes, not hours. On this PR's head I've since accumulated a real burn-in: several 30–60 min continuous agentic generations plus 24 synthetic 4k-token runs — zero aborts. Single-GPU: can't say — my setup is a 5-GPU layer split (2× Turing + 3× Volta), I've never run this workload single-GPU. From my side the workaround is solid and I'd be fine with merging it as is. If deterministic iteration is needed for root-causing, @blakemartz's 2/2 repro is probably the more efficient target than my box — but if patches land here anyway, I can run the occasional soak test. |
@pwilkin was there a reason you explicitly queried device in #17793 for solve_tri? I would tend to agree with @heislera763's assessment. |
|
@blakemartz mind running https://github.com/ORippler/llama.cpp/tree/osimons/cublas-workspace-rootcause to root-cause further based on your stable repro? Invoke like so and And post logs? |
There was a problem hiding this comment.
Confirmed with @blakemartz that this is due to cublas not reclaiming per-cudaGraph allocations made to the default workspace in cublas < 12.4. This PR introduces cublasSetWorkspace() as the documented workaround for older cublas versions.
@pwilkin please chime in on the need for explicit device query #26574 (comment)
| } | ||
| if (cublas_handles[i] != nullptr) { | ||
| CUBLAS_CHECK(cublasDestroy(cublas_handles[i])); | ||
| if (cublas_handles[i][j] != nullptr) { | ||
| CUBLAS_CHECK(cublasDestroy(cublas_handles[i][j])); | ||
| } | ||
| if (cublas_workspaces[i][j] != nullptr) { | ||
| CUDA_CHECK(cudaFree(cublas_workspaces[i][j])); | ||
| } |
There was a problem hiding this comment.
teardown should happen cublas_handle -> workspace -> stream, not stream -> handle -> workspace. Morever, the current device should be set. But this can happen in a follow-up PR, as existing logic already did the teardown in the incorrect way
|
Do we expect this to have an effect on modern RTX hardware, or is it just related to issues with V100? |
|
Sorry, missed this one - yeah, per-stream should be fine. |
|
Data point from the reporter side of #26554, on the V100 question specifically. My box is a mixed 5-GPU layer split, pinned per process via
Every abort I logged reported Caveat so this isn't over-read: device 0 is simply where the failing FP32 GEMM landed given my The stronger data point against a V100-only theory is @blakemartz's: 2x RTX 4090 (Ada), no Volta in the machine at all, deterministic repro 2/2 before the patch and 0/2 after. So Volta, Turing and Ada are all covered by reports now. Nobody has reported from Hopper or Blackwell in either direction, so I wouldn't extrapolate there. |
* provide static workspace for cuBLAS handles * account for concurrent streams when using GGML_CUDA_GRAPH_OPT * drop cublas_handle overloads and remove direct cublasSetStream calls * Update ggml/src/ggml-cuda/common.cuh --------- Co-authored-by: Oliver Simons <osimons@nvidia.com>
Conflicts resolved: - common/CMakeLists.txt: keep both the fork's jsonl.cpp/.h and upstream's new json.cpp/.h (common_json wrapper, ggml-org#27511). - ggml/src/ggml-cuda/out-prod.cu: take upstream's removal of the redundant cublasSetStream (ggml-org#26574 binds the cuBLAS handle to its stream at creation). The fork's conditional lda for dequantized (quantized) src0 is unchanged; upstream's unconditional lda decl is dropped as it would redeclare it. - ggml/src/ggml-metal/ggml-metal.metal: upstream split the monolithic source into kernels/*.metal (ggml-org#26561). The fork's kernels were ported into the new layout: * quantize_q3_K / quantize_q4_K / quantize_mxfp4 -> kernels/quantize.h * kernel_cpy_* q3_K/q4_K/mxfp4 instantiations -> kernels/quantize.metal * adamw gclip, out_prod, out_prod_id, get_rows_back, repeat_back, cross_entropy_loss{,_back} -> kernels/misc.metal * rms_norm_back -> kernels/norm.metal * soft_max_back -> kernels/softmax.metal No CMake or enum changes are needed: kernel->library routing is built from each compiled library's functionNames. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* provide static workspace for cuBLAS handles * account for concurrent streams when using GGML_CUDA_GRAPH_OPT * drop cublas_handle overloads and remove direct cublasSetStream calls * Update ggml/src/ggml-cuda/common.cuh --------- Co-authored-by: Oliver Simons <osimons@nvidia.com>
* provide static workspace for cuBLAS handles * account for concurrent streams when using GGML_CUDA_GRAPH_OPT * drop cublas_handle overloads and remove direct cublasSetStream calls * Update ggml/src/ggml-cuda/common.cuh --------- Co-authored-by: Oliver Simons <osimons@nvidia.com>
out-prod.cu called cublasSetStream() on the shared cuBLAS handle, which resets the handle's workspace to cuBLAS's default pool. Upstream removed this call in PR ggml-org#26574 together with the static workspace; the OUT_PROD dequantization patch re-added the line by accident from rebases. After the first OUT_PROD (training backward), every cuBLAS call then allocated with cudaMallocAsync, so a CUDA graph captured later contained graph memory allocations. On the qvac windows CI GPUs those fail, and finetuning aborted with "MUL_MAT failed: CUDA error: out of memory". The handle is already bound to ctx.stream() with a fixed 4 MiB workspace on Ada, so the call is not needed. Signed-off-by: Korcan Hussein <korcan.hussein@collabora.com>
out-prod.cu called cublasSetStream() on the shared cuBLAS handle, which resets the handle's workspace to cuBLAS's default pool. Upstream removed this call in PR ggml-org#26574 together with the static workspace; the OUT_PROD dequantization patch re-added the line by accident from rebases. After the first OUT_PROD (training backward), every cuBLAS call then allocated with cudaMallocAsync, so a CUDA graph captured later contained graph memory allocations. On the qvac windows CI GPUs those fail, and finetuning aborted with "MUL_MAT failed: CUDA error: out of memory". The handle is already bound to ctx.stream() with a fixed 4 MiB workspace on Ada, so the call is not needed. Signed-off-by: Korcan Hussein <korcan.hussein@collabora.com>
cublasSetStream() resets a cuBLAS handle to the default workspace pool, and cuBLAS calls captured into a CUDA graph may then add memory nodes (cudaMallocAsync/cudaFreeAsync). The decode graph is re-captured every 256 KV positions, and each re-capture left ~80 MiB behind on the main GPU. Bind a per-(device, stream) workspace after every cublasSetStream(), as mainline does since ggml-org/llama.cpp#26574.
Overview
This change assigns a static workspace buffer per-device for cublas, allocating the nvidia recommended workspace size according to the device arch.
You will also notice that cublas handle usage has been slightly adjusted so that we automatically set the workspace after every SetStream, because according to nvidia docs: "The cublasSetStream() function unconditionally resets the cuBLAS library workspace back to the default workspace pool" (I'm not sure why nvidia set it up to behave this way, but that's how it works)
This is an attempt to address the issues found in #26119 and probably also #26554, which would have both been affected by my change in #25749. There seems to be a correlation between these crashes and Volta/Turing + graphs, but reading through the docs I'm unsure why it wouldn't effect other archs, maybe the dynamic workspace allocation works differently on these older archs?
Changes are primarily based on information from: https://docs.nvidia.com/cuda/cublas/index.html#cublassetworkspace
Additional information
Anecdotally, I haven't had issues running this branch for a few days, and the user from #26119 says that this patch fixed the crash they were seeing.
Also, I saw no apparent performance changes
master:
This PR:
Requirements