Repository navigation
RTX 4090D 48GB support? (RuntimeError: CUDA error: mapping of buffer object failed) #36
Description
NVIDIA Open GPU Kernel Modules Version
570.124.04-p2p
Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.
- I confirm that this does not happen with the proprietary driver package.
Operating System and Version
Ubuntu 24.04.2 LTS
Kernel Release
6.8.0-57-generic
Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.
- I am running on a stable kernel release.
Hardware: GPU
NVIDIA GeForce RTX 4090D (48GB)
Describe the bug
In a system with four RTX 4090D GPUs (modded Chinese variant with 48GB VRAM) an error 'mapping of buffer object failed' occurs when attempting p2p communication. I've been able to reproduce the same error with several programs (tabbyAPI, a simple cuda test program, cuda samples etc). The GPUs are connected to the CPU directly without switches.
Obvious stuff out of the way:
- IOMMU off in bios
- Pcie ACS off in bios
- Virtualization disabled (SR-IOV & CPU virtualization)
- Secure boot disabled
- Resizable bar enabled
- nouveau driver blacklisted
General system information:
- Mainboard: asrock rack romd8-2t
- CPU: epyc 7532
- 256GB RAM
- 4x "tronizm" (obscure chinese brand) RTX 4090D 48GB
The cards also purport to be capable of p2p operation (according to cuda samples deviceQuery):
<other 3 devices truncated>
Device 3: "NVIDIA GeForce RTX 4090 D"
CUDA Driver Version / Runtime Version 12.8 / 12.0
CUDA Capability Major/Minor version number: 8.9
Total amount of global memory: 48519 MBytes (50875924480 bytes)
MapSMtoCores for SM 8.9 is undefined. Default to use 128 Cores/SM
MapSMtoCores for SM 8.9 is undefined. Default to use 128 Cores/SM
(114) Multiprocessors, (128) CUDA Cores/MP: 14592 CUDA Cores
GPU Max Clock rate: 2520 MHz (2.52 GHz)
Memory Clock rate: 10501 Mhz
Memory Bus Width: 384-bit
L2 Cache Size: 75497472 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total shared memory per multiprocessor: 102400 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 1536
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 2 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device supports Managed Memory: Yes
Device supports Compute Preemption: Yes
Supports Cooperative Kernel Launch: Yes
Supports MultiDevice Co-op Kernel Launch: Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 193 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
> Peer access from NVIDIA GeForce RTX 4090 D (GPU0) -> NVIDIA GeForce RTX 4090 D (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU0) -> NVIDIA GeForce RTX 4090 D (GPU2) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU0) -> NVIDIA GeForce RTX 4090 D (GPU3) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU1) -> NVIDIA GeForce RTX 4090 D (GPU0) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU1) -> NVIDIA GeForce RTX 4090 D (GPU2) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU1) -> NVIDIA GeForce RTX 4090 D (GPU3) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU2) -> NVIDIA GeForce RTX 4090 D (GPU0) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU2) -> NVIDIA GeForce RTX 4090 D (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU2) -> NVIDIA GeForce RTX 4090 D (GPU3) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU3) -> NVIDIA GeForce RTX 4090 D (GPU0) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU3) -> NVIDIA GeForce RTX 4090 D (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU3) -> NVIDIA GeForce RTX 4090 D (GPU2) : Yes
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.8, CUDA Runtime Version = 12.0, NumDevs = 4
Result = PASS
Also nvidia-smi output:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.04 Driver Version: 570.124.04 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4090 D On | 00000000:01:00.0 Off | Off |
| 30% 43C P0 53W / 425W | 1MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 4090 D On | 00000000:81:00.0 Off | Off |
| 30% 42C P0 54W / 425W | 1MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA GeForce RTX 4090 D On | 00000000:82:00.0 Off | Off |
| 30% 35C P0 58W / 425W | 1MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 3 NVIDIA GeForce RTX 4090 D On | 00000000:C1:00.0 Off | Off |
| 30% 40C P0 42W / 425W | 1MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
To Reproduce
- Install nvidia drivers 570.124.04 from .run file without kernel modules.
- Install kernel modules from here:
https://github.com/aikitoria/open-gpu-kernel-modules/tree/570.124.04-p2p(as I understand, this is just a more up-to-date fork of this repository with patches for more recent driver versions). - Install cuda 12-8 and cudnn 12-8.
- Compile and run nvidia sample
simpleP2P
Simple p2p output:
[./simpleP2P] - Starting...
Checking for multiple GPUs...
CUDA-capable device count: 4
Checking GPU(s) for support of peer to peer memory access...
> Peer access from NVIDIA GeForce RTX 4090 D (GPU0) -> NVIDIA GeForce RTX 4090 D (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU0) -> NVIDIA GeForce RTX 4090 D (GPU2) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU0) -> NVIDIA GeForce RTX 4090 D (GPU3) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU1) -> NVIDIA GeForce RTX 4090 D (GPU0) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU1) -> NVIDIA GeForce RTX 4090 D (GPU2) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU1) -> NVIDIA GeForce RTX 4090 D (GPU3) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU2) -> NVIDIA GeForce RTX 4090 D (GPU0) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU2) -> NVIDIA GeForce RTX 4090 D (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU2) -> NVIDIA GeForce RTX 4090 D (GPU3) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU3) -> NVIDIA GeForce RTX 4090 D (GPU0) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU3) -> NVIDIA GeForce RTX 4090 D (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 4090 D (GPU3) -> NVIDIA GeForce RTX 4090 D (GPU2) : Yes
Enabling peer access between GPU0 and GPU1...
CUDA error at simpleP2P.cu:129 code=205(cudaErrorMapBufferObjectFailed) "cudaDeviceEnablePeerAccess(gpuid[1], 0)"
As we can see, the GPUs purport to be be capable of p2p communication, but p2p programs fall over at runtime.
A test program derived from the content of this gist exhibits the following error:
device(type='cuda', index=0)
device(type='cuda', index=1)
device(type='cuda', index=2)
device(type='cuda', index=3)
cuda:0 -> cuda:1
cuda:2 -> cuda:3
cuda:1 -> cuda:0
cuda:3 -> cuda:2
cuda:1 -> cuda:2
cuda:3 -> cuda:0
cuda:2 -> cuda:1
cuda:0 -> cuda:3
cuda:2 -> cuda:0
cuda:3 -> cuda:1
cuda:0 -> cuda:2
cuda:1 -> cuda:3
All 12 GPU transfer combinations are present
GPU cuda:0 can access: GPU cuda:1: ✓, GPU cuda:2: ✓, GPU cuda:3: ✓,
GPU cuda:1 can access: GPU cuda:0: ✓, GPU cuda:2: ✓, GPU cuda:3: ✓,
GPU cuda:2 can access: GPU cuda:0: ✓, GPU cuda:1: ✓, GPU cuda:3: ✓,
GPU cuda:3 can access: GPU cuda:0: ✓, GPU cuda:1: ✓, GPU cuda:2: ✓,
0%| | 0/12 [00:00<?, ?it/s]Testing cuda:0 to cuda:1
0%| | 0/12 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/ac/local-projects/cuda-p2p-testing/./run.py", line 76, in <module>
_ = warm_up.to(gpu2)
^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: mapping of buffer object failed
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Despite the error message, setting CUDA_LAUNCH_BLOCKING has no effect on the error.
Bug Incidence
Always
nvidia-bug-report.log.gz
More Info
I wasn't expecting these GPUs to be well behaved exactly and had already accepted that I probably wouldn't get good p2p speeds. However, if this patch can be adapted to these cards somehow, then that truly would be excellent!
I expect the firmware has been tampered with in some way to support the additional memory so that may also play a role here.
Reacted by Devin Cofer and Anton Sokolchenko
Reacted by Chihaya, showgood163 and Devin Cofer
Activity
LuosjDD commentedon Apr 7, 2025
BAR1 Memory Usage Total : 32768 MiB Used : 1 MiB Free : 32767 MiBIt looks like the BAR1 size is less than 48G.
ac101m commentedon Apr 7, 2025
Yeah, I noticed that...
The supported values only go up to 32G, probably like the base 4090D. I wonder if there is a way to override the supported values and bump it to 64G?
Ph0rk0z commentedon Apr 7, 2025
Maybe kernel modules assign bar size by card SKU and it needs another patch.
ac101m commentedon Apr 7, 2025
I'm currently poking around the driver code looking for things I might be able to hack. I did find this file that seems like it might be promising, but I haven't spent much time really looking at it yet.
ac101m commentedon Apr 7, 2025
It looks like rebar options are reported by the card through UEFI to the kernel. The driver picks the largest supported size and tries to enable that. If it fails, it tries the next lowest value repeatedly until something works. I tried modifying the driver to enter this loop one level greater than the supported value reported by
pci_rebar_get_possible_sizesbut as I should probably have expected, this did not work.I did learn elsewhere though that there are UEFI hacks to enable rebar on cards older than ampere (2000 series), without a supporting vbios. This might be worth a try if I can't get a modified vbios for these cards. Will post back here if I manage to figure anything out.
Ph0rk0z commentedon Apr 8, 2025
You can certainly try: terminatorul/NvStrapsReBar@bfa61f5
Add your GPU to the list and see if you can customize. I only used regular ReBar with 3090s.
Alternatively, ask your vendor for a vbios update if they are able. It's in their interest to fix this too.
xvim commentedon May 10, 2025
were you ultimately able to get around the 32GB BAR space limitation and successfully enable P2P on the RTX 4090D 48GB?
lambo111-x86 commentedon May 30, 2025
+1
kirbyzhou commentedon Jun 11, 2025
+1
ChiikawaSama commentedon Jul 4, 2025
+1
ac101m commentedon Jul 9, 2025
Nope, haven't looked into it much yet.
ChiikawaSama commentedon Jul 13, 2025
can we just simply force pci_rebar_get_possible_sizes to give 48GB back?
ac101m commentedon Jul 13, 2025
BAR is only adjusted in power-of-two increments, so the magic number is 64GB rather than 48, but you can certainly try! Though it's a kernel function so you'd have to build a kernel with your change.
NvStrapsRebar may also be able to do this at the bios level.
ZiliangMiao commentedon Aug 1, 2025
+1
7 remaining items
ac101m commentedon Aug 26, 2025
@aluo-x Not so fast! Another user (just above) seems to have gotten something working. @kezewang What steps did you take to get p2p working on your system?
ilovesouthpark commentedon Sep 5, 2025
all the 4090d/4090 48GB i found are not with resizebar vbios which is the precondition for p2p patch, how you can bypass this or you have one vbios supports resizebar for 4090d 48GB? Thanks for sharing.
alicepolice commentedon Sep 12, 2025
+1 still waiting for a solution，dual 4090 48G
BluffeyTest commentedon Dec 16, 2025
I tried it，it's bullshit！even cannot make！
BluffeyTest commentedon Dec 17, 2025
+1 still waiting for a solution，4 * 4090 48G
LuosjDD commentedon Dec 17, 2025
According to what I've heard, it can only be modified through vbios; the only hope is insider leaks
BluffeyTest commentedon Dec 17, 2025
how to do?
ChihayaK commentedon May 14, 2026
I manged to make it work also. But it still in very early stage. I dont know if it really works under load and what are the performance improvements.
But seems promising to me.
p2pBandwidthLatencyTest
nccl-tests
nvidia-smi
Edit: Kind of bad news is that I still did not mange to make write work >32GB without error. Still the current working siutations might limit the usable vram to just <= 32GB.
Edit2: I think to correctly handling the >32GB write needs to patch the cuda side of thing. At least that is how I partially solve it, but it is quite intrusive to exsiting software stack. And the performance, not great, at least without optimization. Dont think it is worth to dive in to this rabbit hole. And only using 32GB of vram is not acceptable for me, so I think a real solution is still somehow we get a 64GB bar vbios.
152334H commentedon May 14, 2026
@ChihayaK what is the result of
sudo lspci -vvs 85:00.0 | grep -E "Region|Resizable BAR|BAR [0-9]"on your machine?ChihayaK commentedon May 14, 2026
I dont have a special variant of this gpu. It is indeed 32GB.
iox commentedon Jun 7, 2026
@ChihayaK Can you share some additional details of what you did?
I have two 4080 super modded with 32GB vram each, but BAR1 only reports 16GB.
Ph0rk0z commentedon Jun 13, 2026
Problem is also on the 3080 20gb. Owners have reported no rebar at all.
ChihayaK commentedon Jun 15, 2026
Not possible to alter barsize. That effort was made to adapt the limiation. So a vbios mod is still needed to unlock full potential. Or only 16Gigs of vram can be use while p2p and we all know that is not the point here.
Duanyll commentedon Jul 14, 2026
This works on the modded 48G 4090s now — with one caveat worth stating up front.
With cuMem (modern NCCL / PyTorch,
NCCL_CUMEM_ENABLE=1, which is the default) only NCCL's comm buffers go into the 32G BAR1, so model/activations can use the full 48G while P2P is active. Verified on 4× and 8× 4090-48G: 38 GB resident per GPU + NCCL all-reducevia P2P/CUMEM, data correct; ~22–26 GB/s peer copy, ~20–25 GB/s all-reduce busbw (needsiommu=pt+ ACS off).The catch @ChihayaK : the legacy
cudaDeviceEnablePeerAccesspath (whatsimpleP2Puses) opens the whole device, so it's still capped at ~32 GB — that's exactly your wall, and it doesn't go away for legacy apps. Use a cuMem/NCCL workload instead.Please check my fork (based on the tinygrad/aikitoria mod): https://github.com/Duanyll/open-gpu-kernel-modules (branches
595.71.05-p2p-48g/590.48.01-p2p-48g; seedocs/48g-4090-p2p/)Raw test logs: https://github.com/Duanyll/open-gpu-kernel-modules/tree/590.48.01-p2p-48g/docs/48g-4090-p2p/hw-logs-2026-07
ChihayaK commentedon Aug 5, 2026
Yes, that i pretty much what i ended up with, the part i did not quite solve is that after some testing, i notice that in some cases the memory write will fail silently. And it casues unknown data coruptions without any error, and a lot of work is trying to check if I can at least throw an error when this kind of thing happens and not quite work with a fast solution, thats where i decided to stop. Great work tho!
Edit: typo