Repository navigation
P2P over 2 nodes (pcie over fiber) #34
Description
NVIDIA Open GPU Kernel Modules Version
Na
Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.
- I confirm that this does not happen with the proprietary driver package.
Operating System and Version
Ubuntu
Kernel Release
Na
Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.
- I am running on a stable kernel release.
Hardware: GPU
4090
Describe the bug
If I connect two nodes with two mellanox cards (pcie over fiber) and I have 4090 gpus in both servers, will the p2p communication work correctly over NCCL and RDMA?
To Reproduce
Q
Bug Incidence
Once
nvidia-bug-report.log.gz
Q
More Info
If I connect two nodes with two mellanox cards (pcie over fiber) and I have 4090 gpus in both servers, will the p2p communication work correctly over NCCL and RDMA?
Activity
chardog commentedon Mar 18, 2025
If I connect two nodes with two mellanox cards (pcie over fiber) and I have 4090 gpus in both servers, will the p2p communication work correctly over NCCL and RDMA?
zvorinji commentedon Mar 24, 2025
No. 1) NCCL will block it. But you can get around this by modifying it. 2) Once you've modified it, you will run into the issue where the NIC can't access the memory. Local protection error/bad address. I think the address needs to be translated into a BAR1 address (somehow/somewhere) for this to work. It's possible you also need to modify page sizes and modify the peermem module to incorporate a BAR1 workaround (given lack of GPUDirect RDMA support).
nwpumabian commentedon Jun 11, 2025
Is it possible to perform RDMA reads on a single 4090 card? Can gdrcopy run using RDMA on a single 4090?
zvorinji commentedon Jun 21, 2025
No.
bendchen commentedon Sep 17, 2025
I suspect this might be the reason:
Peer-to-Peer (P2P) is limited to GPUs within the same host.
NVIDIA GPU P2P allows direct access to video memory between GPUs within the same physical server via PCIe or NVLink.
P2P is not available for GPUs across servers because there is no direct physical PCIe/NVLink connection.