Skip to content

Support RTX 5090 #29

New issue
New issue
Open
Open
Support RTX 5090#29
Labels
bugSomething isn't working

Description

@hxssgaa
hxssgaa
opened on Feb 2, 2025

NVIDIA Open GPU Kernel Modules Version

565.57.01-p2p

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

  • I confirm that this does not happen with the proprietary driver package.

Operating System and Version

ubuntu 22.04

Kernel Release

NA

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

  • I am running on a stable kernel release.

Hardware: GPU

RTX 5090

Describe the bug

It would be great to enable P2P on RTX 5090 since it's supporting PCIE-5 which will be twice as fast compared to PCIE4. In fact PCIE-5 could reach the speed of first generation NVLinK. Thus supporting P2P for RTX 5090 could theoretically scale to 8 GPUs without much performance degradtion for distributed training / tensor-parallelism inference.

To Reproduce

Unable to support P2P on RTX 5090 at the moment.

Bug Incidence

Always

nvidia-bug-report.log.gz

NA

More Info

No response

Activity

  1. hxssgaa
    added
    bugSomething isn't working
    on Feb 2, 2025

  2. ilovesouthpark commented on Feb 3, 2025

    @ilovesouthpark
    ilovesouthpark
    on Feb 3, 2025

    i am also very intersting in this but first all 5090 should be available for us :) if you got one please check if large bar is still there if not it does not make any sense to try.

  3. ilovesouthpark commented on Feb 6, 2025

    @ilovesouthpark
    ilovesouthpark
    on Feb 6, 2025

    01:00.0 VGA compatible controller: NVIDIA Corporation Device 2b87 (rev a1) (prog-if 00 [VGA controller])
    Memory at 80000000 (32-bit, non-prefetchable) [size=64M]
    Memory at 4000000000 (64-bit, prefetchable) [size=32G]
    Memory at 4800000000 (64-bit, prefetchable) [size=32M]
    5090D
    Thanks for sharing from CHH user.

  4. fulloo5 commented on Mar 7, 2025

    @fulloo5
    fulloo5
    on Mar 7, 2025

    The 5090 seems to have some issues with cuda and NCCL Lib adaptation. At present, it seems to cause illegal memory access error when NCCL P2P runs.

  5. lambo111-x86 commented on Mar 7, 2025

    @lambo111-x86
    lambo111-x86
    on Mar 7, 2025

    Sat Mar 8 06:06:45 2025
    +-----------------------------------------------------------------------------------------+
    | NVIDIA-SMI 570.86.10 Driver Version: 570.86.10 CUDA Version: 12.8 |
    |-----------------------------------------+------------------------+----------------------+
    | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
    | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
    | | | MIG M. |
    |=========================================+========================+======================|
    | 0 NVIDIA GeForce RTX 5090 D Off | 00000000:8A:00.0 Off | N/A |
    | 54% 71C P1 574W / 575W | 4663MiB / 32607MiB | 100% Default |
    | | | N/A |
    +-----------------------------------------+------------------------+----------------------+
    | 1 NVIDIA GeForce RTX 5090 D Off | 00000000:C3:00.0 Off | N/A |
    | 78% 84C P1 575W / 575W | 4621MiB / 32607MiB | 100% Default |
    | | | N/A |
    +-----------------------------------------+------------------------+----------------------+

    +-----------------------------------------------------------------------------------------+
    | Processes: |
    | GPU GI CI PID Type Process name GPU Memory |
    | ID ID Usage |
    |=========================================================================================|
    | 0 N/A N/A 2334 G /usr/lib/xorg/Xorg 4MiB |
    | 0 N/A N/A 3843 G ./GUI.for.SingBox 6MiB |
    | 0 N/A N/A 3861 G ...bkit2gtk-4.1/WebKitWebProcess 16MiB |
    | 0 N/A N/A 104924 C ...quai/rigel-1.21.0-linux/rigel 4598MiB |
    | 1 N/A N/A 2334 G /usr/lib/xorg/Xorg 4MiB |
    | 1 N/A N/A 104924 C ...quai/rigel-1.21.0-linux/rigel 4598MiB |
    +-----------------------------------------------------------------------------------------+
    root@lam-MU72-SU0-00:~# lspci -vvvs c3:00.0|grep -i bar
    Vector table: BAR=0 offset=00b90000
    PBA: BAR=0 offset=00ba0000
    Capabilities: [134 v1] Physical Resizable BAR
    BAR 1: current size: 256MB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
    Capabilities: [140 v1] Virtual Resizable BAR
    BAR 2: current size: 256MB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB 256TB 512TB 1PB 2PB 4PB 8PB 16PB 32PB 64PB 128PB 256PB 512PB 1EB 2EB 4EB 8EB

    I do got 2pcs of 5090D and if anyone can make it P2P~

  6. LuosjDD commented on Mar 14, 2025

    @LuosjDD
    LuosjDD
    on Mar 14, 2025

    The 5090 seems to have some issues with cuda and NCCL Lib adaptation. At present, it seems to cause illegal memory access error when NCCL P2P runs.

    I updated the cuda version to 12.8.1, but the simpleP2P and p2pBandwidthLatencyTest still have problems.

  7. huanzhang12 commented on Mar 31, 2025

    @huanzhang12
    huanzhang12
    on Mar 31, 2025

    The problem is not about CUDA and NCCL. The driver needs to be modified to fully unlock P2P capability in RTX 5090.

    I made a quick fix to the driver to enable P2P support on RTX 5090. In gmmuFieldSetAperture, I change the aperture value to GMMU_APERTURE_SYS_NONCOH when it is set to GMMU_APERTURE_PEER. In nv_ioremap_wc, I force-disabled writing combining to remove a kernel warning about PAT, but this may not be necessary. More investigation is needed to make the patch better, but at least we have something working now.

    diff --git a/kernel-open/common/inc/nv-linux.h b/kernel-open/common/inc/nv-linux.h
index e9daf8a9..fa60e5d4 100644
--- a/kernel-open/common/inc/nv-linux.h
+++ b/kernel-open/common/inc/nv-linux.h
@@ -531,6 +531,7 @@ static inline void *nv_ioremap_cache(NvU64 phys, NvU64 size)
 static inline void *nv_ioremap_wc(NvU64 phys, NvU64 size)
 {
     void *ptr = NULL;
+    return nv_ioremap_nocache(phys, size);
 #if IS_ENABLED(CONFIG_INTEL_TDX_GUEST) && defined(NV_IOREMAP_DRIVER_HARDENED_WC_PRESENT)
     ptr = ioremap_driver_hardened_wc(phys, size);
 #elif defined(NV_IOREMAP_WC_PRESENT)
diff --git a/src/nvidia/inc/libraries/mmu/gmmu_fmt.h b/src/nvidia/inc/libraries/mmu/gmmu_fmt.h
index 31ec4249..581dc897 100644
--- a/src/nvidia/inc/libraries/mmu/gmmu_fmt.h
+++ b/src/nvidia/inc/libraries/mmu/gmmu_fmt.h
@@ -342,7 +342,14 @@ gmmuFieldSetAperture
     NvU8                      *pMem
 )
 {
-    nvFieldSetEnum(&pAperture->_enum, value, pMem);
+    GMMU_APERTURE new_value;
+    if (value == GMMU_APERTURE_PEER) {
+        new_value = GMMU_APERTURE_SYS_NONCOH;
+    }
+    else {
+        new_value = value;
+    }
+    nvFieldSetEnum(&pAperture->_enum, new_value, pMem);
 }
 
 /*!

    The patch was applied to open driver 570.133.07 (you still need to apply the patch by tinygrad first). In addition, as usually required by P2P, disable IOMMU and PCIe ACS in your BIOS.

    Some quick tests:

    $ ./p2pBandwidthLatencyTest
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 5090, pciBusID: 1, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 5090, pciBusID: 41, pciDeviceID: 0, pciDomainID:0
Device: 2, NVIDIA GeForce RTX 5090, pciBusID: 81, pciDeviceID: 0, pciDomainID:0
Device: 3, NVIDIA GeForce RTX 5090, pciBusID: c1, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CAN Access Peer Device=2
Device=0 CAN Access Peer Device=3
Device=1 CAN Access Peer Device=0
Device=1 CAN Access Peer Device=2
Device=1 CAN Access Peer Device=3
Device=2 CAN Access Peer Device=0
Device=2 CAN Access Peer Device=1
Device=2 CAN Access Peer Device=3
Device=3 CAN Access Peer Device=0
Device=3 CAN Access Peer Device=1
Device=3 CAN Access Peer Device=2

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2     3
     0	     1     1     1     1
     1	     1     1     1     1
     2	     1     1     1     1
     3	     1     1     1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 1516.99  31.81  31.58  32.20 
     1  32.05 1548.61  32.22  32.30 
     2  32.14  32.41 1550.10  32.40 
     3  32.03  31.76  31.70 1547.03 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2      3 
     0 1515.52  56.46  56.44  56.44 
     1  56.46 1536.38  56.47  56.47 
     2  56.46  56.47 1540.97  56.44 
     3  56.46  56.44  56.44 1539.46 
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 1527.32  33.98  33.93  33.96 
     1  33.97 1540.88  34.25  33.98 
     2  33.96  34.27 1537.85  34.08 
     3  34.07  34.15  34.01 1545.43 
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 1527.32 111.38 111.31 111.39 
     1 111.38 1537.07 111.39 111.39 
     2 111.39 111.34 1535.58 111.39 
     3 111.39 111.39 111.39 1535.56 
P2P=Disabled Latency Matrix (us)
   GPU     0      1      2      3 
     0   2.08  12.63  12.69  12.61 
     1  12.60   2.07  12.68  12.64 
     2  12.60  12.56   2.08  12.72 
     3  12.73  12.74  12.65   2.09 

   CPU     0      1      2      3 
     0   2.16   5.65   5.58   5.58 
     1   5.57   1.95   5.53   5.48 
     2   5.47   5.43   1.95   5.49 
     3   5.47   5.41   5.48   1.96 
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2      3 
     0   2.10   0.51   0.51   0.49 
     1   0.53   2.08   0.59   0.58 
     2   0.52   0.52   2.08   0.51 
     3   0.59   0.49   0.50   2.07 

   CPU     0      1      2      3 
     0   2.11   1.55   1.55   1.57 
     1   1.61   2.01   1.56   1.61 
     2   1.61   1.57   2.01   1.55 
     3   1.61   1.56   1.62   2.01 

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

    NCCL 2.26.2-1:

    $ NCCL_P2P_LEVEL=SYS ./all_reduce_perf -b 64M -e 8G -f 2 -g 4
# nThread 1 nGpus 4 minBytes 67108864 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   3743 on huan-genoa-server device  0 [0000:01:00] NVIDIA GeForce RTX 5090
#  Rank  1 Group  0 Pid   3743 on huan-genoa-server device  1 [0000:41:00] NVIDIA GeForce RTX 5090
#  Rank  2 Group  0 Pid   3743 on huan-genoa-server device  2 [0000:81:00] NVIDIA GeForce RTX 5090
#  Rank  3 Group  0 Pid   3743 on huan-genoa-server device  3 [0000:c1:00] NVIDIA GeForce RTX 5090
#
#                                                              out-of-place                       in-place          
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       
    67108864      16777216     float     sum      -1   2209.2   30.38   45.57      0   2165.7   30.99   46.48      0
   134217728      33554432     float     sum      -1   4387.4   30.59   45.89      0   4290.7   31.28   46.92      0
   268435456      67108864     float     sum      -1   8778.9   30.58   45.87      0   8622.8   31.13   46.70      0
   536870912     134217728     float     sum      -1    17471   30.73   46.09      0    17227   31.16   46.75      0
  1073741824     268435456     float     sum      -1    34876   30.79   46.18      0    34448   31.17   46.75      0
  2147483648     536870912     float     sum      -1    69601   30.85   46.28      0    68874   31.18   46.77      0
  4294967296    1073741824     float     sum      -1   139182   30.86   46.29      0   137664   31.20   46.80      0
  8589934592    2147483648     float     sum      -1   278357   30.86   46.29      0   275357   31.20   46.79      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 46.4009 
#

  8. legezywzh commented on Apr 29, 2025

    @legezywzh
    legezywzh
    on Apr 29, 2025

    The problem is not about CUDA and NCCL. The driver needs to be modified to fully unlock P2P capability in RTX 5090.

    I made a quick fix to the driver to enable P2P support on RTX 5090. In gmmuFieldSetAperture, I change the aperture value to GMMU_APERTURE_SYS_NONCOH when it is set to GMMU_APERTURE_PEER. In nv_ioremap_wc, I force-disabled writing combining to remove a kernel warning about PAT, but this may not be necessary. More investigation is needed to make the patch better, but at least we have something working now.

    diff --git a/kernel-open/common/inc/nv-linux.h b/kernel-open/common/inc/nv-linux.h
    index e9daf8a9..fa60e5d4 100644
    --- a/kernel-open/common/inc/nv-linux.h
    +++ b/kernel-open/common/inc/nv-linux.h
    @@ -531,6 +531,7 @@ static inline void *nv_ioremap_cache(NvU64 phys, NvU64 size)
    static inline void *nv_ioremap_wc(NvU64 phys, NvU64 size)
    {
    void *ptr = NULL;

    • return nv_ioremap_nocache(phys, size);
      #if IS_ENABLED(CONFIG_INTEL_TDX_GUEST) && defined(NV_IOREMAP_DRIVER_HARDENED_WC_PRESENT)
      ptr = ioremap_driver_hardened_wc(phys, size);
      #elif defined(NV_IOREMAP_WC_PRESENT)
      diff --git a/src/nvidia/inc/libraries/mmu/gmmu_fmt.h b/src/nvidia/inc/libraries/mmu/gmmu_fmt.h
      index 31ec4249..581dc897 100644
      --- a/src/nvidia/inc/libraries/mmu/gmmu_fmt.h
      +++ b/src/nvidia/inc/libraries/mmu/gmmu_fmt.h
      @@ -342,7 +342,14 @@ gmmuFieldSetAperture
      NvU8 *pMem
      )
      {
    • nvFieldSetEnum(&pAperture->_enum, value, pMem);
    • GMMU_APERTURE new_value;
    • if (value == GMMU_APERTURE_PEER) {
    •    new_value = GMMU_APERTURE_SYS_NONCOH;
    • }
    • else {
    •    new_value = value;
    • }
    • nvFieldSetEnum(&pAperture->_enum, new_value, pMem);
      }

    /*!
    The patch was applied to open driver 570.133.07 (you still need to apply the patch by tinygrad first). In addition, as usually required by P2P, disable IOMMU and PCIe ACS in your BIOS.

    Some quick tests:

    $ ./p2pBandwidthLatencyTest
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 5090, pciBusID: 1, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 5090, pciBusID: 41, pciDeviceID: 0, pciDomainID:0
Device: 2, NVIDIA GeForce RTX 5090, pciBusID: 81, pciDeviceID: 0, pciDomainID:0
Device: 3, NVIDIA GeForce RTX 5090, pciBusID: c1, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CAN Access Peer Device=2
Device=0 CAN Access Peer Device=3
Device=1 CAN Access Peer Device=0
Device=1 CAN Access Peer Device=2
Device=1 CAN Access Peer Device=3
Device=2 CAN Access Peer Device=0
Device=2 CAN Access Peer Device=1
Device=2 CAN Access Peer Device=3
Device=3 CAN Access Peer Device=0
Device=3 CAN Access Peer Device=1
Device=3 CAN Access Peer Device=2

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2     3
     0	     1     1     1     1
     1	     1     1     1     1
     2	     1     1     1     1
     3	     1     1     1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 1516.99  31.81  31.58  32.20 
     1  32.05 1548.61  32.22  32.30 
     2  32.14  32.41 1550.10  32.40 
     3  32.03  31.76  31.70 1547.03 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2      3 
     0 1515.52  56.46  56.44  56.44 
     1  56.46 1536.38  56.47  56.47 
     2  56.46  56.47 1540.97  56.44 
     3  56.46  56.44  56.44 1539.46 
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 1527.32  33.98  33.93  33.96 
     1  33.97 1540.88  34.25  33.98 
     2  33.96  34.27 1537.85  34.08 
     3  34.07  34.15  34.01 1545.43 
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 1527.32 111.38 111.31 111.39 
     1 111.38 1537.07 111.39 111.39 
     2 111.39 111.34 1535.58 111.39 
     3 111.39 111.39 111.39 1535.56 
P2P=Disabled Latency Matrix (us)
   GPU     0      1      2      3 
     0   2.08  12.63  12.69  12.61 
     1  12.60   2.07  12.68  12.64 
     2  12.60  12.56   2.08  12.72 
     3  12.73  12.74  12.65   2.09 

   CPU     0      1      2      3 
     0   2.16   5.65   5.58   5.58 
     1   5.57   1.95   5.53   5.48 
     2   5.47   5.43   1.95   5.49 
     3   5.47   5.41   5.48   1.96 
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2      3 
     0   2.10   0.51   0.51   0.49 
     1   0.53   2.08   0.59   0.58 
     2   0.52   0.52   2.08   0.51 
     3   0.59   0.49   0.50   2.07 

   CPU     0      1      2      3 
     0   2.11   1.55   1.55   1.57 
     1   1.61   2.01   1.56   1.61 
     2   1.61   1.57   2.01   1.55 
     3   1.61   1.56   1.62   2.01 

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

    NCCL 2.26.2-1:

    $ NCCL_P2P_LEVEL=SYS ./all_reduce_perf -b 64M -e 8G -f 2 -g 4
# nThread 1 nGpus 4 minBytes 67108864 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   3743 on huan-genoa-server device  0 [0000:01:00] NVIDIA GeForce RTX 5090
#  Rank  1 Group  0 Pid   3743 on huan-genoa-server device  1 [0000:41:00] NVIDIA GeForce RTX 5090
#  Rank  2 Group  0 Pid   3743 on huan-genoa-server device  2 [0000:81:00] NVIDIA GeForce RTX 5090
#  Rank  3 Group  0 Pid   3743 on huan-genoa-server device  3 [0000:c1:00] NVIDIA GeForce RTX 5090
#
#                                                              out-of-place                       in-place          
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       
    67108864      16777216     float     sum      -1   2209.2   30.38   45.57      0   2165.7   30.99   46.48      0
   134217728      33554432     float     sum      -1   4387.4   30.59   45.89      0   4290.7   31.28   46.92      0
   268435456      67108864     float     sum      -1   8778.9   30.58   45.87      0   8622.8   31.13   46.70      0
   536870912     134217728     float     sum      -1    17471   30.73   46.09      0    17227   31.16   46.75      0
  1073741824     268435456     float     sum      -1    34876   30.79   46.18      0    34448   31.17   46.75      0
  2147483648     536870912     float     sum      -1    69601   30.85   46.28      0    68874   31.18   46.77      0
  4294967296    1073741824     float     sum      -1   139182   30.86   46.29      0   137664   31.20   46.80      0
  8589934592    2147483648     float     sum      -1   278357   30.86   46.29      0   275357   31.20   46.79      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 46.4009 
#

    @huanzhang12 while you were testing simpleP2P, did you see any error messages in dmesg like below:
    [4115826.157667] NVRM: iovaspaceDestruct_IMPL: 4 left-over mappings in IOVAS 0x3e00
    [4115826.254141] NVRM: nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace.c:592
    [4115826.254149] NVRM: nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace.c:601
    [4115826.254315] NVRM: nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace.c:592
    [4115826.254318] NVRM: nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace.c:601
    [4115826.255854] NVRM: nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace.c:592
    [4115826.255857] NVRM: nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace.c:601
    [4115826.255860] NVRM: nvAssertFailedNoLog: Assertion failed: Sysmemdesc outlived its attached pGpu @ mem_desc.c:1509
    [4115826.255932] NVRM: nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace.c:592
    [4115826.255935] NVRM: nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace.c:601

  9. huanzhang12 commented on Apr 29, 2025

    @huanzhang12
    huanzhang12
    on Apr 29, 2025

    @legezywzh I saw that error message with 575 driver. The driver structure changed and a new patch would be needed.
    I am currently using 570.133.20 where the patch should still work.

  10. legezywzh commented on Apr 29, 2025

    @legezywzh
    legezywzh
    on Apr 29, 2025

    @legezywzh I saw that error message with 575 driver. The driver structure changed and a new patch would be needed. I am currently using 570.133.20 where the patch should still work.

    @huanzhang12 Thanks for the quick reply. So with 570.133.20, you don't see these error messages?

  11. huanzhang12 commented on Apr 30, 2025

    @huanzhang12
    huanzhang12
    on Apr 30, 2025

    @huanzhang12 Thanks for the quick reply. So with 570.133.20, you don't see these error messages?

    No errors on the 570 branch. I upgraded to 570.144 and it worked fine too.

  12. xxxzsgxxx commented on May 6, 2025

    @xxxzsgxxx
    xxxzsgxxx
    on May 6, 2025

    The problem is not about CUDA and NCCL. The driver needs to be modified to fully unlock P2P capability in RTX 5090.

    I made a quick fix to the driver to enable P2P support on RTX 5090. In gmmuFieldSetAperture, I change the aperture value to GMMU_APERTURE_SYS_NONCOH when it is set to GMMU_APERTURE_PEER. In nv_ioremap_wc, I force-disabled writing combining to remove a kernel warning about PAT, but this may not be necessary. More investigation is needed to make the patch better, but at least we have something working now.

    diff --git a/kernel-open/common/inc/nv-linux.h b/kernel-open/common/inc/nv-linux.h
    index e9daf8a9..fa60e5d4 100644
    --- a/kernel-open/common/inc/nv-linux.h
    +++ b/kernel-open/common/inc/nv-linux.h
    @@ -531,6 +531,7 @@ static inline void *nv_ioremap_cache(NvU64 phys, NvU64 size)
    static inline void *nv_ioremap_wc(NvU64 phys, NvU64 size)
    {
    void *ptr = NULL;

    • return nv_ioremap_nocache(phys, size);
      #if IS_ENABLED(CONFIG_INTEL_TDX_GUEST) && defined(NV_IOREMAP_DRIVER_HARDENED_WC_PRESENT)
      ptr = ioremap_driver_hardened_wc(phys, size);
      #elif defined(NV_IOREMAP_WC_PRESENT)
      diff --git a/src/nvidia/inc/libraries/mmu/gmmu_fmt.h b/src/nvidia/inc/libraries/mmu/gmmu_fmt.h
      index 31ec4249..581dc897 100644
      --- a/src/nvidia/inc/libraries/mmu/gmmu_fmt.h
      +++ b/src/nvidia/inc/libraries/mmu/gmmu_fmt.h
      @@ -342,7 +342,14 @@ gmmuFieldSetAperture
      NvU8 *pMem
      )
      {
    • nvFieldSetEnum(&pAperture->_enum, value, pMem);
    • GMMU_APERTURE new_value;
    • if (value == GMMU_APERTURE_PEER) {
    •    new_value = GMMU_APERTURE_SYS_NONCOH;
    • }
    • else {
    •    new_value = value;
    • }
    • nvFieldSetEnum(&pAperture->_enum, new_value, pMem);
      }

    Thanks for your patch !!

    I am using the patched driver 570.133.07 on an AMD Turin CPU with RTX 5090. When I disable the Resizable BAR in the BIOS, turn off IOMMU, and disable PCIe ACS, I can see P2P working successfully. However, when running some tests, CUDA errors occur.

    Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
Cuda failure p2pBandwidthLatencyTest.cu:189: 'mapping of buffer object failed'

    But when I enable Resizable BAR again, keeping all other settings the same as before, the nvidia-smi tool fails to recognize the device, showing "No devices were found." However, when checking with lspci -vv, the NVIDIA device is recognized by the system.

    Is the Re-Sizise BAR enabled inecessary?

    Any help would be greatly appreciated!

  13. lambo111-x86 commented on May 30, 2025

    @lambo111-x86
    lambo111-x86
    on May 30, 2025

    @huanzhang12 Thanks for the quick reply. So with 570.133.20, you don't see these error messages?

    No errors on the 570 branch. I upgraded to 570.144 and it worked fine too.

    Hi Docter,may u got some idea about P2P working on 4090 48G?I trid,but just failed..seems like the bar1 size just can be set like 32G,But 4090 48G may need bar 1 size up to 48G?

  14. Panchovix commented on Jun 14, 2025

    @Panchovix
    Panchovix
    on Jun 14, 2025

    @huanzhang12 I just tried applying the patch to 570.X but I get

    pancho@fedora:~/cuda-samples/build/Samples/0_Introduction/simpleP2P$ ./simpleP2P 
[./simpleP2P] - Starting...
Checking for multiple GPUs...
CUDA-capable device count: 2

Checking GPU(s) for support of peer to peer memory access...
> Peer access from NVIDIA GeForce RTX 5090 (GPU0) -> NVIDIA GeForce RTX 5090 (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU1) -> NVIDIA GeForce RTX 5090 (GPU0) : Yes
Enabling peer access between GPU0 and GPU1...
CUDA error at /home/pancho/cuda-samples/Samples/0_Introduction/simpleP2P/simpleP2P.cu:130 code=205(cudaErrorMapBufferObjectFailed) "cudaDeviceEnablePeerAccess(gpuid[1], 0)" 

    pancho@fedora:~/cuda-samples/build/Samples/5_Domain_Specific/p2pBandwidthLatencyTest$ ./p2pBandwidthLatencyTest 
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 5090, pciBusID: 1, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 5090, pciBusID: 3, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1
     0       1     1
     1       1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 1732.32  24.64 
     1  24.66 1765.60 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
Cuda failure /home/pancho/cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest.cu:192: 'mapping of buffer object failed'

    I have other GPUs in the system (4090, 3090, A6000) and P2P between those work fine.

    Here are the modified files, if they help (remaned to .txt)

    /open-gpu-kernel-modules-570.153-5090/kernel-open/common/inc/nv-linux.h

    nv-linux.txt

    /open-gpu-kernel-modules-570.153-5090/src/nvidia/inc/libraries/mmu/gmmu_fmt.h

    gmmu_fmt.txt

  15. nwpumabian commented on Jun 27, 2025

    @nwpumabian
    nwpumabian
    on Jun 27, 2025

    Is it possible to perform RDMA reads on a single 5090 card? Can gdrcopy run using RDMA on a single 5090?

  16. lnykww commented on Jul 17, 2025

    @lnykww
    lnykww
    on Jul 17, 2025

    The problem is not about CUDA and NCCL. The driver needs to be modified to fully unlock P2P capability in RTX 5090.

    I made a quick fix to the driver to enable P2P support on RTX 5090. In gmmuFieldSetAperture, I change the aperture value to GMMU_APERTURE_SYS_NONCOH when it is set to GMMU_APERTURE_PEER. In nv_ioremap_wc, I force-disabled writing combining to remove a kernel warning about PAT, but this may not be necessary. More investigation is needed to make the patch better, but at least we have something working now.

    diff --git a/kernel-open/common/inc/nv-linux.h b/kernel-open/common/inc/nv-linux.h
    index e9daf8a9..fa60e5d4 100644
    --- a/kernel-open/common/inc/nv-linux.h
    +++ b/kernel-open/common/inc/nv-linux.h
    @@ -531,6 +531,7 @@ static inline void *nv_ioremap_cache(NvU64 phys, NvU64 size)
    static inline void *nv_ioremap_wc(NvU64 phys, NvU64 size)
    {
    void *ptr = NULL;

    • return nv_ioremap_nocache(phys, size);
      #if IS_ENABLED(CONFIG_INTEL_TDX_GUEST) && defined(NV_IOREMAP_DRIVER_HARDENED_WC_PRESENT)
      ptr = ioremap_driver_hardened_wc(phys, size);
      #elif defined(NV_IOREMAP_WC_PRESENT)
      diff --git a/src/nvidia/inc/libraries/mmu/gmmu_fmt.h b/src/nvidia/inc/libraries/mmu/gmmu_fmt.h
      index 31ec4249..581dc897 100644
      --- a/src/nvidia/inc/libraries/mmu/gmmu_fmt.h
      +++ b/src/nvidia/inc/libraries/mmu/gmmu_fmt.h
      @@ -342,7 +342,14 @@ gmmuFieldSetAperture
      NvU8 *pMem
      )
      {
    • nvFieldSetEnum(&pAperture->_enum, value, pMem);
    • GMMU_APERTURE new_value;
    • if (value == GMMU_APERTURE_PEER) {
    •    new_value = GMMU_APERTURE_SYS_NONCOH;
    • }
    • else {
    •    new_value = value;
    • }
    • nvFieldSetEnum(&pAperture->_enum, new_value, pMem);
      }

    /*!
    The patch was applied to open driver 570.133.07 (you still need to apply the patch by tinygrad first). In addition, as usually required by P2P, disable IOMMU and PCIe ACS in your BIOS.

    Some quick tests:

    $ ./p2pBandwidthLatencyTest
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 5090, pciBusID: 1, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 5090, pciBusID: 41, pciDeviceID: 0, pciDomainID:0
Device: 2, NVIDIA GeForce RTX 5090, pciBusID: 81, pciDeviceID: 0, pciDomainID:0
Device: 3, NVIDIA GeForce RTX 5090, pciBusID: c1, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CAN Access Peer Device=2
Device=0 CAN Access Peer Device=3
Device=1 CAN Access Peer Device=0
Device=1 CAN Access Peer Device=2
Device=1 CAN Access Peer Device=3
Device=2 CAN Access Peer Device=0
Device=2 CAN Access Peer Device=1
Device=2 CAN Access Peer Device=3
Device=3 CAN Access Peer Device=0
Device=3 CAN Access Peer Device=1
Device=3 CAN Access Peer Device=2

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2     3
     0	     1     1     1     1
     1	     1     1     1     1
     2	     1     1     1     1
     3	     1     1     1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 1516.99  31.81  31.58  32.20 
     1  32.05 1548.61  32.22  32.30 
     2  32.14  32.41 1550.10  32.40 
     3  32.03  31.76  31.70 1547.03 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2      3 
     0 1515.52  56.46  56.44  56.44 
     1  56.46 1536.38  56.47  56.47 
     2  56.46  56.47 1540.97  56.44 
     3  56.46  56.44  56.44 1539.46 
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 1527.32  33.98  33.93  33.96 
     1  33.97 1540.88  34.25  33.98 
     2  33.96  34.27 1537.85  34.08 
     3  34.07  34.15  34.01 1545.43 
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 1527.32 111.38 111.31 111.39 
     1 111.38 1537.07 111.39 111.39 
     2 111.39 111.34 1535.58 111.39 
     3 111.39 111.39 111.39 1535.56 
P2P=Disabled Latency Matrix (us)
   GPU     0      1      2      3 
     0   2.08  12.63  12.69  12.61 
     1  12.60   2.07  12.68  12.64 
     2  12.60  12.56   2.08  12.72 
     3  12.73  12.74  12.65   2.09 

   CPU     0      1      2      3 
     0   2.16   5.65   5.58   5.58 
     1   5.57   1.95   5.53   5.48 
     2   5.47   5.43   1.95   5.49 
     3   5.47   5.41   5.48   1.96 
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2      3 
     0   2.10   0.51   0.51   0.49 
     1   0.53   2.08   0.59   0.58 
     2   0.52   0.52   2.08   0.51 
     3   0.59   0.49   0.50   2.07 

   CPU     0      1      2      3 
     0   2.11   1.55   1.55   1.57 
     1   1.61   2.01   1.56   1.61 
     2   1.61   1.57   2.01   1.55 
     3   1.61   1.56   1.62   2.01 

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

    NCCL 2.26.2-1:

    $ NCCL_P2P_LEVEL=SYS ./all_reduce_perf -b 64M -e 8G -f 2 -g 4
# nThread 1 nGpus 4 minBytes 67108864 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   3743 on huan-genoa-server device  0 [0000:01:00] NVIDIA GeForce RTX 5090
#  Rank  1 Group  0 Pid   3743 on huan-genoa-server device  1 [0000:41:00] NVIDIA GeForce RTX 5090
#  Rank  2 Group  0 Pid   3743 on huan-genoa-server device  2 [0000:81:00] NVIDIA GeForce RTX 5090
#  Rank  3 Group  0 Pid   3743 on huan-genoa-server device  3 [0000:c1:00] NVIDIA GeForce RTX 5090
#
#                                                              out-of-place                       in-place          
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       
    67108864      16777216     float     sum      -1   2209.2   30.38   45.57      0   2165.7   30.99   46.48      0
   134217728      33554432     float     sum      -1   4387.4   30.59   45.89      0   4290.7   31.28   46.92      0
   268435456      67108864     float     sum      -1   8778.9   30.58   45.87      0   8622.8   31.13   46.70      0
   536870912     134217728     float     sum      -1    17471   30.73   46.09      0    17227   31.16   46.75      0
  1073741824     268435456     float     sum      -1    34876   30.79   46.18      0    34448   31.17   46.75      0
  2147483648     536870912     float     sum      -1    69601   30.85   46.28      0    68874   31.18   46.77      0
  4294967296    1073741824     float     sum      -1   139182   30.86   46.29      0   137664   31.20   46.80      0
  8589934592    2147483648     float     sum      -1   278357   30.86   46.29      0   275357   31.20   46.79      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 46.4009 
#

    @huanzhang12
    What model of CPU was used to generate this test result? Does it include a PCIe switch?

    TY!

  17. huanzhang12 commented on Jul 17, 2025

    @huanzhang12
    huanzhang12
    on Jul 17, 2025

    @lnykww AMD EPYC 9534, no PCIe switch, all GPUs directly connected to the CPU via PCIe 5.0.

  18. JzhJay commented on Jul 24, 2025

    @JzhJay
    JzhJay
    on Jul 24, 2025

    The problem is not about CUDA and NCCL. The driver needs to be modified to fully unlock P2P capability in RTX 5090.

    I made a quick fix to the driver to enable P2P support on RTX 5090. In gmmuFieldSetAperture, I change the aperture value to GMMU_APERTURE_SYS_NONCOH when it is set to GMMU_APERTURE_PEER. In nv_ioremap_wc, I force-disabled writing combining to remove a kernel warning about PAT, but this may not be necessary. More investigation is needed to make the patch better, but at least we have something working now.

    diff --git a/kernel-open/common/inc/nv-linux.h b/kernel-open/common/inc/nv-linux.h
    index e9daf8a9..fa60e5d4 100644
    --- a/kernel-open/common/inc/nv-linux.h
    +++ b/kernel-open/common/inc/nv-linux.h
    @@ -531,6 +531,7 @@ static inline void *nv_ioremap_cache(NvU64 phys, NvU64 size)
    static inline void *nv_ioremap_wc(NvU64 phys, NvU64 size)
    {
    void *ptr = NULL;

    • return nv_ioremap_nocache(phys, size);
      #if IS_ENABLED(CONFIG_INTEL_TDX_GUEST) && defined(NV_IOREMAP_DRIVER_HARDENED_WC_PRESENT)
      ptr = ioremap_driver_hardened_wc(phys, size);
      #elif defined(NV_IOREMAP_WC_PRESENT)
      diff --git a/src/nvidia/inc/libraries/mmu/gmmu_fmt.h b/src/nvidia/inc/libraries/mmu/gmmu_fmt.h
      index 31ec4249..581dc897 100644
      --- a/src/nvidia/inc/libraries/mmu/gmmu_fmt.h
      +++ b/src/nvidia/inc/libraries/mmu/gmmu_fmt.h
      @@ -342,7 +342,14 @@ gmmuFieldSetAperture
      NvU8 *pMem
      )
      {
    • nvFieldSetEnum(&pAperture->_enum, value, pMem);
    • GMMU_APERTURE new_value;
    • if (value == GMMU_APERTURE_PEER) {
    •    new_value = GMMU_APERTURE_SYS_NONCOH;
    • }
    • else {
    •    new_value = value;
    • }
    • nvFieldSetEnum(&pAperture->_enum, new_value, pMem);
      }

    /*!
    The patch was applied to open driver 570.133.07 (you still need to apply the patch by tinygrad first). In addition, as usually required by P2P, disable IOMMU and PCIe ACS in your BIOS.

    Some quick tests:

    $ ./p2pBandwidthLatencyTest
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 5090, pciBusID: 1, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 5090, pciBusID: 41, pciDeviceID: 0, pciDomainID:0
Device: 2, NVIDIA GeForce RTX 5090, pciBusID: 81, pciDeviceID: 0, pciDomainID:0
Device: 3, NVIDIA GeForce RTX 5090, pciBusID: c1, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CAN Access Peer Device=2
Device=0 CAN Access Peer Device=3
Device=1 CAN Access Peer Device=0
Device=1 CAN Access Peer Device=2
Device=1 CAN Access Peer Device=3
Device=2 CAN Access Peer Device=0
Device=2 CAN Access Peer Device=1
Device=2 CAN Access Peer Device=3
Device=3 CAN Access Peer Device=0
Device=3 CAN Access Peer Device=1
Device=3 CAN Access Peer Device=2

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2     3
     0	     1     1     1     1
     1	     1     1     1     1
     2	     1     1     1     1
     3	     1     1     1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 1516.99  31.81  31.58  32.20 
     1  32.05 1548.61  32.22  32.30 
     2  32.14  32.41 1550.10  32.40 
     3  32.03  31.76  31.70 1547.03 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2      3 
     0 1515.52  56.46  56.44  56.44 
     1  56.46 1536.38  56.47  56.47 
     2  56.46  56.47 1540.97  56.44 
     3  56.46  56.44  56.44 1539.46 
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 1527.32  33.98  33.93  33.96 
     1  33.97 1540.88  34.25  33.98 
     2  33.96  34.27 1537.85  34.08 
     3  34.07  34.15  34.01 1545.43 
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 1527.32 111.38 111.31 111.39 
     1 111.38 1537.07 111.39 111.39 
     2 111.39 111.34 1535.58 111.39 
     3 111.39 111.39 111.39 1535.56 
P2P=Disabled Latency Matrix (us)
   GPU     0      1      2      3 
     0   2.08  12.63  12.69  12.61 
     1  12.60   2.07  12.68  12.64 
     2  12.60  12.56   2.08  12.72 
     3  12.73  12.74  12.65   2.09 

   CPU     0      1      2      3 
     0   2.16   5.65   5.58   5.58 
     1   5.57   1.95   5.53   5.48 
     2   5.47   5.43   1.95   5.49 
     3   5.47   5.41   5.48   1.96 
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2      3 
     0   2.10   0.51   0.51   0.49 
     1   0.53   2.08   0.59   0.58 
     2   0.52   0.52   2.08   0.51 
     3   0.59   0.49   0.50   2.07 

   CPU     0      1      2      3 
     0   2.11   1.55   1.55   1.57 
     1   1.61   2.01   1.56   1.61 
     2   1.61   1.57   2.01   1.55 
     3   1.61   1.56   1.62   2.01 

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

    NCCL 2.26.2-1:

    $ NCCL_P2P_LEVEL=SYS ./all_reduce_perf -b 64M -e 8G -f 2 -g 4
# nThread 1 nGpus 4 minBytes 67108864 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   3743 on huan-genoa-server device  0 [0000:01:00] NVIDIA GeForce RTX 5090
#  Rank  1 Group  0 Pid   3743 on huan-genoa-server device  1 [0000:41:00] NVIDIA GeForce RTX 5090
#  Rank  2 Group  0 Pid   3743 on huan-genoa-server device  2 [0000:81:00] NVIDIA GeForce RTX 5090
#  Rank  3 Group  0 Pid   3743 on huan-genoa-server device  3 [0000:c1:00] NVIDIA GeForce RTX 5090
#
#                                                              out-of-place                       in-place          
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       
    67108864      16777216     float     sum      -1   2209.2   30.38   45.57      0   2165.7   30.99   46.48      0
   134217728      33554432     float     sum      -1   4387.4   30.59   45.89      0   4290.7   31.28   46.92      0
   268435456      67108864     float     sum      -1   8778.9   30.58   45.87      0   8622.8   31.13   46.70      0
   536870912     134217728     float     sum      -1    17471   30.73   46.09      0    17227   31.16   46.75      0
  1073741824     268435456     float     sum      -1    34876   30.79   46.18      0    34448   31.17   46.75      0
  2147483648     536870912     float     sum      -1    69601   30.85   46.28      0    68874   31.18   46.77      0
  4294967296    1073741824     float     sum      -1   139182   30.86   46.29      0   137664   31.20   46.80      0
  8589934592    2147483648     float     sum      -1   278357   30.86   46.29      0   275357   31.20   46.79      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 46.4009 
#

    @huanzhang12 Hi doctor, I also managed to hack 5090 p2p, but when I ran all reduce performance test on all gpus, I got the following error:
    `test@test:~/nccl-tests/build$ NCCL_DEBUG=INFO NCCL_P2P_LEVEL=SYS NCCL_NVLS_ENABLE=0 ./all_reduce_perf -b 8 -e 8192M -f 2 -g 8

    nThread 1 nGpus 8 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0

    Using devices

    Rank 0 Group 0 Pid 259034 on test device 0 [0000:01:00] NVIDIA GeForce RTX 5090

    Rank 1 Group 0 Pid 259034 on test device 1 [0000:21:00] NVIDIA GeForce RTX 5090

    Rank 2 Group 0 Pid 259034 on test device 2 [0000:41:00] NVIDIA GeForce RTX 5090

    Rank 3 Group 0 Pid 259034 on test device 3 [0000:61:00] NVIDIA GeForce RTX 5090

    Rank 4 Group 0 Pid 259034 on test device 4 [0000:81:00] NVIDIA GeForce RTX 5090

    Rank 5 Group 0 Pid 259034 on test device 5 [0000:a1:00] NVIDIA GeForce RTX 5090

    Rank 6 Group 0 Pid 259034 on test device 6 [0000:c1:00] NVIDIA GeForce RTX 5090

    Rank 7 Group 0 Pid 259034 on test device 7 [0000:e1:00] NVIDIA GeForce RTX 5090

    test:259034:259034 [0] NCCL INFO Bootstrap: Using ens29f0:10.30.9.23<0>
    test:259034:259034 [0] NCCL INFO cudaDriverVersion 12080
    test:259034:259034 [0] NCCL INFO NCCL version 2.25.1+cuda12.8
    test:259034:259124 [0] NCCL INFO ncclMaxSharedMem 82240 exceeds device/fn maxSharedMem 79856
    test:259034:259126 [2] NCCL INFO ncclMaxSharedMem 82240 exceeds device/fn maxSharedMem 79856
    test:259034:259129 [5] NCCL INFO ncclMaxSharedMem 82240 exceeds device/fn maxSharedMem 79856
    test:259034:259130 [6] NCCL INFO ncclMaxSharedMem 82240 exceeds device/fn maxSharedMem 79856
    test:259034:259125 [1] NCCL INFO ncclMaxSharedMem 82240 exceeds device/fn maxSharedMem 79856
    test:259034:259131 [7] NCCL INFO ncclMaxSharedMem 82240 exceeds device/fn maxSharedMem 79856
    test:259034:259127 [3] NCCL INFO ncclMaxSharedMem 82240 exceeds device/fn maxSharedMem 79856
    test:259034:259128 [4] NCCL INFO ncclMaxSharedMem 82240 exceeds device/fn maxSharedMem 79856
    test:259034:259129 [5] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. Using internal network plugin.
    test:259034:259129 [5] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ens29f0:10.30.9.23<0>
    test:259034:259129 [5] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
    test:259034:259129 [5] NCCL INFO Using network IB
    test:259034:259128 [4] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
    test:259034:259128 [4] NCCL INFO Using network IB
    test:259034:259131 [7] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
    test:259034:259131 [7] NCCL INFO Using network IB
    test:259034:259125 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
    test:259034:259125 [1] NCCL INFO Using network IB
    test:259034:259124 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
    test:259034:259124 [0] NCCL INFO Using network IB
    test:259034:259130 [6] NCCL INFO Using network IB
    test:259034:259126 [2] NCCL INFO Using network IB
    test:259034:259127 [3] NCCL INFO Using network IB
    test:259034:259128 [4] NCCL INFO ncclCommInitAll comm 0x6143aa643650 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId 81000 commId 0x74ba504098b9f4b6 - Init START
    test:259034:259125 [1] NCCL INFO ncclCommInitAll comm 0x6143aa4b6bd0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 21000 commId 0x74ba504098b9f4b6 - Init START
    test:259034:259129 [5] NCCL INFO ncclCommInitAll comm 0x6143aa6c79d0 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId a1000 commId 0x74ba504098b9f4b6 - Init START
    test:259034:259124 [0] NCCL INFO ncclCommInitAll comm 0x6143aa432810 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 1000 commId 0x74ba504098b9f4b6 - Init START
    test:259034:259130 [6] NCCL INFO ncclCommInitAll comm 0x6143aa74bd50 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId c1000 commId 0x74ba504098b9f4b6 - Init START
    test:259034:259127 [3] NCCL INFO ncclCommInitAll comm 0x6143aa5bf2d0 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId 61000 commId 0x74ba504098b9f4b6 - Init START
    test:259034:259126 [2] NCCL INFO ncclCommInitAll comm 0x6143aa53af50 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId 41000 commId 0x74ba504098b9f4b6 - Init START
    test:259034:259131 [7] NCCL INFO ncclCommInitAll comm 0x6143aa7d00d0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e1000 commId 0x74ba504098b9f4b6 - Init START
    test:259034:259129 [5] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
    test:259034:259130 [6] NCCL INFO Bootstrap timings total 0.017805 (create 0.000091, send 0.000284, recv 0.000750, ring 0.000380, delay 0.000000)
    test:259034:259129 [5] NCCL INFO Bootstrap timings total 0.018053 (create 0.000083, send 0.000269, recv 0.000580, ring 0.000394, delay 0.000000)
    test:259034:259131 [7] NCCL INFO Bootstrap timings total 0.017680 (create 0.000092, send 0.000292, recv 0.000660, ring 0.000356, delay 0.000000)
    test:259034:259126 [2] NCCL INFO Bootstrap timings total 0.017746 (create 0.000094, send 0.000290, recv 0.000515, ring 0.000369, delay 0.000000)
    test:259034:259124 [0] NCCL INFO Bootstrap timings total 0.017958 (create 0.000088, send 0.000288, recv 0.000269, ring 0.000371, delay 0.000000)
    test:259034:259127 [3] NCCL INFO Bootstrap timings total 0.017854 (create 0.000094, send 0.000287, recv 0.000365, ring 0.000369, delay 0.000000)
    test:259034:259125 [1] NCCL INFO Bootstrap timings total 0.018253 (create 0.000105, send 0.000281, recv 0.000856, ring 0.000407, delay 0.000000)
    test:259034:259128 [4] NCCL INFO Bootstrap timings total 0.018625 (create 0.000096, send 0.000193, recv 0.000779, ring 0.000396, delay 0.000000)
    test:259034:259131 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to SYS
    test:259034:259131 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff
    test:259034:259131 [7] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
    test:259034:259130 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff
    test:259034:259127 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff
    test:259034:259124 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff
    test:259034:259129 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff
    test:259034:259128 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff
    test:259034:259126 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff
    test:259034:259125 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff
    test:259034:259131 [7] NCCL INFO comm 0x6143aa7d00d0 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0
    test:259034:259129 [5] NCCL INFO comm 0x6143aa6c79d0 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0
    test:259034:259131 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6
    test:259034:259126 [2] NCCL INFO comm 0x6143aa53af50 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0
    test:259034:259127 [3] NCCL INFO comm 0x6143aa5bf2d0 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0
    test:259034:259127 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2
    test:259034:259127 [3] NCCL INFO P2P Chunksize set to 131072
    test:259034:259125 [1] NCCL INFO comm 0x6143aa4b6bd0 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0
    test:259034:259129 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4
    test:259034:259129 [5] NCCL INFO P2P Chunksize set to 131072
    test:259034:259131 [7] NCCL INFO P2P Chunksize set to 131072
    test:259034:259124 [0] NCCL INFO comm 0x6143aa432810 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0
    test:259034:259128 [4] NCCL INFO comm 0x6143aa643650 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0
    test:259034:259128 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3
    test:259034:259128 [4] NCCL INFO P2P Chunksize set to 131072
    test:259034:259130 [6] NCCL INFO comm 0x6143aa74bd50 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0
    test:259034:259125 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0
    test:259034:259125 [1] NCCL INFO P2P Chunksize set to 131072
    test:259034:259124 [0] NCCL INFO Channel 00/04 : 0 1 2 3 4 5 6 7
    test:259034:259124 [0] NCCL INFO Channel 01/04 : 0 1 2 3 4 5 6 7
    test:259034:259126 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1
    test:259034:259126 [2] NCCL INFO P2P Chunksize set to 131072
    test:259034:259130 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5
    test:259034:259130 [6] NCCL INFO P2P Chunksize set to 131072
    test:259034:259124 [0] NCCL INFO Channel 02/04 : 0 1 2 3 4 5 6 7
    test:259034:259124 [0] NCCL INFO Channel 03/04 : 0 1 2 3 4 5 6 7
    test:259034:259124 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1
    test:259034:259124 [0] NCCL INFO P2P Chunksize set to 131072
    test:259034:259124 [0] NCCL INFO Check P2P Type intraNodeP2pSupport 1 directMode 1
    test:259034:259140 [1] NCCL INFO [Proxy Service] Device 1 CPU core 39
    test:259034:259134 [3] NCCL INFO [Proxy Service] Device 3 CPU core 170
    test:259034:259135 [7] NCCL INFO [Proxy Service] Device 7 CPU core 183
    test:259034:259138 [3] NCCL INFO [Proxy Service UDS] Device 3 CPU core 9
    test:259034:259136 [5] NCCL INFO [Proxy Service] Device 5 CPU core 189
    test:259034:259146 [0] NCCL INFO [Proxy Service] Device 0 CPU core 55
    test:259034:259139 [7] NCCL INFO [Proxy Service UDS] Device 7 CPU core 31
    test:259034:259141 [6] NCCL INFO [Proxy Service] Device 6 CPU core 183
    test:259034:259137 [4] NCCL INFO [Proxy Service] Device 4 CPU core 0
    test:259034:259142 [5] NCCL INFO [Proxy Service UDS] Device 5 CPU core 47
    test:259034:259143 [2] NCCL INFO [Proxy Service] Device 2 CPU core 79
    test:259034:259144 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 71
    test:259034:259145 [6] NCCL INFO [Proxy Service UDS] Device 6 CPU core 87
    test:259034:259149 [4] NCCL INFO [Proxy Service UDS] Device 4 CPU core 154
    test:259034:259147 [2] NCCL INFO [Proxy Service UDS] Device 2 CPU core 60
    test:259034:259148 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 20
    test:259034:259128 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
    test:259034:259128 [4] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
    test:259034:259131 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
    test:259034:259131 [7] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
    test:259034:259129 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
    test:259034:259129 [5] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
    test:259034:259126 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
    test:259034:259126 [2] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
    test:259034:259125 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
    test:259034:259125 [1] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
    test:259034:259130 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
    test:259034:259130 [6] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
    test:259034:259127 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
    test:259034:259127 [3] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
    test:259034:259124 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
    test:259034:259124 [0] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
    test:259034:259124 [0] NCCL INFO CC Off, workFifoBytes 1048576
    test:259034:259131 [7] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so libnccl-net.so. Using internal tuner plugin.
    test:259034:259131 [7] NCCL INFO ncclCommInitAll comm 0x6143aa7d00d0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e1000 commId 0x74ba504098b9f4b6 - Init COMPLETE
    test:259034:259131 [7] NCCL INFO Init timings - ncclCommInitAll: rank 7 nranks 8 total 0.96 (kernels 0.73, alloc 0.10, bootstrap 0.02, allgathers 0.00, topo 0.08, graphs 0.01, connections 0.02, rest 0.01)
    test:259034:259129 [5] NCCL INFO ncclCommInitAll comm 0x6143aa6c79d0 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId a1000 commId 0x74ba504098b9f4b6 - Init COMPLETE
    test:259034:259128 [4] NCCL INFO ncclCommInitAll comm 0x6143aa643650 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId 81000 commId 0x74ba504098b9f4b6 - Init COMPLETE
    test:259034:259129 [5] NCCL INFO Init timings - ncclCommInitAll: rank 5 nranks 8 total 0.96 (kernels 0.73, alloc 0.10, bootstrap 0.02, allgathers 0.00, topo 0.08, graphs 0.01, connections 0.02, rest 0.01)
    test:259034:259130 [6] NCCL INFO ncclCommInitAll comm 0x6143aa74bd50 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId c1000 commId 0x74ba504098b9f4b6 - Init COMPLETE
    test:259034:259126 [2] NCCL INFO ncclCommInitAll comm 0x6143aa53af50 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId 41000 commId 0x74ba504098b9f4b6 - Init COMPLETE
    test:259034:259124 [0] NCCL INFO ncclCommInitAll comm 0x6143aa432810 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 1000 commId 0x74ba504098b9f4b6 - Init COMPLETE
    test:259034:259124 [0] NCCL INFO Init timings - ncclCommInitAll: rank 0 nranks 8 total 0.96 (kernels 0.74, alloc 0.09, bootstrap 0.02, allgathers 0.00, topo 0.08, graphs 0.01, connections 0.02, rest 0.00)
    test:259034:259128 [4] NCCL INFO Init timings - ncclCommInitAll: rank 4 nranks 8 total 0.96 (kernels 0.73, alloc 0.10, bootstrap 0.02, allgathers 0.00, topo 0.08, graphs 0.01, connections 0.01, rest 0.01)
    test:259034:259130 [6] NCCL INFO Init timings - ncclCommInitAll: rank 6 nranks 8 total 0.96 (kernels 0.74, alloc 0.08, bootstrap 0.02, allgathers 0.00, topo 0.08, graphs 0.01, connections 0.02, rest 0.00)
    test:259034:259125 [1] NCCL INFO ncclCommInitAll comm 0x6143aa4b6bd0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 21000 commId 0x74ba504098b9f4b6 - Init COMPLETE
    test:259034:259125 [1] NCCL INFO Init timings - ncclCommInitAll: rank 1 nranks 8 total 0.96 (kernels 0.74, alloc 0.09, bootstrap 0.02, allgathers 0.00, topo 0.08, graphs 0.01, connections 0.02, rest 0.01)
    test:259034:259127 [3] NCCL INFO ncclCommInitAll comm 0x6143aa5bf2d0 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId 61000 commId 0x74ba504098b9f4b6 - Init COMPLETE
    test:259034:259127 [3] NCCL INFO Init timings - ncclCommInitAll: rank 3 nranks 8 total 0.96 (kernels 0.75, alloc 0.08, bootstrap 0.02, allgathers 0.00, topo 0.08, graphs 0.01, connections 0.02, rest 0.00)
    test:259034:259126 [2] NCCL INFO Init timings - ncclCommInitAll: rank 2 nranks 8 total 0.96 (kernels 0.74, alloc 0.08, bootstrap 0.02, allgathers 0.00, topo 0.08, graphs 0.01, connections 0.02, rest 0.01)

    out-of-place in-place

    size count type redop root time algbw busbw #wrong time algbw busbw #wrong

    (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)

    test:259034:259151 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/direct pointer
    test:259034:259152 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/direct pointer
    test:259034:259151 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/direct pointer
    test:259034:259150 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/direct pointer
    test:259034:259153 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/direct pointer
    test:259034:259152 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/direct pointer
    test:259034:259151 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/direct pointer
    test:259034:259155 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/direct pointer
    test:259034:259156 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/direct pointer
    test:259034:259154 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/direct pointer
    test:259034:259152 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/direct pointer
    test:259034:259150 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/direct pointer
    test:259034:259153 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/direct pointer
    test:259034:259151 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/direct pointer
    test:259034:259152 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/direct pointer
    test:259034:259155 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/direct pointer
    test:259034:259157 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/direct pointer
    test:259034:259156 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/direct pointer
    test:259034:259154 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/direct pointer
    test:259034:259150 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/direct pointer
    test:259034:259157 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/direct pointer
    test:259034:259154 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/direct pointer
    test:259034:259153 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/direct pointer
    test:259034:259156 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/direct pointer
    test:259034:259155 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/direct pointer
    test:259034:259150 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/direct pointer
    test:259034:259157 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/direct pointer
    test:259034:259154 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/direct pointer
    test:259034:259153 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/direct pointer
    test:259034:259156 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/direct pointer
    test:259034:259155 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/direct pointer
    test:259034:259157 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/direct pointer
    test:259034:259157 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
    test:259034:259152 [5] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
    test:259034:259155 [2] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
    test:259034:259153 [4] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
    test:259034:259151 [6] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
    test:259034:259154 [3] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
    test:259034:259156 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
    test:259034:259150 [7] NCCL INFO Connected all rings, use ring PXN 0 GDR 1

    test:259034:259034 [7] enqueue.cc:1500 NCCL WARN Cuda failure 1 'invalid argument'
    test:259034:259034 [7] NCCL INFO group.cc:242 -> 1
    test:259034:259034 [7] NCCL INFO group.cc:470 -> 1
    test:259034:259034 [7] NCCL INFO group.cc:573 -> 1
    test:259034:259034 [7] NCCL INFO group.cc:106 -> 1
    test: Test NCCL failure common.cu:411 'unhandled cuda error (run with NCCL_DEBUG=INFO for details) / '
    .. test pid 259034: Test failure common.cu:609
    .. test pid 259034: Test failure all_reduce.cu:90
    .. test pid 259034: Test failure common.cu:642
    .. test pid 259034: Test failure common.cu:1202
    .. test pid 259034: Test failure common.cu:937`

    Do you have any idea what's going on here?

  19. Panchovix commented on Jul 24, 2025

    @Panchovix
    Panchovix
    on Jul 24, 2025

    @JzhJay how did you manage to make P2P in general work with P2P sample?

  20. PoppingXu commented on Jul 24, 2025

    @PoppingXu
    PoppingXu
    on Jul 24, 2025

    I try this 570.148.08 ang p2p is ok，but p2pbandWitchtest has errors.

    I use intel 8458P. How can I solve this

    Device 0: NVIDIA GeForce RTX 5090
    Device 1: NVIDIA GeForce RTX 5090
    Device 2: NVIDIA GeForce RTX 5090
    Device 3: NVIDIA GeForce RTX 5090
    Device 4: NVIDIA GeForce RTX 5090
    Device 5: NVIDIA GeForce RTX 5090
    Device 6: NVIDIA GeForce RTX 5090
    Device 7: NVIDIA GeForce RTX 5090

    Running host_to_device_memcpy_ce.
    memcpy CE CPU(row) -> GPU(column) bandwidth (GB/s)
    0 1 2 3 4 5 6 7
    0 56.30 56.33 56.35 56.35 56.33 56.38 56.40 56.41

    SUM host_to_device_memcpy_ce 450.84

    Running device_to_host_memcpy_ce.
    memcpy CE CPU(row) <- GPU(column) bandwidth (GB/s)
    0 1 2 3 4 5 6 7
    0 57.32 57.33 57.34 57.35 57.29 57.31 57.34 57.33

    SUM device_to_host_memcpy_ce 458.62

    Running host_to_device_bidirectional_memcpy_ce.
    memcpy CE CPU(row) <-> GPU(column) bandwidth (GB/s)
    0 1 2 3 4 5 6 7
    0 50.94 50.94 50.93 50.90 50.84 50.98 50.92 50.86

    SUM host_to_device_bidirectional_memcpy_ce 407.31

    Running device_to_host_bidirectional_memcpy_ce.
    memcpy CE CPU(row) <-> GPU(column) bandwidth (GB/s)
    0 1 2 3 4 5 6 7
    0 50.84 50.86 50.84 50.86 50.83 50.79 50.79 50.84

    SUM device_to_host_bidirectional_memcpy_ce 406.65

    Running device_to_device_memcpy_read_ce.
    [CUDA_ERROR_MAP_FAILED] mapping of buffer object failed

  21. ec-jt commented on Aug 3, 2025

    @ec-jt
    ec-jt
    on Aug 3, 2025

    @huanzhang12 what gpu variant/vbios version?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions