Hi,
Can you share the platform being used to run the cards?
We do have an ROCm bandwidth test utility, which could be used to see if the bridge is working. It's in the ROCm install.
$ rocm-bandwidth-test -t
or
For stress testing to, install rocm-bandwidth-test (i.e. yum install rocm-bandwidth-test) . It will allow you to test bandwidth between the GPUS:
/opt/rocm/bin/rocm-bandwidth-test ?
Supported arguments:
-h Prints the help screen
-q Query version of the test
-v Run the test in validation mode
-l Run test to collect Latency data
-c Time the operation using CPU Timers
-e Prints the list of ROCm devices enabled on platform
-i Initialize copy buffer with specified 'long double' pattern
-t Prints system topology and allocatable memory info
-m List of buffer sizes to use, specified in Megabytes
-b List devices to use in bidirectional copy operations
-s List of source devices to use in copy unidirectional operations
-d List of destination devices to use in unidirectional copy operations
-a Perform Unidirectional Copy involving all device combinations
-A Perform Bidirectional Copy involving all device combinations
NOTE: Mixing following options is illegal/unsupported
Case 1: rocm_bandwidth_test -a with {lm}{1,}
Case 2: rocm_bandwidth_test -b with {clv}{1,}
Case 3: rocm_bandwidth_test -A with {clmv}{1,}
Case 4: rocm_bandwidth_test -s x -d y with {lmv}{2,}
Another option:
Use rocm-smi (/opt/rocm/bin/rocm-smi) to show GPU topology and print information on how the GPUs are connected (PCIE or XGMI):
I.e.
rocm-smi --showtopo
…..
========================== Link Type between two GPUs ==========================
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5
GPU0 0 PCIE PCIE PCIE PCIE PCIE
GPU1 PCIE 0 PCIE PCIE PCIE PCIE
GPU2 PCIE PCIE 0 PCIE PCIE PCIE
GPU3 PCIE PCIE PCIE 0 PCIE PCIE
GPU4 PCIE PCIE PCIE PCIE 0 PCIE
GPU5 PCIE PCIE PCIE PCIE PCIE 0
…
Regards,
Guy
