General Discussions

[vmetodiev]

Dear AMD Community,

From what I know regarding the PCIe protocol organization, a direct peer-to-peer communication between endpoints is possible only in the presence of an intermediate PCIe switch. It could either be a discrete IC or the functionality could be integrated inside the root complex as well. Correct me if I'm wrong...

Now, because I need a peer-to-peer DirectGMA between a Kintex 7 and Radeon Pro, may you please advise what requirement I have to meet from the point of view of motherboard and chipset selection - in order to get a native PCIe switching functionality?

Best,

Varban

[fsadough]

From what I know regarding the PCIe protocol organization, a direct peer-to-peer communication between endpoints is possible only in the presence of an intermediate PCIe switch.

In order for dGPU0 to send P2P to dGPU1 (without going through the RC), there must be an external Switch connecting dGPU0 and dGPU1.

                                          ---> SW-DS0 – dGPU0

RP -> SW-US -+

                                  ---> SW-DS1 – dGPU1

Also, the ACS settings in both SW-DS0 and SW-DS1, must allow the direct P2P path to be used (and not force the dGPU requests up to the RP).

• Note: The internal Switch within the SOC15 asic only has a single downstream port, so it won’t support any P2P path (ie. the TLP will have to go up to the external Switch, between the two SOC15 asics.

It could either be a discrete IC or the functionality could be integrated inside the root complex as well.

It would need to be an external Switch (ie. not in the RC or not in the SOC15 asic, and it must contain SW-DS ports that connect to the two SOC15 asics (ie. dGPUs).

Correct me if I'm wrong...

Now, because I need a peer-to-peer DirectGMA between a Kintex 7 and Radeon Pro, may you please advise what requirement I have to meet from the point of view of motherboard and chipset selection - in order to get a native PCIe switching functionality?

For two asics to use PCIe direct P2P path (ie. not go up to the RC):

• If the Kintex 7 and Radeon Pro are on two separate add-in cards (AICs), then the MB slot that each are plugged into, must be driven by the same external Switch (eg. a Switch soldered on the MB).
• If the Kintex 7 and Radeon Pro are on the same add-in card, then the AIC must have an external Switch (eg. a Switch soldered on the AIC), that connects to both asic.

[vmetodiev]

Hi @fsadough ,

I really appreciate your thorough and informative response, thank you!

Now, about the configuration - it will be based on two separate cards.

Do you have any reference designs for some PCIe switch ICs that you can share or can you recommend me a specific raiser with integrated PCIe switch onboard?

In both cases, I won't mind if the ICs support older PCIe version (v2.0 or v3.0). 

Best,

Varban

[fsadough · Accepted solution]

We don’t have reference MB designs with a Switch, but MB vendors are permitted to make their own designs, in order to create more PCIe slots (eg. MBs with 6 PCIe connectors).

• Each MB vendor is permitted to place the Switch at any place they wish (eg. on the MB, or on a riser card).

The Switch vendors can provide their own reference card which is normally just a Switch riser card, that plugs into a MB slot, and into which cards can be plugged into.