With AMD's Naples server dies allegedly sporting 128 PCI-E 3.0 lanes, it occured to out group to revisit the question of maximum number of GPUs one can leverage in such a system, without having to jump through flaming hoops. 8 channel DDR4 sounds like a sound foundation to decent main memory bandwidth. With P2P transfer between the GPUs depending on the use case, one might be content with
Some configurations will require extenders such as these Magma extenders. Now I recall that shoving that many GPUs into a single system is no small feat, due to the issue of BIOS wanting to allocate memory for every PCIE device with only 32 bits, several hundred MBs per device. With Naples around the corner and it having such a ridiculous amount of PCIE lanes:
I recommend raising this issue on the ROCm github site
One of the focuses of the project is supporting systems with large number of GPUs.