we have a typical NUMA-system with two Intel Xeon CPUs, each with a local main memory system. To each of the CPUs, an I/O-hub 5520 is connected. Each 5520 in turn connects to a set of PCIe-slots. We use multiple HD 6990 cards, so that one card is connected to the one 5520, and the other card is connected to the other.
Now I want to use calResCreate2D() to create resources in either memory. The memory for these resources has been allocated using VirtualAllocExNuma(), thereby placing it into the desired memory system. I can now place CPU-threads on cores close to this data structure for fast access.
My question is: if such thread performs a calDeviceOpen(..., id), how do I find the ids of the GPUs connected to the memory via the shortest path? In essence, I want to avoid that a DMA goes through one 5520, then through one CPU, then through the other CPU, then through its memory, then back the same way.
Related to that: how can I identify GPUs being on the same card?