To answer the various questions:
- I am running Ubuntu 8.04 amd64
- I have not tried the beta drivers that ship with the OpenCL 2 beta, however I doubt they will fix the pb...
- I was doing something wrong when I installed the 9.11 drivers. I have noticed that fglrx.ko seems to fail to fully reinitialize the hw because sometimes my cards are in such a state that even restarting X.org, rmmod'ing and modprobe'ing fglrx is not sufficient to be able to start X.org (it segfaults). I had to hard reset the box for it to work. So the 9.11 drivers now work but I experience the same symptoms as before: one of the 5 GPUs runs more slowly than the others.
- It is not due to PCI-E link width differences. Not only my workload is not bandwidth intensive (purely ALU-bound), but also all my cards are on x1 PCI-E links (via flexible PCI-E adapters). So it cannot explain the 5th outlier GPU.
I bought a 3rd HD5970 and tried running 3 of them on one motherboard but I observed the same weird performance symptoms.
The 9.11 Linux drivers detect 6 GPU devices. My CAL IL code can use them. But some of the GPU devices seem to be operating more slowly than the others. More precisely, 3 of the 6 GPU devices (the 2nd GPU of each HD5970) take longer than the 3 others to run my "ilperf" benchmark tool whose source code I posted in this thread:
'ilperf' Source code
The 3 slow GPUs report ~700 GFLOPS instead of the theoretical 2320 GFLOPS, while the 3 other GPUs report almost 2320 GFLOPS as expected (more precisely 2270).
I can't verify the shader clocks with aticonfig because the tool report an error against the HD5970 (aticonfig: No supported adapters detected). My guess is that for some reason the HD5970 is throttling the shader clock to around 220 MHz instead of 725 MHz. This would explain the ~700 GFLOPS.
When I modify ilperf to only execute on the first 2 devices (2 GPUs of the first card), the same assymetric performance can be observed: 1st device report ~2320 GFLOPS, 2nd device reports ~700 GFLOPS.
Any idea, anyone? This is an interesting pb.
what about only one 5970 in system? if it is problem with too much GPU or it just that second core is underclocked. and try newest catalyst 9.12 with hotfix.
nou: as I said earlier, with 1 or 2 HD5970 in my system, all GPU devices run at full speed as expected.
Micah: thanks for reporting this to the team!
In the mean time I will give a try to the 9.12 drivers, and maybe the drivers that ship with the SDK 2.0 (if any - I haven't looked at how this SDK is packaged yet)
We need to set up a machine with 3x5970 cards for a project. Are there any specific motherboards that would be recommended for this ? I was thinking of some high end Asus.
Micah: Also when can we expect to have drivers that will support all GPUs (Preferably Linux)
For >4 GPUs, I would suggest a dual quad core xeon motherboard like this one:
Though I haven't tried it. I have dual 5970, and my experience has been that one wants at least 1 core dedicated per GPU ... and at least 1 free to do other OS things ... at least for my applications.
I have tried 2 5970 and 1 5770, and it freezes as it did for my attempted 4x 4850x2 system way back when (Ubuntu 9.10). If 3 5970 is a want rather than a need, I would suggest 4x5870 ... and an asus p6t7 or an msi 790fx-gd70 ... The former is 4xPCIe 16x but I have not tested. The latter is 4xPCIe 8x or 2xPCIe 16x and its working great right now!
But please inform if you get >2 5970 working at some point. I've given up.
Originally posted by: MicahVillmow zpdixon, They are working on a solution for this currently and it will part of a future catalyst release when it is fixed.