I have several HD5870 GPUs. Until recently, I used to get around 330 Gflop/s of performance from my double precision application on each GPU. However, since some time until now, several of the GPUs are starting to show a loss of performance.
Right now, 5 out of 20 GPUs are significantly and consistently under-performing although they are still returning the correct numerical result. To illustrate my problem, I can point out that I have one GPU performing at 85 Gflop/s (roughly 25 percent of the original performance), some other GPUs are performing as low as 150 and 250 Gflop/s. All of my test are done on identical systems (hardware and software), with the same linux OS and drivers, and with no other application running on the system.
For troubleshooting, I tried swapping the GPUs between different computers and the performance-loss problem followed the GPU. So it seems clear to me that the performance issue is related to the GPU.
Has any one experienced a similar performance degradation? I really don't know what could I do and using google to search for the problem has not given me any results so far. Any advice or suggestions will be greatly appreciated :-)