12 Replies Latest reply on Nov 19, 2012 1:02 PM by narada

    GPU Computing - # of GPUs supported

    ardo
      Issues with GPU support when using over 4 gpus.

       

      I currently have a desktop with 6 GPUs ( 3x HD 5970s ) in non-crossfire mode. Unfortunately, it seems that Windows 7 64bit only supports up to 4 GPUs. I have not been able to find a reliable source to deny or confirm this.

      If windows 7 has this limitation, is there a Linux flavor that supports more than 4 GPUs?

      In-case you are wondering, this is not for gaming but high-speed single precision computing. With this current setup ( if I can find 6gpu support ) I am looking to reach 13.8 Teraflops.

      Also, my motherboard does support 3 16x pci-xpress gen2 slots... and I have a 1500w powersupply plugged into a 20amp outlet. Windows is able to detect all 6 cores.. although, 2 of which displays the warning "Drivers failed to load". I am using the latest drivers ( as of today, and the most recent version of ati stream sdk - 2.2 ) 

      To recap:

      - Can windows support 6 GPUs?

      - If not, does Linux?

      Thank you.



        • GPU Computing - # of GPUs supported
          nou

          here http://forums.amd.com/forum/messageview.cfm?catid=328&threadid=122135 zpdixon stated that he have 5 GPUs in the 64bit Linux.

          i think it is worth to try. an post your results.

          • GPU Computing - # of GPUs supported

            The video and system BIOS map the video memory aperture below 4GB for compatibility reasons, which limits the number of GPUs that can reside in any given system.  That number is typically in the 3-4 GPU range, depending on the system and other peripherals.  Linux won’t behave any differently in this regard.

             

            The onlys solution to your problem is to reduce the GPU aperture size, but that is controlled by the OEMs that manufacturethe cards.  AMD provides the chips that go on the boards.

             

            Best of luck on your high-speed single precision computing work!  I wish I had better news to give you.

            • Re: GPU Computing - # of GPUs supported
              narada

              I'm using 6 gpus (three 7990s each of which is a dual 7970) for opencl on linux and it works fine. Windows wouldn't go above 3 or 4 depending on what code you were running. It was experiencing crashes at the driver or opencl runtime level. Regarding linux - all I had to do was install a stock linux (ubuntu/debian) and then install the official amd driver and amd app sdk. So I can definitely say that 6 works on linux! However I've been unable to get my motherboard (Asus Z9PE-D8 WS) to boot with 7 or 8 gpus. With 7 qcode says 'Internal CPU error' and with 8 qcode says 'PCI Bus Enumeration'. The motherboard has 4 x16 slots and 7 pcie slots in total. I think the reason behind it not working with 7 or 8 is most likely related to the link nou posted.

                • Re: GPU Computing - # of GPUs supported
                  captian-n

                  Under Windows 7 x64 I get 5 GPUs running until Catalyst version 12.6. Later Versions seems to be restricted to 4 GPUs. This seems to be the same kind of restriction like the the 2GB limit per GPU, with first drivers all 3GB of my 7950 cards were available. But AMD gives no strategic hints if they want support sometimes more GPUs and more memory. May be it should be a feature of the FirePro cards to separate the cheap from the expensive cards or there are stability problems. Who knows???

                • Re: GPU Computing - # of GPUs supported
                  pwvdendr

                  I have 8x HD7970 running on an MSI big bang marshall B3 motherboard, and 6x HD7970 on a MSI Z77A-GD65 motherboard. Not a single problem under 64-bit Ubuntu. But I had similar issues under windows, so I assume there is a driver problem. Your situation may be different, but giving linux a try can't hurt.

                    • Re: GPU Computing - # of GPUs supported
                      narada

                      pwvdendr: Thanks for the info and the motherboard models. I'm on the lookout for new multi gpu motherboards that support at least 8 after having been let down by the asus mobo which doesn't boot with more than 6. Out of curiosity - do you have 8 air cooled cards in the msi big bang? Spacing looks pretty tight. How did you manage to fit 8 so close to each other? Mine has seven slots but usually air cooled cards will be dual or triple slot and water cooled cards will be single slot but the cooling pipes will extend that to dual or triple slot for each card anyway.

                       

                      captian-n: I am surprised to hear what you are saying. I was using catalyst 12.10 and 12.11 beta on windows and wasn't able to go higher than 3 or 4. Any access to the opencl api would result in a crash. But if you were able to with earlier versions then surely we've regressed. 12.11 beta was unstable as hell on windows but stable on linux so not really an option on windows to be honest.

                        • Re: GPU Computing - # of GPUs supported
                          pwvdendr

                          narada wrote:

                           

                          pwvdendr: Thanks for the info and the motherboard models. I'm on the lookout for new multi gpu motherboards that support at least 8 after having been let down by the asus mobo which doesn't boot with more than 6. Out of curiosity - do you have 8 air cooled cards in the msi big bang? Spacing looks pretty tight. How did you manage to fit 8 so close to each other? Mine has seven slots but usually air cooled cards will be dual or triple slot and water cooled cards will be single slot but the cooling pipes will extend that to dual or triple slot for each card anyway.

                          PCI-e extender cables (with built-in molex cable not to fry the motherboard).

                          https://dl.dropbox.com/u/3060536/GPGPU-front.jpg

                          https://dl.dropbox.com/u/3060536/GPGPU-side.jpg

                           

                          captian-n: I am surprised to hear what you are saying. I was using catalyst 12.10 and 12.11 beta on windows and wasn't able to go higher than 3 or 4.

                          Not sure on later versions, but with 12.3 or 12.4 (not sure which one) I also got 5 working under windows. But 5 out of 8 was still not acceptable, of course.

                          • Re: GPU Computing - # of GPUs supported
                            captian-n

                            4 GPU runs more or less stable with latest beta7 of 12.11. Only sometimes very rarely the complete desktop freezes at that point where all 4 GPU simultaneous write the data back to host. Linux is no option for me at moment because it is part of a commercial software which is Windows based. The only thing I can do is change the vendor, but means from bad to worse.   May be I will make a try with Linux later.

                              • Re: GPU Computing - # of GPUs supported
                                narada

                                We're also somewhat restricted to Windows for commercial reasons but if Linux is dramatically better i.e. works with 8 as opposed to 4 then we may have ground to ask specific teams to consider migrating to Linux. I'll try the beta next time I'm on windows but the last time (still with 12.11 beta) I couldn't even work on windows due to serious instability. Oddly the beta seems perfectly stable on Linux. We will also try vendor alternatives and see if 8 can be reached on windows.