14 Replies Latest reply on Sep 7, 2011 1:37 PM by himanshu.gautam

    8 GPUs in one PC

    Gunter

      Hi,

      we are planning to build a PC with 4 HD 6990 in it, for a total of 8 GPUs. In the first instance, this PC will run Windows 7 Enterprise 64-bit, and Stream SDK 2.3.

      Is this number of GPUs supported by the driver/SDK? If not, what is the limit?

      Thx

        • 8 GPUs in one PC
          genaganna

           

          Originally posted by: Gunter Hi,

          we are planning to build a PC with 4 HD 6990 in it, for a total of 8 GPUs. In the first instance, this PC will run Windows 7 Enterprise 64-bit, and Stream SDK 2.3.

          Is this number of GPUs supported by the driver/SDK? If not, what is the limit?

          It is possible. You will face bandwidth issue if all PCIE are not working at 16x.  Please use latest driver and SDK.  Presently available SDK is 2.5 and driver is 11.8.

            • 8 GPUs in one PC
              settle

               

              Originally posted by: genaganna

              It is possible. You will face bandwidth issue if all PCIE are not working at 16x.  Please use latest driver and SDK.  Presently available SDK is 2.5 and driver is 11.8.

              I have a AMD Radeon HD5970 and wasn't aware that using both GPUs on these kinds of cards was officially supported yet.  If there was news about this would you please post a link.  If it's unofficially supported, please let me know what I can do to make it work.

                • 8 GPUs in one PC
                  genaganna

                   

                  Originally posted by: settle

                   

                  I have a AMD Radeon HD5970 and wasn't aware that using both GPUs on these kinds of cards was officially supported yet.  If there was news about this would you please post a link.  If it's unofficially supported, please let me know what I can do to make it work.

                   

                  OpenCL should work without any issue on x2 cards. Are you facing any issue?

                  What are you looking in link?

                    • 8 GPUs in one PC
                      rick.weber

                      Will you see both devices on a 6990? The release notes don't really say whether this issue has been fixed or not. We're looking at building a similar machine for research, but don't want to waste money on 6990s if we can't use them all. Also, do they make power supplies big enough to drive 4 6990s and 2 magney cours (my guess is you'd want 2-2.5kW).

                        • 8 GPUs in one PC
                          settle

                           

                          Originally posted by: rick.weber Will you see both devices on a 6990? The release notes don't really say whether this issue has been fixed or not.


                          If you check the system requirements for APP SDK 2.5 you can find 5970 listed but not 6990.

                          http://developer.amd.com/sdks/AMDAPPSDK/pages/DriverCompatibility.aspx

                            • 8 GPUs in one PC
                              genaganna

                               

                              Originally posted by: settle

                               

                              I'm using Catalyst 11.8 and APP SDK 2.5 on my openSUSE 11.4 x86_64 machine and I see two GPU devices corresponding to my 5970.  My concern though is if both devices are fully supported or will they conflict as I don't know how they're sharing the global memory.

                               

                              I was also hoping that that the driver would abstract away the two devices with 20 compute units each into a single device with 40 compute units.  Is there any reason why this route wasn't chosen?  Nothing in the OpenCL specifications seems to conflict with this and I think it would strongly appeal to developers.  Maybe it has to do with that the device has 4 GB of global memory but each GPU can only access 2 GB each, but I'm not sure if that's true or not.

                               

                              OpenCL treats two devices as separte. User is responisble to estabish communication b/w them.  Functionally there is no difference b/w X2 cards and devices connected with crossfire.

                                • 8 GPUs in one PC
                                  settle

                                   

                                  Originally posted by: genagannaOpenCL treats two devices as separte. User is responisble to estabish communication b/w them.  Functionally there is no difference b/w X2 cards and devices connected with crossfire.


                                   

                                  Doesn't OpenCL treat dual CPU socket systems as a single CPU device (maybe I misunderstood this part)? However, if that's true then it would make sense to try to treat a dual GPU socket board as a single GPU device.

                                  http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5782450&tag=1

                                    • 8 GPUs in one PC
                                      genaganna

                                       

                                      Originally posted by: settle

                                       

                                       Doesn't OpenCL treat dual CPU socket systems as a single CPU device (maybe I misunderstood this part)? However, if that's true then it would make sense to try to treat a dual GPU socket board as a single GPU device.

                                       

                                      http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5782450&tag=1

                                       

                                      Yes OpenCL treat dual CPU socket system as a single CPU device and X2 card as two separate devices. Good question? I will clarify as early as possible.

                                        • 8 GPUs in one PC
                                          jeff_golds

                                           

                                          Originally posted by: genaganna

                                           

                                          Yes OpenCL treat dual CPU socket system as a single CPU device and X2 card as two separate devices. Good question? I will clarify as early as possible.

                                           

                                          X2 cards will be treated as separate GPU devices.  Dual CPU systems have a much simpler memory architecture that make it much easier to share data between them.  Dual GPU cards have distinct memory for each GPU and no easy mechanism for sharing data.

                                          Jeff

                                            • 8 GPUs in one PC
                                              settle

                                               

                                              Originally posted by: jeff_golds

                                              X2 cards will be treated as separate GPU devices.  Dual CPU systems have a much simpler memory architecture that make it much easier to share data between them.  Dual GPU cards have distinct memory for each GPU and no easy mechanism for sharing data.

                                              In the AMD Accelerated Parallel Processing OpenCL™ Programming Guide (v1.3c) in Appendix D Device Parameters, Hemlock 5970 is listed as L2 Cache Size / GPU = 2 x 512k and Frame Buffer = 2 GB.  Written that way seems to imply that the two GPU dies share a single global memory pool, similar to two CPU dies share a single host memory pool.  I'm not a GPU hardware expert but I'm very interested in learning more.  Maybe the Frame Buffer should read 2 x 1 GB or 2 x 2 GB if they have distinct global memory pools?  If you could describe the X2 card memory architecture a little more I'd really appreciate it.  Thanks!

                                  • 8 GPUs in one PC
                                    settle

                                     

                                    Originally posted by: genaganna

                                     

                                    OpenCL should work without any issue on x2 cards. Are you facing any issue?

                                     

                                    What are you looking in link?

                                     

                                    Going back as far as this thread it was an issue on all x2 cards, but I'm not aware of when/if it was fixed.  I didn't find it anywhere in the Catalyst driver pdf files or APP SDK pdf files.

                                    http://forums.amd.com/forum/messageview.cfm?catid=390&threadid=140518&highlight_key=y&keyword1=2x%20GPU