21 Replies Latest reply on Aug 4, 2010 12:39 PM by d.a.a.

    About HD5970 and 4 boards on one motherboard

    riza.guntur

      I've read from some source about 5970, will it support 4 boards on one motherboard? I mean like 8 GPUs on one motherboard, will it detected properly by CAL runtime this time?

        • About HD5970 and 4 boards on one motherboard
          eduardoschardong

          The card has been launched, and so?

          May I order 4 of them? Will it work? Motherboard recommendations?

           

            • About HD5970 and 4 boards on one motherboard
              riza.guntur

              you want to buy it eduardo?

              I need nome infos since I want to put "but the 5970 can be installed 4 cards in a board, unlike 4870x2" on my paper

                • About HD5970 and 4 boards on one motherboard
                  eduardoschardong

                  Yes... If the 8 gpu per node works and avaliability allows...

                   

                  Performance enough for show.

                   

                    • About HD5970 and 4 boards on one motherboard
                      emuller

                      Just buy 4 5970s and give it a try ...

                      lol ;-)

                      No but seriously, I did that with 4870x2's.  No luck.  I had a msi gd70-790fx and an asrock x58 deluxe running 2 4870x2's and a nvidia gtx260 all doing brook or cuda work.  The minute I go over 4 ATI gpus, the thing crashes.  I was rather exhaustive on my search to get this working, to no avail. It would be great if someone from AMD could comment on this issue.  If 8 GPUs is not going to be supported, I would go with 4 5870s. If it is, I would naturally go with 4 5970s.  I already have a big enough power supply.  Then again, there's the option of getting 2 5970s and putting 2 fermi's in the other 2 slots ... the nvidia architecture is good for some problems ... maybe the two pairs of cards could cooperate ;-)

                       

                       

                       

                       

                       

                       

                        • About HD5970 and 4 boards on one motherboard
                          zpdixon
                          I have a rather encouraging news. I have 5 GPUs working together: two 5970s and one 5870, on 64-bit Linux, with sdk 1.4, and the 9.10 drivers. Although I notice a weird symptom. One of the 5 GPUs seems to run more slowly than the others. For example the "ilperf" program I posted in another thread runs a loop of MAD instructions on all 5 GPUs in parallel. It reports the expected GFLOPS perf for all GPUs except it reports about 800 GFLOPS for one of the 5. The GPU affected by this perf pb seems to be picked at random depending on the order of the video cards in the PCI-E slots. Although in my tests it seems to always be one of the 2 GPUs on one of the 5970s. I can't verify that the 9.11 drivers fix the pb because they make X.org segfault.
                            • About HD5970 and 4 boards on one motherboard
                              eduardoschardong

                              Thank you zpdixon, good news, it would be nice if someone from AMD answer this.

                               

                              BTW, for the driver team, why they are threated as graphics devices in Windows (and then problems like remote desktop)? Tesla don't even have a video output and, IIRC, the first stream processors didn't have too, for this system ok, one of the GPUs will be the graphic, but production it's more like a server with a rage or so for Graphics and as many as possible GPUs on PCIe slots...

                               

                              • About HD5970 and 4 boards on one motherboard
                                emuller

                                @ zpdixon

                                Thanks for the very encouraging news.  Which linux are you running, and have you tried the beta drivers provided with OpenCL 2.0beta4 ? 

                                You say 9.11 crashes X.org ... is this the case also for <=4 GPUs?

                                 

                                 

                                • About HD5970 and 4 boards on one motherboard
                                  riza.guntur

                                   

                                  Originally posted by: zpdixon I have a rather encouraging news. I have 5 GPUs working together: two 5970s and one 5870, on 64-bit Linux, with sdk 1.4, and the 9.10 drivers. Although I notice a weird symptom. One of the 5 GPUs seems to run more slowly than the others. For example the "ilperf" program I posted in another thread runs a loop of MAD instructions on all 5 GPUs in parallel. It reports the expected GFLOPS perf for all GPUs except it reports about 800 GFLOPS for one of the 5. The GPU affected by this perf pb seems to be picked at random depending on the order of the video cards in the PCI-E slots. Although in my tests it seems to always be one of the 2 GPUs on one of the 5970s. I can't verify that the 9.11 drivers fix the pb because they make X.org segfault.


                                  Maybe other slot run in slower speed, for example 4x or 8x

                                  Because when my motherboard got short circuit, it detected as ax, the card run super slow, less than half normal performance. Though after reboot it is fine again

                                  Can you clarify this?

                                    • About HD5970 and 4 boards on one motherboard
                                      zpdixon

                                      To answer the various questions:

                                      - I am running Ubuntu 8.04 amd64

                                      - I have not tried the beta drivers that ship with the OpenCL 2 beta, however I doubt they will fix the pb...

                                      - I was doing something wrong when I installed the 9.11 drivers. I have noticed that fglrx.ko seems to fail to fully reinitialize the hw because sometimes my cards are in such a state that even restarting X.org, rmmod'ing and modprobe'ing fglrx is not sufficient to be able to start X.org (it segfaults). I had to hard reset the box for it to work. So the 9.11 drivers now work but I experience the same symptoms as before: one of the 5 GPUs runs more slowly than the others.

                                      - It is not due to PCI-E link width differences. Not only my workload is not bandwidth intensive (purely ALU-bound), but also all my cards are on x1 PCI-E links (via flexible PCI-E adapters). So it cannot explain the 5th outlier GPU.

                                        • About HD5970 and 4 boards on one motherboard
                                          zpdixon

                                          I bought a 3rd HD5970 and tried running 3 of them on one motherboard but I observed the same weird performance symptoms.

                                          The 9.11 Linux drivers detect 6 GPU devices. My CAL IL code can use them. But some of the GPU devices seem to be operating more slowly than the others. More precisely, 3 of the 6 GPU devices (the 2nd GPU of each HD5970) take longer than the 3 others to run my "ilperf" benchmark tool whose source code I posted in this thread:

                                          'ilperf' Source code


                                          The 3 slow GPUs report ~700 GFLOPS instead of the theoretical 2320 GFLOPS, while the 3 other GPUs report almost 2320 GFLOPS as expected (more precisely 2270).

                                          I can't verify the shader clocks with aticonfig because the tool report an error against the HD5970 (aticonfig: No supported adapters detected). My guess is that for some reason the HD5970 is throttling the shader clock to around 220 MHz instead of 725 MHz. This would explain the ~700 GFLOPS.

                                          When I modify ilperf to only execute on the first 2 devices (2 GPUs of the first card), the same assymetric performance can be observed: 1st device report ~2320 GFLOPS, 2nd device reports ~700 GFLOPS.

                                          Any idea, anyone? This is an interesting pb.

                            • About HD5970 and 4 boards on one motherboard
                              MicahVillmow
                              zpdixon, I have reported this to the correct team to look at but since it is the holidays I don't expect it to be worked on until the new year. Most likely CAL is not correctly clocking the second device up when executing on that device.
                              • About HD5970 and 4 boards on one motherboard
                                MicahVillmow
                                zpdixon,
                                They are working on a solution for this currently and it will part of a future catalyst release when it is fixed.