25 Replies Latest reply on Oct 21, 2010 3:59 PM by ThomasUCF

    still multiple GPU

    ThomasUCF

       

        Hi everyone:

          I just want to make sure before I do it. As now I can not use multiple GPU (2 HD 5970s) to do calculation, someone told me that if I use older version driver ( say version 10.1) then I can do it. Another way is to do it in Linux system( It seems that I can not find the second HD5970 when crossfire is disabled in windows 7). Do they worth a try?

        Thanks a lot.

      ThomasUCF

        • still multiple GPU
          afo

          Hi,

          I also have 2xHD5970, so we can share experiences:

          a) In winXP32 I can only see 3 GPU with all drivers and all SDK;(just to check: did you enabled all the adapters with CCC?) This happens to me with opencl and cal, so it is a driver issue. I always have crossfire disabled in windows and linux and no ribbon cable between boards because in other threads this leads to bad results with the second GPU.

          b) In winXP32 I can use 2 GPU with no bad results at the same time with no performance penalty. They are GPU[0] and GPU[2]; with GPU[1] clEnqueueKernelRange fails with no resources, but SimpleMultiDevice passes. In single mode I can use the 3 GPU with no bad results.

          c) In linuxX86_64 I can see the four GPU, but aticonfig is  very difficult; I didn't managed to install with 2xHD5970; I installed first with a HD4350 and when everything worked fine, I upgraded to 2xHD5970

          d)In linuxX86_64 (OpenSuse 11.2) I can use the 4 GPU with no bad results at the same time, but I see a performance penalty about 1.5/2  times slow when I launch 3/4 instances of my application at the same time. Doing some time profilling shows that kernel time increases. My guess is that because I call from 3/4 different contexts in 3/4 different applications, there is some ovehead in the ati opencl runtime; currently I am experimenting what happens if I create one context with more than one device.

          hope this helps and best regards,

          Alfonso

            • still multiple GPU
              ThomasUCF

               

                Hi afo:

                       Thanks for sharing your experience.

                       In win 7 64 bit with crossfire disabled, I can not found device[2],device[3], maybe I should also remove the ribbon cable.

                     Also the simpleMultiDevice is wrong at several places, for example in the kernel * input and * output are reversed. I modified the sample and see that only device[0] works. I can not even assign buffer to other devices.

                    ThomasUCF

                • still multiple GPU
                  empty_knapsack

                   

                  Originally posted by: ThomasUCF

                   

                     In win 7 64 bit with crossfire disabled, I can not found device[2],device[3], maybe I should also remove the ribbon cable.



                  Did you attached monitor/dummy plug to second GPU?

                    • still multiple GPU
                      himanshu.gautam

                      Hi thomasUCF,

                      Can you also point out any other discrepencies in any other sample?

                      The pointed error is already known and will be fixed in this release.

                       

                        • still multiple GPU
                          ThomasUCF

                           

                            Hi Himanshu:

                                      I found some other problems in simpleMultiDevice.

                                     In the function Device::verifyResults(), the code: 

                                       for(int i = 0; i < width; i++)
                              {
                                  error += input - verificationOutput;
                              }
                              error /= width;

                              if(error < 0.001)
                              {
                                  std::cout << "Passed!\n";
                                  verificationCount++;
                              }
                              else
                              {
                                  std::cout << "Failed!\n";
                              }

                              return SDK_SUCCESS;

                           

                          Here input should be output, but output is zero in gpu 1,2,3 as these gpus can't do calculation. But the program will always show passed! because verificationOutput is infinity, so if you substract infinity, you will always get error < 0.01, so it shows passed! everytime even the gpu1,2,3 is not doing anything.

                          Thanks.

                           

                        • still multiple GPU
                          ThomasUCF

                           No I didn't. Do I have to ?

                            • still multiple GPU
                              himanshu.gautam

                              Hi ThomasUCF,

                              Thanks for quoting the bugs in simpleMultiDevice Sample.

                              You need to have a display/dummy plug connected to both the GPUs in Windows environment.

                              Thanks

                                • still multiple GPU
                                  ThomasUCF

                                   

                                     Hi Himanshu:

                                            Thanks, by dummy plug do you mean only a plug without connecting display? I did it but still I can only find two GPUs instead of four. Or is it because I removed the crossfire connector which connects the two HD5970s?

                                           And when I connect each HD5870 with a real display, it turns out I can not run the sample code, the Nbody or the simpleMultidevice for example.

                                         My system is i7, win 7 64 bit,2 HD5970s. Thank you so much.

                                   

                                  Regards,

                                  ThomasUCF

                                   

                          • still multiple GPU
                            ThomasUCF

                               Hi afo:

                                   I'm sorry, but what do you mean by "enable all the adapters with CCC"?

                             When I turn off the crossfire and take away the ribbon, I can only find two GPUs instead of four. Is it the window problem or should I switch to Linux system?

                               Thanks a lot.

                             ThomasUCF

                              • still multiple GPU
                                empty_knapsack

                                You're need to attach monitor/dummy plug for each ATI GPUs to make Windows recognize them. And dummy plug means the emulation of monitor (google for it, you can take VGA-DVI switch and use some resistors to make it). After this you're you need to extend desktop for all GPUs you're have to make CAL layer to recognize them, in other words GPU must be active to be used, when it isn't attached to desktop it isn't active. After that you'll finally see all 4 GPUs available... and won't be able to use them for OpenCL anyway because of OpenCL issues.

                                 

                                From ATI side it'll nice to finally add this to some FAQ.

                                  • still multiple GPU
                                    Illusio

                                     

                                    Originally posted by: empty_knapsack You're need to attach monitor/dummy plug for each ATI GPUs to make Windows recognize them.

                                     

                                    Are you sure about that? I recently bought a cheap nVidia card to be able to test things on that platform as well, and saw a bunch of advice on the net stating that you needed a dummy plug, but in reality it wasn't necessary. You can force windows to enable displays without monitors in the display settings by pressing the "detect" button and then selecting one of the displays without monitors that appear, and telling windows that you have a standard monitor connected. I'd suspect it works similarly with secondary ATI cards?

                                    Of course, this won't solve any of the 5970 specific issues.

                                     

                                      • still multiple GPU
                                        empty_knapsack

                                        Illusio, yes, it work for NVIDIA GPUs but not works for ATI ones. Try to make your NVIDIA GPU primary one/attached to monitor and then any "detects" for ATI GPU without attached monitor/dummy plug will fail. You can attach dummy plug to ATI GPU, detect it, remove plug and it will work... till system reboot or display turn off due to inactivity. In second case you can even face the whole system lock up once you'll try to activate monitor back.



                                        Actually, if they'll be separate driver for GPUs treating them as other devices not video output ones it'll solve all problems including ones with watch dog timer. AFAIR NVIDIA did that for Tesla, mainly to make able running CUDA applications as system services. Doubt we'll get anything similar for ATI GPUs seeing the current number of problems still not resolved.

                                      • still multiple GPU
                                        ThomasUCF

                                         Hi enmpty_knapsack:

                                               Now I just want to use both of the HD5970s to do calculation in the same system(or half of each since they're crossfired internally).  I tried a lot but still I can only use one of them. Any suggestions?

                                         Thanks a lot.

                                        ThomasUCF

                                          • still multiple GPU
                                            afo

                                            Hi,

                                            Maybe this helps: go to ccc and in information center/graphics hardware you need to see a primary graphics adapter, an enabled adapter and 2 disabled adapters; do you see that?

                                            If not, you need to go to display manager and enable the odd monitors (1,3,5,7). The cards must have a dummy vga connected to them in their dvi connections with a dvi to vga adapter.

                                            Play with this until you are done. After that, with cal/findnumdevices you will see the number of GPUs available (in winxp32 withour crossfire I have 3 and with crossfire I have 4).

                                            hope this helps,

                                            Alfonso

                                             

                                              • still multiple GPU
                                                ThomasUCF

                                                   Hi afo:

                                                          I see one primary, two linked, and one disabled adapter. Does that look OK?

                                                          Thanks.

                                                  ThomasUCF

                                                  • still multiple GPU
                                                    afo

                                                    Hi,

                                                    I am not sure... In winXP32 it shows 1 primary, 1 enabled and 2 disabled with or without crossfire (enabled/disabled by software + ribbon cable plugged/unplugged), so there is a difference between the drivers for winXP32 and Windows7_64. This sounds to me that it is not possible to disable crossfire en W7_64, so you are working with the "primary" and "disabled" GPUs, the linked ones are not available to you with this S.O....

                                                    So my best advice for you is to switch to linux; with openSuse 11.2 x86_64 the only troubles I found are:

                                                    1) It was not possible to do the install with 2xHD5970, I installed S.O.+cat+sdk with HD4350, and when everything worked fine, I upgraded to 2xHD5970.

                                                    2) There is a performance penalty of around 1.5x if you launch 4 opencl applications at the same time (each one with its own context and targetting a different GPU).

                                                    by the way: programming and debugging is easier en windows because you have the profilling tools, so I use a dualboot system, when the application works fine in single mode in windows, I transfer it to linux to do the final work (so the code has a lot of "#ifdef _MSC_VER")

                                                    best regards,

                                                    Alfonso

                                                      • still multiple GPU
                                                        ThomasUCF

                                                          Thanks Alfonso:

                                                                    That's very helpful.

                                                                    So in Linux system, can I make the four GPUs work on the same problem? Or can I transfer data between the 4 GPUs' memory in one single application?

                                                            Thanks a lot.

                                                        ThomasUCF

                                                          • still multiple GPU
                                                            afo

                                                            Hi,

                                                            You have pros an cons...

                                                            It is easier to divide your input data by the number of GPUs, and launch 4 instances of your application, each one working with a different set of input data independiently (I don't know if this is possible in your application). But you have a performance penalty because the opencl runtime has to deal with 4 different opencl applications at the same time.

                                                            If you have the GPUs in the same context (creating a context with more than one GPU) you have limits to the memory that you can alloc to your buffers, so the memory transfers start to impact to your application.

                                                            Anyway, is not a good idea to exchange data between the GPUs, because the host has to act as a router, reading data from one GPU and writing it to the other; so the performance drops a lot.

                                                            Those are just my experiences, maybe there is another way to do things that avoid the problems that I mentioned.

                                                            best regards,

                                                            Alfonso

                                                              • still multiple GPU
                                                                ThomasUCF

                                                                  Hi Alfonso:

                                                                       Thanks a lot, that's rally helpful.

                                                                      They've been talking about crossfired internally in HD 5970, can I use both of them in Linux?  If not, and plus the 1.5X performance penalty,it is not much improvement for 2 HD5970s than a single HD5870.

                                                                      Looking forward for your reply.

                                                                 ThomasUCF

                                                                  • still multiple GPU
                                                                    afo

                                                                    hi,

                                                                    In linux you can use both processors of a HD5970 at the same time with no bad results (provide you use a different context for each GPU), so if you have 2xHD5970 you can use 4 GPUs.

                                                                    About 1.5x: this means that if you run one instance of your application in, say GPU[0], and the applications takes 20 seconds to finish, when you ran 4 instances in GPU[0] to GPU[3] at the same time, each one will take around 30 seconds and they will finish at the same time too. so the improvement from single HD5870 will be: 4x(2/3)=8/3=2.666x improvement  (4 stands for 4 GPUs instead of 1; 2/3 stands for 1.5times slower=2/3 performance).

                                                                    best regards,

                                                                    Alfonso

                                                              • still multiple GPU
                                                                ThomasUCF

                                                                 Hi afo:

                                                                      I am now switching to Linux, but I met a lot of problems.

                                                                      I have to state that I'm new to Linux, yesterday I tried to install the driver for HD5970, but after done so, the screen seemed as if no driver is installed.

                                                                And what do you mean by" It was not possible to do the install with 2xHD5970, I installed S.O.+cat+sdk with HD4350, and when everything worked fine, I upgraded to 2xHD5970."?   Is it also the driver problem so that I must use  HD 4350 first?

                                                                      Thank you very much.

                                                                 

                                                                ThomasUCF

                                                                  • still multiple GPU
                                                                    afo

                                                                    Hi,

                                                                    No, is not exactly a driver problem, but when I did aticonfig --initial --adapter=all, the configuration generated was not compatible with the monitor, so I didn't have image.

                                                                    So, my recomendation is to get a cheap an mature board, like HD4350/4550 because they are capable to do opencl, Install linux with this board, get the system running, install ati driver, chech that the system is using it, and then install sdk, build the samples and check that everything goes fine. Then upgrade to 1xHD5970, check that opencl samples are running fine, and then upgrade to 2xHD5970. At that point you can start working in your application.

                                                                    best regards,

                                                                    Alfonso