cancel
Showing results for 
Search instead for 
Did you mean: 

Archives Discussions

ThomasUCF
Journeyman III

still multiple GPU

 

  Hi everyone:

    I just want to make sure before I do it. As now I can not use multiple GPU (2 HD 5970s) to do calculation, someone told me that if I use older version driver ( say version 10.1) then I can do it. Another way is to do it in Linux system( It seems that I can not find the second HD5970 when crossfire is disabled in windows 7). Do they worth a try?

  Thanks a lot.

ThomasUCF

0 Likes
25 Replies
afo
Adept I

Hi,

I also have 2xHD5970, so we can share experiences:

a) In winXP32 I can only see 3 GPU with all drivers and all SDK;(just to check: did you enabled all the adapters with CCC?) This happens to me with opencl and cal, so it is a driver issue. I always have crossfire disabled in windows and linux and no ribbon cable between boards because in other threads this leads to bad results with the second GPU.

b) In winXP32 I can use 2 GPU with no bad results at the same time with no performance penalty. They are GPU[0] and GPU[2]; with GPU[1] clEnqueueKernelRange fails with no resources, but SimpleMultiDevice passes. In single mode I can use the 3 GPU with no bad results.

c) In linuxX86_64 I can see the four GPU, but aticonfig is  very difficult; I didn't managed to install with 2xHD5970; I installed first with a HD4350 and when everything worked fine, I upgraded to 2xHD5970

d)In linuxX86_64 (OpenSuse 11.2) I can use the 4 GPU with no bad results at the same time, but I see a performance penalty about 1.5/2  times slow when I launch 3/4 instances of my application at the same time. Doing some time profilling shows that kernel time increases. My guess is that because I call from 3/4 different contexts in 3/4 different applications, there is some ovehead in the ati opencl runtime; currently I am experimenting what happens if I create one context with more than one device.

hope this helps and best regards,

Alfonso

0 Likes

 

  Hi afo:

         Thanks for sharing your experience.

         In win 7 64 bit with crossfire disabled, I can not found device[2],device[3], maybe I should also remove the ribbon cable.

       Also the simpleMultiDevice is wrong at several places, for example in the kernel * input and * output are reversed. I modified the sample and see that only device[0] works. I can not even assign buffer to other devices.

      ThomasUCF

0 Likes

Originally posted by: ThomasUCF

 

   In win 7 64 bit with crossfire disabled, I can not found device[2],device[3], maybe I should also remove the ribbon cable.



Did you attached monitor/dummy plug to second GPU?

0 Likes

Hi thomasUCF,

Can you also point out any other discrepencies in any other sample?

The pointed error is already known and will be fixed in this release.

 

0 Likes

 

  Hi Himanshu:

            I found some other problems in simpleMultiDevice.

           In the function Device::verifyResults(), the code: 

             for(int i = 0; i < width; i++)
    {
        error += input - verificationOutput;
    }
    error /= width;

    if(error < 0.001)
    {
        std::cout << "Passed!\n";
        verificationCount++;
    }
    else
    {
        std::cout << "Failed!\n";
    }

    return SDK_SUCCESS;

 

Here input should be output, but output is zero in gpu 1,2,3 as these gpus can't do calculation. But the program will always show passed! because verificationOutput is infinity, so if you substract infinity, you will always get error < 0.01, so it shows passed! everytime even the gpu1,2,3 is not doing anything.

Thanks.

 

0 Likes

 No I didn't. Do I have to ?

0 Likes

Hi ThomasUCF,

Thanks for quoting the bugs in simpleMultiDevice Sample.

You need to have a display/dummy plug connected to both the GPUs in Windows environment.

Thanks

0 Likes

 

   Hi Himanshu:

          Thanks, by dummy plug do you mean only a plug without connecting display? I did it but still I can only find two GPUs instead of four. Or is it because I removed the crossfire connector which connects the two HD5970s?

         And when I connect each HD5870 with a real display, it turns out I can not run the sample code, the Nbody or the simpleMultidevice for example.

       My system is i7, win 7 64 bit,2 HD5970s. Thank you so much.

 

Regards,

ThomasUCF

 

0 Likes

   Hi afo:

       I'm sorry, but what do you mean by "enable all the adapters with CCC"?

 When I turn off the crossfire and take away the ribbon, I can only find two GPUs instead of four. Is it the window problem or should I switch to Linux system?

   Thanks a lot.

 ThomasUCF

0 Likes

You're need to attach monitor/dummy plug for each ATI GPUs to make Windows recognize them. And dummy plug means the emulation of monitor (google for it, you can take VGA-DVI switch and use some resistors to make it). After this you're you need to extend desktop for all GPUs you're have to make CAL layer to recognize them, in other words GPU must be active to be used, when it isn't attached to desktop it isn't active. After that you'll finally see all 4 GPUs available... and won't be able to use them for OpenCL anyway because of OpenCL issues.

 

From ATI side it'll nice to finally add this to some FAQ.

0 Likes

Originally posted by: empty_knapsack You're need to attach monitor/dummy plug for each ATI GPUs to make Windows recognize them.

 

Are you sure about that? I recently bought a cheap nVidia card to be able to test things on that platform as well, and saw a bunch of advice on the net stating that you needed a dummy plug, but in reality it wasn't necessary. You can force windows to enable displays without monitors in the display settings by pressing the "detect" button and then selecting one of the displays without monitors that appear, and telling windows that you have a standard monitor connected. I'd suspect it works similarly with secondary ATI cards?

Of course, this won't solve any of the 5970 specific issues.

 

0 Likes

Illusio, yes, it work for NVIDIA GPUs but not works for ATI ones. Try to make your NVIDIA GPU primary one/attached to monitor and then any "detects" for ATI GPU without attached monitor/dummy plug will fail. You can attach dummy plug to ATI GPU, detect it, remove plug and it will work... till system reboot or display turn off due to inactivity. In second case you can even face the whole system lock up once you'll try to activate monitor back.



Actually, if they'll be separate driver for GPUs treating them as other devices not video output ones it'll solve all problems including ones with watch dog timer. AFAIR NVIDIA did that for Tesla, mainly to make able running CUDA applications as system services. Doubt we'll get anything similar for ATI GPUs seeing the current number of problems still not resolved.

0 Likes

Interesting. I think I'll pass on attempting to lock up my system though.

 

0 Likes

 Hi enmpty_knapsack:

       Now I just want to use both of the HD5970s to do calculation in the same system(or half of each since they're crossfired internally).  I tried a lot but still I can only use one of them. Any suggestions?

 Thanks a lot.

ThomasUCF

0 Likes

Hi,

Maybe this helps: go to ccc and in information center/graphics hardware you need to see a primary graphics adapter, an enabled adapter and 2 disabled adapters; do you see that?

If not, you need to go to display manager and enable the odd monitors (1,3,5,7). The cards must have a dummy vga connected to them in their dvi connections with a dvi to vga adapter.

Play with this until you are done. After that, with cal/findnumdevices you will see the number of GPUs available (in winxp32 withour crossfire I have 3 and with crossfire I have 4).

hope this helps,

Alfonso

 

0 Likes

   Hi afo:

          I see one primary, two linked, and one disabled adapter. Does that look OK?

          Thanks.

  ThomasUCF

0 Likes

Hi,

I am not sure... In winXP32 it shows 1 primary, 1 enabled and 2 disabled with or without crossfire (enabled/disabled by software + ribbon cable plugged/unplugged), so there is a difference between the drivers for winXP32 and Windows7_64. This sounds to me that it is not possible to disable crossfire en W7_64, so you are working with the "primary" and "disabled" GPUs, the linked ones are not available to you with this S.O....

So my best advice for you is to switch to linux; with openSuse 11.2 x86_64 the only troubles I found are:

1) It was not possible to do the install with 2xHD5970, I installed S.O.+cat+sdk with HD4350, and when everything worked fine, I upgraded to 2xHD5970.

2) There is a performance penalty of around 1.5x if you launch 4 opencl applications at the same time (each one with its own context and targetting a different GPU).

by the way: programming and debugging is easier en windows because you have the profilling tools, so I use a dualboot system, when the application works fine in single mode in windows, I transfer it to linux to do the final work (so the code has a lot of "#ifdef _MSC_VER")

best regards,

Alfonso

0 Likes

  Thanks Alfonso:

            That's very helpful.

            So in Linux system, can I make the four GPUs work on the same problem? Or can I transfer data between the 4 GPUs' memory in one single application?

    Thanks a lot.

ThomasUCF

0 Likes

Hi,

You have pros an cons...

It is easier to divide your input data by the number of GPUs, and launch 4 instances of your application, each one working with a different set of input data independiently (I don't know if this is possible in your application). But you have a performance penalty because the opencl runtime has to deal with 4 different opencl applications at the same time.

If you have the GPUs in the same context (creating a context with more than one GPU) you have limits to the memory that you can alloc to your buffers, so the memory transfers start to impact to your application.

Anyway, is not a good idea to exchange data between the GPUs, because the host has to act as a router, reading data from one GPU and writing it to the other; so the performance drops a lot.

Those are just my experiences, maybe there is another way to do things that avoid the problems that I mentioned.

best regards,

Alfonso

0 Likes

  Hi Alfonso:

       Thanks a lot, that's rally helpful.

      They've been talking about crossfired internally in HD 5970, can I use both of them in Linux?  If not, and plus the 1.5X performance penalty,it is not much improvement for 2 HD5970s than a single HD5870.

      Looking forward for your reply.

 ThomasUCF

0 Likes

hi,

In linux you can use both processors of a HD5970 at the same time with no bad results (provide you use a different context for each GPU), so if you have 2xHD5970 you can use 4 GPUs.

About 1.5x: this means that if you run one instance of your application in, say GPU[0], and the applications takes 20 seconds to finish, when you ran 4 instances in GPU[0] to GPU[3] at the same time, each one will take around 30 seconds and they will finish at the same time too. so the improvement from single HD5870 will be: 4x(2/3)=8/3=2.666x improvement  (4 stands for 4 GPUs instead of 1; 2/3 stands for 1.5times slower=2/3 performance).

best regards,

Alfonso

0 Likes

 Hi Alfonso:

 

    Thanks a lot, that's very helpful.

ThomasUCF

0 Likes

 Hi afo:

      I am now switching to Linux, but I met a lot of problems.

      I have to state that I'm new to Linux, yesterday I tried to install the driver for HD5970, but after done so, the screen seemed as if no driver is installed.

And what do you mean by" It was not possible to do the install with 2xHD5970, I installed S.O.+cat+sdk with HD4350, and when everything worked fine, I upgraded to 2xHD5970."?   Is it also the driver problem so that I must use  HD 4350 first?

      Thank you very much.

 

ThomasUCF

0 Likes

Hi,

No, is not exactly a driver problem, but when I did aticonfig --initial --adapter=all, the configuration generated was not compatible with the monitor, so I didn't have image.

So, my recomendation is to get a cheap an mature board, like HD4350/4550 because they are capable to do opencl, Install linux with this board, get the system running, install ati driver, chech that the system is using it, and then install sdk, build the samples and check that everything goes fine. Then upgrade to 1xHD5970, check that opencl samples are running fine, and then upgrade to 2xHD5970. At that point you can start working in your application.

best regards,

Alfonso

0 Likes

Hi Alfonso:

     That sounds frightening for a Linux beginner, thanks any way, but are there simpler and more straight forward ways?

ThomasUCF

0 Likes