cancel
Showing results for 
Search instead for 
Did you mean: 

Archives Discussions

pwvdendr
Adept II

maximum number of GPUs?

Is there a maximum number of GPUs that can be used for OpenCL computing?

I heared there is a hard limit of 8 with the current linux drivers, but I'm not sure on windows. Technically it's not so difficult to build a compute cluster consisting of 7 dual GPUs on a single mother board (say 7x HD6990 or soon even 7x HD7990). But I just want to verify that the drivers will actually support this under windows, since I heared they don't support it under linux (only up to 8).

54 Replies
Skysnake
Adept II

Re: maximum number of GPUs?

How far i know, there is still the problem with to few PCI-E adress bits.

But i am also very interested in a solution. How far i know aktive PCI-E riser cards could be a solution

0 Likes
davibu
Journeyman III

Re: maximum number of GPUs?

Up to now, I have never seen anyone use more than 8 GPUs on a single system (8xSingle GPU cards or 4xDual GPUs cards). You can check LuxMark results database for few examples: http://www.luxrender.net/luxmark/top/top20/Sala/GPU

I assume it is an hardware limit (not a software one).

0 Likes
pwvdendr
Adept II

Re: maximum number of GPUs?

Could you give a reference or more details? I know you need PCI-e extender cables to connect them, for space reasons (is this the same as riser cards?) but I don't see how this would affect address bits in any way.

0 Likes
jross
Adept I

Re: maximum number of GPUs?

I don't believe it's limited by  PCI address bits (at least not within reason).  I think the only hardware limit you have to worry about is BIOS memory space.  I could be wrong, but I believe graphics devices grab a larger chunk of bios memory than, say, your USB controller.

It is a software issue.  People have made several attempts at adding more than 4 dual-GPU or 8 single-GPU cards to a single machine without much success.  Until you can demonstrate to AMD that they will be making a lot of money in the 8+ GPU workstation business, they're probably not going to dedicate driver developer time to it.

Windows and Linux (X Server) could also have software issues.  Hardly anybody ever tries 8+ GPUs.

However, I don't think we've ever received a straight answer from AMD on whether they're working on the support.  I'm still very interested in hearing if it's possible.

0 Likes
Meteorhead
Challenger

Re: maximum number of GPUs?

It is true. GPU cards take hold of 256MB of BIOS memory space, and if you've got 8 GPUs, that's 2048MB just for GPUs. You have SATA controllers, sound card, ethernet controller, USB hub... and many other things that take up BIOS memory. In general it is almost impossible to free up more memory for a 9th GPU, but definately not 10th.

Only solution would be to create 64-bit BIOS, or solve the issue of GPUs taking up so much BIOS memory.

We would've built such a serious machine (16-32 GPUs per node), but it's impossible.

0 Likes
Skysnake
Adept II

Re: maximum number of GPUs?

Sorry guys,

you are all wrong. Look here http://fastra2.ua.ac.be/

13 GPUs in one machine

I try to talk since 1,5 year with AMD over such a project with AMD GPUs, but i have no luck Not at the 2011 CeBIT nor on the 2012 CeBIT. I have seen the AMD guys, but they looked pretty busy at the XFX exhibition stand

So if anybody from AMD read this, send me a mail, if you are interested

0 Likes
Meteorhead
Challenger

Re: maximum number of GPUs?

I have to say: excellent work done Skysnake! I have read through the specs and technical detail stuff, and it looks real neat. The only thing I ma unsure of, how can flexible PCI-E risers solve that if you screw cards into the proper place on the back of the chassis, then the PCI-E connectors of the videocards will actually insert into a proper slot on the board, and there is no room for the flexi riser. If you could make a picture of that (or explain) how that's done, that would be cool.

We are in the design phase of creating similar, GPU-packed machine, and the most suitable board (the well known TYAN board) is dropped due to bad experiences with support and QoS from partners of ours. This ASUS motherboard is among the best for single CPU solutions, but it cannot hold enough RAM and it doesn't have enough "real" PCI-E lanes. Dual socket boards that hold lot of PCI-E slots is extremely rare, and also it would be nice to have 100+GB RAM for more than 12 GPUs. It would be interesting to have more VRAM then RAM.

It sounds somewhat disencouriging to hear that AMD is not interested in building such a machine. It would show nice on just any website to show that it is reality to build such beasts. Our institute would have applications that could utilize such single-node/many-GPU workers. We would consider using PCI-E extender boxes (CUBIX), but the very same problems arise that arised in the FASTRA I-II.

Skysnake, it is mentioned on the site that it is not verified that the I/O port space is really neccessary but is not verified. It would be good to know, since it seems that is the only reason the 13 GPU limit exists. (Although I don't know how can all this be put together with fglrx, but I fear a lot more complications would arise, but I may be wrong) With PCI-E 3.0 coming about, 32 GPUs is still viable from a bandwidth point of view.

0 Likes
Skysnake
Adept II

Re: maximum number of GPUs?

Sorry, it is not my project so no pictures from me, but you can find the picture you search on the website

http://fastra2.ua.ac.be/?page_id=38

http://fastra2.ua.ac.be/wp-content/gallery/fastra2/thumbs/thumbs_IMG_0422.JPG

The Riser cables are just for physical reaseons. You cannot put 2 Slot cards into 1Slot space   At the Fastra2 project they solved the BIOS/Driver problem with a single card that is different to the other one. You use this card for booting.

And since this night, i perhaps know how you can solve the booting problem easely But i have to confirm this in the next days/weeks...

But then still it is not clear for me, if there are some more driver/BIOS problems and so on. I don´t think that there will be a OpenCL problem, but you will never know until you do it.

Meteorhead schrieb:

We are in the design phase of creating similar, GPU-packed machine, and the most suitable board (the well known TYAN board) is dropped due to bad experiences with support and QoS from partners of ours. This ASUS motherboard is among the best for single CPU solutions, but it cannot hold enough RAM and it doesn't have enough "real" PCI-E lanes. Dual socket boards that hold lot of PCI-E slots is extremely rare, and also it would be nice to have 100+GB RAM for more than 12 GPUs. It would be interesting to have more VRAM then RAM.

Can you say, what you are doing? I am just a physics/computer science student at the University of Heidelberg.

I know a solution for your problem, but the product is not released yet, so i am not able to say something about this, but you should have 7-9 PCI-E 16x slots and more than enough bandwith and DIMM-Slots. I have talked with the company at the CeBIT, and they could give me perhaps such a board, but then i still need 2 CPUs for something about 500-1k each -.-

100+ GB is not possible. How far i know 8 PCI-E Slots are the maximum and so eaven if you use Dual-GPUs (7990 what is not released now) with the maximum of 6 GB RAM, you will need a 9th card for this. But 8 could be possible, but very very very expensive. (more or less not FirePro/Tesla )

It sounds somewhat disencouriging to hear that AMD is not interested in building such a machine. It would show nice on just any website to show that it is reality to build such beasts. Our institute would have applications that could utilize such single-node/many-GPU workers. We would consider using PCI-E extender boxes (CUBIX), but the very same problems arise that arised in the FASTRA I-II.

Yeah, at the 2011 CeBIT, AMD was only in the Reseller area, and they don´t let me in -.- So i have only a card with a email adress, but absolutely no response from there. So i have talked this year with there board partners, and they are much more interested. A BIG thanks! to XFX at this position. They sponsored me a card, so i am able to have a look at the ne GCN architecture.

Skysnake, it is mentioned on the site that it is not verified that the I/O port space is really neccessary but is not verified. It would be good to know, since it seems that is the only reason the 13 GPU limit exists. (Although I don't know how can all this be put together with fglrx, but I fear a lot more complications would arise, but I may be wrong) With PCI-E 3.0 coming about, 32 GPUs is still viable from a bandwidth point of view.

Bandwith is alwas such a point. There are applications out there where 16x PCI-E 3.0 is still not fast enough and other, where 8 PCI-E 1.0 lanes are enough... So there is no easy answer to something like this. The same for the question if a HD5870 or a Tesla card is a better solution. It depends. In most cases the Tesla is faster, but there are also problems, where the Tesla only can see the back lights of the old VLIW5/4 cards. Have a look on Bit coining and so on.

I really don´t know what the problem is with the BIOS/driver, but i know, that SuperMicro use active riser cards for there 6 GPU machine. So with there solution it should be perhabs possible to use 6 dual-GPU cards, but i don´t know. I think they have never done something like this, because they see no solution for the thermal problem on one side, and the power supply in a rack on the other side.

Btw. i see no thermal problem, when you do it right, and also it should be possible to get the cards into 1 slot.

And btw. who do you call Skysnake?

0 Likes
Marix
Adept II

Re: maximum number of GPUs?

All wouldn't be a problem if GPU vendors finally did their homework and provided drivers that work with EFI boot (without any BIOS image loading tricks).