cancel
Showing results for 
Search instead for 
Did you mean: 

Archives Discussions

rotor
Journeyman III

Opteron 8-GPU systems?

Hi Gaurav,

Can you give us a hint of which pci-expansion card/box do you use and which interface do you use to hook up the pc-expansion card to the system? And as other asked: do you lost bandwith in the expansion card?

Thanks,

Roto

0 Likes
gaurav_garg
Adept I

Opteron 8-GPU systems?

PCIe host adapter card is PCIe x16 2.0. The PCIe lanes are dynamically assigned to each GPU in expansion system. So, you will get full bandwidth as long as you are using a single GPU per expansion system. But, bandwidth is divided in case multiple GPUs are used.

0 Likes
jross
Adept I

Opteron 8-GPU systems?

@rotor
I'm only aware of a single vendor for an expansion box: One Stop Systems

Additionally, just because the host adapter card is PCIe x16 2.0, it doesn't mean the motherboard supports it.  Using the NVIDIA nForce Professional 3600 and 3050 chipset, you will have two slots at PCIe x16 1.0. or half the bandwidth per adapter.  The card is backwards compatible.  Just sayin'

@gaurav.garg
Was there a particular bootup process? X server configuration?  Special device permissions? Runlevel?

0 Likes
rotor
Journeyman III

Opteron 8-GPU systems?

Thanks Jross,

I really like the OSS 2U GPU/SSD server of the One Stop. For there expansion system, if they hook up 4 GPUs over only 1 PCIe 16 lanes link, the bandwidth theoretically will be decreased 4 times if all 4 cards do transferring data at once. It's also worth to mention about the delay of long communication between the host and the expansion box at the initialization.

Back to the Nvidia chipset. If I use that chip set does it mean that I have to make a custom design mainboard to handle the chip set? I have known so far that the consumer-level workstation mother board now just have up to 3 PCIe 16 lanes which can handles up to only 3 GPUs.

Thanks,

Roto

 

0 Likes
rotor
Journeyman III

Opteron 8-GPU systems?

Originally posted by: gaurav.garg PCIe host adapter card is PCIe x16 2.0. The PCIe lanes are dynamically assigned to each GPU in expansion system. So, you will get full bandwidth as long as you are using a single GPU per expansion system. But, bandwidth is divided in case multiple GPUs are used.

 

Thanks Gaugrav for the information.

So you used PCIe expansion box to handle the multiple GPUs then hook up the box through a PCIe host adapter? I think this is a good solution but if we aims at high bandwith applications, it would not be enough to make us happy

Roto

0 Likes
jross
Adept I

Opteron 8-GPU systems?

@rotor
PCIe would split 16 lanes by the number of GPUs.  Additionally, since it's PCIe 1.0, the bandwidth is decreased by two.  So each of the four boards would have 1/8th the bandwidth of a dedicated PCIe 2.0 x16 slot.  That may or may not be a problem depending on your application.  Look into gamer-grade motherboards instead of workstation-class for lots of PCIe slots.  Unfortunately, most of the information I've seen and personal experience suggests it's not very simple to build a functional system with more than 4 ATI GPUs.  There seem to be a lot of driver/kernel/BIOS hacks required to make it work and there's no single recipe out there.  Only a few people claim it works.  Details of those systems are very slim, and for the most part, I agree with empty_knapsacks first comment.  This sitation may change in the future, however.

0 Likes
gaurav_garg
Adept I

Opteron 8-GPU systems?

@gaurav.garg
Was there a particular bootup process? X server configuration?  Special device permissions? Runlevel?


No, it was normal bootup without any hacks from ourside. We didn't configure x server manually, it was configured by aticonfig. We used default runlevel, 6.

We were initially installing ATI catalyst 10.2 and were facing similar issues with 8 GPUs that other users have posted on this forum. But, catalyst 10.4 got installed smoothly without any hacks from ourside.

0 Likes
alxvry
Journeyman III

Opteron 8-GPU systems?

Arakgeeta,

Were you able to find any 4GPU Opteron Systems?  From what I understand, Tyan now has Opteron platforms with this capability:  http://www.tyan.com/product_SKU_spec.aspx?ProductType=MB&pid=687&SKU=600000213

0 Likes
moozoo
Adept III

Opteron 8-GPU systems?

Originally posted by: jross @rotor There seem to be a lot of driver/kernel/BIOS hacks required to make it work and there's no single recipe out there.  Only a few people claim it works.  Details of those systems are very slim, and for the most part, I agree with empty_knapsacks first comment.  This sitation may change in the future, however.


This url has some details on the driver/kernel/BIOS hacks

http://fastra2.ua.ac.be/?page_id=214

I'm hoping that a motherboard having a 64 bit EFI BIOS would resolve this by having an option to allocate PCIe address space above the 4Gb limit during startup.

Its possible that the I/O port space issue might need to be address by GPU designs that require less space.

 

0 Likes
moozoo
Adept III

Opteron 8-GPU systems?

Originally posted by: jross @rotor There seem to be a lot of driver/kernel/BIOS hacks required to make it work and there's no single recipe out there.  Only a few people claim it works.  Details of those systems are very slim, and for the most part, I agree with empty_knapsacks first comment.  This sitation may change in the future, however.


This url has some details on the driver/kernel/BIOS hacks

http://fastra2.ua.ac.be/?page_id=214

I'm hoping that a motherboard having a 64 bit EFI BIOS would resolve this by having an option to allocate PCIe address space above the 4Gb limit during startup.

Its possible that the I/O port space issue might need to be address by GPU designs that require less space.

0 Likes