The cards finally arrived:
These are the 'workstation edition' of the Powercolor 7990 Devil13; to the best of my knowledge we are the only customer to have these. Currently testing them with the original air coolers prior to fitting the custom nickel-copper waterblocks and integrating them all into the same machine;
Host machine has dual overclocked 8-core Xeons, 128 GB DDR3, quad SSDs, 10gig ethernet, dual PSUs, quad triple-fan radiators etc. I can happily confirm that the 'impossible to disable crossfire on dual-GPU cards' bug that made the 5970 and 6990 unusable for GPGPU has been fixed for the 7990; there was lots of flickering and Catalyst Control Center crashed when we disabled CrossfireX, but after opening it up again CrossfireX was finally disabled and so far dual-device tests are working fine. Will be running 8-GPU scaling and overclocking tests next week; if it works well then we will make this spec our standard developer workstation for the next project.
That looks really sweet. Looking forward to your scaling and overclocking results.
What types of applications are you building these machines for? Do you sell them?
The project is a production deployment of a C++ to OpenCL auto-parallelising cross-compiler, based on supercompilation techniques. These workstations are used to run the supercompiler itself (which is extremely compute intensive) plus testing of the application code, which consists of many different workflows and codebases currently running on several thousand 12-core Xeon blades. The production severs are currently planned as standard 8 x (next year's) Firestream 4U rackmount boxes, but we do have a team looking at the possibility of liquid cooling for those as well.
I don't know if they'll be available as stand-alone machines - currently they come as part of a GPGPU consulting / software license / dev hardware package - but certainly I'll let you know if they go on sale anywhere. Certainly the series production version will be a bit tidier inside than the engineering prototype in the photo!
Build in progress. Will clearly have to truncate the brackets with the bandsaw in order to get four cards onto the motherboard (the 10GBASE-T NIC will have to go on a flexible PCIe riser, mounted elsewhere in the case);
Had to remove the backplates to physically fit the top card into the slot; the clearance on the ASUS Z9PE-D8 WS is very tight. Will mount individual passive heatsinks on the rear memory chips. Seems a shame to throw away all this carefully engineered copper and aluminium, but alas Powercolor didn't want to ship bare cards;
Out of curiosity, how are you watercooling 1500+ watts? I'm running a similar machine with 8x HD7970 on an 8-slot motherboard, but I had to go for extender cables and air cooling, because I failed to find reliable water cooling without making the water boil under full load.
Quad radiators each with triple 120mm fans;
This is the loop configuration;
Temperatures and noise are both quite low so far. 8 x 7970 is the expected production config, /if/ we can demonstrate good enough reliability with the AMD Drivers (looking dubious at the moment), the 4 x 7990 is specifically to fit into a desktop machine.
Nice progress. Do you mind sharing what kind of problems you have encountered with the drivers? What version of the drivers and AMD APP SDK are you planning to use? What OS?
I guess the same problems that I experienced. Booting works fine, but the AMD drivers demonstrate problems. Under Windows they will either not recognize all the cards or BSOD'ing with an error in atikmdag.sys. Under Linux they can be made to work: I wrote a small tutorial here how to make it work under Ubuntu. http://devgurus.amd.com/thread/159019#1280510
Using 4 radiators, wow. Yes, that can definitely keep it cool, even with overclocking beyond the 1125MHz that air cooling allows, and probably without sounding like a vacuum cleaner like mine does.
Out of curiosity, how much did the entire watercooling system cost? Looking up some watercooling parts suggests figures beyond the cost of the cards.