Forgive me... I think you misunderstood me. It is not in my interest to have more VRAM than RAM. Infact that would complicate things, since if it would be a simulation software of some physical system, the entire system could not manifest in host memory if there were more VRAM then RAM. But if I had 16 dual GPUs let's say (let's forget about BIOS limitations for a second, bandwidth would be enough), that's 96GB VRAM. That is why it would be good to have a motherboard somewhat more serious than ASUS P6T7. I know it's very elegant to say "forget about BIOS limitations" when that's the only complication out there, and it's extremely to get around just alone this issue. However, if I understood it correctly from your posts, practically all limitations to the number of GPUs hooked up to a single host could be lifted (even with present BIOS of motherboards).
About this EFI boot... could someone enlighten me as to what the issue is or how it operates and what is this homework that should be done?
EFI have much more abilities and get rid of some limitations like for >3TB HDDs. So also some limitations for adressing etc. are no longer there, but as far i know, the System/Board builder doesn´t use them.
But i don´t know if this is really true. Thinks like that are very difficult to say, because nearly nobody do something like this.
It is always the chicken egg problem -.-
Having such a system I am not even sure whether it would make sense to have all of the VRAM mirrorred to system RAM at any point in time. But that's a different matter.
Regarding the EFI boot. AFAIK the drivers by both GPU vendors currently require a normal BIOS boot, the notable exception being their OS X drivers. When doing an EFI boot the driver will fail to operate as the expect the classical video BIOS bootstrapping to have taken place. There is some hack to this from EFI, but nothing I have seen working reliable.
So do I take it correctly, that EFI boot is something similar to what FASTRA II has done, but in a more standardized way? That EFI does not take care of initializing devices, rather then leaving that to the drivers themselves?
Mirroring the contents of VRAM to RAM can be avoided in most cases, but it is a hassle to work around in most cases, and in rare cases not possible at all.
We would've built such a serious machine (16-32 GPUs per node), but it's impossible.
I don't mean to detract from what you want to do, but why not consider building a cluster with 4 nodes with 8 GPUs per node? If your algorithm/application is not I/O intensive then a cluster seems to be to be a reasonable alternative. And if your algorithm/application is I/O intensive, wouldn't that many GPUs sharing the PCIe bus quickly saturate it, leading to a performance bottleneck. Plus the cost of such a specialized system grows exponentially compared to scaling with commodity-built systems. Sorry, maybe I'm too naive about this extreme (8+ GPUs) HPC workstation.
Check out SnuCL for an free and open-source OpenCL framework that abstracts away all the GPUs in a cluster and makes it just as easy to program as a single system.
I believe the FASTRA team modified the Linux kernel so that it ignored (or rather extended) the information that the BIOS handed to the Linux kernel. Basically, the BIOS initialization was wrong and they manually corrected it. It's a level of kernel hacking where most people cannot or wont go.
I'm not familiar with graphics bootstrapping via EFI. If it operates correctly for 8+ GPUs, then there should be no need for a Linux kernel patch.
The VRAM/RAM issue is separate from the rest. Find a system that will support enough 8GB or 16GB DIMMs if it's an issue. You're probably out of luck with the ASUS P6T7.
Just wait a while for the Dual-Sockel 2011 Boards. There will be some with lots of PCI-E 3.0 slots and lots of DIMM-slots also. The RAM Problem is definitly no problem.
Would be great if someone of the AMD guys could write something here.
This thread was singled out in the AMD Developer Central Newsletter yesterday as "one of the popular topics being discussed right now" and in this forum and "you'll notice the AMD moderators to be more responsive".
Out of all of the threads where 8+ AMD GPUs has come up for discussion in these forums, I don't believe the community has received input from AMD. We realize Skyrim multi-GPU driver support is more of a priority, but it would be a really big psychological and marketing win if AMD had a customer show off a 10-20 Tahiti GPU machine that could be used to smack around the competition.