Let's talk virtualization

Discussion created by axero on Oct 25, 2011
Latest reply on Nov 14, 2011 by axero
FLR support and VT-x/AMD-v like extensions for GPUs.

There has long been an ongoing battle on which operating system is superior to the other and with virtualization technology this battle is soon coming to an end. The truth is that no operating system is superior to the other. It is for example well known that Windows has some severe flaws at the low level when you look at things "under the hood" but it is unmatched when it comes to the abundance of software and computer games. It is also well known that ZFS which is found in Solaris based operating systems is a file system that is unmatched in terms of reliability and safety against data corruption, which is a growing concern as larger and more dense storage hardware has become less reliable in the past few years (many more hard drives have failed on me compared to 10 years ago). I'm very concerned about these issues and I can no longer trust a hard drive in a Windows environment to reliably keep my data. Linux has many advantages in terms of system resources efficiency and stability. This list of operating systems and their advantages/disadvantages can go on...

So why should I have to choose? Why can't I take advantage of all of these benefits from these operating systems and get the best of all worlds? The answer is that I can, by virtualization. In the past few years the world has seen exciting development in the Xen community and really powerful extensions that enhance the capabilities of a virtualization such as the Intel VT-x/ AMD-v and the Intel VT-d / AMD-Vi (IOMMU) have become widespread among desktop hardware whereas it has been commonplace among enterprise-level hardware for quite some time by now.

Several platforms such as MacOSX, Linux and Windows are running at the same time on the same machine while you can run e.g. Battlefield3 that requires hardware acceleration even on a virtual machine.

(url to full-size picture:

So it is quite evident that the role of an operating system is going to change considerably in the future. The operating system that runs on-the-metal is going to become a simplistic hypervisor that manages simplistic virtual machines. The operating systems as they are today will shrink into so-called wrappers that merely supply the frameworks required to run a particular piece of software (such as .net, Visual Runtime etc).

So there will be a separation between the hardware and the operating systems by an abstraction layer where different wrappers (that used to be operating systems) share the underlying hardware with each other. There will no longer be a question whether you use Windows, MacOS or Linux. You just use whatever you prefer as a base OS and use whatever is needed to run the applications you want, which in reality could mean that you run several operating systems simultaneously on the very same machine.

This separation has already begun, ZFS is a good example of that. The ZFS file system looks at the hard drives as a storage pool and the user is not concerned with the physical characteristics of the partitions and where the sectors begin or end. I didn't like it at first but later found that this approach is ingenious. So I see it as a natural step that the rest of the hardware will undergo the same transition. I also think a lot can be done with the UEFI framework in this regard.

The latest advancement in the virtualization technology is the set of IOMMU extensions which allows virtual machines to run directly on selected parts of the hardware on the host. This means that I can run say, Linux on-the-metal while playing Crysis 2 on a virtual machine that runs directly on the GPUs. Here's a video showing Unigine Heaven running on a virtual Windows machine inside Ubuntu on a dual GPU setup:

This is called PCI passthrough where PCI slots are passed through to the virtual machine or VGA passthrough where also the VGA-BIOS mappings are sorted out. In another setup I may want to run Windows on-the-metal and pass through a whole hard disk controller to a Solaris machine where I run a secured storage pool with redundancy (e.g. raidz3). For ZFS to give proper protection against data corruption it is an imperative that it runs directly on the hardware and not through a virtualized abstraction layer. There currently is no support for IOMMU on Windows hosts but that will change eventually, our hopes lie with hyper-v, VirtualBox and VMWare.

However, there is a lot to be done and the purpose of my post in these forums is to address this. For PCI passthrough and VGA passthrough to work it is a requirement that the hardware supports function level reset (FLR) which is a feature that allows the hardware to be reset and reinitialized at any time on a running machine (i.e. at function level). FLR is standard on QuadroFX cards and nVidia supply patches that enable FLR on Geforce cards upon request.

Another issue is that current virtualization technologies only support passthrough of entire GPUs to virtual machines and GPUs can currently only be shared through emulation which makes it impossible to run applications that rely on hardware accelerated 3D (such as DirectX games). This situation is pretty much the same as where the virtualization was before the VT-x/AM-v extensions were introduced. The CPU instructions had to be emulated on the VM which severely degraded the performance on that machine. When VT-x/AMD-v came, virtual machines could be run directly on the CPU with almost no overhead at all.

So I would like to suggest similar extensions that allow the GPUs to be shared over several machines just like CPUs can be shared via VT-x/AMD-v.

So my suggestions in short:

  • Work to get FLR support to become a standard feature among hardware
  • Develop Intel VT-x / AMD-v like extensions for GPUs allowing for GPU power to be shared seamlessly among VMs and the host just like the CPU