The new VMSVGA display controller is supposed to be the default in VirtualBox 6 and earlier this year, if I remember correctly, I had a Ubuntu client running succesfully with this display controller. I haven't used this in months and decided to install a clean Ubuntu (also tried Mint) guest with the VMSVGA display controller but it no longer works.
I have a AMD Ryzen 2700X CPU and a AMD Vega 56 GPU. I'm running the latest Windows 10 (fully up to date) and the latest Adrenalin 2019 Edition 19.9.2 Optional display drivers. After installing the VirtualBox Guests Additions the virtual machine no longer loads up succesfully with the VMSVGA display controller.
In the VirtualBox log file I see the following message:
2a20.1ec0: Error (rc=0):
2a20.1ec0: supR3HardenedMonitor_LdrLoadDll: rejecting 'C:\WINDOWS\System32\amdihk64.dll': rcNt=0xc0000190
I believe this to be a "hardening" issue where VirtualBox is rejected the driver DLL file because it isn't signed. I also tried the latest Recommended driver "Adrenalin 2019 Edition 19.5.2" but this didn't work either. I'm using the latest VirtualBox release 6.0.12 but I also tried downgrading to 6.0.6, which I believe was the version I used earlier this year, with no success.
How can I resolve this issue and make use of the VMSVGA display controller?
Also created a post on the VirtualBox forums but no replies yet: virtualbox.org • View topic - VMSVGA not working possible hardening issue with AMD display driver
I have managed to solve this problem. It wasn't an issue with the driver it seems but rather the assigned memory to the display controller in VirtualBox. I increased this from 16 MB to 128 MB (the maximum) and then it worked.