Hi,
So a follow up to my problem as it appears to be resolved now. The issue was not with the AMD drivers but the BIOS from Sapphire.
When I bought the card the first thing I did was to download the latest BIOS and flash it, mostly according to instructions. The card comes with two BIOS switches and I only flashed the left (low power) one, and that is the switch which has caused me the most issues.
Recently I remembered that there was another BIOS switch and tried using that right one instead, and without any other changes I immediately saw huge stability improvements. Still my system could freeze up so I contacted Sapphire, they requested a VBIOS copy and told me to flash it, which I found strange since I had already done that.
Long story short, after re-flashing it, the VBIOS got properly updated, and since the support from Sapphire spoke very little English it was a bit difficult but they did say what I did initially should have worked, so I am a bit confused as to what went wrong the first time, though it seems to have been either my fault or the BIOS updater silently failing.
I am also a bit confused as to why Radeon Software says the BIOS Date is from 2020-02-25, while Sapphire website says release date in January, and why the BIOS Part Number differs from the one used by Sapphire.