Same thing happened for me and my RX580. Updated to 20.9.1 from 20.4.2 and performance went way down. My GPU usage also kept going from 100% to 0% to 100% and so on. Spiking up and down, even in idle.
I decided I had to rollback and used DDU to uninstall the driver and reinstall 20.4.2. Then the problem was gone but when I started No Mans Sky I noticed horrible jagged lines, low res shadows and weird LOD / texture pop in around the camera. I then found that this problem was present across all games. It's like anti-aliasing is not working properly anymore. And like I said the shadows and LOD are off too.
One mistake I made is use DDU without safe mode. I advise you to run DDU in safe mode exclusively. Look up a how-to to do it properly.