In Fortnite,i get stutterings while gaming and the lobby. Also when I'm joining a match it loads slower than my previous rx570. In games like GTA and cs:go I have low fps. I don't know if the problem is the RAM the 5700 XT or the drivers.
My rig: 2700x
asus strix b450-f gaming
5700 XT Saphire Pulse
2400mhz ddr4 corsair lpx
650w 80+gold seasonic focus
No you are not alone. Lots of people reporting driver issues. Many are saying the newly released 20.5.1 driver helps. Install it and choose clean install.
I have pretty much the same set up and I experience the exact same issues, stuttering in the lobby in Fortnite and most of the time low fps in GTA. So I don't think it's your ram. If you find something, please let me know! I might try the newest drivers, I am currently using the previous version. I'll update if it works.