The 7900XT seems to aggressively downclock VRAM when it gets the chance. As a result in older, very CPU bound games where there is very little GPU load, VRAM downclocks and performance takes a nosedive (often with severe stuttering). Games like Starcraft 2, Genshin Impact, etc. If you turn on the metrics overlay in Radeon Software and observe VRAM clock dipping while in-game, you've found your culprit. If you only have one monitor I'm not sure what you can do aside from try to increase GPU load (increase render resolution, higher quality settings, etc).
But if you have multi monitor you can try the new 23.1.2 drivers where VRAM doesn't seem to downclock anymore from 2487MHz. Or you can keep both monitors at high refresh rates at all times, which should keep VRAM clock high as possible.