It is possible that the issue is related to the GPU's memory management. When the VRAM is being heavily utilized, the GPU may try to free up memory by swapping data in and out of the VRAM, which can cause stuttering and lag spikes in the game. This could be a problem with the GPU itself or a driver issue.
To troubleshoot the issue, you could try updating your graphics drivers to the latest version, as this may improve the GPU's memory management. You could also try reducing the graphics settings in the game to reduce the VRAM usage and see if that resolves the issue.
PC Hardware Specialist