So I decided to boot up some modded Minecraft on version 1.7.10 and after about 20-30 mins I was getting horrible stutters while in an interface. I went to restart my game and it crashed upon me trying to exit to desktop. Opened task manager and booted up MC again and could watch the RAM usage climb rather steadily. In a previous forum someone asked if my VRAM was maxing out as well and indeed it is.
I tested on vanilla, as well as version 1.10.2. My VRAM seems to max out first and then my PC starts pulling memory from other sources (i.e my physical RAM). I've tested Forza Horizon 5 and Rust to see if the issue persisted and it seems to only be with Minecraft.
I have an RX570 4GB, i7-9700K, and 32GB of RAM. On the latest x64 version of JAVA. I have also tried the last 3 or 4 drivers.