So after some research, I've noticed that the problem occurs after I Alt+Tab to go from game to other application and back. After that VRAM skyrockets and stays high even in idle Windows. So if any of you have an explanation for that, I'd love to hear it...