I have the same issue on 144Hz monitor in 1440p.
Driver 20.2.1.
Windows 7
Right now I have 3,5 GB taken...
On 144 Hz there is also problem with VRAM clock - it works on full power 1750 MHz even on idle...
I've created a ticket on AMD support - but they just refuse to reply on my questions and instead they want me to shuffle drivers like an idiot...