I'd be useful to know which Operating System you're using...
Now if it's any other version of Windows than 10., you'll need to grab the AMD OCAT Performance Analyser... this will give you a full breakdown of what your GPU is doing at any given moment.
You can get it from GPUOpen.com
If you're on Windows 10, you have a different and easier method... as the Task Manager (right-click the Task Bar > Task Manager) has built-in Performance Information for your GPU (since v1904)
Unfortunately the Resource Manager doesn't yet., not sure why... but still you can look on the Performance Tab, and it should show an aggregated overview of what's going on with your GPU (and if you have multiple it will break down which, with Primary being GPU 0, Secondary GPU 1, etc.)
Switching between the Graph Info (i.e. 3D, Video Encode, TrueAudio. etc) you can see what area of the GPU is utilising the Graphics Memory, but you can also go to the Processes Tab and it'll list "GPU" as well as "CPU" as the selectable columns. You can trace from there what's causing the issue.
It isn't the Drivers, I can tell you that for a fact... it's likely another program that's silently running in the background that has a memory leak.
I know that Edge Chromium had such an issue for a while., although it was fixed quite quickly; but you can look there and disable / shutdown what's causing it.
