I recently purchased a rx570 and had noticed it wasn't performing very well in a lot of my games, so I opened up task manager and it was only using 1 out of the 8 GB of available physical VRAM. I have my drivers up to date and nothing is wrong with any other part.
