I am having the exact same problem since I installed my new RX 6600 XT, I upgraded from a RX 570 4GB and never had that problem.
My guess is that the problem is at current driver version with RDNA architecture GPUs.
Doing Factory Reset also solves the problem temporarily for me, but it is not convenient to do that every few hours.
Specs:
Ryzen 5 5600G
2x8GB Team Force DDR4 3000Mhz
MSI MAG B550M MORTAR WIFI
AsRock Challenger RX 6600 XT at PCIE 3.0, even though the PCIE slot supports 4.0, the processor does not.
AMD Software: Adrenalin Edition Version 22.3.1
Windows and AMD drivers installed at M.2 slot SSD