I tried to investigate this a little further. Re-Installation of the driver is not necessary. There is a reset function for adrenalin software which also does the trick.
I also benchmarked the card with BasemarkGPU the effect is viasable there also. Witch a fresh driver I score about 20000 for Vulcan, 19500 with directX12 and 19000 with OpenGL.
directX and Vulcan degrade to about 1000 points less, OpenGL seems not to affected that much.
Reseting Adrenalin brings me back to higher values (after appling custom performance options in adrenalin, I set: max frequency 2650MHz, 1075mV, VRAM fast timing, 2150MHz).
Maybe it's worth mentioning, that after a reset the performance is also is not completlly idetical. This difference is visable but not more than 1-2%
So the question would be: What changes are made in the software just by using the GPU in a game? These changes must be responsible for performance degradation as is is reversible by a reset of the software...