Found the reason! It does not depend on the drivers in any way. I have an RX580, a problem with audio interrupts (solved, it seems). Everything was done. As a result, I noticed that in idle time the video card switches to PCE-e 16x 1.1 mode, with a small load it returns to 16x 3.0 and at this moment an interruption occurs. Solution: I increased the minimum frequency from 300MHz to 310Mhz. Now 1.1 does not switch and the problems are gone!
Sorry for the grammar, online translator