I have a kernel that executes a few times per second. There are 2 anomalies that I can't figure out.
1 (less important). Every 3-4 seconds the gap between the end of the kernel execution and the start of the next kernel execution gets bigger, then backs to normal. I can reduce the time of the gap by using multiple queues, but anomaly stays.
2 (more important). Every 7-8 seconds the execution time of the kernel gets bigger for one iteration then backs to normal. It doesn't matter if the kernel executes once or ten times per second. A slowdown always comes after 7-8 seconds.