Hi Triviq.
As an youtuber I have faced this issues multiple times, sometimes do still happen, but not to the same extent.
Not sure if this will help you, but for me reducing the audio quality to 16 bit 44khz (Dvd studio quality) did helped me out.
I was concerned that I would notice the downgrade on the audio, but fortunaly I was unable to notice.
If you are a windows 10 user:
1. Go to Control Panel
2. Sound
3. Speakers, select your speakers and go to it's properties
4. Advanced tab and on default format, select 16 bit, 48000hz (DVD Quality)
5. Restart the PC and try out.
It worked for me, but that means nothing.
PS: People also say that you can use 24 bit, as long as it is set to 48000hz you should be ok.
Cheers