I have now tested this and confirmed that it only happens when using my Sapphire 6750xt nitro. I have tried it with two of them.
1070 I tried had no issues.
I have reinstalled drivers.
played with different res and monitors.
I logged usage and didn't notice anything spiking.
I uninstalled drivers and AMD adrenaline.
There has to be some way to see what is causing this