So after a lot more searching and thinking I've finally gotten to the face of the issue. The card runs at 71C~ in idle not doing anything and the only thing I can think of is that the memory clock is completely stable and never seems to downclock like the GPU does. Wanted to install old drivers where this didn't happen but could only install CCC then and not the drivers. So is there a solution other than downclocking the memory on my own?
Only running 1 monitor and W10 x64.