Well, this is odd. I plugged in a Dell DP to HDMI adapter to use the other adapter with a laptop, which refused to show picture. I then switched back to the Club 3D adapter, reset my TV power, and now I'm getting 4k@60 on my desktop again . Not a great solution, but it's solved for now, maybe this will help someone?
I'll update if I find a way to reproduce, but I am SURPRISED it worked at all. One hint, that I will look at if it breaks again, I did mess around with the settings in CRU prior to doing all this the other day now. Something I did there might have tricked the driver to register my display properly after that reset.