I´m having that same exact issue.
Upgraded from a 980ti to an 5700xt Nitro and my monitor is the Alienware AW3418DW.
I managed to get it working at 1440p@85Hz but not higher .
As soon as I try to set it to 100hz my Monitor blacks out and resets after about 10sec to 85hz
I thought it might be an issue due to the monitors gsync module
also it only works on 85hz in windows
ingame it sets automatically to 60hz
this really sucks
maybe its some Nvidia drivers i failed to throw of of my PC.
Going to keep you updated if I find a solution
Greetings