First thing I would do is install the HD 5xxx GPU again and upgrade to Windows 2004.
Then after updating Windows to the latest version, install the RX 570 and see if it works or not.
Post back what is occurring.
To connect to a different GPU Output you will need an Active HDMI>DVI adapter to see if your Monitor is working via HDMI output. Same with the DP output.
But that is not important for the time being. First upgrade to Windows 2004 and see if your problem is the same or not.
EDIT: I was thinking before you upgrade to Windows 2004, remove all traces of the current AMD Driver using DDU in Safe mode with the internet disconnected and deleting C:\AMD folder in your Root Directory. Then install your RX570. It should boot up to Windows with Microsoft Basic Display Adapter since there is no AMD driver installed. Than go to Windows Settings - Display and choose 1920x1080 and see if your monitor works with that resolution using Windows Microsoft Display driver.
If it does, then go to Windows Device Manager under "Display Adapter" and click on your RX570 then click on Driver update and let Windows install whatever version AMD driver it has. See if you continues to get the same result of no video Out of Range or your Monitor is now working with 1920X1080 Resolution with the AMD Driver.
If it is, it might indicate that the latest AMD Driver may not be completely compatible with your old version of Windows.
By the way, you said you install the HD 5xxx GPU and everything worked correctly. That GPU uses very old version of AMD driver, so it is possible when you installed the RX570 without removing the HD5xxx GPU driver and might be causing the problem. The latest AMD driver is not compatible with the HD5xxx GPU card.