I have the same problem with Radeon RX 580. It has two DisplayPort 1.4 ports. Nothing connected to the two HDMI 2.0 ports.
I have two 3 port DisplayPort 1.4 MST Hubs. I understand the RX 580 doesn't support DSC so it cannot do three 4K60 displays to a single MST Hub. However it should be able to do 3 displays set to 2560x1440 60Hz. But it will not. The most number of displays per MST Hub that I can connect is two. I can only connect 3 displays total. This is ridiculous. I even tried using CRU to remove all the 4K timings from the EDIDs in case Windows or AMD drivers were being dumb about reserving bandwidth for resolutions I don't want to use...
Does AMD even test this stuff?
I can boot into Ubuntu, connect 3 displays per MST Hub (6 total), and have all the displays quickly arranged and set to 2560x1440 60Hz. Lucky the driver is open source in Linux so AMD can't ruin it. Do the Linux people just care more? Is AMD hampered by Windows requirements?
The nice thing about Ubuntu's Screen Display UI is that you can change the settings of all the displays before applying them to the GPU(s). It's so much faster (don't need to wait for each display to readjust to the new settings and don't need to click a "Keep Settings" button for every display).
For Windows, AMD needs to make their own Display Settings UI (doesn't Nvidia do that?). It's really silly that I have to use four or five different UIs (display arrangement, resolution, bpc, timing, some of these buried behind another UI). There's too many windows and too much clicking and too much wasted space and time in the current AMD Radeon Pro Software UI.
In Windows, if you select 1440p for a 4K display you might be getting a 1440p framebuffer that is scaled up by the GPU to output at 4K to the display. The Advanced settings tab in Windows Display preferences will show when the framebuffer size does not match the output resolution. The output resolution/timing can be seen in Radeon Software as well. In macOS, using the SwitchResX app, the Current Resolutions tab will indicate which modes are scaled and which modes are actual timings. Apple's UI is terrible - it doesn't show output resolution or pixel format / depth and you have no control over pixel format / depth (except the HDR setting requires at least 10bpc).