I purchased the Radeon RX580 for mid 2012 Mac Pro. Can not get correct resolution.
Support for iMacs are provided by Apple. Go to apple.com. If you bought a card that is not designed for a mac like an off the shelf RX 580 for Windows it may work but may require a special bios. If it was not sold to be mac compatible, if such a bios exists it is likely 3rd party and would void a warranty. If you bought a card that is supposed to be Mac compatible out to the box then ask the support team of the company that made that card. Some cards people can get working with custom information that has to be entered into the terminal, but I don't know what offhand. It will take some research on your part.
The forums at www.tonymacx86.com have a lot of good information.
Another great mac site https://www.mac-forums.com/forums/
Another good mac help forum Mac Support
The apple community forums Official Apple Support Community
if you didn't buy a card specifically made for mac and you don't want to mess with trying to get your card working.
Other World Computing sell a lot of mac ready add in components.
Search Results for video cards at MacSales.com
Need to post your question at Apple Forums concerning Drivers for your Mac Pro and the RX-580.
These are the AMD Apple Drivers for MAC: AMD Drivers and Support for Radeon, Radeon Pro, FirePro, APU, CPU, Ryzen, desktops, laptops
Also the RX-580 doesn't show any Apple compatible drivers in its download page. Radeon™ RX 580 Drivers & Support | AMD
Edit: you can try to use a small utility to make Custom Resolutions called CRU (Custom Resolution Utility). Not sure if it will work on a Apple computer or OS.
Retrieving data ...