Radeon HD 5770 supports PCI Express 2.1 up to x16
Radeon RX 550 supports PCI Express 3.0 up to x16
PCI Express 2.1 x16 is PCI Express 3.0 x8 (in terms of Bandwidth).
While your Motherboard does support PCI Express up to x16., keep in mind this is restricted by whatever your CPU Supports.
As PCI Express 3.0 was in it's early adoption stated in 2011 (when the last LGA 775 Processors were produced)., this is why you'll have few Lanes.
The actual bandwidth should be identical though and more than enough for the RX 550.
If you want to use anything more powerful, I'd strongly recommend a more modern System.