There isn't a huge difference in performance between the RX 480 and the 580. You would essentially be cutting your performance in half in games that scale well with multiple GPUs
To Crossfire or not all depends on what YOU play or plan on playing the most. If most of your preferred games handle poorly with Crossfire, then it wouldn't make much sense to go that route. Since you already have 2 Rx 480's, I'd hold on to them as ajlueke said, you would be dropping potential performance! For games that don't scale well, simply disable the Crossfire mode in the Crimson software for that specific game.
It sucks that people fall for marketing fails. RX 480 = RX 580. The difference is too small. RX 580 is a refresh. Think of it as a car which comes out every year even though they will look the same as the year before. Keep using CFX for now.
Oh you're saying you're sitting on a goldmine then! I would sell the 2 480's and put it towards a Vega - then for this advice, you can send me your extra 580 when you have your shiny new Vega? (just kidding btw)
You'll see no performance boost if you try to CFX that 580 with a 480 and if the games you play benefit more from a single, keep the 580 and vice versa.
Just don't forget to send me whatever you're not keeping so I can sell it to buy a shiny new Rx Vega - (again, just kidding)