Could be a bitrate issue. Pixely look on stream usually can be attributed to low bitrate. It can also depend on what is being streamed, some games for example have a lot going on at the same time (dense grass fields, a lot of particle effects going off) which causes streams to be "pixelated" and there isn't much you can do about it since it's due to low bitrate on platforms side.
When it comes to streaming on AMD GPUs, through OBS specifically, I have some experience with it on an older GPU (RX 480) and it was perfectly fine. I'd say it's most likely some settings that need to be tweaked (most likely bitrate related stuff).