Ocasionally windows will refuse to play a file - VLC can alleviate this...unless it's a null or corrupted file.
Since this is such an old orginal post, I'm going to suggest making sure you're using the h264 encoder for relive - I don't think the R7 supports h265 hardware encoding, but I may be wrong.
Yes, that should be h264 / AVC, mp4.
When testing, they setting it to allow recording of desktop, some games run windowed fullscreen, recording desktop enabled may subvert any nonsense going on that may be causing blank video files.
I have had this problem before on both vega 64 and rx480 cards. Ticking the desktop box seemed to fix the problem.