As it turned out, ReLive Radeon does not support 21:9.
When recording in AVC codec (no others), native resolution 21:9 2560х1080 is compressed to 16:9 1920х800. In the result, the image becomes murky and of low quality regardless of the bitrate.
Question: Why AMD does not fix this problem? Other programs record image record everything with no problems and compression.