I couldn't find a post about this in the past, so I decided to let everyone know how I made it work for me.
Firstly, I play on a 24" 1920x1080 resolution screen with a 8GB RX480.
I simply thought AMD didn't support VSR in OpenGL due to how poor Radeon OpenGL performance still is in a lot of games for me, and the lack of the Radeon Overlay in OpenGL.
But, the way to make OpenGL games (Doom 3 BFG, Wolfenstein The New Order at least) work with VSR is to set your desktop resolution to 3840x2160 or whichever VSR resolution you want to play on before launching the game. Then you need to set the same resolution in your game, but you might need to restart the game after setting the resolution in some cases.
The reason I posted this, is because with almost every DirectX game I could have the desktop resolution at 1920x1080 and then just set the in-game resolution to 3840x2160 and it would scale properly, but not for OpenGL. Sometimes trying it this way with OpenGL causes major performance issues, or it only shows quarter of the screen.
Hope this helps someone, might even apply to Eyefinity