There are problems when capturing games with VSync enabled. It results on low framerate, freezes and stuttering. When removing VSync, game capture is OK.
Vsync disabled, 45 fps cap
One solution you may come up with is disabling Vsync and setting a fps cap (30, 45, 60). Yes, that works. But not for all games. In The Witcher 3, you have to enable VSync to avoid tearing (or have a FreeSync monitor). In-game cap or Radeon Relive FRTC don't prevent tearing. Actually, 60 fps game capture for TW3 isn't possible at this moment due to this Radeon Relive bug.
I already did a clean installation. The issue persists.
Windows 10 Pro 1709