There are problems when capturing games with VSync enabled. It results on low framerate, freezes and stuttering. When removing VSync, game capture is OK.
Vsync disabled, 45 fps cap
One solution you may come up with is disabling Vsync and setting a fps cap (30, 45, 60). Yes, that works. But not for all games. In The Witcher 3, you have to enable VSync to avoid tearing (or have a FreeSync monitor). In-game cap or Radeon Relive FRTC don't prevent tearing. Actually, 60 fps game capture for TW3 isn't possible at this moment due to this Radeon Relive bug.
I already did a clean installation. The issue persists.
Windows 10 Pro 1709
Watch out as Vsnc is more related to Nvidiaworks supported titles. I advise do you manual settings in global settings & use your globals for game control adjust the rest in game. Run Chill with a modest cap of 40 on the low & 70 as my target with my card capped to 60fps. I run with enhanced sync settings Chill enabled & adjusted game settings. I can Capture Witcher 3 on Medium setting with frames capped to 60 in game with V sync enabled. I was doing a Vermintide 2 run the other night didnt effect it on 18.2.1 drivers with a sapphire pulse RX560OC 4G with only volts & fan settings bumped in wattman. Giving it a few more volts in wattman made a huge diffrence with Relive on an non vsnc of free sync monitor is probly 10 years old. I dont run an OC on this card the biggest thing is they lock the volts back from factory to meet the engery requirements. I know 3 or 4 others that have just bumped the volts in their cards & made a huge diffrence with Relive as well.These are what I run for my build as is works every 1 has there own opinions. What works for some doesn't for others these are the things that work for me tho be others with more opinions. Just share what works for me.