Ok so i have tested few things, and it looks like this might be issue with the size, or maybe codec.
When i try to record instant reply and save it which is a little longer i was able to save one of 15 min recording but i had to switch from AVC to HEVC and also i had to switch from 60 to 30fps recording.
And even with that, it takes about 1 min from the moment i click save instant replay, for replay to be saved to the disk and to get notification in upper right corner that replay is saved. During that time if i try to play game actively or go to desktop or switch to steam application, replay is not saved.
So only it's saved if i click to save replay and don't touch my PC for that time until i get notification that replay is saved.
So this might looks like memory allocation problem.
Do anyone know where do relive store instant replay while it's not saved yet? Is it in RAM or somewhere on HDD or does it use default set location in relive (mine is D:/Recordings)
If it use RAM as temporary storage before you click save replay than this would explain the issue of crashing(sort of)
EDIT: I noticed 2 more things now, i found a cache folder and it's in the same folder as you set it up in settings (mine is D:/Recordings), and also when instant replay reach max setup time (i set it to 15 min) it starts to read/write alot of info on HDD and crashes, causing to loss instant replay and cache folder just gets bigger and bigger every time it crash.