I have Dell Studio XPS 8100 (bought in 2010) with a AMD Radeon HD 5450. I would replace the video card with a more powerful (enough to play Fallout 4), for example with Sapphire VGA Radeon R9 380 4 GB. Is this model compatible with my desktop? Otherwise, with what model of video card I could replace it (~300 euro)?
You'd have to take the PSU into consideration.
An HD5450 gpu requires approx. 22 Amps and 300-350 Watts.
An R9 380 gpu requires 28 Amps and 500-550 Watts.
I searched for your desktop system and it seems you currently have a 350W PSU, right ?
So, you must upgrade your PSU first (and make sure it fits as well) before going for an R9 380 gpu, or any modern mid-to-top class gpu for that matter.