I built my system a few days ago, and I am having some problems with compression benchmarks. My system works great for the most part, but the 5800x shuts down due to high temps in compression benchmarks. I have had no problems in other benchmarks like cinebench or gaming. Temps are usually pretty good too. 30-40c idle and 55c under gaming. I do run a little undervolt usually, but the issue is still there running stock. Running a 280mm aio (nzxt z63).
850w Corsair RM
Nzxt n7 B550
When it shuts down are you sure its due to overheating?
The Maximum Operating Temperature for your 5800x is 90c. For the processor to shut down due to overheating, I imagine the processor's temperature must go above 100c. At 90c the processor automatically starts to throttle.
The 5800x has a high TDP Rating of 105 Watts. Which means you will need at least a CPU Cooler rated at 150 watts or higher to keep the processor within its temperature range under any types of loads.
The NZXT CPU Cooler you have should be strong enough to keep your processor from overheating since it is compatible with AMD ThreadRipper Processors which have very high TDP Ratings.
Download and run OCCT CPU and PSU tests and see if your computer shuts down or not. Keep a close eye on your PSU outputs and CPU Cooler fan speeds and CPU and Motherboard Temperatures.
A weak or bad PSU could cause your computer to shut down or if the Motherboard's VRM or GPU is overheating that is as long as your processor's temperature is not going over 90c during your tests.
I would set BIOS back to factory default which will get rid of the slight undervolt you have set by doing a CMOS Clear or through BIOS itself. Then see if it shuts down. This will eliminate any BIOS settings you may have set that might be causing your issue.
If it still shuts down with BIOS set at default then you can go back to setting it the way you had before.