After much research I pulled the trigger on this X670E ProArt board after reading about its bifurcation support. This board has 3 PCIE x16 slots and only slot 1 supports bifurcation which is fine.
I purchased an Asus Hyper M.2 x16 PCIE card with 4 M.2 slots for NVME (along with 4 brand new 2TB NVMEs to populate them). I have been happily using 4 NVMEs on that card with PCI bifurcation enabled with x4x4x4x4 mode. To enable this I had to set PCIE Slot1 to "RAID" mode which supports "up to 4 devices" which is exactly what I wanted.
In addition to the 4xNVMEs via that card I have an NVME connected to m2_1 for primary drive, a second NVME connected to m2_2 for VM use, and 2 sata HDDs (which will eventually become 4 sata HDDs) for more storage and VM use.
Unfortunately after setting up several ZFS pools using these drives, doing weeks of performance tuning on said pools and configurations, and then moving tons of data around, etc I was finally ready to move the GPU from my old box into this new one. However once I plugged in my GPU to PCIEx16 slot 2 my 4 drives turned into 2!
After looking at the BIOS again it seems that PCIEx16 slot 2 was in x8 mode and PCIEx16 slot 1 was still in "RAID" mode but I still only see 2 of the 4 NVME drives!
Does simply using slot 2 reduce the capabilities of slot 1? Is this expected behavior? I absolutely require x4 bifurcation on slot 1 and I need my GPU connected to one of the available PCIEx16 slots. Slot 3 is far too low and won't fit the card. I don't even mind it being x8 speed I just need it to work well.
Any help is appreciated as I'm not sure how to proceed. I've invested not just a ton of money on the many parts I didn't mention which are outside the scope of this post, but I spent many many weekends customizing and building up the various things I need. This is a huge disappointment to say the least.