cancel
Showing results for 
Search instead for 
Did you mean: 

Processors

abriasffxi
Adept I
Adept I

PCI Passthrough does not work (at all) on X399 Threadripper

All,

I created a reddit post but also will place one here.  PCI Passthrough is completely broken on threadripper and the X399 platform.  There is not a single case reported of being succesful.  The tech reviewer who was supposed to post a video showing it has been radio silent.  I can't believe we have to deal with 0day support when a MAJOR use case of threadripper is to have plenty of VM's.

https://www.reddit.com/r/Amd/comments/6vbe6w/threadripper_broken_on_linux_for_pci_passthrough/

Will provide any information necessary if we have dev cooperation but I'll be RMA'ing at the end of my 30 days if theres no patch in place.

Thanks,

Chuck

29 Replies
noodles59
Miniboss
Miniboss

Re: PCI Passthrough does not work (at all) on X399 Threadripper

That's why I usually wait 2 years b4 jumping on new hardware designs - this is sufficient time to patch hardware errata.

0 Kudos
starlightknight
Adept II
Adept II

Re: PCI Passthrough does not work (at all) on X399 Threadripper

I have this issue as well. Due to it and other issues, my Threadripper build has been a giant unmitigated disaster unfortunately. I was extremely excited to do this build. I had hyped up the possibilities that all the cores offered specifically with VMs, Pass-through, Compilation, etc to everyone I knew, and they'd all be anticipating seeing how it turns out. Unfortunately, I had to report to them that my build has been a complete failure due to this.

For a high-end part @ $999 and affiliated motherboard (Zenith) @ $549, this is not acceptable. If I had bought the Core i9 instead, I would not be having this issue. If I go back to my older i7, I don't have this issue. I want to give Team AMD a chance as I appreciate what you guys are doing to up the available core count, but the parts have to work.

Several things greatly concern me right now after I've looked into it further that I am looking for AMD to address:

  • The Ryzen had PCI passthrough broken as well at launch, and you had to fix it with AGESA 1006. Since the Threadripper is two joined 8-core Ryzens, why didn't you learn from this? It is extremely disappointing to see this issue repeated on the higher-end more expensive part, where this functionality is most valuable. For a high end processor, this functionality should've been tested and working day 1.
  • I've been reading about the Nested Page Tables bug (NPT) in kvm_amd that has been around for...... 9 years? Why? This is an absolutely critical bug; AMD engineering should be assisting with fixing this. Having your options be that either the GPU or the CPU performance terrible in a VM is not an acceptable option and botches the entire potential of high core parts for these applciations. See: 196409 – kvm_amd nested pagetable gpu passthrough performance oddities
  • Ryzen had a critical issue with high core loads related to compilation that ended up being a hardware fault. AMD says this is a "performance marginality problem" (what?). Affected users have to get their physical processor exchanged through support. This makes me *really* nervous. One of the first things I thought when I say Threadripper's core count is, man that's going to be great for compiling AOSP. While the article claims Threadripper is not affected, the fact that something like this was just discovered on Ryzen makes me really nervous about anything broken on my expensive Threadripper See: AMD Confirms Linux 'Performance Marginality Problem' On Ryzen - Slashdot
  • My expensive motherboard won't even boot with certain common PCIe cards (USB controllers, for example). The torx screws was too short on the socket making threading near impossible. The memory slots are so sensitive that it took multiple re-seats of every DIMM to get it to boot with 128GB. The boot order of my HDDs is randomly lost with a SATA card installed (bugs reported: ROG Zenith Extreme (X399, socket TR4) - info, experience, updates - Page 8​ ). This has been the hardest and most frustrating build I've done in the last 20 years.

None of these issues are problems on Team Intel. I want to give Team AMD a chance again... I really do. The Threadripper has amazing potential if the issues are fixed, and makes the price much more affordable. But again, the features required to utilize them have to work. Like the original poster, if I don't see both recognition and traction from AMD on these issues soon, then I will be RMA'ing as well at the end of the window and going with the Core i9. I hope that AMD takes these issues seriously and starts to get fixes in place. Especially because, since you're using the same architecture for Epyc, those processors are going to have a hard time in the server market with things like this broken.

With that said, on to information about this issue. Here is what I've collected from my motherboard. Please pass on to engineering:

  • OS: Linux Mint 18.2 Cinnamon Edition (Ubuntu 16.04)
  • CPU: AMD Ryzen Threadripper 1950x
  • MB: Asus Zenith Xtreme (BIOS 0503)
  • MEM: 8x16GB (128GB) Crucial Ballistix BLS4K16G4D240FSC
  • GPU: 2x EVGA 1080Ti FTW3 Hybrid
  • Other: Inatek USB PCIe (KTU3FR-5O2U, disabled due to BIOS bugs), Supermicro AOC-SAS2LP-MV8 8-port SATA, Asus 10GEth PCIe (Bundled w/ Zenith)

Root problem:

  • Pass-through GPU stuck in D3 state, no output. VM hangs. Let me know if there is any more information I can provide.

Kernels tried:

  • 4.8, 4.10, 4.12, 4.13rc6

Versions tried (same result on all of them):

  • QEMU 2.5, Libvirt 2.5
  • QEMU 2.9, Libvirt 2.5
  • QEMU 2.10rc4, Libvirt 3.6.0

Logs:

filthyscum
Adept II
Adept II

Re: PCI Passthrough does not work (at all) on X399 Threadripper

I have the exact same problem.

If I use pci=nommconf in grub, the iommu groups are all bunched up together.

Motherboard is an Asrock X399 Fatality professional gamer (wierd name for a workstation board!).

filthyscum
Adept II
Adept II

Re: PCI Passthrough does not work (at all) on X399 Threadripper

You can also add USB controllers with Fresco Logic chipsets to the "not compatible" list. System will not POST with it installed (chipset init failed).

0 Kudos
starlightknight
Adept II
Adept II

Re: PCI Passthrough does not work (at all) on X399 Threadripper

Yep, that's what my Inateck USB PCIe is. I reported that to Asus on their forums and another user confirmed the issue there as well.

filthyscum
Adept II
Adept II

Re: PCI Passthrough does not work (at all) on X399 Threadripper

I reported the issue to Asrock last week, but have not heard back from them yet.

So far am in the same boat as you, I need IOMMU and passthrough to work otherwise I have to RMA the whole build :\

Edit: Asrock replied, usual test all the slots, test in another machine etc.

0 Kudos
filthyscum
Adept II
Adept II

Re: PCI Passthrough does not work (at all) on X399 Threadripper

I find the lack of official replies overwhelmingly disappointing. To be honest, time is running out on my 30 day open return thing.

0 Kudos
starlightknight
Adept II
Adept II

Re: PCI Passthrough does not work (at all) on X399 Threadripper

Agreed. I filed a support ticket pointing to this (and the other) threads - hopefully that will draw some attention. I want to like the Threadripper and really appreciate what AMD is trying to do by getting higher core counts more affordable, but the parts gotta work.

filthyscum
Adept II
Adept II

Re: PCI Passthrough does not work (at all) on X399 Threadripper

Got any reply to your support ticket?

0 Kudos