We have a system designed to take advantage of PCIe 4 DMA transactions between the host and our custom PCIe device.
the device is a Xilinx-based FPGA, which is the PCI master, and has two endpoints, and we bifurcate the x16 slot to 8x8
The system is currently running on Fedora 32 with kernel 5.8.9-200
Running continuous DMA transfers from the device to host, we see about a 28G/s data rate, which is what we are targeting. However, the same exercise running DMAs from host to the PCI device, the performance is okay for a period (approx 1588746 usec), then drops to about 25%, and remains there until the system is rebooted.
interestingly, if we run the exercise with a single endpoint, this problem persists with the endpoint, while the other is fine until it has run the requisite number of transfers, then it too has the permanent condition.
we tried disabling all cores except one to see if perhaps it was context switched to a die that didn't have direct PCIe slot connection, and we see nothing in system logs, or anything else using the common tools for linux to inspect the pci devices and system.
viewing chipscope on the device side, we see no anomalies, just a drop in DMA requests all of a sudden.
any help would be appreciated.
This is the type of technical question that is best asked at AMD Forum's SERVER GURUS: https://community.amd.com/t5/server-gurus/ct-p/amd-server-gurus
That specific Forum deals with everything to do with Servers and EPYC processors and its associated hardware and software.