How to configure an AXI - PCIe bridge with Zync UltraScale+ MPSoC to connect to external PCIe endpoint?
I my design, I need to connect an external PCIe device to a PCIe x2 port from the PL of MPSoC. My design in Vivado 2021 includes the PS, smartconnect and a "DMA/Bridge Subsystem for PCI Express" core, configured as a Bridge. My PCIe device requests three bars of 16K, 4K and another 16K.
The issue I am facing is that when I generate a project in Vitis, in Bare Metal, the PCIe link comes up, and all the three PCIe BARs are allocated. In petaLinux this is not the case. I tried different configurations: in one of them the boot log says the BARs are successfully allocated, but then the device driver still things they were not allocated, in another, the log says "pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring."
Here are some details of my design:
In the Address Editor I have /xdma_0/S_AXI_B BAR0 configured to 0x00_9000_0000; range: 128M
The AXI-PCIe Bridge is configured as a root port. It has one PCIe BAR of size 256 MB, with PCIe to AXI translation set to 0x90000000. In its AXI:BARs tab it has AXI BAR_0 configured with AXI to PCIe Translation set to 0x0.
This design results in the following petaLinux boot log:
[ 3.367399] xilinx-xdma-pcie a0000000.axi-pcie: host bridge /amba_pl@0/axi-pcie@90000000 ranges:
[ 3.375290] xilinx-xdma-pcie a0000000.axi-pcie: No bus range found for /amba_pl@0/axi-pcie@90000000, using [bus 00-ff]
[ 3.386113] xilinx-xdma-pcie a0000000.axi-pcie: MEM 0x0090000000..0x0097ffffff -> 0x0000000000
[ 3.395122] xilinx-xdma-pcie a0000000.axi-pcie: Using MSI FIFO mode
[ 3.401361] xilinx-xdma-pcie a0000000.axi-pcie: PCIe Link is UP
[ 3.407302] xilinx-xdma-pcie a0000000.axi-pcie: PCI host bridge to bus 0000:00
[ 3.414379] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 3.419829] pci_bus 0000:00: root bus resource [mem 0x90000000-0x97ffffff] (bus address [0x00000000-0x07ffffff])
[ 3.429980] pci 0000:00:00.0: [10ee:9122] type 01 class 0x060000
[ 3.435928] pci 0000:00:00.0: reg 0x10: [mem 0x90000000-0x97ffffff]
[ 3.443407] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 3.450224] pci 0000:01:00.0: [1e60:2864] type 00 class 0x0b4000
[ 3.456108] pci 0000:01:00.0: reg 0x10: [mem 0x90000000-0x90003fff 64bit pref]
[ 3.463271] pci 0000:01:00.0: reg 0x18: [mem 0x90000000-0x90000fff 64bit pref]
[ 3.470450] pci 0000:01:00.0: reg 0x20: [mem 0x90000000-0x90003fff 64bit pref]
[ 3.477774] pci 0000:01:00.0: PME# supported from D3hot
[ 3.482848] pci 0000:01:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[ 3.498801] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[ 3.504271] pci 0000:01:00.0: BAR 0: no space for [mem size 0x00004000 64bit pref]
[ 3.511781] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x00004000 64bit pref]
[ 3.519650] pci 0000:01:00.0: BAR 4: no space for [mem size 0x00004000 64bit pref]
[ 3.527172] pci 0000:01:00.0: BAR 4: failed to assign [mem size 0x00004000 64bit pref]
[ 3.535046] pci 0000:01:00.0: BAR 2: no space for [mem size 0x00001000 64bit pref]
[ 3.542569] pci 0000:01:00.0: BAR 2: failed to assign [mem size 0x00001000 64bit pref]
[ 3.550441] pci 0000:00:00.0: PCI bridge to [bus 01]
When I change the AXI to PCIe translation address to be set to 0x8000000 (right after the PCIe BAR), the log looks better, but still says that the bridge configuration is invalid, and the driver doesn't recognize that the BARs were allocated. In addition, the address range allocated from the PCIe BARs are not in the 0x90000000 - 0x97FFFFFF range.
[ 3.368327] xilinx-xdma-pcie a0000000.axi-pcie: host bridge /amba_pl@0/axi-pcie@90000000 ranges:
[ 3.376245] xilinx-xdma-pcie a0000000.axi-pcie: No bus range found for /amba_pl@0/axi-pcie@90000000, using [bus 00-ff]
[ 3.387063] xilinx-xdma-pcie a0000000.axi-pcie: MEM 0x0090000000..0x0097ffffff -> 0x0008000000
[ 3.396074] xilinx-xdma-pcie a0000000.axi-pcie: Using MSI FIFO mode
[ 3.402315] xilinx-xdma-pcie a0000000.axi-pcie: PCIe Link is UP
[ 3.408257] xilinx-xdma-pcie a0000000.axi-pcie: PCI host bridge to bus 0000:00
[ 3.415331] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 3.420780] pci_bus 0000:00: root bus resource [mem 0x90000000-0x97ffffff] (bus address [0x08000000-0x0fffffff])
[ 3.430931] pci 0000:00:00.0: [10ee:9122] type 01 class 0x060000
[ 3.436879] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x07ffffff]
[ 3.444356] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 3.451179] pci 0000:01:00.0: [{my_device_properties}] type 00 class 0x0b4000
[ 3.457059] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit pref]
[ 3.464223] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit pref]
[ 3.471402] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref]
[ 3.478726] pci 0000:01:00.0: PME# supported from D3hot
[ 3.483799] pci 0000:01:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[ 3.499756] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[ 3.505221] pci 0000:01:00.0: BAR 0: assigned [mem 0x00100000-0x00103fff 64bit pref]
[ 3.512920] pci 0000:01:00.0: BAR 4: assigned [mem 0x00104000-0x00107fff 64bit pref]
[ 3.520617] pci 0000:01:00.0: BAR 2: assigned [mem 0x00108000-0x00108fff 64bit pref]
[ 3.528315] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 3.533232] pci 0000:00:00.0: bridge window [mem 0x00100000-0x001fffff 64bit pref]
root@My_Project:~/my_device# insmod my_device_pci.ko
[ 202.510773] my_device: Init module. driver version 1.2.3
[ 202.516019] my_device 0000:01:00.0: Probing on: {my_device_properties}...
[ 202.521337] my_device 0000:01:00.0: Probing: Allocate memory for device extension, 11600
[ 202.529099] pci 0000:00:00.0: can't enable device: BAR 0 [mem 0x00000000-0x07ffffff] not claimed
[ 202.537881] pci 0000:00:00.0: Error enabling bridge (-22), continuing
[ 202.544322] my_device 0000:01:00.0: enabling device (0000 -> 0002)
[ 202.550157] my_device 0000:01:00.0: Probing: Device enabled
[ 202.555384] my_device 0000:01:00.0: BAR 0: can't reserve [mem 0x00100000-0x00103fff 64bit pref]
[ 202.563725] my_device 0000:01:00.0: Probing: Error allocating bars -16
[ 202.569903] my_device 0000:01:00.0: Probing: Failed init pcie resources
[ 202.576179] my_device: probe of 0000:01:00.0 failed with error -16
Any help with the right way to configure the bridge or the tweaks to perform on petaLinux configuration will be highly appreciated.
Side question: why we even need to configure any AXI BAR on the Bridge, if there is no other AXI communicating device except for the PS?
