How to use pxb-pcie in correct way?

[Marcin Juszkiewicz]

For quite a while I am experimenting with PCI Express setup on SBSA-Ref 
system. And finally decided to write.

We want to play with NUMA setup and "pxb-pcie" can be assigned to NUMA 
node other than cpu0 one. But adding it makes other cards dissapear...

When I boot sbsa-ref I have plain PCIe setup:

(qemu) info pci
  Bus  0, device   0, function 0:
    Host bridge: PCI device 1b36:0008
      PCI subsystem 1af4:1100
      id ""
  Bus  0, device   1, function 0:
    Ethernet controller: PCI device 8086:10d3
      PCI subsystem 8086:0000
      IRQ 255, pin A
      BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      BAR2: I/O at 0xffffffffffffffff [0x001e].
      BAR3: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id ""
  Bus  0, device   2, function 0:
    Display controller: PCI device 1234:1111
      PCI subsystem 1af4:1100
      BAR0: 32 bit prefetchable memory at 0x80000000 [0x80ffffff].
      BAR2: 32 bit memory at 0x81084000 [0x81084fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x00007ffe].
      id ""

Adding extra PCIe card works fine - both just "igb" and "igb" with 
"pcie-root-port".

But adding "pcie-root-port" + "igb" and then "pxb-pcie" makes "igb" 
dissapear:

../code/qemu/build/qemu-system-aarch64
-monitor telnet::45454,server,nowait
-serial stdio
-device pcie-root-port,id=ULyWl,slot=0,chassis=0
-device igb,bus=ULyWl
-device pxb-pcie,bus_nr=1

(qemu) info pci
  Bus  0, device   0, function 0:
    Host bridge: PCI device 1b36:0008
      PCI subsystem 1af4:1100
      id ""
  Bus  0, device   1, function 0:
    Ethernet controller: PCI device 8086:10d3
      PCI subsystem 8086:0000
      IRQ 255, pin A
      BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      BAR2: I/O at 0xffffffffffffffff [0x001e].
      BAR3: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id ""
  Bus  0, device   2, function 0:
    Display controller: PCI device 1234:1111
      PCI subsystem 1af4:1100
      BAR0: 32 bit prefetchable memory at 0x80000000 [0x80ffffff].
      BAR2: 32 bit memory at 0x81085000 [0x81085fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x00007ffe].
      id ""
  Bus  0, device   3, function 0:
    PCI bridge: PCI device 1b36:000c
      IRQ 255, pin A
      BUS 0.
      secondary bus 1.
      subordinate bus 1.
      IO range [0xf000, 0x0fff]
      memory range [0xfff00000, 0x000fffff]
      prefetchable memory range [0xfffffffffff00000, 0x000fffff]
      BAR0: 32 bit memory at 0x81084000 [0x81084fff].
      id "ULyWl"
  Bus  0, device   4, function 0:
    Host bridge: PCI device 1b36:000b
      PCI subsystem 1af4:1100
      id ""


If I add "igb" directly (without root port) then it appears correctly:

(qemu) info pci
  Bus  0, device   0, function 0:
    Host bridge: PCI device 1b36:0008
      PCI subsystem 1af4:1100
      id ""
  Bus  0, device   1, function 0:
    Ethernet controller: PCI device 8086:10d3
      PCI subsystem 8086:0000
      IRQ 255, pin A
      BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      BAR2: I/O at 0xffffffffffffffff [0x001e].
      BAR3: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id ""
  Bus  0, device   2, function 0:
    Display controller: PCI device 1234:1111
      PCI subsystem 1af4:1100
      BAR0: 32 bit prefetchable memory at 0x80000000 [0x80ffffff].
      BAR2: 32 bit memory at 0x810c4000 [0x810c4fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x00007ffe].
      id ""
  Bus  0, device   3, function 0:
    Ethernet controller: PCI device 8086:10c9
      PCI subsystem 1af4:1100
      IRQ 255, pin A
      BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      BAR2: I/O at 0xffffffffffffffff [0x001e].
      BAR3: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
      id ""
  Bus  0, device   4, function 0:
    Host bridge: PCI device 1b36:000b
      PCI subsystem 1af4:1100
      id ""


When I add "pcie-root-port" with "igb" followed by "pcie-root-port" and 
"pxb-pcie" then no IGB again:


-device pcie-root-port,id=RjKXs,slot=0,chassis=0
-device igb,bus=RjKXs
-device pcie-root-port,chassis=7
-device pxb-pcie,bus_nr=1


(qemu) info pci
  Bus  0, device   0, function 0:
    Host bridge: PCI device 1b36:0008
      PCI subsystem 1af4:1100
      id ""
  Bus  0, device   1, function 0:
    Ethernet controller: PCI device 8086:10d3
      PCI subsystem 8086:0000
      IRQ 255, pin A
      BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      BAR2: I/O at 0xffffffffffffffff [0x001e].
      BAR3: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id ""
  Bus  0, device   2, function 0:
    Display controller: PCI device 1234:1111
      PCI subsystem 1af4:1100
      BAR0: 32 bit prefetchable memory at 0x80000000 [0x80ffffff].
      BAR2: 32 bit memory at 0x81086000 [0x81086fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x00007ffe].
      id ""
  Bus  0, device   3, function 0:
    PCI bridge: PCI device 1b36:000c
      IRQ 255, pin A
      BUS 0.
      secondary bus 1.
      subordinate bus 1.
      IO range [0xf000, 0x0fff]
      memory range [0xfff00000, 0x000fffff]
      prefetchable memory range [0xfffffffffff00000, 0x000fffff]
      BAR0: 32 bit memory at 0x81085000 [0x81085fff].
      id "RjKXs"
  Bus  0, device   4, function 0:
    PCI bridge: PCI device 1b36:000c
      IRQ 255, pin A
      BUS 0.
      secondary bus 2.
      subordinate bus 2.
      IO range [0xf000, 0x0fff]
      memory range [0xfff00000, 0x000fffff]
      prefetchable memory range [0xfffffffffff00000, 0x000fffff]
      BAR0: 32 bit memory at 0x81084000 [0x81084fff].
      id ""
  Bus  0, device   5, function 0:
    Host bridge: PCI device 1b36:000b
      PCI subsystem 1af4:1100
      id ""


Same (no "igb") with more complex structure behind "pxb-pcie" bridge:

../code/qemu/build/qemu-system-aarch64 


-monitor telnet::45454,server,nowait 


-serial stdio 


-device pcie-root-port,id=PrZUz,slot=0,chassis=0 


-device igb,bus=PrZUz 


-device pcie-root-port,chassis=7 


-device pxb-pcie,id=pxb1,bus_nr=1 


-device pcie-pci-bridge,id=pci,bus=pxb1 


-device es1370,bus=pci,addr=9,id=es1370 


-device e1000,bus=pci,addr=10,id=e1000 


-device pcie-root-port,id=root_port_for_ahci,bus=pxb1,chassis=10,slot=0 


-device ahci,bus=root_port_for_ahci,id=ahci 



  Bus  1, device   0, function 0:
    PCI bridge: PCI device 1b36:000e
      IRQ 255, pin A
      BUS 1.
      secondary bus 2.
      subordinate bus 2.
      IO range [0x1000, 0x1fff]
      memory range [0x81100000, 0x811fffff]
      prefetchable memory range [0xfffffffffff00000, 0x000fffff]
      BAR0: 64 bit memory at 0x81200000 [0x812000ff].
      id "pci"
  Bus  2, device   9, function 0:
    Audio controller: PCI device 1274:5000
      PCI subsystem 4942:4c4c
      IRQ 255, pin A
      BAR0: I/O at 0xffffffffffffffff [0x00fe].
      id "es1370"
  Bus  2, device  16, function 0:
    Ethernet controller: PCI device 8086:100e
      PCI subsystem 1af4:1100
      IRQ 255, pin A
      BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      BAR1: I/O at 0xffffffffffffffff [0x003e].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "e1000"
  Bus  1, device   1, function 0:
    PCI bridge: PCI device 1b36:000c
      IRQ 255, pin A
      BUS 1.
      secondary bus 3.
      subordinate bus 3.
      IO range [0x0000, 0x0fff]
      memory range [0x81000000, 0x810fffff]
      prefetchable memory range [0xfffffffffff00000, 0x000fffff]
      BAR0: 32 bit memory at 0x81201000 [0x81201fff].
      id "root_port_for_ahci"
  Bus  3, device   0, function 0:
    SATA controller: PCI device 8086:2922
      PCI subsystem 1af4:1100
      IRQ 255, pin A
      BAR4: I/O at 0xffffffffffffffff [0x001e].
      BAR5: 32 bit memory at 0xffffffffffffffff [0x00000ffe].
      id "ahci"
  Bus  0, device   0, function 0:
    Host bridge: PCI device 1b36:0008
      PCI subsystem 1af4:1100
      id ""
  Bus  0, device   1, function 0:
    Ethernet controller: PCI device 8086:10d3
      PCI subsystem 8086:0000
      IRQ 255, pin A
      BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      BAR2: I/O at 0xffffffffffffffff [0x001e].
      BAR3: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id ""
  Bus  0, device   2, function 0:
    Display controller: PCI device 1234:1111
      PCI subsystem 1af4:1100
      BAR0: 32 bit prefetchable memory at 0x80000000 [0x80ffffff].
      BAR2: 32 bit memory at 0x81386000 [0x81386fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x00007ffe].
      id ""
  Bus  0, device   3, function 0:
    PCI bridge: PCI device 1b36:000c
      IRQ 255, pin A
      BUS 0.
      secondary bus 1.
      subordinate bus 3.
      IO range [0x0000, 0x1fff]
      memory range [0x81000000, 0x812fffff]
      prefetchable memory range [0xfffffffffff00000, 0x000fffff]
      BAR0: 32 bit memory at 0x81385000 [0x81385fff].
      id "PrZUz"
  Bus  1, device   0, function 0:
    PCI bridge: PCI device 1b36:000e
      IRQ 255, pin A
      BUS 1.
      secondary bus 2.
      subordinate bus 2.
      IO range [0x1000, 0x1fff]
      memory range [0x81100000, 0x811fffff]
      prefetchable memory range [0xfffffffffff00000, 0x000fffff]
      BAR0: 64 bit memory at 0x81200000 [0x812000ff].
      id "pci"
  Bus  2, device   9, function 0:
    Audio controller: PCI device 1274:5000
      PCI subsystem 4942:4c4c
      IRQ 255, pin A
      BAR0: I/O at 0xffffffffffffffff [0x00fe].
      id "es1370"
  Bus  2, device  16, function 0:
    Ethernet controller: PCI device 8086:100e
      PCI subsystem 1af4:1100
      IRQ 255, pin A
      BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      BAR1: I/O at 0xffffffffffffffff [0x003e].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "e1000"
  Bus  1, device   1, function 0:
    PCI bridge: PCI device 1b36:000c
      IRQ 255, pin A
      BUS 1.
      secondary bus 3.
      subordinate bus 3.
      IO range [0x0000, 0x0fff]
      memory range [0x81000000, 0x810fffff]
      prefetchable memory range [0xfffffffffff00000, 0x000fffff]
      BAR0: 32 bit memory at 0x81201000 [0x81201fff].
      id "root_port_for_ahci"
  Bus  3, device   0, function 0:
    SATA controller: PCI device 8086:2922
      PCI subsystem 1af4:1100
      IRQ 255, pin A
      BAR4: I/O at 0xffffffffffffffff [0x001e].
      BAR5: 32 bit memory at 0xffffffffffffffff [0x00000ffe].
      id "ahci"
  Bus  0, device   4, function 0:
    PCI bridge: PCI device 1b36:000c
      IRQ 255, pin A
      BUS 0.
      secondary bus 4.
      subordinate bus 4.
      IO range [0xf000, 0x0fff]
      memory range [0xfff00000, 0x000fffff]
      prefetchable memory range [0xfffffffffff00000, 0x000fffff]
      BAR0: 32 bit memory at 0x81384000 [0x81384fff].
      id ""
  Bus  0, device   5, function 0:
    Host bridge: PCI device 1b36:000b
      PCI subsystem 1af4:1100
      id "pxb1"

What happens with missing card? It does not have to be "igb" - "ac97" 
gets lost too.

And only first added one dissapears. If I add two cards (with root-port 
entries) before "pxb-pcie" then second one is present.