Re: [Qemu-devel] [Qemu-arm] [RFC PATCH 12/14] hw/arm/bcm2836: Add the BCM2838 which uses a GICv2

[Philippe Mathieu-Daudé]

Hi Luc,

On 9/5/19 10:41 AM, Luc Michel wrote:
> On 9/4/19 7:13 PM, Philippe Mathieu-Daudé wrote:
>> The BCM2838 is improvement of the BCM2837:
>> - Cortex-A72 instead of the A53
>> - peripheral block and local soc controller are mapped differently,
>> - GICv2
>> - PCIe block
>> - exhanced MMU to address over 4GiB of SDRAM
>>
>> See https://www.raspberrypi.org/forums/viewtopic.php?t=244479&start=25
>> and https://patchwork.kernel.org/patch/11053097/
>>
>> This patch starts mapping the GICv2 but interrupt lines are NOT
>> wired (yet).
>>
>> This is enough to start running the Ubuntu kernel8.img from [1].
>>
>> Extract the kernel with:
>>
>>    $ mkdir bootpart
>>    $ guestfish \
>>        --ro \
>>        -a ubuntu-18.04.3-preinstalled-server-arm64+raspi4.img \
>>        -m /dev/sda1
>>    Welcome to guestfish, the guest filesystem shell for
>>    editing virtual machine filesystems and disk images.
>>
>>    ><fs> ls /
>>    COPYING.linux
>>    LICENCE.broadcom
>>    System.map
>>    armstub8-gic.bin
>>    bcm2710-rpi-3-b-plus.dtb
>>    bcm2710-rpi-3-b.dtb
>>    bcm2710-rpi-cm3.dtb
>>    bcm2711-rpi-4-b.dtb
>>    bcm2837-rpi-3-b-plus.dtb
>>    bcm2837-rpi-3-b.dtb
>>    cmdline.txt
>>    config.txt
>>    fixup4.dat
>>    fixup4cd.dat
>>    fixup4db.dat
>>    fixup4x.dat
>>    kernel8.img
>>    overlays
>>    start4.elf
>>    start4cd.elf
>>    start4db.elf
>>    start4x.elf
>>    ><fs> copy-out / bootpart/
>>    ><fs> q
>>
>> Then some progress can be noticed running:
>>
>>    $ qemu-system-aarch64 -d unimp,guest_errors,int,in_asm \
>>        -M raspi4 \
>>        -kernel bootpart/kernel8.img \
>>        -dtb bootpart/bcm2711-rpi-4-b.dtb \
>>        -initrd bootpart/boot/initrd.img \
>>        -append \
>>           "earlycon=pl011,0xfe201000 console=ttyAMA0 console=tty1 loglevel=8"
>>
>> Not very interesting, but it runs until configuring the GIC.
>> (remove 'in_asm' if too verbose).
>>
>> TODO:
>>
>> - wire IRQs to the GIC :)
>>
>> - map the SPI bootrom from [3] (boot sequence: [4])
>>
>> - per [2] we could try booting without using the GIC, adding "enable_gic=0"
>>    in config.txt. this variable is parsed by the firmware:
>>
>>    $ fgrep -r enable_gic bootpart
>>    Binary file bootpart/start4x.elf matches
>>    Binary file bootpart/start4.elf matches
>>    Binary file bootpart/start4db.elf matches
>>    Binary file bootpart/start4cd.elf matches
>>    bootpart/config.txt:enable_gic=1
>>
>>    the stub [5] doesn't seem to check a register for it.
>>    maybe it falls back to kernel7l?
>>
>> - decompile start4.elf to check how 'enable_gic' is used
>>    using vc4 toolchain from [6]
>>
>> [1] https://github.com/TheRemote/Ubuntu-Server-raspi4-unofficial/releases
>> [2] 
>> https://jamesachambers.com/raspberry-pi-ubuntu-server-18-04-2-installation-guide/
>> [3] 
>> https://www.raspberrypi.org/documentation/hardware/raspberrypi/booteeprom.md
>> [4] 
>> https://raspberrypi.stackexchange.com/questions/10442/what-is-the-boot-sequence
>> [5] 
>> https://github.com/raspberrypi/tools/commit/7f4a937e1bacbc111a22552169bc890b4bb26a94#diff-8c41083e9fa0c98f1c3015e11b897444
>> [6] https://github.com/christinaa/rpi-open-firmware
>>
>> Signed-off-by: Philippe Mathieu-Daudé <address@hidden>
>> ---
>>   hw/arm/bcm2836.c         | 75 ++++++++++++++++++++++++++++++++++++++++
>>   include/hw/arm/bcm2836.h |  3 ++
>>   2 files changed, 78 insertions(+)
>>
>> diff --git a/hw/arm/bcm2836.c b/hw/arm/bcm2836.c
>> index 019e67b906..d89d7cd71d 100644
>> --- a/hw/arm/bcm2836.c
>> +++ b/hw/arm/bcm2836.c
>> @@ -21,6 +21,7 @@ struct BCM283XInfo {
>>       const char *cpu_type;
>>       hwaddr peri_base; /* Peripheral base address seen by the CPU */
>>       hwaddr ctrl_base; /* Interrupt controller and mailboxes etc. */
>> +    hwaddr gic_base;
>>       int clusterid;
>>   };
>>   
>> @@ -40,9 +41,25 @@ static const BCM283XInfo bcm283x_socs[] = {
>>           .ctrl_base = 0x40000000,
>>           .clusterid = 0x0,
>>       },
>> +    {
>> +        .name = TYPE_BCM2838,
>> +        .cpu_type = ARM_CPU_TYPE_NAME("cortex-a72"),
>> +        .peri_base = 0xfe000000,
>> +        .ctrl_base = 0xff800000,
>> +        .gic_base = 0x40000,
>> +    },
>>   #endif
>>   };
>>   
>> +#define GIC_NUM_IRQS                256
>> +
>> +#define GIC_BASE_OFS                0x0000
>> +#define GIC_DIST_OFS                0x1000
>> +#define GIC_CPU_OFS                 0x2000
>> +#define GIC_VIFACE_THIS_OFS         0x4000
>> +#define GIC_VIFACE_OTHER_OFS(cpu)  (0x5000 + (cpu) * 0x200)
>> +#define GIC_VCPU_OFS                0x6000
>> +
>>   static void bcm2836_init(Object *obj)
>>   {
>>       BCM283XState *s = BCM283X(obj);
>> @@ -55,6 +72,11 @@ static void bcm2836_init(Object *obj)
>>                                   info->cpu_type, &error_abort, NULL);
>>       }
>>   
>> +    if (info->gic_base) {
>> +        sysbus_init_child_obj(obj, "gic", &s->gic, sizeof(s->gic),
>> +                              TYPE_ARM_GIC);
>> +    }
>> +
>>       sysbus_init_child_obj(obj, "control", &s->control, sizeof(s->control),
>>                             TYPE_BCM2836_CONTROL);
>>   
>> @@ -115,6 +137,59 @@ static void bcm2836_realize(DeviceState *dev, Error 
>> **errp)
>>   
>>       sysbus_mmio_map(SYS_BUS_DEVICE(&s->control), 0, info->ctrl_base);
>>   
>> +    /* bcm2838 GICv2 */
>> +    if (info->gic_base) {
>> +        object_property_set_uint(OBJECT(&s->gic), 2, "revision", &err);
>> +        if (err) {
>> +            error_propagate(errp, err);
>> +            return;
>> +        }
>> +
>> +        object_property_set_uint(OBJECT(&s->gic),
>> +                                 BCM283X_NCPUS, "num-cpu", &err);
>> +        if (err) {
>> +            error_propagate(errp, err);
>> +            return;
>> +        }
>> +
>> +        object_property_set_uint(OBJECT(&s->gic),
>> +                                 32 + GIC_NUM_IRQS, "num-irq", &err);
>> +        if (err) {
>> +            error_propagate(errp, err);
>> +            return;
>> +        }
>> +
>> +        object_property_set_bool(OBJECT(&s->gic),
>> +                                 true, "has-virtualization-extensions", 
>> &err);
>> +        if (err) {
>> +            error_propagate(errp, err);
>> +            return;
>> +        }
>> +
>> +        object_property_set_bool(OBJECT(&s->gic), true, "realized", &err);
>> +        if (err) {
>> +            error_propagate(errp, err);
>> +            return;
>> +        }
>> +
>> +        sysbus_mmio_map(SYS_BUS_DEVICE(&s->gic), 0,
>> +                        info->ctrl_base + info->gic_base + GIC_DIST_OFS);
>> +        sysbus_mmio_map(SYS_BUS_DEVICE(&s->gic), 1,
>> +                        info->ctrl_base + info->gic_base + GIC_CPU_OFS);
>> +        sysbus_mmio_map(SYS_BUS_DEVICE(&s->gic), 2,
>> +                        info->ctrl_base + info->gic_base + 
>> GIC_VIFACE_THIS_OFS);
>> +        sysbus_mmio_map(SYS_BUS_DEVICE(&s->gic), 3,
>> +                        info->ctrl_base + info->gic_base + GIC_VCPU_OFS);
>> +
>> +        for (n = 0; n < BCM283X_NCPUS; n++) {
>> +            sysbus_mmio_map(SYS_BUS_DEVICE(&s->gic), 4 + n,
>> +                            info->ctrl_base + info->gic_base
>> +                            + GIC_VIFACE_OTHER_OFS(n));> +        }
>> +
>> +        /* TODO wire IRQs!!! */
> 
> I think as a bare minimum, you must wire:
>   - the ARM generic timer IRQs going out of all the CPUs, into their
> respective PPI. Looking at [1], they seem to be connected to the usual
> PPIs (at least we have the same mapping in the vexpress, the xynqmp and
> the virt board).

Done :)

>   - The PMU interrupts

Still TODO.

>   - Possibly the GICv2 maintenance interrupt, but I can't find the IRQ
> number in the DTS. This is related to the virtualization extension. It
> should not prevent Linux from booting if it's not connected (I think KVM
> does not even use the GICv2 maintenance interrupts anyway).

Done, this is a GIC-400 so it is fixed as PPI IRQ#9 :)

>   - Finally, Connect the four GICv2 output (irq, fiq, virq, vfiq) to
> their respective CPU inputs.

Done.

> [1]
> https://github.com/raspberrypi/linux/blob/rpi-5.3.y/arch/arm/boot/dts/bcm2838.dtsi
> 
> 
> Here is a snippet of the virt board, quickly adapted foc this SoC (I
> didn't test it):

Thanks! It helped :)

> #define BCM2838_ARCH_TIMER_VIRT_IRQ   11
> #define BCM2838_ARCH_TIMER_S_EL1_IRQ  13
> #define BCM2838_ARCH_TIMER_NS_EL1_IRQ 14
> #define BCM2838_ARCH_TIMER_NS_EL2_IRQ 10
> 
> #define BCM2838_PMU_SPI_BASE          16
> [...]
> 
>  for (i = 0; i < BCM283X_NCPUS; i++) {
>         DeviceState *cpudev = DEVICE(qemu_get_cpu(i));
>         int ppibase = GIC_NUM_IRQS + i * GIC_INTERNAL + GIC_NR_SGIS;
>         int irq;
> 
>         /* Mapping from the output timer irq lines from the CPU to the
>          * GIC PPI inputs.
>          */
>         const int timer_irq[] = {
>             [GTIMER_PHYS] = BCM2838_ARCH_TIMER_NS_EL1_IRQ,
>             [GTIMER_VIRT] = BCM2838_ARCH_TIMER_VIRT_IRQ,
>             [GTIMER_HYP]  = BCM2838_ARCH_TIMER_NS_EL2_IRQ,
>             [GTIMER_SEC]  = BCM2838_ARCH_TIMER_S_EL1_IRQ,
>         };
> 
>         for (irq = 0; irq < ARRAY_SIZE(timer_irq); irq++) {
>             qdev_connect_gpio_out(cpudev, irq,
>                                   qdev_get_gpio_in(gicdev,
>                                                    ppibase +
> timer_irq[irq]));
>         }
> 
>         /* I don't know the maintenance IRQ number for the this SoC */
> #if 0
>         qemu_irq irq = qdev_get_gpio_in(gicdev,
>                                         ppibase + ARCH_GIC_MAINT_IRQ);
>         sysbus_connect_irq(SYS_BUS_DEVICE(gicbusdevs->gic), i + 4 *
> smp_cpus, irq);
> #endif
> 
>         /* PMU interrupt */
>         qdev_connect_gpio_out_named(cpudev, "pmu-interrupt", 0,
>                                     qdev_get_gpio_in(gicdev,
> BCM2838_PMU_SPI_BASE + i));
> 
>         /* Connect the GICv2 outputs to the CPU */
>         sysbus_connect_irq(SYS_BUS_DEVICE(gicbusdevs->gic), i,
>                            qdev_get_gpio_in(cpudev, ARM_CPU_IRQ));
>         sysbus_connect_irq(SYS_BUS_DEVICE(gicbusdevs->gic), i + smp_cpus,
>                            qdev_get_gpio_in(cpudev, ARM_CPU_FIQ));
>         sysbus_connect_irq(SYS_BUS_DEVICE(gicbusdevs->gic), i + 2 *
> smp_cpus,
>                            qdev_get_gpio_in(cpudev, ARM_CPU_VIRQ));
>         sysbus_connect_irq(SYS_BUS_DEVICE(gicbusdevs->gic), i + 3 *
> smp_cpus,
>                            qdev_get_gpio_in(cpudev, ARM_CPU_VFIQ));
>     }

With this and few other changes I get some hopeful kernel output:

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 4.19.67-v8+ (james@james-ubuntu2) (gcc
version 9.1.0 (GCC)) #1 SMP PREEMPT Mon Aug 26 21:26:31 MDT 2019
[    0.000000] Machine model: Raspberry Pi 4 Model B
[    0.000000] earlycon: pl11 at MMIO 0x00000000fe201000 (options '')
[    0.000000] bootconsole [pl11] enabled
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 8 MiB at 0x000000003b800000
[    0.000000] On node 0 totalpages: 245760
[    0.000000]   DMA32 zone: 3840 pages used for memmap
[    0.000000]   DMA32 zone: 0 pages reserved
[    0.000000]   DMA32 zone: 245760 pages, LIFO batch:63
[    0.000000] random: get_random_bytes called from
start_kernel+0xa0/0x46c with crng_init=0
[    0.000000] percpu: Embedded 24 pages/cpu s57368 r8192 d32744 u98304
[    0.000000] pcpu-alloc: s57368 r8192 d32744 u98304 alloc=24*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: enabling workaround for EL2 vector hardening
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 241920
[    0.000000] Kernel command line: rw earlycon=pl011,0xfe201000
console=ttyAMA0 loglevel=8 root=/dev/mmcblk0p2 fsck.repair=yes
net.ifnames=0 rootwait rdinit=/sbin/init
[    0.000000] Dentry cache hash table entries: 131072 (order: 8,
1048576 bytes)
[    0.000000] Inode-cache hash table entries: 65536 (order: 7, 524288
bytes)
[    0.000000] Memory: 923932K/983040K available (8060K kernel code,
966K rwdata, 2560K rodata, 960K init, 916K bss, 50916K reserved, 8192K
cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] ftrace: allocating 28504 entries in 112 pages
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=64 to
nr_cpu_ids=4.
[    0.000000]  Tasks RCU enabled.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GIC: Using split EOI/Deactivate mode
[    0.000000] arch_timer: cp15 timer(s) running at 54.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff
max_cycles: 0xc743ce346, max_idle_ns: 440795203123 ns
[    0.001377] sched_clock: 56 bits at 54MHz, resolution 18ns, wraps
every 4398046511102ns
[    0.070247] Console: colour dummy device 80x25
[    0.085191] Calibrating delay loop (skipped), value calculated using
timer frequency.. 108.00 BogoMIPS (lpj=216000)
[    0.087407] pid_max: default: 32768 minimum: 301
[    0.118255] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes)
[    0.120037] Mountpoint-cache hash table entries: 2048 (order: 2,
16384 bytes)
[    0.462960] ASID allocator initialised with 32768 entries
[    0.475367] rcu: Hierarchical SRCU implementation.
[    0.540749] EFI services will not be available.
[    0.563715] smp: Bringing up secondary CPUs ...
[    0.624297] Detected PIPT I-cache on CPU1
[    0.628830] CPU1: Booted secondary processor 0x0000000001 [0x410fd083]
[    0.698386] Detected PIPT I-cache on CPU2
[    0.699334] CPU2: Booted secondary processor 0x0000000002 [0x410fd083]
[    0.752144] Detected PIPT I-cache on CPU3
[    0.753057] CPU3: Booted secondary processor 0x0000000003 [0x410fd083]
[    0.759037] smp: Brought up 1 node, 4 CPUs
[    0.762310] SMP: Total of 4 processors activated.
[    0.763888] CPU features: detected: 32-bit EL0 Support
[    0.819489] CPU: All CPU(s) started at EL2
[    0.826062] alternatives: patching kernel code
[    0.948345] devtmpfs: initialized
[    1.235286] Enabled cp15_barrier support
[    1.237025] Enabled setend support
[    1.253695] clocksource: jiffies: mask: 0xffffffff max_cycles:
0xffffffff, max_idle_ns: 7645041785100000 ns
[    1.255510] futex hash table entries: 1024 (order: 4, 65536 bytes)
[    1.275781] pinctrl core: initialized pinctrl subsystem
[    1.347547] DMI not present or invalid.
[    1.378896] NET: Registered protocol family 16
[    1.439424] cpuidle: using governor menu
[    1.450693] vdso: 2 pages (1 code @ (____ptrval____), 1 data @
(____ptrval____))
[    1.452529] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.

Regards.

Phil.