Re: QEMU-KVM offers OPAL firmware interface? OpenBSD guest support?

[Mark Kettenis]

> From: Cédric Le Goater <clg@kaod.org>
> Date: Sat, 28 Aug 2021 10:46:52 +0200
> 
> Hello,
> 
> On 8/27/21 7:00 PM, Greg Kurz wrote:
> > On Fri, 27 Aug 2021 18:48:04 +0200
> > Mark Kettenis <mark.kettenis@xs4all.nl> wrote:
> > 
> >>> Date: Fri, 27 Aug 2021 18:02:59 +0200
> >>> From: Greg Kurz <groug@kaod.org>
> >>>
> >>> On Fri, 27 Aug 2021 15:12:31 +0000
> >>> Joseph <joseph.mayer@protonmail.com> wrote:
> >>>
> >>>> On Friday, August 27th, 2021 at 11:01 PM, Greg Kurz <groug@kaod.org> 
> >>>> wrote:
> >>>>> Linux knows how to drive both powernv and pseries platforms.
> >>>> ..
> >>>>> OpenBSD might have to implement proper guest-side pseries
> >>>>> support to run as a guest under an hypervisor on POWER. I don't
> >>>>> know OpenBSD but this likely a huge effort.
> >>>>>
> >>>>> More details in the "Linux on POWER Architecture Reference":
> >>>>> https://openpowerfoundation.org/wp-content/uploads/2020/07/LoPAR-20200611.pdf
> >>>>> and under the arch/powerpc/platforms/pseries/ directory in the linux
> >>>>> kernel sources.
> >>>>
> >>>> Hi Greg,
> >>>>
> >>>> Thanks for following up.
> >>>>
> >>>> (Meanwhile I posted this Q to KVM-PPC and QEMU-PPC too. On the latter 
> >>>> it's 
> >>>> https://lists.nongnu.org/archive/html/qemu-ppc/2021-08/msg00416.html and 
> >>>> on the further it didn't register yet.)
> >>>>
> >>>> First I believe KVM-QEMU and "PowerKVM" are different - the latter was
> >>>> a KVM fork maintained by IBM. The further is just the KVM and QEMU
> >>>> repo. Did IBM their contributions so PowerKVM was upstreamed, you tell
> >>>> me.
> >>>>
> >>>
> >>> Yes you're right that PowerKVM was an IBM internal fork of KVM and
> >>> QEMU, but it was discontinued ages ago. All the relevant bits have
> >>> been merged upstream and all development since then happens upstream.
> >>> So it doesn't make a big difference now to use PowerKVM instead of
> >>> QEMU-KVM on POWER.
> >>>
> >>>> Are you saying that KVM-QEMU has all relevant Power9 support
> >>>> already, for a Linux host OpenBSD as a guest on bare metal
> >>>> ("powernv" mode)?
> >>>>
> >>>
> >>> There's some confusion here. In bare metal, you just have a single
> >>> instance of the OS running unvirtualized directly on the host.
> >>> KVM-QEMU isn't involved at all in this case. According to the OpenBSD
> >>> statement, you can already install and run it in this mode on an
> >>> OpenPOWER system.
> >>
> >> Correct.
> >>
> >>>> In this case why would there be any relevance in OpenBSD implementing
> >>>> pseries.
> >>>>
> >>>
> >>> If you want to also run OpenBSD inside a VM, then OpenBSD must
> >>> implement proper support to be able to run in the paravirtualized
> >>> PAPR environment provided by KVM-QEMU on POWER. The OpenBSD statement
> >>> seem to indicate this is missing. Nothing special "should" be needed
> >>> on the KVM-QEMU side.
> >>
> >> Indeed.  OpenBSD/powerpc64 currently does not implement PAPR
> >> environment support.  I've looked at it at some point and I don't
> >> think adding support for it would be impossible.  But I only have
> >> powernv hardware.  I suppose I could run KVM-QEMU on Linux on that but
> >> my time is limited.
> >>
> > 
> > Yeah, KVM-QEMU can run on powernv hardware. So you could use it
> > as a development environment to implement PAPR support in OpenBSD.
> > This is clearly not something achievable if you only have limited
> > time though.
> > 
> >> It is possible to use QEMU to emulate a powernv machine and in
> >> principle you can run OpenBSD in that environment on a Linux host.
> >> But that's emulation and not virtualization.
> > 
> > Yeah, no KVM in this case and it will be extremely slow.
> 
> KVM is supported under the QEMU PowerNV machine. I would not say it 
> is fast (clearly not) but guests start under the emulated hypervisor.
> The emulated hypervisor it self can sustain a network bandwidth of 
> 10MB/s running on a small laptop which is quite fast for emulation.
> 
> Anyhow, the QEMU PowerNV is a dev/test platform and not a production 
> one 
> 
> We do use the pseries XIVE emulation in prod on some P9 systems.

OpenBSD probably uses XIVE in a different way than Linux though.

> >> It doesn't seem to work
> >> though and the installer hangs after printing its first message.
> 
> Can you send the command line ?

Sure.  This is what I'm using:

qemu-system-ppc64 -M powernv9 -cpu power9 -m 2G -smp 1 \
                  -serial mon:stdio \
                  -device ich9-ahci,id=ahci0,bus=pcie.0 \
                  -device qemu-xhci,id=usb0,bus=pcie.2 \
                  -bios /usr/local/share/qemu/skiboot.lid \
                  -kernel ./pnor.BOOTKERNEL \
                  -drive id=disk,file=miniroot70.img,if=none \
                  -device ide-hd,bus=ahci0.0,drive=disk \
                  -drive id=disk1,file=power.img,if=none \
                  -device ide-hd,bus=ahci0.1,drive=disk1 \
                  -device e1000e,bus=pcie.1

miniroot70.img is the OpenBSD installer image that can be found at:

  http://cdn.openbsd.org/pub/OpenBSD/snapshots/powerpc64/miniroot70.img

power.img is the disk image to install on; you can probably leave that out.

Not sure where I got pnor.BOOTKERNEL from anymore.

This command boots into the installer, but hangs at the "Welcome to
the OpenBSD installer" message.  It is supposed to print a prompt
after that, but that never happens.

> >> I suspect QEMU's emulation of the XIVE interrupt controller isn't 100%
> >> faithful.  
> 
> It is for pseries. PowerNV takes some shortcuts but it has enough support.  
> 
> >> It's also really slow when running on an amd64 machine, 
> 
> Compared to real HW. Yes :)
> 
> >> so I didn't investigate further.
> 
> Here are some docs :
> 
>   https://qemu.readthedocs.io/en/latest/system/ppc/powernv.html
> 
> But if you have a real POWER9 system available, I would just install 
> the latest debian, ubunutu or fedora distros to run KVM guests. All 
> is merged. Working your way through the PAPR interface is going to 
> take some time, though. see the pseries platform under Linux. 
> 
> I thought the BSD folks were working on POWER9 baremetal support,
> PowerNV platform, on top of OPAL/skiboot. Is that completed ?

Yup. OpenBSD boots fine on the Raptor Talos/Blackbird machines that we have.