Re: [PATCH qemu v20] spapr: Implement Open Firmware client interface

[David Gibson]

On Mon, Jun 07, 2021 at 12:21:21AM +0200, BALATON Zoltan wrote:
> On Fri, 4 Jun 2021, David Gibson wrote:
> > On Wed, Jun 02, 2021 at 02:29:29PM +0200, BALATON Zoltan wrote:
> > > On Wed, 2 Jun 2021, David Gibson wrote:
> > > > On Thu, May 27, 2021 at 02:42:39PM +0200, BALATON Zoltan wrote:
> > > > > On Thu, 27 May 2021, David Gibson wrote:
> > > > > > On Tue, May 25, 2021 at 12:08:45PM +0200, BALATON Zoltan wrote:
> > > > > > > On Tue, 25 May 2021, David Gibson wrote:
> > > > > > > > On Mon, May 24, 2021 at 12:55:07PM +0200, BALATON Zoltan wrote:
> > > > > > > > > On Mon, 24 May 2021, David Gibson wrote:
> > > > > > > > > > On Sun, May 23, 2021 at 07:09:26PM +0200, BALATON Zoltan 
> > > > > > > > > > wrote:
> > > > > > > > > > > On Sun, 23 May 2021, BALATON Zoltan wrote:
> > > > > > > > > > > and would using sc 1 for hypercalls on pegasos2 cause 
> > > > > > > > > > > other
> > > > > > > > > > > problems later even if the assert could be removed?
> > > > > > > > > > 
> > > > > > > > > > At least in the short term, I think you probably can remove 
> > > > > > > > > > the
> > > > > > > > > > assert.  In your case the 'sc 1' calls aren't truly to a 
> > > > > > > > > > hypervisor,
> > > > > > > > > > but a special case escape to qemu for the firmware 
> > > > > > > > > > emulation.  I think
> > > > > > > > > > it's unlikely to cause problems later, because nothing on a 
> > > > > > > > > > 32-bit
> > > > > > > > > > system should be attempting an 'sc 1'.  The only thing I 
> > > > > > > > > > can think of
> > > > > > > > > > that would fail is some test case which explicitly verified 
> > > > > > > > > > that 'sc
> > > > > > > > > > 1' triggered a 0x700 (SIGILL from userspace).
> > > > > > > > > 
> > > > > > > > > OK so the assert should check if the CPU has an HV bit. I 
> > > > > > > > > think there was a
> > > > > > > > > #detine for that somewhere that I can add to the assert then 
> > > > > > > > > I can try that.
> > > > > > > > > What I wasn't sure about is that sc 1 would conflict with the 
> > > > > > > > > guest's usage
> > > > > > > > > of normal sc calls or are these going through different paths 
> > > > > > > > > and only sc 1
> > > > > > > > > will trigger vhyp callback not affecting notmal sc calls?
> > > > > > > > 
> > > > > > > > The vhyp shouldn't affect normal system calls, 'sc 1' is 
> > > > > > > > specifically
> > > > > > > > for hypercalls, as opposed to normal 'sc' (a.k.a. 'sc 0'), and 
> > > > > > > > the
> > > > > > > > vhyp only intercepts the hypercall version (after all Linux on 
> > > > > > > > PAPR
> > > > > > > > certainly uses its own system calls, and hypercalls are active 
> > > > > > > > for the
> > > > > > > > lifetime of the guest there).
> > > > > > > > 
> > > > > > > > > (Or if this causes
> > > > > > > > > an otherwise unnecessary VM exit on KVM even when it works 
> > > > > > > > > then maybe
> > > > > > > > > looking for a different way in the future might be needed.
> > > > > > > > 
> > > > > > > > What you're doing here won't work with KVM as it stands.  There 
> > > > > > > > are
> > > > > > > > basically two paths into the vhyp hypercall path: 1) from TCG, 
> > > > > > > > if we
> > > > > > > > interpret an 'sc 1' instruction we enter vhyp, 2) from KVM, if 
> > > > > > > > we get
> > > > > > > > a KVM_EXIT_PAPR_HCALL KVM exit then we also go to the vhyp path.
> > > > > > > > 
> > > > > > > > The second path is specific to the PAPR (ppc64) implementation 
> > > > > > > > of KVM,
> > > > > > > > and will not work for a non-PAPR platform without substantial
> > > > > > > > modification of the KVM code.
> > > > > > > 
> > > > > > > OK so then at that point when we try KVM we'll need to look at 
> > > > > > > alternative
> > > > > > > ways, I think MOL OSI worked with KVM at least in MOL but will 
> > > > > > > probably make
> > > > > > > all syscalls exit KVM but since we'll probably need to use KVM PR 
> > > > > > > it will
> > > > > > > exit anyway. For now I keep this vhyp as it does not run with KVM 
> > > > > > > for other
> > > > > > > reasons yet so that's another area to clean up so as a proof of 
> > > > > > > concept
> > > > > > > first version of using VOF vhyp will do.
> > > > > > 
> > > > > > Eh, since you'll need to modify KVM anyway, it probably makes just 
> > > > > > as
> > > > > > much sense to modify it to catch the 'sc 1' as MoL's magic thingy.
> > > > > 
> > > > > I'm not sure how KVM works for this case so I also don't know why and 
> > > > > what
> > > > > would need to be modified. I think we'll only have KVM PR working as 
> > > > > newer
> > > > > POWER CPUs having HV (besides being rare among potential users) are 
> > > > > probably
> > > > > too different to run the OSes that expect at most a G4 on pegasos2 so 
> > > > > likely
> > > > > it won't work with KVM HV.
> > > > 
> > > > Oh, it definitely won't work with KVM HV.
> > > > 
> > > > > If we have KVM PR doesn't sc already trap so we
> > > > > could add MOL OSI without further modification to KVM itself only 
> > > > > needing
> > > > > change in QEMU?
> > > > 
> > > > Uh... I guess so?
> > > > 
> > > > > I also hope that MOL OSI could be useful for porting some
> > > > > paravirt drivers from MOL for running Mac OS X on Mac emulation but I 
> > > > > don't
> > > > > know about that for sure so I'm open to any other solution too.
> > > > 
> > > > Maybe.  I never know much about MOL to begin with, and anything I did
> > > > know was a decade or more ago so I've probably forgotten.
> > > 
> > > That may still be more than what I know about it since I never had any
> > > knowledge about PPC KVM and don't have any PPC hardware to test with so 
> > > I'm
> > > mostly guessing. (I could test with KVM emulated in QEMU and I did set up 
> > > an
> > > environment for that but that's a bit slow and inconvenient so I'd leave 
> > > KVM
> > > support to those interested and have more knowledge and hardware for it.)
> > 
> > Sounds like a problem for someone else another time, then.
> 
> So now that it works on TCG with vhyp I tried what it would do on KVM PR
> with the sc 1 but I could only test that on QEMU itself running in a Linux
> guest. First I've hit missing this callback:
> 
> https://git.qemu.org/?p=qemu.git;a=blob;f=target/ppc/kvm.c;h=104a308abb5700b2fe075397271f314d7f607543;hb=HEAD#l856
> 
> that I can fix by providing a callback in pegasos2.c that does what the else
> clause would do returning POWERPC_CPU(current_cpu)->env.spr[SPR_SDR1] (I
> guess that's the correct thing to do if it works without vhyp).

For your case, yes that's right.  Again vhyp is designed for the case
where the (hash) MMU is owned by the hypervisor.  But due to a gross
hack the way we communicate the userspace address of the hash table to
KVM PR is via the SDR1 register, which is why we need that hook.

> After getting past this, the host QEMU crashed on the first sc 1 call with
> this error:
> 
> qemu: fatal: Trying to deliver HV exception (MSR) 8 with no HV support

> NIP 0000000000000148   LR 0000000000000590 CTR 0000000000000000 XER 
> 0000000000000000 CPU#0
> MSR 000000000000d032 HID0 0000000060000000  HF 00004012 iidx 0 didx 0
> TB 00000203 876006644638 DECR 422427
> GPR00 0000000000000680 000000000000fe90 0000000000008e00 000000000000f005
> GPR04 000000000000fe9c 0000000000000001 0000000000000e78 0000000000000000
> GPR08 000000000000fe98 000000000000fe9c 0000000000000001 0000000000000000
> GPR12 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> GPR16 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> GPR20 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> GPR24 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> GPR28 0000000000000000 0000000000000000 0000000000008e9c 000000000000fe90
> CR 20000000  [ E  -  -  -  -  -  -  -  ]             RES ffffffffffffffff
> FPR00 bff0000000000000 0000000000000000 0000000000000000 0000000000000000
> FPR04 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> FPR08 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> FPR12 3ff553f7ced91687 0000000000000000 0000000000000000 0000000000000000
> FPR16 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> FPR20 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> FPR24 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> FPR28 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> FPSCR 0000000082004000
>  SRR0 00000000000001d4  SRR1 300000000000d032    PVR 00000000003c0301 VRSAVE 
> 00000000ffffffff
> SPRG0 000000003fe00000 SPRG1 c00000000ff60000  SPRG2 c00000000ff60000  SPRG3 
> 0000000000000000
> SPRG4 0000000000000000 SPRG5 0000000000000000  SPRG6 0000000000000000  SPRG7 
> 0000000000000000
>  SDR1 000000003f000006   DAR f00000000090abf0  DSISR 0000000042000000
> Aborted (core dumped)
> 
> (vof.bin looks like this:
> 
>  100:   3c 40 00 00     lis     r2,0
>  104:   60 42 8e 00     ori     r2,r2,36352
>  108:   48 00 00 cc     b       0x1d4
>  10c:   3c 40 00 00     lis     r2,0
>  110:   60 42 8e 00     ori     r2,r2,36352
>  114:   94 21 ff 90     stwu    r1,-112(r1)
>  118:   93 e1 00 68     stw     r31,104(r1)
>  11c:   7f e8 02 a6     mflr    r31
>  120:   48 00 02 8d     bl      0x3ac
>  124:   60 00 00 00     nop
>  128:   7f e8 03 a6     mtlr    r31
>  12c:   83 e1 00 68     lwz     r31,104(r1)
>  130:   38 21 00 70     addi    r1,r1,112
>  134:   4e 80 00 20     blr
>  138:   7c 64 1b 78     mr      r4,r3
>  13c:   3c 60 00 00     lis     r3,0
>  140:   60 63 f0 05     ori     r3,r3,61445
>  144:   44 00 00 22     sc      1
>  148:   4e 80 00 20     blr
> 
> so I think it's the sc 1 at 0x144) The error is coming from here:
> 
> https://git.qemu.org/?p=qemu.git;a=blob;f=target/ppc/excp_helper.c;h=fd147e2a37662456d30f7ab74b23bfb036260ced;hb=HEAD#l830
> 
> What does this mean? What would a real CPU do with this and where it could
> be catched to use as hypercall method on CPUs without HV or what else should
> we do if we wanted this to work with KVM PR too in the future?

The interesting bit is actually how we're getting to that part of
powerpc_excp.  I guess we must be getting a KVM exit for that 'sc 1',
but I don't know what type.  If we can figure out that would be where
we'd need to intercept it and send it to the vhyp handler instead of
actually trying to enter the hypercall vector on the emulated CPU,
which doesn't have one.

-- 
David Gibson                    | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
                                | _way_ _around_!
http://www.ozlabs.org/~dgibson

signature.asc
Description: PGP signature