Re: [RFC PATCH] hw/arm/virt: Support NMI injection

[Gavin Shan]

On 1/31/20 8:39 PM, Marc Zyngier wrote:
> On 2020-01-31 06:59, Gavin Shan wrote:
> > On 1/29/20 8:04 PM, Marc Zyngier wrote:
> > > On 2020-01-29 02:44, Alexey Kardashevskiy wrote:
> > > > On 28/01/2020 17:48, Gavin Shan wrote:
> > > > > but a NMI is injected
> > > > > through LAPIC on x86. So I'm not sure what architect (system reset on
> > > > > ppc or injecting NMI on x86) aarch64 should follow.
> > > >
> > > > I'd say whatever triggers in-kernel debugger or kdump but I am not
> > > > familiar with ARM at all :)
> > >
> > > All that is completely OS specific, and has no relation to the architecture.
> > > As I mentioned in another part of the thread, the closest thing to this
> > > would be to implement SDEI together with an IMPDEF mechanism to enter it
> > > (or even generate a RAS error).
> > >
> > > On the other hand, SDEI is pretty horrible, and means either KVM or QEMU
> > > acting like a firmware for the guest. To say that I'm not keen is a massive
> > > understatement.
> > >
> > >          M.
> >
> > Marc, could you please explain a bit about "IMPDEF mechanism"? I'm not sure if
> > it means a non-standard SDEI event should be used, corresponding to the HMP/QMP
> > "nmi" command.
>
> SDEI doesn't quite describe *why* and *how* you enter it. You just get there by
> some mean (SError, Group-0 interrupt, or IMPlementation DEFined mechanism).
> It is then for the SDEI implementation to sort it out and enter the OS using the
> registered entry point.

Thanks for the explanation, which make things much clearer.

> > Also, If I'm correct, you agree that a crash dump should be triggered on arm64
> > guest once HMP/QMP "nmi" command is issued?
>
> No, I don't agree. QEMU and KVM are OS agnostic, and there is nothing in the 
> ARMv8
> architecture that talks about "crash dumps".  If your "nmi" command generates
> a SDEI event and that event gets dispatched to the guest, it is entirely the 
> guest's
> responsibility to do whatever it wants. We should stay clear of assuming 
> behaviours.

Ok. Thank you for your clarification.

> > I also dig into SDEI a bit. It seems the SDEI support in QEMU isn't upstream 
> > yet:
> >
> > https://patchew.org/QEMU/address@hidden/
>
> And I'm glad. SDEI, as I said, is absolutely horrible. I'm also very fortunate
> to have been CC'd on this series on an email address I cannot read.
> This would have huge impacts on both QEMU and KVM, and this needs more than
> a knee jerk reaction to support a QEMU command.
>
> And to be honest, if what you require is the guest kernel to panic, why don't
> you just implement a QEMU-specific driver in Linux that does just that?
> Some kind of watchdog driver, maybe?

Marc, sorry for the delay and didn't come to you in time because I wanted to 
figure
out the mechanism, which helps to get similar output as x86/ppc: NMI (or reset 
exception)
is received as indication to errors, possibly panic and restart the guest 
kernel.

The mechanism I figured out is to inject SError to guest, as below snippet 
shows. It
helps to get a panic and guest rebooting, which looks similar to what x86/ppc 
have.
I can post the patch as RFC if it's right direction to go :)

Note: I'm still investigating the code to see how SError can be injected when 
TCG
is used. I think we need same function when TCG is enabled, or it's something 
for
future.

static void do_inject_nmi_kvm(CPUState *cpu, Error **errp)
{
    struct kvm_vcpu_events events;
    int ret;

     :
    memset(&events, 0, sizeof(events));
    events.exception.serror_pending = 1;
    ret = kvm_vcpu_ioctl(CPU(cpu), KVM_SET_VCPU_EVENTS, &events);
     :
}

# echo 1 > /proc/sys/kernel/panic
# (qemu) nmi
[  812.510613] SError Interrupt on CPU0, code 0xbf000000 -- SError
[  812.510617] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 
5.5.0-rc2-00187-gf72202430e30 #2
[  812.510617] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[  812.510618] pstate: 40400085 (nZcv daIf +PAN -UAO)
[  812.510619] pc : cpu_do_idle+0x48/0x58
[  812.510619] lr : arch_cpu_idle+0x30/0x238
[  812.510620] sp : ffff8000112c3e80
[  812.510620] pmr_save: 000000f0
[  812.510621] x29: ffff8000112c3e80 x28: 0000000040e10018
[  812.510622] x27: 000000033e50d340 x26: 0000000000000000
[  812.510623] x25: 0000000000000000 x24: ffff8000112ca21c
[  812.510624] x23: ffff800010f98cb8 x22: ffff8000112c98c8
[  812.510625] x21: ffff8000112ca1f8 x20: 0000000000000001
[  812.510626] x19: ffff800010f86018 x18: 0000000000000000
[  812.510627] x17: 0000000000000000 x16: 0000000000000000
[  812.510628] x15: 0000000000000000 x14: 0000000000000000
[  812.510629] x13: 0000000000000000 x12: ffff0002fe640100
[  812.510630] x11: ffffffffffffffff x10: 00000000000009f0
[  812.510631] x9 : ffff800010088928 x8 : ffff8000112d28d0
[  812.510632] x7 : ffff8002ed63a000 x6 : 0000002fe2092876
[  812.510633] x5 : 00ffffffffffffff x4 : ffff8002ed63a000
[  812.510634] x3 : 0000000000001bce x2 : 00000000000000f0
[  812.510635] x1 : 0000000000000000 x0 : 0000000000000060
[  812.510636] Kernel panic - not syncing: Asynchronous SError Interrupt
[  812.510637] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 
5.5.0-rc2-00187-gf72202430e30 #2
[  812.510638] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[  812.510638] Call trace:
[  812.510639]  dump_backtrace+0x0/0x1a8
[  812.510639]  show_stack+0x1c/0x28
[  812.510640]  dump_stack+0xbc/0x100
[  812.510640]  panic+0x168/0x370
[  812.510640]  nmi_panic+0x68/0x98
[  812.510641]  arm64_serror_panic+0x84/0x90
[  812.510641]  do_serror+0x88/0x140
[  812.510642]  el1_error+0x8c/0x108
[  812.510642]  cpu_do_idle+0x48/0x58
[  812.510643]  default_idle_call+0x44/0x4c
[  812.510643]  do_idle+0x228/0x2d0
[  812.510644]  cpu_startup_entry+0x28/0x48
[  812.510644]  rest_init+0xdc/0xe8
[  812.510645]  arch_call_rest_init+0x14/0x1c
[  812.510645]  start_kernel+0x494/0x4c0
[  812.510675] SMP: stopping secondary CPUs
[  812.510676] Kernel Offset: disabled
[  812.510676] CPU features: 0x04402,22000438
[  812.510677] Memory Limit: none
       :
<guest is rebooted>

Thanks,
Gavin