Re: [Qemu-devel] [visorchipset] invalid opcode: 0000 [#1] PREEMPT SMP

[Jet Chen]

Thanks Borislav.
As I never test this issue on the latest version of qemu, qemu guys may want to 
reproduce it on their side.

Although every reproduce detail can be found in this mail thread, I would like 
to give a summary here.

- kernel code base:
        
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master

commit 12e364b9f08aa335dc7716ce74113e834c993765
Author:     Ken Cox <address@hidden>
AuthorDate: Tue Mar 4 07:58:07 2014 -0600
Commit:     Greg Kroah-Hartman <address@hidden>
CommitDate: Tue Mar 4 16:58:21 2014 -0800

    staging: visorchipset driver to provide registration and other services


- reproduce script (original one in the first message of this thread can't 
manage to reproduce. should remove -enable-kvm option):
------------------------------------------------------------
#!/bin/bash

kernel=$1

kvm=(
        qemu-system-x86_64 -cpu kvm64
        -kernel $kernel
        -smp 2
        -m 256M
        -net nic,vlan=0,macaddr=00:00:00:00:00:00,model=virtio
        -net user,vlan=0
        -net nic,vlan=1,model=e1000
        -net user,vlan=1
        -boot order=nc
        -no-reboot
        -watchdog i6300esb
        -serial stdio
        -display none
        -monitor null
)

append=(
        debug
        sched_debug
        apic=debug
        ignore_loglevel
        sysrq_always_enabled
        panic=10
        prompt_ramdisk=0
        earlyprintk=ttyS0,115200
        console=ttyS0,115200
        console=tty0
        vga=normal
        root=/dev/ram0
        rw
)

"address@hidden" --append "${append[*]}"
------------------------------------------------------------------

- dmesg log:

[   24.135101] FPGA image file name: xlinx_fpga_firmware.bit
[   24.137595] GPIO INIT FAIL!!
[   24.141283] driver version 1.0.0.0 loaded
[   24.142539] chipset driver version 1.0.0.0 loadedinvalid opcode: 0000 [#1] 
PREEMPT \
SMP  [   24.144793] Modules linked in:
[   24.145303] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 
3.14.0-rc5-00621-g12e364b #1
[   24.145303] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   24.145303] task: ffff88001157a010 ti: ffff88001157c000 task.ti: 
ffff88001157c000
[   24.145303] RIP: 0010:[<ffffffff81e37115>]  [<ffffffff81e37115>] \
visorchipset_init+0x7b/0x8c5 [   24.145303] RSP: 0000:ffff88001157de58  EFLAGS: 
\
00000286 [   24.145303] RAX: 000000000000070b RBX: 0000000000000004 RCX: \
4000000000000000 [   24.145303] RDX: a70aba7500000000 RSI: ffff88001157de5c 
RDI: \
ffff88001157de58 [   24.145303] RBP: ffff88001157de90 R08: 0000000000000002 
R09: \
ffff88001157de60 [   24.145303] R10: ffff88001157de64 R11: 0000000000000000 
R12: \
ffff88001157de5c [   24.145303] R13: ffff88001157de60 R14: ffff88001157de64 
R15: \
0000000000000000 [   24.145303] FS:  0000000000000000(0000) 
GS:ffff880012600000(0000) \
knlGS:0000000000000000 [   24.145303] CS:  0010 DS: 0000 ES: 0000 CR0: \
000000008005003b [   24.145303] CR2: ffff880002992000 CR3: 0000000001c07000 
CR4: \
00000000003006f0 [   24.145303] DR0: 0000000000000000 DR1: 0000000000000000 
DR2: \
0000000000000000 [   24.145303] DR3: 0000000000000000 DR6: 0000000000000000 
DR7: \
0000000000000000 [   24.145303] Stack:
[   24.145303]  00000800000306c1 078bfbf982d82203 ffffffff81e3709a 
0000000000000000
[   24.145303]  00000000000001df 0000000000000000 0000000000000000 
ffff88001157df08
[   24.145303]  ffffffff810002b2 ffffffff810b2600 ffff88001157df08 
ffffffff810b27db
[   24.145303] Call Trace:
[   24.145303]  [<ffffffff81e3709a>] ? visorchannel_init+0x1d/0x1d
[   24.145303]  [<ffffffff810002b2>] do_one_initcall+0x8e/0x138
[   24.145303]  [<ffffffff810b2600>] ? param_array_set+0xef/0xf5
[   24.145303]  [<ffffffff810b27db>] ? parse_args+0x180/0x248
[   24.145303]  [<ffffffff81dfbf86>] kernel_init_freeable+0x108/0x199
[   24.145303]  [<ffffffff81dfb73a>] ? do_early_param+0x8a/0x8a
[   24.145303]  [<ffffffff8173f08e>] ? rest_init+0xc2/0xc2
[   24.145303]  [<ffffffff8173f097>] kernel_init+0x9/0xda
[   24.145303]  [<ffffffff8176024c>] ret_from_fork+0x7c/0xb0
[   24.145303]  [<ffffffff8173f08e>] ? rest_init+0xc2/0xc2
[   24.145303] Code: 8d 65 cc 4c 8d 6d d0 4c 8d 75 d4 79 21 48 ba 00 00 00 00 
75 ba \
0a a7 48 b9 00 00 00 00 00 00 00 40 bb 04 00 00 00 b8 0b 07 00 00 <0f> 01 c1 8b 
35 c2 \
c4 b4 00 48 c7 c7 f5 93 b4 81 31 c0 e8 3b 21  [   24.145303] RIP  \
[<ffffffff81e37115>] visorchipset_init+0x7b/0x8c5 [   24.145303]  RSP \
<ffff88001157de58> [   24.187247] ---[ end trace 62b5721899a66a6c ]---
[   24.188157] Kernel panic - not syncing: Fatal exception


kernel kconfig & full dmesg log please check attachment in this mail.

Thanks,
Jet


On 04/13/2014 07:51 PM, Borislav Petkov wrote:
> Should we perhaps CC qemu-devel here for an opinion.
> 
> Guys, this mail should explain the issue but in case there are
> questions, the whole thread starts here:
> 
> http://lkml.kernel.org/r/address@hidden
> 
> Thanks.
> 
> On Sat, Apr 12, 2014 at 01:35:49AM +0800, Jet Chen wrote:
>> On 04/12/2014 12:33 AM, H. Peter Anvin wrote:
>>> On 04/11/2014 06:51 AM, Romer, Benjamin M wrote:
>>>>
>>>>> I'm still confused where KVM comes into the picture.  Are you actually
>>>>> using KVM (and thus talking about nested virtualization) or are you
>>>>> using Qemu in JIT mode and running another hypervisor underneath?
>>>>
>>>> The test that Fengguang used to find the problem was running the linux
>>>> kernel directly using KVM. When the kernel was run with "-cpu Haswell,
>>>> +smep,+smap" set, the vmcall failed with invalid op, but when the kernel
>>>> is run with "-cpu qemu64", the vmcall causes a vmexit, as it should.
>>>
>>> As far as I know, Fengguang's test doesn't use KVM at all, it runs Qemu
>>> as a JIT.  Completely different thing.  In that case Qemu probably
>>> should *not* set the hypervisor bit.  However, the only thing that the
>>> hypervisor bit means is that you can look for specific hypervisor APIs
>>> in CPUID level 0x40000000+.
>>>
>>>> My point is, the vmcall was made because the hypervisor bit was set. If
>>>> this bit had been turned off, as it would be on a real processor, the
>>>> vmcall wouldn't have happened.
>>>
>>> And my point is that that is a bug.  In the driver.  A very serious one.
>>>  You cannot call VMCALL until you know *which* hypervisor API(s) you
>>> have available, period.
>>>
>>>>> The hypervisor bit is a complete red herring. If the guest CPU is
>>>>> running in VT-x mode, then VMCALL should VMEXIT inside the guest
>>>>> (invoking the guest root VT-x), 
>>>>
>>>> The CPU is running in VT-X. That was my point, the kernel is running in
>>>> the KVM guest, and KVM is setting the CPU feature bits such that bit 31
>>>> is enabled.
>>>
>>> Which it is because it wants to export the KVM hypercall interface.
>>> However, keying VMCALL *only* on the HYPERVISOR bit is wrong in the extreme.
>>>
>>>> I don't think it's a red herring because the kernel uses this bit
>>>> elsewhere - it is reported as X86_FEATURE_HYPERVISOR in the CPU
>>>> features, and can be checked with the cpu_has_hypervisor macro (which
>>>> was not used by the original author of the code in the driver, but
>>>> should have been). VMWare and KVM support in the kernel also check for
>>>> this bit before checking their hypervisor leaves for an ID. If it's not
>>>> properly set it affects more than just the s-Par drivers.
>>>>
>>>>> but the fact still remains that you
>>>>> should never, ever, invoke VMCALL unless you know what hypervisor you
>>>>> have underneath.
>>>>
>>>> From the standpoint of the s-Par drivers, yes, I agree (as I already
>>>> said). However, VMCALL is not a privileged instruction, so anyone could
>>>> use it from user space and go right past the OS straight to the
>>>> hypervisor. IMHO, making it *lethal* to the guest is a bad idea, since
>>>> any user could hard-stop the guest with a couple of lines of C.
>>>
>>> Typically the hypervisor wants to generate a #UD inside of the guest for
>>> that case.  The guest OS will intercept it and SIGILL the user space
>>> process.
>>>
>>>     -hpa
>>>
>>
>> Hi Ben,
>>
>> I re-tested this case with/without option -enable-kvm.
>>
>> qemu-system-x86_64 -cpu Haswell,+smep,+smap                  invalid op
>> qemu-system-x86_64 -cpu kvm64                                        invalid 
>> op
>> qemu-system-x86_64 -cpu Haswell,+smep,+smap -enable-kvm      everything OK
>> qemu-system-x86_64 -cpu kvm64 -enable-kvm                    everything OK
>>
>> I think this is probably a bug in QEMU.
>> Sorry for misleading you. I am not experienced in QEMU usage. I don't 
>> realize I need try this case with different options Until read Peter's reply.
>>
>> As Peter said, QEMU probably should *not* set the hypervisor bit. But based 
>> on my testing, I think KVM works properly in this case.
>>
>> Thanks,
>> Jet
>>
>

dmesg-quantal-f4-128:20140407182830:x86_64-randconfig-br0-04050702:3.14.0-rc5-00621-g12e364b:1
Description: Text document

config-3.14.0-rc5-00621-g12e364b
Description: Text document