[Qemu-devel] Re: virtio-blk XFS corruption

[Peter Lieven]

Am 25.09.2010 um 20:11 schrieb Peter Lieven:

> 
> Am 25.09.2010 um 17:58 schrieb Christoph Hellwig:
> 
>> On Sat, Sep 25, 2010 at 05:40:34PM +0200, Peter Lieven wrote:
>>> 
>>> Am 25.09.2010 um 17:37 schrieb Christoph Hellwig:
>>> 
>>>> FYI, qemu 0.12.2 is missing:
>>> 
>>> you mean 0.12.4 not 0.12.2, don't you?
>> 
>> Yes, sorry. (but 0.12.2 is of course missing it, too..)
>> 
>>>> which in the past was very good at triggering XFS guest corruption.
>>>> Please try with the patch applied or even better latests qemu from git.
>>>> 
>>> 
>>> i'm just trying with 0.12.5. 
>>> 
>>> i'm not so familiar with git. is there a command to pull only patches
>>> that are marked as stable and will be in the next official release?
>> 
>> All the qemu stable releases are tagged and you can check do
>> 
>>      git-checkout v0.12.5
>> 
>> but that's not the main git HEAD which would also be interesting.
>> 
> 
> with v0.12.5 no xfs error, but the machine hangs after a few minutes...
> 
> (gdb)  thread apply all backtrace full
> 
> Thread 5 (Thread 0x7f15131c7950 (LWP 3579)):
> #0  0x00007f15136c1cd7 in ioctl () from /lib/libc.so.6
> No symbol table info available.
> #1  0x000000000042b9f1 in kvm_run (env=0x1d4cef0) at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:921
>       r = 0
>       kvm = (kvm_context_t) 0x1d2b7b0
>       run = (struct kvm_run *) 0x7f1514c6f000
>       fd = 16
> #2  0x000000000042cf4e in kvm_cpu_exec (env=0x1d4cef0) at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
>       r = 0
> #3  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d4cef0) at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
>       run_cpu = 1
> #4  0x000000000042d819 in ap_main_loop (_env=0x1d4cef0) at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
>       env = (struct CPUX86State *) 0x1d4cef0
>       signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 
> 15 times>}}
>       data = (struct ioperm_data *) 0x0
> #5  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
> No symbol table info available.
> #6  0x00007f15136c9fcd in clone () from /lib/libc.so.6
> No symbol table info available.
> #7  0x0000000000000000 in ?? ()
> No symbol table info available.
> 
> Thread 4 (Thread 0x7f15129c6950 (LWP 3580)):
> #0  0x00007f15136c1cd7 in ioctl () from /lib/libc.so.6
> No symbol table info available.
> #1  0x000000000042b9f1 in kvm_run (env=0x1d67010) at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:921
>       r = 0
>       kvm = (kvm_context_t) 0x1d2b7b0
>       run = (struct kvm_run *) 0x7f1514c6c000
>       fd = 17
> #2  0x000000000042cf4e in kvm_cpu_exec (env=0x1d67010) at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
>       r = 0
> #3  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d67010) at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
>       run_cpu = 1
> #4  0x000000000042d819 in ap_main_loop (_env=0x1d67010) at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
>       env = (struct CPUX86State *) 0x1d67010
>       signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 
> 15 times>}}
>       data = (struct ioperm_data *) 0x0
> #5  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
> No symbol table info available.
> #6  0x00007f15136c9fcd in clone () from /lib/libc.so.6
> No symbol table info available.
> #7  0x0000000000000000 in ?? ()
> No symbol table info available.
> 
> ---Type <return> to continue, or q <return> to quit---
> Thread 3 (Thread 0x7f15121c5950 (LWP 3581)):
> #0  0x00007f151464da94 in __lll_lock_wait () from /lib/libpthread.so.0
> No symbol table info available.
> #1  0x00007f1514649190 in _L_lock_102 () from /lib/libpthread.so.0
> No symbol table info available.
> #2  0x00007f1514648a7e in pthread_mutex_lock () from /lib/libpthread.so.0
> No symbol table info available.
> #3  0x000000000042b7dc in post_kvm_run (kvm=0x1d2b7b0, env=0x1d74d90) at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:858
> No locals.
> #4  0x000000000042ba68 in kvm_run (env=0x1d74d90) at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:930
>       r = -1
>       kvm = (kvm_context_t) 0x1d2b7b0
>       run = (struct kvm_run *) 0x7f1514c69000
>       fd = 18
> #5  0x000000000042cf4e in kvm_cpu_exec (env=0x1d74d90) at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
>       r = 0
> #6  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d74d90) at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
>       run_cpu = 1
> #7  0x000000000042d819 in ap_main_loop (_env=0x1d74d90) at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
>       env = (struct CPUX86State *) 0x1d74d90
>       signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 
> 15 times>}}
>       data = (struct ioperm_data *) 0x0
> #8  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
> No symbol table info available.
> #9  0x00007f15136c9fcd in clone () from /lib/libc.so.6
> No symbol table info available.
> #10 0x0000000000000000 in ?? ()
> No symbol table info available.
> 
> Thread 2 (Thread 0x7f15119c4950 (LWP 3582)):
> #0  0x00007f151464da94 in __lll_lock_wait () from /lib/libpthread.so.0
> No symbol table info available.
> #1  0x00007f1514649190 in _L_lock_102 () from /lib/libpthread.so.0
> No symbol table info available.
> #2  0x00007f1514648a7e in pthread_mutex_lock () from /lib/libpthread.so.0
> No symbol table info available.
> #3  0x000000000042b7dc in post_kvm_run (kvm=0x1d2b7b0, env=0x1d82b10) at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:858
> No locals.
> #4  0x000000000042ba68 in kvm_run (env=0x1d82b10) at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:930
>       r = -1
>       kvm = (kvm_context_t) 0x1d2b7b0
>       run = (struct kvm_run *) 0x7f1514c66000
>       fd = 19
> #5  0x000000000042cf4e in kvm_cpu_exec (env=0x1d82b10) at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
>       r = 0
> #6  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d82b10) at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
>       run_cpu = 1
> ---Type <return> to continue, or q <return> to quit---
> #7  0x000000000042d819 in ap_main_loop (_env=0x1d82b10) at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
>       env = (struct CPUX86State *) 0x1d82b10
>       signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 
> 15 times>}}
>       data = (struct ioperm_data *) 0x0
> #8  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
> No symbol table info available.
> #9  0x00007f15136c9fcd in clone () from /lib/libc.so.6
> No symbol table info available.
> #10 0x0000000000000000 in ?? ()
> No symbol table info available.
> 
> Thread 1 (Thread 0x7f1514c766f0 (LWP 3576)):
> #0  0x00007f15136c2742 in select () from /lib/libc.so.6
> No symbol table info available.
> #1  0x000000000040c25a in main_loop_wait (timeout=1000) at 
> /usr/src/qemu-kvm-0.12.5/vl.c:3994
>       ioh = (IOHandlerRecord *) 0x0
>       rfds = {fds_bits = {83886728, 0 <repeats 15 times>}}
>       wfds = {fds_bits = {0 <repeats 16 times>}}
>       xfds = {fds_bits = {0 <repeats 16 times>}}
>       ret = 3
>       nfds = 26
>       tv = {tv_sec = 0, tv_usec = 999994}
> #2  0x000000000042dd9d in kvm_main_loop () at 
> /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:2126
>       fds = {24, 25}
>       mask = {__val = {268443712, 0 <repeats 15 times>}}
>       sigfd = 26
> #3  0x000000000040c98c in main_loop () at /usr/src/qemu-kvm-0.12.5/vl.c:4212
>       r = 0
> #4  0x000000000041055a in main (argc=39, argv=0x7fffe23b2838, 
> envp=0x7fffe23b2978) at /usr/src/qemu-kvm-0.12.5/vl.c:6256
>       gdbstub_dev = 0x0
>       boot_devices_bitmap = 12
>       i = 0
>       snapshot = 0
>       linux_boot = 0
>       initrd_filename = 0x0
>       kernel_filename = 0x0
>       kernel_cmdline = 0x58958c ""
>       boot_devices = "dc", '\0' <repeats 30 times>
>       ds = (DisplayState *) 0x1d9ff00
>       dcl = (DisplayChangeListener *) 0x0
>       cyls = 0
>       heads = 0
>       secs = 0
>       translation = 0
>       hda_opts = (QemuOpts *) 0x0
>       opts = (QemuOpts *) 0x1d2adc0
>       optind = 39
> ---Type <return> to continue, or q <return> to quit---
>       r = 0x7fffe23b4bf7 "-no-kvm-clock"
>       optarg = 0x0
>       loadvm = 0x0
>       machine = (QEMUMachine *) 0x862720
>       cpu_model = 0x7fffe23b4b80 "qemu64,model_id=Intel(R) Xeon(R) CPU", ' ' 
> <repeats 11 times>, "E5430  @ 2.66GHz,-tsc,-nx"
>       fds = {-499439560, 32767}
>       tb_size = 0
>       pid_file = 0x7fffe23b4b3f "/var/run/qemu/vm-244.pid"
>       incoming = 0x0
>       fd = 0
>       pwd = (struct passwd *) 0x0
>       chroot_dir = 0x0
>       run_as = 0x0
>       env = (struct CPUX86State *) 0x0
>       show_vnc_port = 0
>       params = {0x58d2a6 "order", 0x58d2ac "once", 0x58d2b1 "menu", 0x0}
> 

sorry people, i have to apologize. i was testing 0.12.5 on an old dell with a 
xeon e5430. obviously, something is broken on these systems.
if i run the same tests on a recent xeon L5640 everything runs flawlessly. no 
hangs and no kvm emulation errors.

my stress tests with xfs using a diablo newsserver have also not generated an 
xfs corruption within the last 24 hours with
0.12.5. with 0.12.4 it appeared always after a few hours.

i will keep you posted if anything changes.

thanks,
peter