[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Qemu-devel] [Bug 735454] Re: live kvm migration with non-shared storage
Sebastian J. Bronner
[Qemu-devel] [Bug 735454] Re: live kvm migration with non-shared storage corrupts file system
Tue, 15 Mar 2011 12:56:46 -0000
** Attachment added: "A set of scripts to exercise the file system"
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
live kvm migration with non-shared storage corrupts file system
Status in QEMU:
Description of problem:
Migrating a kvm guest on non-shared lvm-storage using block migration
(-b flag) results in a corrupted file system if that guest is under
considerable I/O load.
Version-Release number of selected component (if applicable):
The error can be reproduced consistently.
Steps to Reproduce:
1. create a guest using lvm-based storage
2. create an LV on the destination node for the guest to be migrated
3. place the attached scripts somewhere on the guest's system
4. run 'runlots'
5. migrate the guest using the -b flag
6. if the migration doesn't complete in an appropriate amount of time
(45 minutes for our 100GB image), it will be necessary to stop the
test scripts: type 'killall python'
7. attempt to shut down the guest, forcing it off if necessary
8. access the partitions of the LV on the node: 'partprobe /dev/mapper
9. run fsck: 'fsck -n -f /dev/mapper/<volume-name>p1'
You should see a big mess of errors, that go beyond what can be
accounted for by an unclean shutdown.
Expected is a clean bill of health from fsck.
I suspect that there is some sort of race condition in the live
synchronization algorithm for dirty blocks used for block migration.
The only safe way to migrate guests in this scenario is by suspending
them just prior to the migration. That way they are first suspended,
then everything is transferred, and finally resumed on the target
node. When the I/O load is low, the migration works live, as well.
However, this is too risky to use on production systems because there
is no way to tell when the I/O load is too high for a successful live
Using this workaround is very dissatisfying because for a guest with a
100GB filesystem, the migration takes 45 minutes on our systems,
meaning that we have a downtime of 45 minutes. Having migrated other
guests with 0 downtime got us hooked.
The attached scripts to simulate high I/O load are somewhat artificial in
nature. However, the bug is motivated by a real-world scenario: We migrated a
productive mail-server that subsequently became buggy, finally crashed and
corrupted several of our customers' e-mails. Unfortunately a bug of this nature
can't be tested on non-productive systems, because they don't reach the
necessary load levels. The scripts reliably reproduce the failure experienced
by our mail-server.
|[Prev in Thread]
||[Next in Thread]|