qemu-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] RFC: moving fsfreeze support from the userland guest ag


From: Michael Roth
Subject: Re: [Qemu-devel] RFC: moving fsfreeze support from the userland guest agent to the guest kernel
Date: Wed, 27 Jul 2011 11:07:13 -0500
User-agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:5.0) Gecko/20110624 Thunderbird/5.0

On 07/27/2011 10:24 AM, Andrea Arcangeli wrote:
Hello everyone,

I've been thinking at the current design of the fsfreeze feature used
by libvirt.

It currently relays on an userland agent in the guest talking to qemu
with some vmchannel communication. The guest agent would walk the
filesystems in the guest and call fsfreeze ioctl on them.

The fsfreeze is an optional feature, it's not required to do safe
snapshots, after fsfreeze (regardless if available or not) QEMU must
still block all I/O for all qemu blkdevices before the image is saved,
to allow safe snapshotting of non-linux guests. Then if a VM is
restarted in the snapshot it becomes identical to a fault tolerance
fallback with nfs or drdb in a highly available
configuration. Fsfreeze just provides some further (minor) benefit on
top of that (which probably won't be available for non-linux guests
any time soon).

The benefits this optional fsfreeze feature provides to the snapshot
are:

1) more peace of mind by not relaying on the kernel journal reply code
when snapshotting journaled/cow filesystems like ext4/btrfs/xfs

2) all dirty outstanding cache is flushed, which reduces the chances
of running into userland journaling data reply bugs if userland is
restarted on the snapshot

3) allows safe live snapshotting of not jorunaled fs like vfat/ext2 on
linux (not so common, and vfat on non-linux guest won't benefit)

4) allows to mount the snapshotted image readonly without requiring
metadata journal reply

Problem is that having a daemon in guest userland is not my
preference, considering it can be done with a virtio-fsfreeze.ko
kernel module in guest without requiring any userland modification to
the guest (and no interprocess communication through vmchannel
or similar way).

This means a kernel upgrade in the guest that adds the
virtio-fsfreeze.ko virtio paravirt driver would be enough to be able
to provide fsfreeze during snapshots.

A virtio-fsfreeze.ko would certainly be more developer friendly, you
could just build the kernel and even boot it with -kernel bzImage
(after building it with VIRTIO_FSFREEZE=y). Then it'd just work
without any daemon or vmchannel or any other change to the guest
userland.

I could see some advantage in not having to modify qemu if libvirt was
talking directly to the guest agent, so to avoid any knowledge into
qemu about FSFREEZE. But it's not even like that, I see FSFREEZE guest
agent patches floating around. So if qemu has to be modified and be
aware of the fsfreeze feature in the userland guest agent (and not
just asked to block all I/O which doesn't require any guest knowledge
and in turn it'd remain agnostic about fsfreeze) I think it'd be
better if the fsfreeze qemu code would just go into a virtio backend.

There is also an advantage in reliability as there's no more need to
worry about mlocking the memory of the userland guest agent, making
sure no lib is calling any I/O function to be able to defreeze the
filesystems later, making sure the oom killer or a wrong kill -9
$RANDOM isn't killing the agent by mistake while the I/O is blocked
and the copy is going. The guest kernel is a more reliable and natural
place to call fsfreeze through a virtio-fsfreeze guest driver without
having to spend time into worrying about the reliability of the
guest-agent feature. It'd surely also waste less memory in the guest
(not that the agent takes much memory but a few kbytes of .text of a
kernel module for this surely would takes a fraction of the mlocked
RAM the agent would take, the RAM saving is the least interesting
part of course).

If there was no hypervisor behind the kernel, it could only be the
userland starting a fsfreeze, so we shouldn't be fooled into thinking
userland is the best place where to start a fsfreeze invocation, it's
most certainly not, but on the host (without virt) there's no other
thing that could possibly ask for it. But here we have an hypervisor
behind the guest kernel that asks for it, so starting the fsfreeze
through a virtio-fsfreeze.ko kernel module loaded into the guest
kernel (or linked into the guest kernel) sounds a cleaner and more
reliable solution (maybe simpler too).

I'd be certainly a more friendly solution for developers to test or
run it, libvirt would talk only with qemu, and qemu would only talk
with the guest kernel without requiring any modification to the guest
userland. My feeling is that usually what feels much simpler to use
for developers tends to be a better solution (not guaranteed) and to
me a virtio-fsfreeze.ko solution would look much simpler to use.

There are drawbacks, like the fact respinning an update to the
fsfreeze code, would then require an upgrade of the guest kernel,
instead of a package update. But there are avantages too in terms of
coverage, as an updated kernel would also run on top of an older guest
userland that may not have a agent package to install through a
repository.

In any case if the virtio-fsfreeze.ko doesn't register into qemu
virtio-fsfreeze backend, the qemu monitor command should still just
work and allow snapshotting by just only blocking all I/O, that is
more than enough for a not-buggy guest capable of fault tolerance
against power loss.

I understand an agent may be needed for other features but I think
whenever a feature is better suited for not requiring userland guest
support, it shouldn't. To me requiring modifications to the guest
userland, looks the least transparent and most intrusive possible way
to implement a libvirt feature so it should be used when it has
advantages and I see mostly disadvantages here.


One thing worth mentioning is that the current host-side interface to the guest agent is not what we're hoping to build libvirt interfaces around. It's a standalone, out-of-band tool for now, but when QMP is converted to QAPI the guest agent interfaces will be exposed to the host transparently to the host as normal QMP commands. libvirt should be able to tell the difference from a guest-agent induced fsfreeze or a guest kernel induced fsfreeze (except perhaps to identify extended capabilities in a particular case):

http://wiki.qemu.org/Features/QAPI/GuestAgent

Another thing to note is that snapshotting is not necessarily something that should be completely transparent to the guest. One of the planned future features for the guest agent (mentioned in the snapshot wiki, and a common use case that I've seen come up elsewhere as well in the context of database applications), is a way for userspace applications to register callbacks to be made in the event of a freeze (dumping application-managed caches to disk and things along that line). The implementation of this would likely be a directory where application can place scripts in that get called in the event of a freeze, something that would require a user-space daemon anyway.

Also, in terms of supporting older guests, the proposed guest tools ISO (akin to virtualbox/vmware guest tools):

http://lists.gnu.org/archive/html/qemu-devel/2011-06/msg02239.html

would give us a distribution channel that doesn't require any involvement from distro maintainers. A distro-package to boot strap the agent would be still be preferable, but the ISO approach seems to work well in practice. And for managed environments getting custom packages installed generally isn't as much of a problem as requiring reboots or kernel changes.

This is just a suggestions, I think the agent should work too.

Thanks a lot,
Andrea





reply via email to

[Prev in Thread] Current Thread [Next in Thread]