Re: [Qemu-block] [Qemu-devel] QCOW2 support for LZO compression

[Daniel P. Berrange]

On Mon, Jun 26, 2017 at 11:20:33AM +0200, Peter Lieven wrote:
> 
> Am 26.06.2017 um 10:28 schrieb Kevin Wolf:
> > [ Cc: qemu-devel; don't post to qemu-block only! ]
> > 
> > Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
> > > Hi,
> > > 
> > > I am currently working on optimizing speed for compressed QCOW2
> > > images. We use them for templates and would also like to use them for
> > > backups, but the latter is almost infeasible because using gzip for
> > > compression is horribly slow. I tried to experiment with different
> > > options to deflate, but in the end I think its better to use a
> > > different compression algorithm for cases where speed matters. As we
> > > already have probing for it in configure and as it is widely used I
> > > would like to use LZO for that purpose. I think it would be best to
> > > have a flag to indicate that compressed blocks use LZO compression,
> > > but I would need a little explaination which of the feature fields I
> > > have to use to prevent an older (incompatible) Qemu opening LZO
> > > compressed QCOW2 images.
> > > 
> > > I also have already some numbers. I converted a fresh Debian 9 Install
> > > which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
> > > compressed QCOW2.  With GZIP compression the result is 356MB whereas
> > > the LZO version is 452MB. However, the current GZIP variant uses 35
> > > seconds for this operation where LZO only needs 4 seconds. I think is
> > > is a good trade in especially when its optional so the user can
> > > choose.
> > > 
> > > What are your thoughts?
> > We had a related RFC patch by Den earlier this year, which never
> > received many comment and never got out of RFC:
> > 
> > https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html
> 
> I was not aware of that one. Thanks for pointing out.
> 
> > 
> > So he chose a different algorithm (zstd). When I asked, he posted a
> > comparison of algorithms (however a generic one and not measured in the
> > context of qemu) that suggests that LZO would be slightly faster, but
> > have a considerable worse compression ratio with the settings that were
> > benchmarked.
> 
> My idea to choose LZO was that it is widely available and available in
> any distro you can think of. We already have probing for it in configure.
> My concern with ZSTD would be that it seems there are no packages
> available for most distros and that it seems to be multi-threaded. I don't
> know if this will cause any trouble?

As a distro maintainer I'd always prefer option to use a library that is
already widely available. While ZSTD could certainly be added to distros,
it means the QEMU maintainer will end up having to package it & become
the defacto long term maintainer of it long term, which is an extra burden.

WRT to making compression algorithms configurable, I think it is important
to ensure we don't add lots of optional algorithms. An important factor is
portability of images - we don't want to end up with each distro's build
of QEMU enabling a different sub-set of compression algorithms, as that is
going to harm interoperability for distributed images. This again makes me
prefer a compression format whose library is widely available, as that makes
it highly likely that the distro will choose to enable it during build.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|