Re: [Qemu-block] [PATCH v1] [RFC] qcow2: add compression type feature

[Denis Plotnikov]

On 17.05.2019 2:25, John Snow wrote:
> 
> 
> On 5/16/19 9:48 AM, Denis Plotnikov wrote:
>> The patch adds some preparation parts for incompatible compression type
>> feature into QCOW2 header that indicates that *all* compressed clusters
>> must be (de)compressed using a certain compression type.
>>
>> It is implied that the compression type is set on the image creation and
>> can be changed only later by image conversion, thus the only compression
>> algorithm is used for the image.
>>
>> The plan is to add support for ZSTD and then may be something more effective
>> in the future.
>>
>> ZSTD compression algorithm consumes 3-5 times less CPU power with a
>> comparable compression ratio with zlib. It would be wise to use it for
>> data compression e.g. for backups.
>>
>> The default compression is ZLIB.
>>
> 
> (Merely a curiosity:)
> 
> Since this is coming from Virtuozzo, I trust that you've had good luck
> with ZSTD already in R&D. What do the compression ratios look like in
> practice? It's touted as "comparable to zlib" which certainly does sound
> quite nice for streaming compression of backups.
> 
> I suppose in the worst case it ought to be faster than bandwidth speeds,
> so no harm in utilizing it.
Yes, we did some research on it. Actually, there is a patch in the 
mailing list (please, take a look: ) which applies ZSTD compression in 
the migration.
Here are the numbers from there:

host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
migration to the same host
VM: 2xVCPU, 8G RAM total
5G RAM used, memory populated with postgreqsl data
produced by pgbench performance benchmark

Threads: 1 compress – 1 decompress

zstd provides slightly less compression ratio with almost the same
CPU usage but copes with RAM compression roughly 2 times faster

compression type              zlib       |      zstd
---------------------------------------------------------
compression level          1       5     |   1       5
compression ratio          6.92    7.05  |   6.69    6.89
cpu idle, %                82      83    |   86      80
time, sec                  49      71    |   26      31
time diff to zlib, sec                   |  -25     -41
time diff to zlib, %                     |  -47%    -56%

I general ZSTD provides better compression ratio on big dependent chunks 
of data. Than bigger the data size then better ZSTD it compresses.

Since, in our cases (migration: 4K RAM block, qcow2: cluster) we
have to compress independent chunks the ability of ZSTD to find better 
compression solution is restricted.

Although, the compression ratio is pretty much the same in both cases 
the experiments shown that ZSTD does the compression much faster (x2).

Which is obviously good for us.

Here is other comparison of ZLIB vs ZSTD without any application in qemu 
which shows that ZSTD works faster

zlib = max compression level
zstd = compression level 5 (max 22)

cycles consumed for compression:
                                                
        4k txt  4K ram  4K bin  64K txt  64K ram  64K bin
zlib      400K    344K    1.3M      13M       5M    92.3M
zstd      350K    235K    312K     3.3M     1.3M     2.4M
Diff,%     -12     -32     -77      -75      -73      -97
                                                
size after compression in bytes:

        4k txt  4K ram  4K bin  64K txt  64K ram  64K bin
zlib      1542    3599    1403  17386     64735     20609
zstd      1568    3540    1250  17656     65546     20023
Diff,%      -2       2      11     -2        -1         3

Data sources for test files [we took 4K and 64K chunks from there]:

txt = linux/Documentation/memory-barriers.txt
ram = /boot/initramfs-4.20.0-rc6+.img
bin = x86_64-softmmu/qemu-system-x86_64

Increasing of ZSTD compression ratio didn't give any significant 
improvements of the out size but slowed down the pace of ZSTD

Denis

> 
>> Signed-off-by: Denis Plotnikov <address@hidden>

-- 
Best,
Denis