On Thu, Dec 7, 2017 at 10:15 AM Gianluca Cecchi <
address@hidden> wrote:
Trying harder...
It is not. file size is what you get from stat:
$ truncate -s 1g empty
$ stat empty
File: 'empty'
Size: 1073741824 Blocks: 0 IO Block: 4096 regular file
...
$ qemu-img info empty
image: empty
file format: raw
virtual size: 1.0G (1073741824 bytes)
disk size: 0
The value "disk size" used by qemu-img is confusing and not useful
when you want to transfer the file to another host.
I don't know why qemu-img display this value instead of the actual
file size, adding qemu-block mailing list in case someone can explain
this.
When you upload or download this file, you will transfer 1g of zeros.
Yes this is raw sparse file.
That bug is not related. It was about converting raw sparse format
to qcow2.
In 4.2 we use qemu-img map and the same algorithm used in qemu to
estimate the required size needed to convert raw volume to qcow2 volume.
For transferring images over http, there is no way to avoid sending
unused blocks, except compression.
You can use rsync to transfer images if you like, but we support
only http.
I tested creating qcow2 files with virt manager, and it seems to use
the preallocation=metadata by default (behind your back), I guess
for improving performance in some cases. oVirt doe not use this
option so we don't have this issue when downloading files.
Here example of image created by virt-manager:
# ls -lsh fedora26-2.img
3.4M -rw-------. 1 root root 21G Dec 7 21:36 fedora26-2.img
# qemu-img info fedora26-2.img
image: fedora26-2.img
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 3.3M
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: true
refcount bits: 16
corrupt: false
And here is same image as ovirt create images:
# qemu-img create -f qcow2 test.qcow2 20g
Formatting 'test.qcow2', fmt=qcow2 size=21474836480 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
# qemu-img info test.qcow2
image: test.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 196K
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
# ls -lhs test.qcow2
196K -rw-r--r--. 1 root root 193K Dec 7 21:40 test.qcow2
So to download this image you need to transfer only 196k of data, not 20g.
We can get the same results as virt-manager if we use preallocation=metadata:
# qemu-img create -f qcow2 -o preallocation=metadata test-preallocation-metadata.qcow2 20g
Formatting 'test-preallocation-metadata.qcow2', fmt=qcow2 size=21474836480 encryption=off cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16
# qemu-img info test-preallocation-metadata.qcow2
image: test-preallocation-metadata.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 3.3M
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
# ls -lhs test-preallocation-metadata.qcow2
3.4M -rw-r--r--. 1 root root 21G Dec 7 21:42 test-preallocation-metadata.qcow2
To optimize the import of images with unneeded preallocation, you can
convert them to unallocated and compressed images as I showed
in the previous mail.
Here is an example with the preallocated images, like the one from virt-manager.
I'll create file with some data - I'm using data that will compress well
to make the example more convincing :-)
# dd if=/dev/zero bs=8M count=128 | tr "\0" "\1" > image.raw
128+0 records in
128+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.88171 s, 571 MB/s
Add some empty space:
# truncate -s 10g image.raw
# ls -lhs image.raw
1.1G -rw-r--r--. 1 root root 10G Dec 7 22:01 image.raw
Now lest create a qcow2 preallocated image like virt-manger does:
# qemu-img convert -f raw -O qcow2 -o preallocation=metadata image.raw image-preallocated-metadata.qcow2
# qemu-img info image-preallocated-metadata.qcow2
image: image-preallocated-metadata.qcow2
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 1.0G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
# ls -lhs image*
1.1G -rw-r--r--. 1 root root 11G Dec 7 22:02 image-preallocated-metadata.qcow2
1.1G -rw-r--r--. 1 root root 10G Dec 7 22:01 image.raw
Finally, create a compressed qcow2, optimized for upload:
# qemu-img convert -c -f qcow2 -O qcow2 image-preallocated-metadata.qcow2 image-preallocated-metadata-optimized.qcow2
# qemu-img info image-preallocated-metadata-optimized.qcow2
image: image-preallocated-metadata-optimized.qcow2
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 1.6M
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
# ls -lhs image*
1.6M -rw-r--r--. 1 root root 1.7M Dec 7 22:04 image-preallocated-metadata-optimized.qcow2
1.1G -rw-r--r--. 1 root root 11G Dec 7 22:02 image-preallocated-metadata.qcow2
1.1G -rw-r--r--. 1 root root 10G Dec 7 22:01 image.raw
All the images contain exactly the same data.
Now, how can we optimize your image on upload? we can't.
When uploading from the UI, our code run in the browser
and we don't have access to your file system.
When using the oVirt SDK, you write the code for uploading so you can
do the conversion if want to save upload time and bandwidth and can
spend the time to do the conversion.
We can provide an upload command line tool that will convert
and compress images before upload, and will use the oVirt SDK
to upload the images.
But I think it will be simpler and easier to maintain to support
compression when uploading or downloading data. This is
already supported by browsers and common tools like curl,
and it works with any file format without conversion.
I hope I convinced you this time ;-)
Nir