Re: [Qemu-block] [VDSM] travis tests fail consistently since Apr 14

[Eric Blake]

On 06/11/2018 06:07 PM, Nir Soffer wrote:

> > Correct image files should be
> > aligned to 512 byte sectors, so something is wrong with your image to
> > start with (hard disks don't have half sectors).

Ideally, a guest will never see an unaligned image size: qemu should 
round UP when creating an image, and then truncate DOWN when opening an 
unaligned image that it cannot resize.  qemu-img DOES round the guest 
virtual size up on creation; and thus, for the raw format, a file 
created by qemu-img will be aligned, and you can only get an unaligned 
image by manual efforts.

But for qcow2 files, the qcow2 spec is clear that an incomplete final 
cluster (whether or not that last cluster end is also sector-aligned) is 
well-formed with the tail reading as zeroes; if the last cluster is 
guest-visible, qemu-img will have rounded it up to sector (but not 
necessarily cluster) alignment.  But if the last cluster is something 
else, like the refcount, L1, or L2 table, then yes, it is very common 
that the consumed disk size is not sector aligned.

$ qemu-img create -f qcow2 test3 123
Formatting 'test3', fmt=qcow2 size=123 cluster_size=65536 
lazy_refcounts=off refcount_bits=16
$ qemu-img info test3
image: test3
file format: qcow2
virtual size: 512 (512 bytes)
disk size: 196K
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
$ ls -l test3
-rw-r--r--. 1 eblake eblake 196616 Jun 11 20:32 test3
$ echo $((196616/512*512))
196608

So, you can see that the guest-visible size was rounded up (123 became 
512), which is now sector-aligned but not cluster-aligned; and that the 
final cluster occupies only 8 bytes at the moment (the difference 
between 196616 and 196608).

Maybe we should improve qemu to always create cluster-aligned qcow2 
images, but that's a bigger audit.

> Right. We indeed forbid uploads on unaligned images in the UI.
>
> But we found that qemu-img creates unaligned images:
>
> $ qemu-img create -f qcow2 empty.raw 10g
> Formatting 'empty.raw', fmt=qcow2 size=10737418240 cluster_size=65536
> lazy_refcounts=off refcount_bits=16
>
> $ ls -l empty.raw
> -rw-r--r--. 1 nsoffer nsoffer 196768 Jun 12 01:59 empty.raw
>
> $ python -c "print 196768 % 512"
> 160
>
> $ qemu-img map -f raw --output json test.raw

Wait; where did test.raw come from; given that your earlier commands 
were dealing with empty.raw?  And mapping a qcow2 file as though it were 
raw is indeed confusing (you don't want to do that for a guest using the 
image).

> The image becomes aligned once you write anything into it, but I still find
> it
> strange that qemu-img create such files. Shouldn't it create always 3
> complete
> clusters?

Because historical versions of qemu do not, we already have to cope with 
unaligned images (when opening them as qcow2; but not when 
transliterating and opening them as raw, since it took this long to find 
the regression); we could make future versions of qemu always create 
cluster-aligned images, but you'd still have to cope with existing 
images that aren't.

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org