Re: [Qemu-devel] [PATCH] No change in userland tools after resizing qcow2 image

[John Snow]

On 07/20/2015 09:56 PM, 김태하 wrote:
>> On 07/19/2015 05:24 AM, Taeha Kim wrote:
>>> Hello,
>>>
>>>
>>> There is no change in userland tools after resizing qcow2 image except
>>> file utility.
>>>
>>> For example when resize qcow2 image, the "file" utility is detectable
>>> increased size.
>>> However, the "ls", “stat”, and “du” utility still don't know how many
>>> change size of image is changed.
>>> The following patch enables to let userland tools - ls, du, stat -
>>> know and apply changed size after resizing qcow2 image created by the
>>> qemu-img tool.
>>> Currently, “file” utility can only know without this patch.
>>>
>>> In addtion, "file" utility sometimes don't know qcow2 image format's
>>> version as follows.
>>>
>>> $ file 100G.qcow2
>>> 100G.qcow2: QEMU QCOW Image (Unknown)
>>>
>>>
>>> So, user can't have any information about qcow2 image size.
>>>
>>> ======================
>>> Signed-off-by: Taeha Kim <address@hidden> diff --git a/block.c
>>> b/block.c index d088ee0..93427f8 100644
>>> --- a/block.c
>>> +++ b/block.c
>>> @@ -2529,6 +2529,9 @@ int bdrv_truncate(BlockDriverState *bs, int64_t
>> offset)
>>>         if (bs->blk) {
>>>             blk_dev_resize_cb(bs->blk);
>>>         }
>>> +       if (ret == 0) {
>>> +           ret = truncate(bs->filename, offset);
>>> +       }
>>>     }
>>>     return ret;
>>> }
>>> =====================
>>>
>>>
>>> $ ./qemu/qemu-img --version
>>> qemu-img version 2.3.91, Copyright (c) 2004-2008 Fabrice Bellard
>>>
>>> The how-to-reproduce is as follows with three reproduction.
>>>
>>> First let’s say that we create a qcow2 image using qemu-img tool as
>>> follows. There is still no problem.
>>>
>>> $ ./qemu/qemu-img create -f qcow2 -o preallocation=metadata 100G.qcow2
>>> 100G Formatting '100G.qcow2', fmt=qcow2 size=107374182400
>>> encryption=off
>>> cluster_size=65536 preallocation='metadata' lazy_refcounts=off
>>> refcount_bits=16
>>>
>>> $ ./qemu/qemu-img check 100G.qcow2
>>> No errors were found on the image.
>>> 1638400/1638400 = 100.00% allocated, 0.00% fragmented, 0.00%
>>> compressed clusters Image end offset: 107390828544
>>>
>>> $ ls -l 100G.qcow2
>>> -rw-r--r--.   1 devjames devjames 107390828544 Jul 15 09:55 100G.qcow2
>>>
>>> $ stat 100G.qcow2
>>>   File: ‘100G.qcow2’
>>>   Size: 107390828544    Blocks: 32408          IO Block: 4096   regular file
>>> Device: fd00h/64768d    Inode: 5778245         Links: 1
>>> Access: (0644/-rw-r--r--)  Uid: ( 1000/devjames)   Gid: ( 1000/devjames)
>>> Context: unconfined_u:object_r:user_home_t:s0
>>> Access: 2015-07-15 09:55:17.269620129 +0900
>>> Modify: 2015-07-15 09:55:17.269620129 +0900
>>> Change: 2015-07-15 09:55:17.269620129 +0900
>>>  Birth: -
>>>
>>> $ du -bb 100G.qcow2
>>> 107390828544    100G.qcow2
>>>
>>> $ file 100G.qcow2
>>> 100G.qcow2: QEMU QCOW Image (v3), 107374182400 bytes
>>>
>>> But, from now on there is a problem.
>>>
>>> Second let’s say we resize the qcow2 image size from 100G to 200G
>>> using current qemu-img without the patch.
>>>
>>> $ ./qemu/qemu-img resize 100G.qcow2 200G Image resized.
>>>
>>> $ ./qemu/qemu-img check 100G.qcow2
>>> No errors were found on the image.
>>> 1638400/3276800 = 50.00% allocated, 0.00% fragmented, 0.00% compressed
>>> clusters Image end offset: 107390894080
>>>
>>> $ ls -l 100G.qcow2
>>> -rw-r--r--. 1 devjames devjames 107390832128 Jul 15 10:02 100G.qcow2
>>>
>>> $ stat 100G.qcow2
>>>   File: ‘100G.qcow2’
>>>   Size: 107390832128    Blocks: 32416          IO Block: 4096   regular file
>>> Device: fd00h/64768d    Inode: 5778245         Links: 1
>>> Access: (0644/-rw-r--r--)  Uid: ( 1000/devjames)   Gid: ( 1000/devjames)
>>> Context: unconfined_u:object_r:user_home_t:s0
>>> Access: 2015-07-15 10:02:04.842425479 +0900
>>> Modify: 2015-07-15 10:02:04.854425682 +0900
>>> Change: 2015-07-15 10:02:04.854425682 +0900
>>>  Birth: -
>>>
>>> $ du -bb 100G.qcow2
>>> 107390832128    100G.qcow2
>>>
>>> $ file 100G.qcow2
>>> 100G.qcow2: QEMU QCOW Image (v3), 214748364800 bytes
>>>
>>> We can see that “ls”, “stat”, and “du” utilities don’t know change
>>> size of qcow2 image except “file” one.
>>>
>>> Third let’s say we resize the qcow2 image size from 100G to 200G using
>>> qemu-img with the patch.
>>>
>>> $ ./qemu/qemu-img resize 100G.qcow2 200G Image resized.
>>>
>>> $ ./qemu/qemu-img check 100G.qcow2
>>> No errors were found on the image.
>>> 1638400/3276800 = 50.00% allocated, 0.00% fragmented, 0.00% compressed
>>> clusters Image end offset: 107390894080
>>>
>>> $ ls -l 100G.qcow2
>>> -rw-r--r--. 1 devjames devjames 214748364800 Jul 15 10:08 100G.qcow2
>>>
>>> $ stat 100G.qcow2
>>>   File: ‘100G.qcow2’
>>>   Size: 214748364800    Blocks: 32416          IO Block: 4096   regular file
>>> Device: fd00h/64768d    Inode: 5778245         Links: 1
>>> Access: (0644/-rw-r--r--)  Uid: ( 1000/devjames)   Gid: ( 1000/devjames)
>>> Context: unconfined_u:object_r:user_home_t:s0
>>> Access: 2015-07-15 10:04:37.595018501 +0900
>>> Modify: 2015-07-15 10:08:01.622484709 +0900
>>> Change: 2015-07-15 10:08:01.622484709 +0900
>>>  Birth: -
>>>
>>> $ du -bb 100G.qcow2
>>> 14748364800    100G.qcow2
>>>
>>> $ file 100G.qcow2
>>> 100G.qcow2: QEMU QCOW Image (v3), 214748364800 bytes
>>>
>>> Now we can see above all four utilities know change size of qcow2 image.
>>> So I made the patch above for the "qemu-img" tool so that "ls",
>>> “stat”, and “du” utility can be applied increased size of the qcow2
>>> image file.
>>>
>>
> 
> Hello,
> 
> Sorry for late response. At last, I found how to add ">" when the email is 
> replied in outlook. :)
> 
>> It seems to me that the real issue here is that the resize command you are
>> issuing doesn't respect your original pre-allocation desires.
>>
> 
> You're right. At first I thought that the change of resizing with 
> pre-allocation needs to be applied in user space.
> And the "preallocation" option doesn't have full, for example, 
> "preallocation=full".
> I expected that this will enable the qcow2 image to fill increased size fully 
> when preallocated image is resized.
> 
>> When changing the size of the qcow2 file, it generally just updates the
>> headers and doesn't pre-allocate those storage blocks. The actual size of
>> the .qcow2 image doesn't take up disk space for empty blocks. That's part
>> of its design. The reason 'file' can tell the "virtual" size of the disk
>> is because it is reading the header to do so.
>>
> 
> If it updates only headers, I think the "preallocation=metadata" is right. 
> But user still cannot know the exact size of the qcow2 image without the 
> qemu-img tool.
> If people who has only qcow2 images will guess or need many time to know the 
> image size. Because currently qemu-img doesn't support shrinking, perhaps the 
> user will have slack space unintentionally.
> 
>> ls and friends are telling you the size of the actual file, as you've
>> noticed.
>>
>> Changing the resize mechanisms to automatically truncate to achieve pre-
>> allocation is wrong, I'm afraid.
> 
> I agree with you. Could you tell me your opinion about adding the resize 
> mechanisms iff use "preallocation=full" or "preallocation=data"?
> Of course, new patch will have other approach with internal functions related 
> to qcow2 structure, not just simple userspace API.
> 
>>
>> I think if anything, we'd want to allow the qemu-img tool to take options
>> for the resize, concerning pre-allocation strategy.
> 
> I didn't understand what you were saying. Could you please explain to me one 
> more time?
> What is exactly pre-allocation strategy? You mean that current resize 
> function cannot change any more, or have to keep "as-is"?
> 

What I mean to say is that the qemu-img tool doesn't currently appear to
accept arguments for its resize function. That is, there's no way to say
e.g.

qemu-img resize -o preallocation=metadata 100G.qcow2 200G

because there is no valid "-o" flag for "resize" currently.

If it is your goal to resize a preallocated image to a new preallocated
image using the qemu-img tool, I think changes to qemu-img are what we want.

>>
>> --js
>>
> 
> Thank you very much.
> 
> Sincerely,
> Taeha
> 
> 
>>> I have created a VM using qcow2 image after patching using this patch.
>>> The 100G.qcow2 has resized 200G is detected and available.
>>> $ qemu-kvm -smp 4 -m 4096 100G.qcow2 -cdrom
>>> ../iso/Fedora-Live-Workstation-x86_64-21-5.iso
>>>
>>> Thanks.
>>>
>>>
>>>
>>> Sincerely yours,
>>> Taeha Kim
>>> Senior Engineer, Network Functions Virtualization Part, Samsung
>>> Electronics
>>>
>