bug#6131: [PATCH]: fiemap support for efficient sparse file copy

[jeff.liu]

Jim Meyering wrote:
> jeff.liu wrote:
> 
>> Jim Meyering wrote:
>>> Jim Meyering wrote:
>>>> Subject: [PATCH 01/10] cp: Add FIEMAP support for efficient sparse file 
>>>> copy
>>> FYI, using those patches, I ran a test for the first time in a few days:
>>>
>>>     check -C tests TESTS=cp/sparse-fiemap VERBOSE=yes
>>>
>>> It failed like this on an ext4 partition using F13:
>>>
>>>     + timeout 10 cp --sparse=always sparse fiemap
>>>     + fail=1
>>>     ++ stat --printf %s sparse
>>>     ++ stat --printf %s fiemap
>>>     + test 1099511628800 = 0
>>>     + fail=1
>>>
>>> That is very odd.  No diagnostic from cp, yet it failed
>>> after creating a zero-length file.
>>>
>>> Here's the corresponding piece of the script:
>>>
>>>     # It takes many minutes to copy this sparse file using the old method.
>>>     # By contrast, it takes far less than 1 second using FIEMAP-copy.
>>>     timeout 10 cp --sparse=always sparse fiemap || fail=1
>>>
>>>     # Ensure that the sparse file copied through fiemap has the same size
>>>     # in bytes as the original.
>>>     test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1
>>>
>>> However, so far I've been unable to reproduce the failure,
>>> running hundreds of iterations:
>>>
>>>     for i in $(seq 300); do printf .; make check -C tests \
>>>       TESTS=cp/sparse-fiemap VERBOSE=yes >& makerr-$i || break; done
>>>
>>> Have any of you heard of a problem whereby a cold cache can cause
>>> such a thing?  "echo 3 > /proc/sys/vm/drop_caches" didn't help.
>> Hi Jim,
>>
>> Have you run `sync' before clean the buffer and caches?  Actually, even run 
>> `sync' first, sometimes,
>> maybe the dirty objects still can not be freed in some cases. :(
> 
> Hi Jeff,
> 
> Thanks for the feedback.
> 
> The test script I ran above does not invoke sync between
> the dd invocation and the cp --sparse.
> If running sync before cp is required in order to avoid a failure,
> then I consider that a bug in cp.
> 
>> I can reproduce this issue on ext4 and btrfs(physical mounted partition) or 
>> just run the
> 
> If you can reproduce the cp failure, then please use my latest
> patch that diagnoses 2nd and subsequent FIEMAP ioctl failure
> and tell me what cp prints when it fails.
Strange!! Today I can not reproduce this issue on my laptop even repeat the 
test script up to 300 times.

Yesterday I also run test against your fiemap-copy git-tree with the latest 
diagnose patch, but it
did not show me the 2nd ioctl failure info because that 1TB sparse file only 
have one extent
allocated on ext4, so the ioctl(2) only hit once.

> 
> I would really like to know how the first ioctl can succeed,
> yet the second one fails.
> 
> Your demonstrations show that *after* the cp, one may have
> to sync in order to see precisely if/how the newly-created
> destination file is fragmented, but that is nothing new.
> Note the commented-out "sync" uses in the test script.
> As far as I can see, your examples do not show cp failing
> like it did for me.
Yeah, I just realized that the behaviour I observed is caused by the delay 
allocation mechanism of
the particular FS.


Thanks,
-Jeff
> 
> 
> 


-- 
With Windows 7, Microsoft is asserting legal control over your computer and is 
using this power to
abuse computer users.