Re: RFC: new cp option: --efficient-sparse=HOW

[Jim Meyering]

Pádraig Brady wrote:
> On 31/01/11 21:46, Jim Meyering wrote:
>> Now that we have can read sparse files efficiently,
>> what if I want to copy a 20PiB sparse file, and yet I want to
>> be sure that it does so efficiently.  Few people can afford
>> to wait around while a normal processor and storage system process
>> that much raw data.  But if it's a sparse file and the src and dest
>> file systems have the right support (FIEMAP ioctl), then it'll be
>> copied in the time it takes to make a few syscalls.
>>
>> Currently, when the efficient sparse copy fails, cp falls back
>> on the regular, expensive, read-every-byte approach.
>>
>> This proposal adds an option, --efficient-sparse=required,
>> to make cp fail if the initial attempt to read the sparse file fails,
>> rather than resorting to the regular (very slow in the above case) copy
>> procedure.
>>
>> The default is --efficient-sparse=auto, and for symmetry,
>> I've provided --efficient-sparse=never, in case someone finds
>> a reason to want to skip the ioctl.
>
> I'm 50:50 on whether we should add this option.
> If a user doesn't want to copy the file if it's going
> to be too slow, then what other options do they have?
> Also the sparseness of a file can change over time.
>
> Given the fairly specialized need for such an option,
> perhaps it's best to foist to other tools, like:
>
> test $(du f) -lt $limit && df -T -t myfs f && cp f f2
>
> or perhaps
>
> timeout $time_limit cp f f2 || { rm f2; echo "oops" >&2; }

[Rats!  I thought I had sent this at 5pm two days ago
 and happen upon the "unsent" buffer only now.  ]

Hi Pádraig,

Thanks for the feedback.
I think we'll want *some* option, eventually, if only to tell
cp to choose how to treat a sparse file:

  1) copy the file (using extents, if possible) and introducing holes
     wherever possible.  This produces essentially the same result as
     cp in coreutils-8.9 and earlier.

  2) copy the file as precisely as possible.  This is possible now,
     when the FIEMAP ioctl works and when we can induce holes in
     a destination file.

For the imminent coreutils-8.10 release, I'll refrain from adding any
new command-line option, and let the default be #1, as it is with the
code now on master.

Does anyone know how to determine if a file system (say the one with ".")
supports the FIEMAP ioctl, but without compiling/running a C program?
I.e., via perl or python?  I've written a tiny C program that works
and a Perl one that is supposed to be identical (at least the buffer
arg has identical content), yet does not work.
The goal is to make the tests more robust by replacing the kludgey
FS-name-based test with an actual syscall-based test.

Jim