bug#7649: sparse files and commands

[Eric Blake]

[please keep the list in the loop, and don't top-post on technical lists]

On 12/15/2010 03:20 PM, address@hidden wrote:
> THANKS, I see what you mean.
> Also food for thought, IF OS supports it, skip file blocks that are all null 
> in a sparse file.
> This way less IO and less processing.

That's what I was alluding to - we are working on adding fiemap support
here:

http://git.savannah.gnu.org/cgit/coreutils.git/log/?h=fiemap-copy

as well as a proposal for a hole iterator on OSs that support it (Linux
supports it for ext4 and btrfs via fiemap ioctl; Solaris supports it via
lseek(,SEEK_HOLE)

http://lists.gnu.org/archive/html/bug-coreutils/2010-07/msg00117.html

Note that an explicit block of all 0s cannot be efficiently skipped;
this only works for sparse files (where the block is completely
represented in the metadata of the file, and is implicitly all 0s).
Also, there has been movement to get Linux to add support for punching
holes into an existing file (right now, the only way to make a file more
sparse is to create a new file with the same contents while leaving
holes where the source had all-0 blocks, but that's obviously not as
efficient as modifying a file in-place to add a hole).

Once hole iteration is implemented (and right now, cp is our guinea
pig), then we can port that effort to a number of other programs (tr,
cmp, tar, ...) to make for more efficient I/O on files where we can
behave differently if we know that a block is sparse.

-- 
Eric Blake   address@hidden    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

signature.asc
Description: OpenPGP digital signature