[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Bug-tar] [PATCH] improved sparse file detection

From: Bernd Schubert
Subject: [Bug-tar] [PATCH] improved sparse file detection
Date: Tue, 24 Aug 2010 17:17:03 +0200
User-agent: KMail/1.13.2 (Linux/2.6.32-24-generic; KDE/4.4.2; x86_64; ; )

Hi all,

for improved stat() performance the Lustre filesystem uses entirely empty 
sparse files on its metadata target (MDT). Now with hundredes of millions of 
sparse file of huge sizes, creating a backup of of the MDT using vanilla 
gnu-tar is basically impossible, as it needs far too much time to detect 
sparse files.

Attached is a patch to improve sparse file detection, if the entire file is 
sparse. All credits go to my colleague Kit Westneat as it was his idea.

Example: Directory with a single 4 TiB  sparse file:


address@hidden@bathl:/tmpa/tests$ time tar cvfS sparse.tar sparse

real    0m14.710s
user    0m7.100s
sys     0m7.450s

(I aborted with ctrl-c, tar was running at 99% CPU time)

Improved tar:

address@hidden@bathl:/tmpa/tests$ time 
/tmpa/devel/Lustre/tar/tar-1.22-13.el6/tar-1.22/src/tar cvfS sparse.tar sparse

real    0m0.006s
user    0m0.000s
sys     0m0.000s

The patch still has a TODO comment, as tar_sparse_scan(file, ...) may call
file->optab->scan_block() and simply did not figure out yet where the 
scan_block() method is assigned (if at all, dead code?).

Any comment is appreciated.


PS: I'm used to linux-style indentation and I'm not sure if I did it the right 
way. If it is wrong, please complain and I will try to reformat it.

Bernd Schubert
DataDirect Networks

Attachment: improve_sparse_detection.patch
Description: Text Data

reply via email to

[Prev in Thread] Current Thread [Next in Thread]