[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Processing files from a tar archive in parallel

From: Benjamin R. Haskell
Subject: Re: Processing files from a tar archive in parallel
Date: Tue, 29 Mar 2011 16:21:53 -0400 (EDT)
User-agent: Alpine 2.01 (LNX 1266 2009-07-14)

On Tue, 29 Mar 2011, Jay Hacker wrote:


That is a great idea. However, can I be sure the file is completely written to disk before tar prints the filename? It seems to print the filename first. Could that not lead to a race condition, or the consumer reaching the "end" of the file before tar has finished writing it?

(See the other fork of this thread.  Yes, it's an issue.)

Might be able to work around it with this (dumb) script:

==> ~/bin/wrapped-untar <==
# usage: wrapped-untar file.tar.gz | parallel do whatever {}
unset prior
tar xvf "$@" | while read filename ; do
    [ -n "$prior" ] && echo "$prior"
[ -n "$prior" ] && echo "$prior"

Assuming `tar` itself doesn't do things in parallel, that just delays the output of the filename until the next filename is printed. And when it stops seeing new filenames, it prints the last one that was queued.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]