Re: Processing files from a tar archive in parallel

[Benjamin R. Haskell]

On Tue, 29 Mar 2011, Jay Hacker wrote:

> Hans,
>
> That is a great idea.  However, can I be sure the file is completely 
> written to disk before tar prints the filename?  It seems to print the 
> filename first.  Could that not lead to a race condition, or the 
> consumer reaching the "end" of the file before tar has finished 
> writing it?

(See the other fork of this thread.  Yes, it's an issue.)

Might be able to work around it with this (dumb) script:

==> ~/bin/wrapped-untar <==
#!/bin/sh
# usage: wrapped-untar file.tar.gz | parallel do whatever {}
unset prior
tar xvf "$@" | while read filename ; do
    [ -n "$prior" ] && echo "$prior"
        prior="$filename"
done
[ -n "$prior" ] && echo "$prior"
=========================

Assuming `tar` itself doesn't do things in parallel, that just delays 
the output of the filename until the next filename is printed.  And when 
it stops seeing new filenames, it prints the last one that was queued.

--
Best,
Ben