[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Processing files from a tar archive in parallel

From: Jay Hacker
Subject: Re: Processing files from a tar archive in parallel
Date: Tue, 29 Mar 2011 16:14:21 -0400


That is a great idea.  However, can I be sure the file is completely
written to disk before tar prints the filename?  It seems to print the
filename first.  Could that not lead to a race condition, or the
consumer reaching the "end" of the file before tar has finished
writing it?

On Tue, Mar 29, 2011 at 11:20 AM, Hans Schou <> wrote:
> On Tue, 29 Mar 2011, Jay Hacker wrote:
>> I have a large gzipped tar archive containing many small files; just
>> untarring it takes a lot of time and space.  I'd like to be able to process
>> each file in the archive, ideally without untarring the whole thing first,
>> and I'd like to process several files in parallel.  Is there a recipe for
>> this with GNU Parallel?
> tar xvf big-file.tar.gz | parallel echo "Proc this file {}"
> Parallel will start when the first file is untared.
> /hans
> --
> Horsebakken 78, DK-2400 København NV

reply via email to

[Prev in Thread] Current Thread [Next in Thread]