Re: Processing files from a tar archive in parallel

[Ole Tange]

On Tue, Mar 29, 2011 at 11:41 PM, Hans Schou <chlor@schou.dk> wrote:
> On Tue, 29 Mar 2011, Ole Tange wrote:
>
>> While I loved Hans' idea, it does indeed have a race condition. This
>> should run 'ls -l' on each file after decompressing and clearly fails
>> now and then:
>>
>> $ tar xvf ../i.tgz | parallel ls -l > ls-l
>> ls: cannot access 1792: No such file or directory
>>
>> But you could unpack in a new dir and use:
>>
>> http://www.gnu.org/software/parallel/man.html#example__gnu_parallel_as_dir_processor
>
> A solution could be to print #1 file name when you got #2 file name. As tar
> files is a stream (sequence), #1 is written to disk when #2 name comes to
> the output. Something like:
>  tar xvf x.tgz | ./print-1-when-got-2.pl | parall....

tar xvf ../i.tgz | perl -ne 'print $last;$last=$_;END{print $last}' |
parallel ls -l >ls-l

Yep that seems to work too.


/Ole