Re: Processing files from a tar archive in parallel

[Hans Schou]

On Tue, 29 Mar 2011, Ole Tange wrote:

> While I loved Hans' idea, it does indeed have a race condition. This
> should run 'ls -l' on each file after decompressing and clearly fails
> now and then:
>
> $ tar xvf ../i.tgz | parallel ls -l > ls-l
> ls: cannot access 1792: No such file or directory
>
> But you could unpack in a new dir and use:
> http://www.gnu.org/software/parallel/man.html#example__gnu_parallel_as_dir_processor

A solution could be to print #1 file name when you got #2 file name. 
As tar files is a stream (sequence), #1 is written to disk when #2 
name comes to the output. Something like:
  tar xvf x.tgz | ./print-1-when-got-2.pl | parall....



Joke of the week:
You could also use strace and look for 'close', try:
  strace -eopen,close tar xf x.tgz

Pipe the output to a perl script and print the file name when you get 
the close command ;-)

/hans
--
Horsebakken 78, DK-2400 København NV