Re: Processing files from a tar archive in parallel

[Ole Tange]

On Tue, Mar 29, 2011 at 11:41 PM, Cook, Malcolm <MEC@stowers.org> wrote:
> ooops, more like:
>
>        tar -t big-file.tar.gz  | parallel tar -O -x -f big-file.tar.gz '|' 
> someCommandThatReadsFromStdIn

You probably mean:

  tar -tf big-file.tar.gz  | parallel tar -O -x -f big-file.tar.gz {}
'|' someCommandThatReadsFromStdIn

I am afraid this is highly inefficient. If big-file.tar.gz contains
1000 files, then to extract the last file the second tar will have to
read through the first 999 files (as you cannot seek into tar.gz
files). In total you will be reading big-file.tar.gz 1000*999/2 times.


/Ole