Re: Processing files from a tar archive in parallel

[Benjamin R. Haskell]

On Tue, 29 Mar 2011, Hans Schou wrote:

> On Tue, 29 Mar 2011, Benjamin R. Haskell wrote:
>
> > On Tue, 29 Mar 2011, Hans Schou wrote:
> > > tar xvf big-file.tar.gz | parallel echo "Proc this file {}"
>
> > Hans, you left off the 'z' in 'tar zxvf':
>
> No, its a new feature in tar (Ubuntu 10). If you specify a file, tar 
> will test compression type it self and unzip it.

Oy.  I guess that's preferable to -z vs -j vs -J.  Good to know.


> > Jay, you probably also want to 'rm' the files as you go, since space sounds 
> > like an issue.
>
> And write a script which can handle one file, if you have a lot of commands 
> to run on one file - or decitions.

+1

Any thoughts on the race condition, though?

--
Best,
Ben