[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Processing files from a tar archive in parallel
From: |
Hans Schou |
Subject: |
Re: Processing files from a tar archive in parallel |
Date: |
Tue, 29 Mar 2011 18:02:28 +0200 (CEST) |
User-agent: |
Alpine 2.00 (DEB 1167 2008-08-23) |
On Tue, 29 Mar 2011, Benjamin R. Haskell wrote:
On Tue, 29 Mar 2011, Hans Schou wrote:
tar xvf big-file.tar.gz | parallel echo "Proc this file {}"
Hans, you left off the 'z' in 'tar zxvf':
No, its a new feature in tar (Ubuntu 10). If you specify a file, tar
will test compression type it self and unzip it.
Try this at home:
tar zcf htmls.tar.gz $(find /usr/share/doc -name '*.html')
And then do the thing:
tar xvf htmls.tar.gz | parallel tidy -miq
Jay, you probably also want to 'rm' the files as you go, since space
sounds like an issue.
And write a script which can handle one file, if you have a lot of
commands to run on one file - or decitions.
/hans
--
Horsebakken 78, DK-2400 København NV
- Re: Processing files from a tar archive in parallel, (continued)
- Re: Processing files from a tar archive in parallel, Ole Tange, 2011/03/29
- RE: Processing files from a tar archive in parallel, Cook, Malcolm, 2011/03/29
- RE: Processing files from a tar archive in parallel, Cook, Malcolm, 2011/03/29
- Re: Processing files from a tar archive in parallel, Ole Tange, 2011/03/29
- RE: Processing files from a tar archive in parallel, Cook, Malcolm, 2011/03/30
- Re: Processing files from a tar archive in parallel, Jay Hacker, 2011/03/30
- Re: Processing files from a tar archive in parallel, Hans Schou, 2011/03/29
- Re: Processing files from a tar archive in parallel, Ole Tange, 2011/03/29
- Re: Processing files from a tar archive in parallel, Benjamin R. Haskell, 2011/03/30
Re: Processing files from a tar archive in parallel, Benjamin R. Haskell, 2011/03/30
Re: Processing files from a tar archive in parallel, Benjamin R. Haskell, 2011/03/30