[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: Processing files from a tar archive in parallel
From: |
Cook, Malcolm |
Subject: |
RE: Processing files from a tar archive in parallel |
Date: |
Wed, 30 Mar 2011 08:29:45 -0500 |
Ole - thanks for the education - Malcolm
> -----Original Message-----
> From: ole.tange@gmail.com [mailto:ole.tange@gmail.com] On
> Behalf Of Ole Tange
> Sent: Tuesday, March 29, 2011 5:03 PM
> To: Cook, Malcolm
> Cc: Jay Hacker; parallel@gnu.org
> Subject: Re: Processing files from a tar archive in parallel
>
> On Tue, Mar 29, 2011 at 11:41 PM, Cook, Malcolm
> <MEC@stowers.org> wrote:
> > ooops, more like:
> >
> > tar -t big-file.tar.gz | parallel tar -O -x -f
> big-file.tar.gz
> > '|' someCommandThatReadsFromStdIn
>
> You probably mean:
>
> tar -tf big-file.tar.gz | parallel tar -O -x -f
> big-file.tar.gz {} '|' someCommandThatReadsFromStdIn
>
> I am afraid this is highly inefficient. If big-file.tar.gz
> contains 1000 files, then to extract the last file the second
> tar will have to read through the first 999 files (as you
> cannot seek into tar.gz files). In total you will be reading
> big-file.tar.gz 1000*999/2 times.
>
>
> /Ole
>
- Processing files from a tar archive in parallel, Jay Hacker, 2011/03/29
- Re: Processing files from a tar archive in parallel, Hans Schou, 2011/03/29
- Re: Processing files from a tar archive in parallel, Jay Hacker, 2011/03/29
- Re: Processing files from a tar archive in parallel, Hans Schou, 2011/03/29
- Re: Processing files from a tar archive in parallel, Ole Tange, 2011/03/29
- RE: Processing files from a tar archive in parallel, Cook, Malcolm, 2011/03/29
- RE: Processing files from a tar archive in parallel, Cook, Malcolm, 2011/03/29
- Re: Processing files from a tar archive in parallel, Ole Tange, 2011/03/29
- RE: Processing files from a tar archive in parallel,
Cook, Malcolm <=
- Re: Processing files from a tar archive in parallel, Jay Hacker, 2011/03/30
- Re: Processing files from a tar archive in parallel, Hans Schou, 2011/03/29
- Re: Processing files from a tar archive in parallel, Ole Tange, 2011/03/29
- Re: Processing files from a tar archive in parallel, Benjamin R. Haskell, 2011/03/30
Re: Processing files from a tar archive in parallel, Benjamin R. Haskell, 2011/03/30
Re: Processing files from a tar archive in parallel, Benjamin R. Haskell, 2011/03/30