[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
--pipepart implemented
From: |
Ole Tange |
Subject: |
--pipepart implemented |
Date: |
Wed, 9 Apr 2014 05:22:05 +0200 |
I have now implemented ---pipepart. It works like --pipe except that
it is way more efficient and works only on physical files. It does not
(and will never) support -N or -L as we cannot tell what line we are
on if we seek into the middle of the file.
parallel -a bigfile1 --pipepart --block 1g wc -c
This works similar to:
cat bigfile1 | parallel --pipe --block 1g wc -c
You can have multiple bigfiles:
parallel -a bigfile1 -a bigfile2 --pipepart --block 1g wc -c
This will work similar to:
cat bigfile1 bigfile2 | parallel --pipe --block 1g wc -c
Performance on my 8 core machine is a kickass 3.3 GB/s(!) compared to
350 MB/s using --pipe.
I am sure there are bugs, so start hunting.
git clone git://git.savannah.gnu.org/parallel.git
/Ole
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- --pipepart implemented,
Ole Tange <=