[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
feature request: gzip/bzip support for sort
From: |
Craig Macdonald |
Subject: |
feature request: gzip/bzip support for sort |
Date: |
Sat, 13 Jan 2007 17:40:00 +0000 |
User-agent: |
Thunderbird 1.5.0.9 (X11/20061206) |
On some occasions, I have the need to sort extremely large files, but
which compress well using programs such as gzip or bzip.
I can emulate the sorting of a gzipped files while keeping input
compressed using shell pipes, eg
zcat in.gz | sort | gzip > out.gz
However, if there is not enough temporary space available for sort to
store to (ie, usually /tmp), then sort will fail.
In a similar vein, I can sort multiple large gzipped files, where each
file is small enough to sort in available temporary space, but
the complete file is not, by making use of fifos.
for i in *.gz;
do
zcat $i | sort | gzip > $i.out
mkfifo $i.out.fifo
zcat $i.out > $i.out.fifo &
done
sort -m *.out.fifo | gzip > out.gz
However, things would be made easier if sort, like tar, could support
the use of compressors and decompressors, (a) for input and (b) for the
temporary files. Are there any difficulties in adding such features to
sort that I haven't envisaged?
Many thanks
Craig
- feature request: gzip/bzip support for sort,
Craig Macdonald <=
- Re: feature request: gzip/bzip support for sort, Jim Meyering, 2007/01/13
- Re: feature request: gzip/bzip support for sort, Dan Hipschman, 2007/01/13
- Re: feature request: gzip/bzip support for sort, Dan Hipschman, 2007/01/13
- Re: feature request: gzip/bzip support for sort, Dan Hipschman, 2007/01/14
- Re: feature request: gzip/bzip support for sort, Dan Hipschman, 2007/01/15
- Re: feature request: gzip/bzip support for sort, Dan Hipschman, 2007/01/15
- Re: feature request: gzip/bzip support for sort, Paul Eggert, 2007/01/16