[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

feature request: gzip/bzip support for sort

From: Craig Macdonald
Subject: feature request: gzip/bzip support for sort
Date: Sat, 13 Jan 2007 17:40:00 +0000
User-agent: Thunderbird (X11/20061206)

On some occasions, I have the need to sort extremely large files, but which compress well using programs such as gzip or bzip.

I can emulate the sorting of a gzipped files while keeping input compressed using shell pipes, eg
   zcat in.gz | sort | gzip > out.gz
However, if there is not enough temporary space available for sort to store to (ie, usually /tmp), then sort will fail.

In a similar vein, I can sort multiple large gzipped files, where each file is small enough to sort in available temporary space, but
the complete file is not, by making use of fifos.
for i in *.gz;
   zcat $i | sort | gzip > $i.out
   mkfifo $i.out.fifo
   zcat $i.out > $i.out.fifo &
sort -m *.out.fifo | gzip > out.gz

However, things would be made easier if sort, like tar, could support the use of compressors and decompressors, (a) for input and (b) for the temporary files. Are there any difficulties in adding such features to sort that I haven't envisaged?

Many thanks


reply via email to

[Prev in Thread] Current Thread [Next in Thread]