[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

sort --parallel

From: Nathan Watson-Haigh
Subject: sort --parallel
Date: Thu, 18 Aug 2011 10:13:09 +0930

I’ve just discovered that a more recent version of coreutils (8.12) than the one I currently have installed (5.97) has the --parallel option. However, when I try to sort a large file I don’t see any speedup when using --parallel=8 over --parallel=1. In addition, I only see < 100% CPU usage. I’m on a 32 core system with 128GB RAM and would like to sort a stream consisting of several 100million lines in a smaller amount of time.


I’m also investigating GNU parallel, any comments on pros/cons of each? E.g. does GNU sort parallelise the merge part? My limited experience with GNU parallel is that it only parallelises the sort but then a single thread is used to do the merge across all the smaller sorted files.




AWRI logo 

Nathan Watson-Haigh
Senior Bioinformatician | The Australian Wine Research Institute
Waite Precinct, Hartley Grove cnr Paratoo Road, Urrbrae (Adelaide) SA 5064 | Map
PO Box 197, Glen Osmond SA 5064,
( +61 8 83136836 (direct) | ( +61 8 83136600 | Ê +61 8 83136601
8 Events

This communication, including attachments, is intended only for the addressee(s) and contains information which might be confidential and/or the copyright of The Australian Wine Research Institute (AWRI) or a third party. If you are not the intended recipient of this communication please immediately delete and destroy all copies and contact the sender. If you are the intended recipient of this communication you should not copy, disclose or distribute any of the information contained herein without the consent of the AWRI and the sender. Any views expressed in this communication are those of the individual sender except where the sender specifically states them to be the views of the AWRI. No representation is made that this communication, including attachments, is free of viruses. Virus scanning is recommended and is the responsibility of the recipient.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]