[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Merging (-m/--merge) slower than sorting irrespective of --parallel opti

Subject: Merging (-m/--merge) slower than sorting irrespective of --parallel option
Date: Fri, 22 Feb 2013 11:55:30 -0500

Using sort (GNU coreutils) 8.21
Linux xxxx 2.6.18-274.12.1.el5 #1 SMP Tue Nov 29 13:37:46 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
CentOS release 5.7 (Final)
16 cores / Pentium III Xeon (Cascades) / .18um / X5550 @ 2.67GHz / L1 data cache: 32K, 8-way, 64 byte lines / data TLB: 4K pages, 4-way, 64 entries / L2 unified cache: 256K, 8-way

Pre-sorting every file in the coreutils-8.21 source distro then using "sort --files0-from" on the sorted files unexpectedly outperforms "sort --merge --files0-from" by more than 100% (i.e. "sort --merge" is more than 2x SLOWER than non-merge.)  The gap narrows but still exists when disabling parallelism (--parallel=1.)  FWIW ~20% of the lines being sorted are identical (isolated newlines), and the same relative performance is observed with other datasets I tried (I used the coreutils sourcetree as it is readily available.)

SO: is this normal?

Here's what I did:
$ export LC_ALL=C
$ export LANG=C
$ cd /tmp/coreutils-8.21
$ mkdir /tmp/coreutils-8.21-sorted
$ find . -type d | sort | while read d ; do mkdir /tmp/coreutils-8.21-sorted/$d ; done
$ find . -type f | while read f ; do sort $f > /tmp/coreutils-8.21-sorted/$f ; done
$ cd /tmp/coreutils-8.21-sorted
$ find . -type f -print0 >lst
             total       used       free     shared    buffers     cached
Mem:      24674780    5897556   18777224          0     780368    4337448
-/+ buffers/cache:     779740   23895040
Swap:     26738680          0   26738680


time sort -m --files0-from lst >/dev/null

real    0m2.410s
user    0m1.466s
sys     0m0.943s

$ time sort --files0-from lst >/dev/null

real    0m0.813s
user    0m1.891s
sys     0m0.274s

Disabling parallelism,

$ time sort --parallel=1 -m --files0-from lst >/dev/null

real    0m2.545s
user    0m1.561s
sys     0m0.984s

$ time sort --parallel=1 --files0-from lst >/dev/null

real    0m1.748s
user    0m1.600s
sys     0m0.148s

And using -s:

$ time sort --parallel=1 -m -s --files0-from lst >/dev/null

real    0m2.502s
user    0m1.577s
sys     0m0.925s

$ time sort --parallel=1 -s --files0-from lst >/dev/null

real    0m1.826s
user    0m1.620s
sys     0m0.206s

Anyone have any idea why these non-intuitive results?  Why does "sort -m" appear so slow even with --parallel=1?
Should this go to address@hidden?

Thank you!

reply via email to

[Prev in Thread] Current Thread [Next in Thread]