[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH] sort: Add --threads option, which parallelizes internal sort
From: |
Ralf Wildenhues |
Subject: |
Re: [PATCH] sort: Add --threads option, which parallelizes internal sort. |
Date: |
Sat, 20 Jun 2009 10:06:03 +0200 |
User-agent: |
Mutt/1.5.20 (2009-06-15) |
Hi Glen,
* glen lenker wrote on Fri, Jun 19, 2009 at 01:03:10PM CEST:
> On Fri, Jun 12, 2009 at 12:54 AM, Ralf Wildenhues wrote:
> >
> > No, I did not specify the amount of RAM. The system I tested on has
> > plenty of RAM, way more than is needed for the sort. Specifying
> > something like 2G of RAM does not make any visible difference.
>
> When testing on gcc16, 16G of RAM, sort chooses to use only 1G of
> RAM for a 256M file, and ends up hitting the external merge code. It
> was through experimentation with gprof that I found that anything less
> than 2G causes sort to do an external sort on gcc16.
Hmm, ok. I did retest with 2G at the time, but I didn't check logs for
whether the external merge code was used. I can try to recheck sometime
(systems are busy ATM).
> > > > 160.22user 1.34system 2:41.61elapsed 99%CPU
> > > > 159.83user 1.45system 1:27.12elapsed 185%CPU
> > > > 159.84user 1.56system 0:52.26elapsed 308%CPU
> > > > 160.67user 1.53system 0:36.26elapsed 447%CPU
> > > >
> > > > This seems to be what I would expect from a good implementation.
> >
> > Yes, 56% efficiency going from 1 to 8 threads sounds like a much better
> > number, also your system overhead looks very sane compared to what I
> > saw. Seems like it's a system-specific issue after all. Which Linux
> > kernel version and which pthread (glibc) version are you using?
>
> This was on gcc16.
> Linux gcc16 2.6.18-6-vserver-amd64 #1 SMP Thu Dec 25 21:35:11 UTC 2008
> x86_64 GNU/Linux
> glibc 2.3.6.ds1-13etch9
here:
Linux 2.6.24-23-generic
GNU C Library stable release version 2.7, by Roland McGrath et al.
> > Glen, could you look at this, too? I mean just timings of 1 vs 2
> > threads for different file sizes? Thanks.
>
> The data was again cat'ed instances of the dictionary, put onto tmpfs. I
> made sure that each time it had enough memory.
>
> 1st run
> memory 1 thread 2 threads speedup
> 16M 7.27 4.04 ~1.8
> 32M 16.15 9.19 ~1.75
> 64M 35.06 19.61 ~1.78
> 128M 75.47 40.99 ~1.84
> 256M 162.30 87.27 ~1.86
>
> 2nd run
> memory 1 thread 2 threads speedup
> 16M 7.27 4.05 ~1.8
> 32M 16.15 8.86 ~1.82
> 64M 35.07 19.18 ~1.83
> 128M 75.22 41.28 ~1.82
> 256M 161.76 87.77 ~1.84
Well, that is a small increase and probably below noise level,
but at least there is no decrease, and it looks fairly consistent,
so I think that is good.
> On this subject, what exactly is the motivation of sort's
> default_sort_size()?
> Shouldn't it try to do an internal sort if possible?
Reading the change logs, I think it was added to avoid overcommitting
memory (and thus hitting the chance of being killed later by an OOM
situation).
Cheers,
Ralf