Re: [Chicken-users] Nursery sizing considered stupid

[Brandon J. Van Every]

Brandon J. Van Every wrote:
> > -- Using a stack size of 131072 bytes.
> >        It shows a 15% performance improvement over the default stack 
> > size.
> >
> > Brilliant!  128K shows a 15% improvement over the default 128K!
> >   
>
> I must have screwed the logic when I shuffled the location of 
> DEFAULT_TARGET_STACK_SIZE around.  I will fix it shortly.

Frighteningly, having examined the testing logic, it is correct.  It's 
actually reporting on 2 different runs of the 128K sample size.  Once 
when testing the default, and once when looping through the increasing 
sample sizes.  This means with 10 samples, the default of 128K shows a 
variance of 15% !!  That means variance on your machine severely sucks, 
even worse than on mine.  10 samples is a bit random on my machine, but 
with a 5% performance threshold the results are stable.  Apparently, 
there's no reason to assume the sampling results will be stable on any 
other machine.

So, I'm bumping the samples to 100, and implementing a low-high variance 
test.  The low-high variance test won't take any more time, and will 
demonstrate how random the sampling actually is.  100 samples is kinda 
slow on my box.  I was hoping to make due with fewer samples and have 
faster build times.  But this apparently won't work in the general 
case.  If 100 samples isn't good enough to create testing stability for 
machines in general, then nsample is junk.  If proven to be junk, then 
we will need to either improve it or scrap it.

I'm still working on the low-high testing method.  It's not in Darcs yet.

I will ask again:

> What era was nsample written in?  Oftentimes, benchmarks get written 
> for stuff, but hardware moves on, rendering them less relevant.  Like, 
> maybe it was a great benchmark for a 486 or something.

I would also like to know if the 300000 bytes requirement for MSVC is 
truth or historical fiction.


Cheers,
Brandon Van Every