[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: our benchmark-suite

From: Ludovic Courtès
Subject: Re: our benchmark-suite
Date: Wed, 16 May 2012 23:01:12 +0200
User-agent: Gnus/5.110018 (No Gnus v0.18) Emacs/24.0.93 (gnu/linux)


Andy Wingo <address@hidden> skribis:

> On Wed 25 Apr 2012 22:39, address@hidden (Ludovic Courtès) writes:
>>> So, those are the problems: benchmarks running for inappropriate,
>>> inconsistent durations;
>> I don’t really see such a problem.  It doesn’t matter to me if
>> ‘’ takes 2mn while ‘’ takes 40s, since I’m not
>> comparing them.
> Running a benchmark for 2 minutes is not harmful to the results, but it
> is a bit needless.  One second is enough.

Well, duration has to be chosen such that the jitter is small enough.
Sometimes it could be 2mn, sometimes 1s.


>>> and benchmarks being optimized out.
>> That should be fixed.
> In what way?  It would make those benchmarks different.
> Thesis: anything for which you would want to turn off the optimizer is
> not a good benchmark anyway.

Yes, it depends on the benchmarks.  For instance, I once added
benchmarks for ‘1+’ and ‘1-’, because I wanted to see the impact of an
optimization to the corresponding VM instructions.

Nowadays peval would optimize those benchmarks out.  Yet, the fact is
that I was interested in the performance of the underlying VM
instructions, regardless of what the compiler might be doing.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]