Re: our benchmark-suite

[Ludovic Courtès]

Hi!

Andy Wingo <address@hidden> skribis:

> On Wed 25 Apr 2012 22:39, address@hidden (Ludovic Courtès) writes:
>
>>> So, those are the problems: benchmarks running for inappropriate,
>>> inconsistent durations;
>>
>> I don’t really see such a problem.  It doesn’t matter to me if
>> ‘arithmetic.bm’ takes 2mn while ‘vlists.bm’ takes 40s, since I’m not
>> comparing them.
>
> Running a benchmark for 2 minutes is not harmful to the results, but it
> is a bit needless.  One second is enough.

Well, duration has to be chosen such that the jitter is small enough.
Sometimes it could be 2mn, sometimes 1s.

[...]

>>> and benchmarks being optimized out.
>>
>> That should be fixed.
>
> In what way?  It would make those benchmarks different.
>
> Thesis: anything for which you would want to turn off the optimizer is
> not a good benchmark anyway.

Yes, it depends on the benchmarks.  For instance, I once added
benchmarks for ‘1+’ and ‘1-’, because I wanted to see the impact of an
optimization to the corresponding VM instructions.

Nowadays peval would optimize those benchmarks out.  Yet, the fact is
that I was interested in the performance of the underlying VM
instructions, regardless of what the compiler might be doing.

Thanks,
Ludo’.