Re: our benchmark-suite

[Ludovic Courtès]

Hi,

Neil Jerram <address@hidden> skribis:

> address@hidden (Ludovic Courtès) writes:
>
>>> My proposal is to rebase the iteration count in 0-reference.bm to run
>>> for 0.5s on some modern machine, and adjust all benchmarks to match,
>>> removing those benchmarks that do not measure anything useful.
>>
>> Sounds good.  However, adjusting iteration counts of the benchmarks
>> themselves should be done rarely, as it breaks performance tracking like
>> <http://ossau.homelinux.net/~neil/bm_master_i.html>.
>>
>>> Finally we should perhaps enable automatic scaling of the iteration
>>> count.  What do folks think about that?
>>>
>>> On the positive side, all of our benchmarks are very clear that they are
>>> a time per number of iterations, and so this change should not affect
>>> users that measure time per iteration.
>>
>> If the reported time is divided by the global iteration count, then
>> automatic scaling of the global iteration count would be good, yes.
>
> For http://ossau.homelinux.net/~neil I do still have all of the raw data
> including iteration counts, so I could easily implement dividing by the
> iteration count, and hence allow for future iteration count changes.
>
> Is there any downside from doing that?  (I don't think so.)

No, I guess.  And as you show, having raw data instead of synthesized
figures gives more freedom.

Thanks,
Ludo’.