Re: Add a script for running timing benchmarks (issue 545950043 by address@hidden)

[Han-Wen Nienhuys]

On Sun, Apr 26, 2020 at 10:04 PM Valentin Villenave
<address@hidden> wrote:
>
> On 4/26/20, address@hidden <address@hidden> wrote:
> > I can host this script somewhere else so it can be referenced in the CG,
> > but I don't think there optimizing our C++ code  is a domain for
> > beginners.
>
> I may be off the mark here, but what about adding your speed test into
> the standard regression test suite? I know the regtests are already
> measured for compilation time, but IIRC there’s no middle-size score
> compilation such as the horn thing, separate from the rest,
> specifically to get an inkling as to which changes in the codebase may
> result in speed gains or unwanted slowdowns (not noticeable
> otherwise). If this means adding 20s or so to `make check’, couldn’t
> that be a bargain worth discussing?

It doesn't work.

1) You need to have a machine doing nothing else. Even web browsing in
the background will mess with shared L3 caches.
2) This always does comparisons. So you need to have a binary of the
previous version available as well. The current setup doesn't produce
that.
3) We used to have a CPU time field in our regtest (look for .profile)
files, but they were too noisy and were switched off.

-- 
Han-Wen Nienhuys - address@hidden - http://www.xs4all.nl/~hanwen