Re: [PATCH] sort: Add --threads option, which parallelizes internal sort.

[Ralf Wildenhues]

* Paul Eggert wrote on Fri, Apr 03, 2009 at 09:57:54PM CEST:
> More important, it's not clear to me what the role of the test suite
> ought to be.  Should the test really fail if it doesn't get enough
> performance improvement with 2 threads?  How do we decide what's
> "enough"?  None of our other tests are performance tests so we are in a
> bit of a new ground here.

I agree with Paul that performance tests in a generic test suite are
very difficult.  They depend too much on the load of the system, for
example.  We have introduced performance tests in the Autoconf and
Automake testsuites recently, but the required error margin is huge.
Basically, the only thing they check for is "is -j4 a few percent faster
than -j1?"

There already are categories in the coreutils testsuite.  Maybe we can
add a cagetory "test that should be run on idle systems"?

> Given the costs involved I'm inclined to think that "make check" should
> focus on correctness tests, and a new target ("make benchmark", say?)
> should be used for performance tests.  But perhaps my view is colored by
> my using such a slow machine.

Well, if relative timings are used, then the age of the machine only
impacts the absolute run time of the testsuite, not the outcome of a
performance test.

Cheers,
Ralf