Re: [Fwd: Re: [avr-gcc-list] GCC-AVR Register optimisations]

[Dave N6NZ]

Andrew Hutchinson wrote:

> For speed/size I was think we need to have sanity checks as gcc versions 
> have  been known to suddenly dump a bunch of unexpected data in the 
> middle of a linked program.
> Of course all the functional tests still give correct results and all 
> looks good!

Yup.  A common compiler testing problem.  Optimizations are always a 
balancing act, since some that improve speed increase code size and so 
forth.

> The second use would be as evaluating impact of a patch - so we can see 
> that if it causes significant degradations in other areas. Qualative 
> perhaps - but the alternative is nothing..

I agree.  If there were a good test set that could be applied to a patch 
in isolation it would be very useful.  Even just a qualitative "dash 
board" report reveals good information, especially after everyone gets 
used to seeing the same data presented with the same format every time.

The test matrix deserves some thought.  The same test might have 
different pass/fail criteria under different options.  For example, 
imagine a suite of 10 tests.  You might say 10/10 must show zero code 
growth under -Os, 7/10 show no speed degradation under -Os, the same 10 
tests must show zero slow down for 10/10 under -O3, 6/10 no code growth 
under -O3.  (Just an example, may not be realistic.)

W.r.t options, we might also want to have a "depth first" test set, and 
a "breadth first" test set.  Depth first being going after specific test 
points with targeted, tests, and breadth first being a few tests under a 
broad range of conditions.

> gcc test suite has full set of answers and framework. We may not pass 
> all the marked applicable tests due to limited memory or unimplemented 
> functions. Yet even with this functional regression can easily be seen.

I'll have to go take a look at the framework.  So far, I've been a 
trusting user.... ./configure; make all; and get a cup of tea.

-dave