Re: [avr-gcc-list] testsuite saga continues

[Paulo Marques]

Wouter van Gulik wrote:
> > Can you elaborate on that? What exactly are you calling a dump log?
>
> Just printing out the last N instruction executed before the abort. so 
> it's easier to spot where the test fails, but not having the penalty of 
> running avrtest_log. Which on my windows machine is much much slower. 
> avrtest is finished almost immediately, while the _log edition takes 
> several seconds. Using the latest CVS version.

The logging version will always be slower. This is not just a matter of
"outputting" the log, it is also a matter of "building" the log.

We can avoid the "output" cost by only printing the last N lines, but we
can not avoid the build cost. The code to do this was there at some
point, but I decided to remove it, because under Linux you probably can
do the same by running "avrtest_log test_program | tail -n N" and it
should run almost as fast as a native solution.

So, I can add a "--tail" option to the log version, but the naked
version will never be able to print any log at all, so that it runs as
fast as possible.

Remember that the main purpose of avrtest is to run gcc's testsuite.
While running the testsuite, having a log is useless, but speed is
important.

BTW, I've done some more optimizations and the version I have now is
almost twice as fast as the one on CVS, doing 30 P4 clocks per AVR
clock, i.e., on my P4 3GHz I can simulate a 100MHz avr :)

> > [...]
> > That probably won't work because (IIRC) smaller parts don't have the 
> > JMP and CALL instructions and only have RJMP / RCALL, so they can't 
> > address a large flash in a natural way.
>
> Aha, yes, I forgot about that... good point... right... ;)
> Do you have an idea how many test compile to a binary bigger then 8KB? 
> Since most test cases look really small, although they may link in huge 
> amounts of library code.

I don't have those numbers right now, but since there are tests that
don't even fit in 128Kb of flash, there are probably some more that
don't fit on 8Kb.

> > One thing we could do is setup our own benchmark suite (not the 
> > testsuite) that could be used to test size / speed regressions in all 
> > architectures. This could also trigger failures if gcc started using 
> > invalid instructions for a particular architecture.
> >
> > We could then have a nice matrix with compiler version / optimization 
> > setting / architecture, to show regressions on every architecture we 
> > support.
>
> Do you already have a format for doing this? XML based?

Nop. I haven't even started to think about the details.

I would give my full support to anyone trying to setup a benchmarking
framework, though ;)

--
Paulo Marques
Software Development Department - Grupo PIE, S.A.
Phone: +351 252 290600, Fax: +351 252 290601
Web: www.grupopie.com