Re: [avr-gcc-list] binutils/.../testsuite/avr

[Klaus Rudolph]

Theodore A. Roth schrieb:

> On Sat, 4 Dec 2004, Klaus Rudolph wrote:
>
>  
>
> > Hi all,
> >
> >    
> >
> > > On the other hand, binutils and gcc just spit out a file. As such, there
> > > should be no need for a simulator for the most part. It's just a matter
> > > of compiling or assembling a chunk of code and then comparing the output
> > > to some expected result. The output used for the compare could either be
> > > the raw intermediate asm file for a C code fragment or possibly a
> > > disassemle of the object file.
> > >
> > >
> > >      
> > I think that is ok for assembler but never for C/C++. If you change
> > something in gcc maybe a better optimizer
> > the code is valid but the comparisson against a older file is not ok. So
> > a simulation is needed
> > to check that the code *do* the same and not *look* the same. :-)
> >    
>
> In that case, you have changed the expected output, so you have to
> change the compare expression. Besides, if you change gcc to make a
> better optimization, the old output file is no longer of interest since
> gcc will no longer generate it. Remember that the compare expression is
> going to be tied to the command line options that you feed to gcc and to
> the version of gcc that you are running. Also remember that we are not
> comparing the previous output to the new output, we are comparing the
> current output with an expected expression of that output. What the
> testsuite does more than anything is insure that if you change the
> output for input X, it doesn't change the output for input Y if X and Y
> are not related in any way.
>  
You are right if we have very basic testcases for this test. but this 
makes a lot of work if we need
a loot of this testcases anyway. If you make changes in register usage 
for example or optimization
you will get other code for a lot of the older "expected files". And you 
have to verify that the
new files did the same and my opinion is that take a look at the results 
from an simulation is more
sensefull. If the simulation give you the result that all things you 
would do are done and the expected
cycles are not increades then the target is reached: the code works.

> If you were to use a simulator and only look at the simulator's results
> instead of the code that gcc generated, you don't really know that your
> change to gcc had any effect. Suppose the change to gcc was in the wrong
> place so your test case didn't even exercise that new code. You've got a
> false positive test result since gcc didn't change the output file, but
> the sim still gives the same output.
>  

If you change the input to a wrong assembler code you must get wrong 
simulation results!
If this is not true the test is wrong. I have no idea why the simulation 
will not exercice the wrong
code. If you have a bad relocation or putted things in wrong sections or 
something that kind you
will lost the needed code inside your test and the testreult is wrong.

> Also, the sim really doesn't do the same thing if you change it's input.
> The end result of the sequence of instructions may be the same, but the
> sequence of instructions is different. So in the end, it's the sequence
> of instructions that we really care about.
>  
No. For example we change the sequence for register allocation we have a 
totally
differnt binary but the result after the simulation must be the same.
And also it doesnt really matter when we transfer things around from 
register to memory and
from eeprom to registers or what else. Important is only the all 
volatile accesses came in the right
order. All other transfers are normally not important.


> If you are changing gcc to alter the asm it spits out, you better darn
> well know what instructions it should spit out and that gives you the
> basis to write the expected output compare expression. If you can't
> write that expected output compare expression, I fail to see how using a
> simulator will prove that gcc did indeed spit out the correct sequence
> of instructions.
>
>  
Really simple answer:

If you want to read the eeprom for example into the usart output 
register the result is: the content
from eeprom is in the uart output register. It is not important to check 
that register r19 is used. It
is also not important that the function was inlined or not. Is is also 
not important that
the stack has been initializised before if no stack was needed :-)
Important is that the number of cycles is not increased to make shure 
that the generated code is
not more bad then the earlier one.

There are maybe some special testcases where you have to check that 
functions are inlined or not inlined
for example. But this is only important if you check the inlining 
algorithm. And yes, you have to check that
not wrong registers are used and maybe overwrite important content, but 
this could be tested with simulation as
well. If wrong reisters are used the content is lost and it is important 
that the content will be check after simulation,
but not the used registers.

Bye
   Klaus