Re: [Gnash-dev] Re: make check

[strk]

My primary goal at this stage is having developers run
'make check' and immediately realize they broke something :)
What makes this impossible at this stage is the fact that we have
too many "expected" failures in the actionscript.all directory,
so exiting with an error now would be meaningless.
Maybe we can exit with error on failures in libbase/, server/
and misc-ming.all for now and later enable error-exiting for
actionscript.all.

Of course I agree on running all tests before exiting, but
the final exit code, if any failure is found, should be an error.

Another option would be completely avoiding to descend in the 
actionscript.all dir for now, since we are not even sure those tests are
correct :)

What do you think ?

On Fri, Oct 13, 2006 at 10:15:01AM -0600, Rob Savoye wrote:
> strk wrote:
> 
> > mmm, I've seen that tests are build on 'make all', we want them
> > to only be built by 'make check' instead, right ?
> > Was there any problem with that ?
> 
>   I see you fixed this, but yes, the test cases shouldn't be built
> unless you're executing the check target.
> 
> > The other issue was to have 'make check' continue even when tests
> > in a subdir fails. I worked around this by having 'actionscript.all'
> > be executed last as it is known to contain failures, but... is there
> > a cleaner way ?
> 
>   The traditional way folks do regression testing is to not have the
> test run stop when there is a FAILure, it just gets logged. When trying
> to reproduce a specific bug, often the test case(s) in just that
> directory are run. So my question is... what are your trying to
> accomplish exactly ?
> 
>   I think what you're saying is you want to see a FAILure right away,
> and not run all the other test cases by default ? Or just that after
> running all the test cases, you want to know what the FAILures were ? I
> can probably make things work however you want to, testing wise, once I
> know what that is. :-)
> 
>   If it'll help, I'll bring back to life a script I wrote for GCC/GDB
> testing that analyzes the result files. We had to do this once we got a
> few hundred thousand test cases because the results were so large you
> couldn't find problems anymore.
> 
>       - rob -
> 
> 
> _______________________________________________
> Gnash-dev mailing list
> address@hidden
> http://lists.gnu.org/mailman/listinfo/gnash-dev

-- 

 /"\    ASCII Ribbon Campaign
 \ /    Respect for low technology.
  X     Keep e-mail messages readable by any computer system.
 / \    Keep it ASCII.