Re: Progress on a Classpath mauve suite?

[Bryce McKinlay]

Roman Kennke wrote:

> Am Montag, den 07.03.2005, 21:49 -0600 schrieb Archie Cobbs:
>  
>
> > In this thread:
> >
> > http://lists.gnu.org/archive/html/classpath/2004-12/threads.html#00137
> >
> > we talked about putting together somehow a list of Mauve tests that
> > all Classpath-based VMs should expect to pass, along with another
> > (complementary) list of those tests a Classpath-based VM should
> > expect to fail.
> >    
>
> Sorry if I am completely naive here, but why should there be tests, that
> a Classpath-based VM should be expected to fail?? Isn't the whole point
> to (ideally) pass all tests? 

Ideally, yes. But in practice, it isn't practical to pass all the tests 
;-). What mauve does need is an easier way to highlight regressions. 
Currently the way people check for regressions is to do before & after 
mauve runs and diff the output to look for new FAILs, which is obviously 
pretty tedious. It would be nice if mauve could do this automatically by 
feeding it a file containing a list of known FAILs for a given VM, so 
that it would only let you know about failures that were previously 
expected to pass (and successes that were expected to fail!). This is 
complicated a little by mauve's design - there isn't a way to uniquely 
identify each "test" (ie each check() call), and sometimes some 
check()'s wont be run if a previous one failed, so fixing one bug can 
sometimes cause more FAILs. This means that it may not be clear if a new 
FAIL is actually a regression or a check that just wasn't run before. We 
could make the tests more course-grained from the point of view of the 
expected-failure machinery - ie track success/failure at the Testlet 
level, but that has the disadvantage of potentially not showing new 
regressions within a given a testlet that was already failing.

Bryce