Re: Progress on a Classpath mauve suite?

[zander]

On Tue, Mar 08, 2005 at 05:07:08PM +0100, Roman Kennke wrote:
> Am Dienstag, den 08.03.2005, 09:11 -0600 schrieb Archie Cobbs:
> > Roman Kennke wrote:
> > >>http://lists.gnu.org/archive/html/classpath/2004-12/threads.html#00137
> > >>
> > >>we talked about putting together somehow a list of Mauve tests that
> > >>all Classpath-based VMs should expect to pass, along with another
> > >>(complementary) list of those tests a Classpath-based VM should
> > >>expect to fail.
> > > 
> > > Sorry if I am completely naive here, but why should there be tests, that
> > > a Classpath-based VM should be expected to fail?? Isn't the whole point
> > > to (ideally) pass all tests? If there would be such a test, I would
> > > rather rewrite the harness.check(cond) into harness.check(!cond) so that
> > > the (expected) failure passes...
> > 
> > Ideally there should be no expected failures. In reality there always
> > are, simply because some feature is unimplemented or some bug has not
> > been tracked down yet. Obviously, any expected failures are good
> > starting points for anyone looking to improve Classpath... another
> > benefit of having (or being able to easily generate) such a list.
> 
> Ah, now I'm understanding. I think maybe you mean the mauve HTML pages
> that I once put together? This basically tested different runtimes
> against the Mauve suite and formatted to Mauve output nicely into HTML
> for displaying in a browser, so it is easily visible which tests fails
> (red) and which don't (green).

At the FOSDEM meeting I brought up this issue since I have been working on
a new mauve-runner (based on the work from David Gilbert) which indeed
generates this kind of output and is written in such a way that a more
structured output of a mauve run is also possible, allowing a 3rth package
to cross-reference different runs. Where each run may be on a different
JVM, for example.

When the processing-power becomes available (developer.classpath.org is
probably best) we intend to move this there and put online the results in
various easy to parse ways.

See also this thread (on the mauve mailinglist)
http://sources.redhat.com/ml/mauve-discuss/2005-q1/msg00020.html

-- 
Thomas Zander