[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Mauve test question
From: |
Stephen Crawley |
Subject: |
Re: Mauve test question |
Date: |
Tue, 04 Jan 2005 16:01:40 +1000 |
address@hidden said:
> The point is that every test has a known state, one of:
> (a) don't even bother trying it (doesn't compile or we know it fails)
> (b) it is a known failure (i.e., a Classpath bug)
> (c) it should pass (i.e., if it fails, it must be a JVM bug)
> (d) it should pass but there can be false negatives depending on the JVM
Ahem ...
I think (d) is mis-characterized. If a test fails but it isn't a VM
or Classpath bug, then >>surely<< it must be a fault in the testcase
and/or the Mauve framework. Please let's be clear about this.
For example:
1) Tests for behavior that is beyond the scope of "the standard".
For example, testcases that test the names of classes returned by
certain factory methods, or that test exceptions message strings
are fundamentally broken.
2) Tests that have been designed to match (Sun acknowledged) buggy
JDK behavior or buggy specs / javadocs.
3) Tests that assume some VM-specific functionality or behavior.
For example, almost any test for GC-related behavior is inevitably
VM-specific, since there is no VM-independent way to force the GC
to run.
4) Tests that depend on environmental conditions. For example,
some existing tests assume that the host machine can open socket
connections to random machines/ports on other network. These
break for me because I am behind a firewall.
IMO, all of the above are actually bugs in the Mauve testcases, and/or
problems in the way that they are organized / classified.
-- Steve
- Re: Mauve test question,
Stephen Crawley <=