Re: Tests, Documentation

[Dalibor Topic]

Hi Graydon,

graydon hoare wrote:
> Sascha Brawer <address@hidden> writes:
>
>
> > First, please write more test cases. I agree that it is boring, dumb, and
> > taxing; testing slows down development. But it's also the most reliable
> > way to make sure that code really works. If anyone needs to change our
> > code in five years, they'll want to know whether their change has any
> > serious impact. Without a large test suite, this is close to impossible.
>
>
> I agree that we should be writing more test cases. but I think of it
> in terms of a technical rather than moral failure: our testing system
> requires mental energy beyond what we are willing to exert, and
> rewards you with a sense of grinding tedium rather than excitement and
> success. I think this is a result of mostly technical issues:
>
>   - mauve should be arranged such that:
>     - it's fetched, configured and built as part of classpath

yes!

>     - it builds and runs with the JVM / compiler combination you
>       configured classpath with

oh yes!

>     - tests can be added inline with a source file, or "right next
>       door" in a similarly-named class or package.

only if you want mauve to have GNU Classpath specific features.

>     - tests are run with "make check" from classpath, using $CWD as
>       the package restriction when in a subdirectory

yes!

>     - it uses simple introspection conventions rather than scanning
>       text in a source tree and writing makefiles for itself

java.lang.reflect? I think that one's out of question for tests for java 
1.0. But I agree that reflection could make writing some tests easier. I 
was thinking about using reflection for generating tests for 
getter/setter pairs, for example.

>   - it should provide sub-frameworks for easily testing:
>     - serializability

yes!

>     - comparability, equality, hashing, toString, etc.

I'm dealing with equality, hashing & toString atm, and hope to have 
something to discuss on mauve-discuss based on some tests I'm writing 
for FieldPosition.

>     - container protocols
>     - threading protocols
>     - property (bean) protocols
>     - event listening / observer protocols
>     - null-pointer sensitivity

Add 'input sensitivity' to that. Many methods should check their input 
for correctness. I believe some of classpath's IO classes 
(BufferedReader ?) have problems with kaffe's regression tests for that.

>   - it should provide (or at least provide a place to hook up)
>     loopback, in-memory versions of:
>     - "remote" web server
>     - filesystem
>     - window system
>     - database server

yes!

>   - it should provide generic facilities for doing:
>     - pseudorandom / antirandom test vectors

definitely.

>     - all pairs / all n-tuples test reduction

test reduction? could you elaborate?

>     - noise-tolerant boundary / range checking

again, culd you elaborate some more on that?

>     - coverage and profiling analysis

yes! when I write a mauve test, I'd like to know what code paths I've 
missed. That should be possible to automatize.

>     - establishing and releasing invariants across methods
>     - establishing and releasing test prerequisites ("fixtures")
>
> these are not too much to ask; existing JUnit-based suites often
> provide some or all these as extensions to the basic TestCase class,
> and they make testing feel much more productive. 
>
> without enhancing the framework -- judging from the tests we've
> accumulated so far -- I don't think the mauve suite is ever going to
> offer you much assurance about whether a change has "serious impact".

Yep. And it is not so much fun to write tests in, if you have to do much 
of the above by hand for every other test you write.

> > Second, please write understandable documentation for every single
> > method. Developers should be able to create free programs without
> > referring to external documents. These could disappear at any time.
>
>
> I think javadoc is a very poor documentation system. it makes the same
> mistake as the rest of the java libraries: that verbosity and
> factoring into ever-smaller parts is a prima facie duty.

It's better than a lot of the rest, i.e. no docs at all. You can process 
documentation in various formats, etc. Of course something CWEB-ish 
would be a great thing, but I think JavaDoc goes a long way into that 
direction without requiring that you learn TeX ;)

> imo it should not be. it only works in your favour up to a point, then
> it works against you. to understand a package or logical concept in
> javadoc requires you to have dozens of browser tabs open, flipping
> back and forth and assembling knowledge in pieces, like a jigsaw
> puzzle. it's a very unpleasant way of reading.

That's only the case if a class is badly designed and has a ton of 
methods, fields & constructors. Of course, the JDK API is full of such 
things. ;) But that doesn't mean that our implementations need to be 
equally messy.

> I would much rather each class -- or each *package* -- had 1 javadoc
> comment, maybe with some diagrams or the like, explaining its
> operation in broad detail and pointing out salient features and how
> they relate to one another. method-by-method operation is not
> particularly useful to document, especially since it's free code and
> you can always just read what the method does.

That's what the top level comment of the class is for. To explain the 
general usage of the class. But you also need to document the methods, 
as the behaviour may be different depending on different states an 
instance can be in.

Example: A closed stream can not be read from. That's easy to see from 
the code, usually. But try to see what some methods in GregorianCalender 
are supposed to do just by reading the code. ;) I assume that 
understanding the Java2D code, if you're not proficient in 2D graphics, 
would not be trivial either.

You shouldn't have to understand how it works in order to use it. That's 
why you have JavaDocs on methods. To describe the user what she needs to 
know in order to use it ;)

cheers,
dalibor topic