Re: [Chicken-users] Unit testing

[Kon Lovett]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Nov 17, 2006, at 10:06 PM, Alejandro Forero Cuervo wrote:

> > Please elaborate on "specify limits for the execution".
>
> Well, I was thinking of setrlimit(2).  Perhaps also of setreuid(2).
> Things like "this expresion should never take more than 5 seconds to
> evaluate and no more than 1MB of memory" would be good.

Ok, resource limits. Have Scheme access via "rlimit" egg. No support  
for EUID/SUID yet, the posix unit uses setuid. Since this is UNIX  
only maybe a sh script using ulimit?

How to set resource limits for Windows? (Interix is not usually  
installed.)

> > As far as "crash" proof goes, that will require something that traps
> > memory exceptions, etc. None of the existing 3 support this.  Also
> > requires a test results storage system that is incremental,
> > transactional, & probably external. "test-infrastructure" constructs
> > its' results in a functional manner, "check" is incremental but not
> > safe, & "testeez" is incremental but not safe.
>
> I was thinking of just forking a separate process that runs the test
> and passes the result back over a pipe(2).

So was I. Still, none of the existing "saves" the results in a  
logging-like manner.

When a system error occurs it would be nice if at least the results  
up to the error are available. Even better if info on the system  
error is passed.

Also, just plain Scheme exceptions should be handled. (Of the 3 only  
test-infrastructure so far does this.)

> > I do maintain "test-infrastructure" but am not the author. Think I
> > will make the necessary thread-safety changes.

This has nothing to do w/ your concerns, just a reminder to myself of  
bad style.

> I'm afraid using threads wouldn't be safe enough.  The tests would
> probably test C code from the eggs.  If I understand how threads in
> Chicken work, a bad low-level error in the tested code could easily
> crash the whole testing process.

Yes. I was only thinking of the "infinite loop" issue. (& checking  
for thread safety is useful anyway.)

> I guess my ideal implementation for defining tests would be along
> these lines:
>
>     (test-compare
>       (car (cons 1 2)) ; expresion
>       1                ; expected value
>       equal?)          ; compare function
>
>     (test-assert
>       (cons 1 2)
>       pair?)

Rather than start from scratch testeez already has forms like the  
above. Changing manner in which test results are captured to a  
"write" rather than a "cons" would be the big change. And, unlike  
check & test-infrastructure it is not dependent on hygienic macros.

> We would have lots of files with all these tests (one for each egg).
> And then we would reference them from another file as this:
>
>     (run-tests
>       "lists.scm"        ; file
>       "Basic list tests" ; description of tests
>       1                  ; max memory tests for each test (MB)
>       5)                 ; max number of seconds for each test

> This would generate a report for each test defined in lists.scm.
> Each test would be run as its own process.  Ideally, the process would
> set its UID to a specific user right before running the code.  Also, I
> think the test files should allow defining symbols that the tests can
> use but that different files shouldn't be able to interfere with each

Please elaborate "defining symbols".

> other (so I would probably also fork a process before actually running
> the code from each file).  I also think run-tests should try to run
> the tests both in compiled as in interpreted mode.

I actually run my tests compiled by default as it is a more stringent  
environment.

> So, of the top of my head, that's it all.

SchemeUnit anyone? (Actually it doesn't address all of your concerns  
either.)

> Alejo.
> http://azul.freaks-unidos.net/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iEYEARECAAYFAkVfqt0ACgkQJJNoeGe+5O41bQCeJr5U/0ustYJs92fmL5HhWPRY
9BUAmwRoVnF452a7CM9D1wkoW3cVKNfc
=WILs
-----END PGP SIGNATURE-----