[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Parallel tests execution [0/4]

From: Akim Demaille
Subject: Re: Parallel tests execution [0/4]
Date: Mon, 20 Oct 2008 05:02:21 -0500

Le 18 oct. 08 à 07:37, Ralf Wildenhues a écrit :

* Akim Demaille wrote on Fri, Oct 17, 2008 at 11:12:45AM CEST:
At some point we were developping more-or-less test driven: many tests
were written way before it was possible to pass them, so we knew that
the tests would fail and they were XFAIL'ed. Yet, it was not acceptable for the executable being tested to die with a SEGV. So simple failures are just exit 1, and hard-errors (or 'unacceptable errors') had an exit
status that could not be saved by XFAIL.

Hmm.  Ah, ok.

So the desired semantics are: we have a test that is marked as XFAIL
currently, but if it does this and that, then that should be a FAIL
because it is worse than the expected failure.


Yes, that sounds interesting.  And it is different from my proposed

We have a failure that should prevent further test from even being

Yes, I have understood that.  And it sounds like an interesting feature.

Now, the implementation of my proposed semantics /almost/ allows to have
your proposed semantics, too: you can run
 make -k check

That's really different: you now require the user, who runs the test suite, to change the way he should run it.

and the driver will run all tests, even after hard failures.  In the
presence of hard failures, however, it will fail to produce a summary
and the test-suite.log file.

Exactly.  This is very wrong.

I'm not sure whether I want to introduce both semantics. I see yours as
reasonable, I'm not sure how applicable mine would be in practice.
Opinions appreciated.

Your semantics can be addressed via dependencies, don't it?

In my test suites I do have the case of tests that cannot succeed under some conditions, but n that case they all individually exit SKIP. That way, even if you run just some of the tests (via explicit make check TESTS=...), they will be ignored, even if you skipped some of the HARD error (in your sense) tests.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]