[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Bug-dejagnu] Possible typo in contrib/sum2junit.sh

From: Jacob Bachmeyer
Subject: Re: [Bug-dejagnu] Possible typo in contrib/sum2junit.sh
Date: Wed, 31 Oct 2018 23:15:02 -0500
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv: Gecko/20090807 MultiZilla/ SeaMonkey/1.1.17 Mnenhy/

Rob Savoye wrote:
On 10/31/18 7:34 AM, Jacob Bachmeyer wrote:
While browsing the DejaGnu sources, I noticed that sum2junit.sh appears
to miscount failing tests due to a possible typo on line 49.  I am not
sufficiently familiar with JUnit to say with certainty or to be sure of
a correct patch, but this patch at least makes the code and comment

  I'm traveling right now, but as the author of that contrib script, it
looks ok. I can test it when I get home in a few days. That script is
for importing test results into Jenkins for CI, and not core DejaGnu

The apparent typo fixed in that patch would only affect the summary counts generated on the "testsuite" tag. Passes and expected failures are counted as passed tests, while expected failures and unexpected passes are counted as failures. Unexpected failures (DejaGnu result "FAIL") do not appear to be counted at all. Expected failures (DejaGnu result "XFAIL") are counted twice. This patch counts FAIL as failure instead of counting XFAIL as both pass and failure.

Further examination finds another suspicious bit of code on line 93: testcases are reported using only either "skipped" or "failure" tags, even for tests that produced a PASS result, which just looks "fishy" to me.

I have not been able to find a DTD or even an actual specification for this format. I found what appears to be an attempt to document the format at <URL:http://llg.cubic.org/docs/junit/> but that page is clearly not authoritative, as it asks for any additional information anyone might have.

Apparently, JUnit XML does not allow any message to be recorded for a test that passes? (This would explain the suspicious use of "failure" tags in the sum2junit script.)

Perhaps Jenkins ignores these details in its XML input? It seems that the sum2junit script groups test results into faked classes, such that a failing test appears to be in class "${tool}-FAIL" and a passing test in class "${tool}-PASS". This will probably work until someone tries to do aggregation of test results across multiple runs or testing systems in Jenkins where every test from a DejaGnu testsuite appears to fail if not skipped. The information is displayed, but any statistics generated in Jenkins will be wrong.

The best answer would probably be to add support for DejaGnu's own XML format (which seems to have a far more sensible data model) to Jenkins, but that is outside the scope of the DejaGnu project and I am not currently using Jenkins, so I will leave that to someone else.

-- Jacob

reply via email to

[Prev in Thread] Current Thread [Next in Thread]