test suite [Was: libobjc2 release?]

[Fred Kiefer]

Am 15.03.2010 10:30, schrieb Richard Frith-Macdonald:
> 
> On 14 Mar 2010, at 12:44, Nicola Pero wrote:
> 
>> But you *do* have a point about spurious/known failures in the test suite.  
>> I would recommend we aim for exactly 0 failures.  We should silence
>> all spurious or known failures so that people can run the test suite with 
>> confidence knowing that any failure can and must be reported. ;-)
> 
> Well of course zero failures is the aim (and I think the base tests are 
> usually very close to zero failures on gnu/linux) ... but what do you do 
> about bugs which nobody has got around to fixing?  It would help if people 
> were interested in contributing and maintaining testcases of course, but I 
> don't think anyone (and I include myself in this) really like writing 
> testcases.
> 
> I've recently cleaned up four of the five failures in the base library 
> testcases, but that still leaves me (on CentOS 5.4, Intel) with this:
> 
> --- Running tests in base ---
> 
> base/NSException/basic.m:
> FAIL: working callStackSymbols ... if this has failed it is probably due to a 
> lack of support for objective-c method names (local symbols) in the 
> backtrace_symbols() function of your libc. If so, you might lobby your 
> operating system provider for a fix.
> 
>     260 COMPLETED
>       1 FAIL
>    5232 PASS
> 
> 
> I added the extra text about the probable reason for the failure, but this is 
> a case I don't really know how to deal with ... there was old, working (at 
> least on some systems) code for this function, but it depended on libbfd and 
> David spotted that libbfd uses the GPL rather than the LGPL and we can't  
> link it with base without making base GPL rather than LGPL.  He wrote new 
> code which works for most things, but not for stack symbols ... because the 
> underlying function (backtrace_symbols()) does not work properly (at least, 
> not on any version of libc I've used).
> We presumably don't want to just ignore the fact that the method no longer 
> works, but as far as I know there's no likelihood of a fix in the near future.
> 
> 
> One possibility that occurs to me is that we could maintain a file listing 
> 'common' failures and change the testsuite script to filter failures through 
> that file (eg with 'fgrep -v -f filename').
> That might then give us output like:
> 
> --- Running tests in base ---
> 
>   260 COMPLETED
>    5232 PASS
>   1 IGNORED (low priority failure ... see log for details)

Looks like you are lucky here :-)
What I get on an OpenSuse 64bit system is this:

./runtests.sh base
--- Running tests in base ---


base/coding/decoding.m:
FAIL: decoding version 2 of class NSValue

base/Functions/NSGeometry1.m:
FAIL: near identical points are equal
FAIL: near identical sizes are equal
FAIL: near identical rects are equal

base/KVC/mutable.m:
ofObject:<Lists: 0x685a90>
context:(null)
ofObject:<Lists: 0x685a90>
context:(null)
ofObject:<Lists: 0x685a90>
context:(null)
ofObject:<Lists: 0x685a90>
context:(null)
ofObject:<Lists: 0x685a90>
context:(null)
ofObject:<Lists: 0x685a90>
context:(null)
ofObject:<Lists: 0x685a90>
context:(null)

base/NSException/basic.m:
FAIL: working callStackSymbols ... if this has failed it is probably due
to a lack of support for objective-c method names (local symbols) in the
backtrace_symbols() function of your libc. If so, you might lobby your
operating system provider for a fix.

base/NSStream/socket.m:
FAIL: read www.google.com https

    259 COMPLETED
      7 context
      6 FAIL
      7 ofObject
   5226 PASS

I know that the first four errors are probably down to numeric issues
and that the "ofObject" and "context" messages are caused by some NSLog
statements that get picked up wrongly by the result grepping mechanism.
In the end it is just the missing gtsl and the one problem you have as
well. But to somebody not in the know this looks different.

As for the IGNORE case, I would prefer the EXPECTED_FAIL that Nicola
suggested.

Fred