Re: GNUstep testfarm reference

[Richard Frith-Macdonald]

On 5 Mar 2011, at 12:34, David Chisnall wrote:

> On 5 Mar 2011, at 09:22, Ivan Vučica wrote:
> 
>> I'm not sure I understand -- would one need to test under Cocoa or under 
>> GNUstep/OSX? If Cocoa testing is needed, I could try and contribute 
>> something without setting up GS/OSX :-D
> 
> 
> Running the tests under Cocoa is the goal.  In theory, the tests should 
> always pass on OS X, because differences between OS X and GNUstep (unless 
> explicitly testing GNUstep extensions) are bugs.  If a test for some Cocoa 
> functionality works under GNUstep, but fails under Cocoa, then the test is 
> wrong and we should fix it (and then fix the GNUstep code that it was 
> testing).

Yes, that's the theory ... but unfortunately we need to address things 
carefully and slowly ...

Some failures under OSX are cases where we are testing for behavior which was 
correct in the past (eg from OPENSTEP or earlier versions of OSX), but which 
have been broken in recent OSX.
In these cases we might want to update the tests (and behavior of gnustep) to 
reflect current OSX behavior ... but need to do it with care in case the change 
breaks existing apps.
Some are cases where the test is wrong (or perhaps just too picky) and again, 
we want to change the test and gnustep ... but again with care ... we need to 
fix any code which depends on the existing behavior before we change the 
behavior.
A few may be cases where OSX is definitely buggy/bad.  In this case we have to 
decide whether to follow the buggy behavior or simply continue to report the 
bug on OSX.

So ... getting all the tests to pass on OSX (or be treated as hopes that Apple 
will fix things) is the goal ... but simply changing the testcases to pass on 
OSX without first dealing with the effect of those changs is not an option :-(