Re: Creating a GNUstep desktop on Solaris

[Alexander Malmberg]

Adam Fedor wrote:
> Perhaps I should be more specific, much as I love to delve into compiler 
> and language internals :-)
> Of course the above code will not compile (I said so, in fact). It's 
> really just pseudo-code for a similar problem that occurs in Objective-C 
> due to function pointer indirection. Thus when the compiler sees:
>
> struct foo {int i; int j;};
>
> @interface Foo(Category)
> - (struct foo) structReturn;
> @end
>
> int main()
> {
> struct foo bar = [(Foo *)nil structReturn];
> }
>
> (which is perfectly legal code), it converts the method call into 
> something like (warning: pseudo-like code here):
>
> struct foo bar = objc_msg_send(nil, @selector(structReturn), args);

With the GNU runtime, you get something like:

IMP _temp = objc_msg_lookup(nil, @selector(...));

struct foo bar = ((funky cast)_temp)(nil, @selector(...), args);

> which in turn performs some magic to make sure the proper type gets 
> returned for the selector.

That's the funky cast part. :) Normally, the function pointer will be 
cast to the "right" function pointer type (i.e. with the correct return 
type and argument types), in the sense that it will match the types that 
the actual function expects. However, there's only one nil method, so it 
can't be correct for all possible combinations of types.

Most of the time, with most ABI, this is harmless. However, structure 
returns are special. In most (it seems) ABI:s, structures are returned 
by copying them into memory reserved by the caller, so the function 
really has an extra hidden argument where the caller passes in this 
address. The nil method doesn't expect to return a structure, so it 
doesn't know about these hidden arguments or other special handling, and 
this breaks things.

I don't know exactly how this breaks solaris, but fwiw, it isn't 
harmless on x86, either. For some reason, the called function is 
responsible for popping the extra argument off the stack (unlike all 
other arguments, which are popped by the caller). Thus, the caller will 
push an extra argument on the stack, but the nil function won't pop it, 
so you wind up with a stack pointer that's off by 4 bytes after the nil 
function returns. (And, of course, the memory where the caller expects 
the structure to be will be completely uninitialized.)

Most of the time, this won't hurt you (stack references usually go 
through the frame pointer, which is unhurt, and the stack pointer is 
restored later), but it isn't hard to construct cases where this will 
cause a crash.

Anyway, the moral of all this is that you should never send messages 
with structure returns if the receiver might be nil.

(To be extra correct, you should never send messages that don't return 
object pointers if the receiver might be nil.)

(I believe it would be possible to hack libobjc so that the nil method 
can detect this and warn even on systems where this won't (always) cause 
a crash. This should make it easier to find these bugs.)

- Alexander Malmberg