Non-rectangular windows

[Christian Stieber]

Hi,

well, I have an application that uses a lot of non-rectangular windows
on the Mac, created by using an NSView as content view that drawRect:-s
an NSImage:

-(void)drawRect:(NSRect)Rectangle

{
  (void)Rectangle;

  if (SkinImage!=nil)
    {
      [[NSColor clearColor] set];
      NSRectFill([self frame]);

      [SkinImage compositeToPoint:NSZeroPoint operation:NSCompositeSourceOver];

#ifdef APPLE
      [[self window] invalidateShadow];
#endif
}

Now, that didn't work with GNUstep. Originally I didn't think much about that
since a lot of things don't work with GNUstep and need to be worked around
in the application, so I started researching the possibilities to create non-
rectangular windows on X11, stumbled across the Shape extension, and tried to
use that --- which didn't work for some reason. Then I found that GNUstep
actually DOES use the Shape extension...

Since then, I've been trying to figure out how to get it to work. I did find
that I need to
  [Defaults setBool:YES forKey:@"XWindowBuffer-shape-hack"]
in my GNUstep-init hacking, so it even considers using Shape, but it still
doesn't seem to do anything.

I did, however, succeed in setting a breakpoint on some function in the
XWindowBuffer.m, set "has_alpha" to 1 in the debugger --- and I get a shaped
window. Since then, I've been trying to figure out how to get GNUstep to set
has_alpha... with no success.

I tried randomly setting all sorts of "setBackgroundColor" and stuff to
NSColor clearColor, looking for anything that looks like it might affect
alpha handling, but no success. On the other hand, I don't really know
where to look for what...

Concerning GNUstep, only method that seems to set has_alpha is needsAlpha,
and it only gets called from some ART file... but there it ends for me, I
haven't been able to determine what any of the stuff there does or when it
does what, so trying to backtrack it to the "real" NS* classes ended long
before I got anywhere near that.

I'm using gnustep-startup-0.19.2.tar.gz, if that helps. My build system
knows how to unpack, modify and build from a startup archive, and last time
I checked that was still the most current one. Since the above experiment
with the debugger was with that version, shape support seems to be there;
I just need to know the magic switch (if all else fails I'll just try to
patch the check for has_alpha out of the library and see what happens then).

Hints, anyone? :-)

Christian