Clang stack alignment on i386-gnu

[Sergey Bugaev]

Hello!

I'm porting the Ladybird browser to build and run on the Hurd. After
having done a bunch of changes and finally getting something built, I
was rewarded with a SIGILL inside innocently looking Qt initialization
code.

Turns out Qt in Debian hurd-i386 was built with -msse2 -mfpmath=sse,
and the code was trying to do some SSE on an improperly aligned
pointer. (Side note, I would expect this to raise a SIGBUS or
something, not a SIGILL. Linux seems to raise SIGSEGV in this case.
The Mach exception is EXC_BAD_INSTRUCTION & EXC_I386_GPFLT.) The
pointer was stack-relative (a local variable), and the Qt code made no
attempt to align the stack pointer, i.e. it was assuming the stack was
already 16-byte aligned before the call, just like on x86_64.

This led me to learn that the 32-bit x86 ABI had been altered and now
actually fully requires 16-byte esp alignment. Since that's the case,
we should support this in glibc properly in various places; but glibc
*does* appear to invoke main () on an aligned stack in my case. But
that alignment is not preserved by the Ladybird code between main ()
and calling into Qt.

I have traced this to Clang (which I'm building Ladybird with) not
preserving the 16-byte esp alignment requirement when targeting the
Hurd; both GCC targeting the Hurd and Clang targeting Linux preserve
the alignment. This piece of code in Clang appears to be the culprit,
but note that I'm entirely unfamiliar with the LLVM code base:

  // Stack alignment is 16 bytes on Darwin, Linux, kFreeBSD, NaCl, and for all
  // 64-bit targets.  On Solaris (32-bit), stack alignment is 4 bytes
  // following the i386 psABI, while on Illumos it is always 16 bytes.
  if (StackAlignOverride)
    stackAlignment = *StackAlignOverride;
  else if (isTargetDarwin() || isTargetLinux() || isTargetKFreeBSD() ||
           isTargetNaCl() || Is64Bit)
    stackAlignment = Align(16);

https://github.com/llvm/llvm-project/blob/7ed96b1c0d9751efcf72591b36edd11a3ea97284/llvm/lib/Target/X86/X86Subtarget.cpp#L311-L318

Ladybird rebuilt with explicit -mstack-alignment=16 starts \o/

Sergey