[PATCH v2 00/20] aarch64-gnu port & GNU/Hurd on AArch64 progress

[Sergey Bugaev]

Hello!

This is v2 of my work on the aarch64-gnu port, aka GNU/Hurd on 64-bit
ARM. v1 is here [0].

[0]: https://sourceware.org/pipermail/libc-alpha/2024-January/153675.html

Last time, Joseph Myers has pointed out that the aarch64-gnu port can
not be merged into glibc until aarch64-gnu support is upstream in all of
glibc's build-time dependencies. That is still not the case, so this
patchset can not be merged yet; but otherwise it should be in a fairly
mergeable state. It does not yet anything to NEWS or
build-many-glibcs.py, though.

I'm porting this, again, to gather some feedback (hopefully more than
last time...), and at Maxim's request, so Linaro can test this on their
CI and ensure this doesn't break existing ports.

The upstreaming status of various aarch64-gnu components is,
specifically:

* Binutils patch to add the aarch64-gnu target is upstream and in the
  2.42 release;
* GCC patches to add aarch64-gnu target (& libgcc host) have been
  reviewed by ARM and Hurd port maintainers and should land upstream
  very soon;
* initial Hurd patches are upstream;
* glibc patches (this patchset) are not yet upstream;
* GNU Mach changes are not upstream, and upstreaming story is unclear;
* GNU MIG needs no changes, it just works.

Last time, there was no AArch64 port of GNU Mach, and so the only
testing I have done was running a simple statically-linked executable on
Linux under GDB -- which, nevertheless, helped me identify and fix a
number of issues.

Since then, however, I have been (some may say, relentlessly) working on
filling in the missing piece, namely porting gnumach (with important
help & contributions by Luca D.). I am happy to report that we now have
an experimental port of gnumach that builds and works on AArch64! While
that may sound impressive, note that various things about it are in an
extremely basic, proof-of-concept state rather than being seriously
production-ready; and also that Mach is a small kernel (indeed, a
microkernel), and it was designed from the start (back in the 80s) to be
portable, so most of the "buisness logic" functionality (virtual memory,
IPC, tasks/threads/scheduler) is explicitly arch-independent.

Despite the scary "WIP proof-of-concept" status, there is enough
functionality in Mach to run userland code, handle exceptions and
syscalls, interact with the MMU to implement all the expected virtual
memory semantics, schedule/switch tasks and threads, and so on.
Moreover, all of gnumach's userspace self-tests pass!

This meant there was enough things in place for me to try running glibc
on it, and the amazing thing is my simple test executable, the same one
I previously tested on Linux w/ GDB, just worked on real Mach without me
having to make any additional changes to the glibc side, or even
recompile it.

But I did not stop there, and got several of the core Hurd servers
working! Namely, these are ext2fs, exec, startup, auth, and proc
servers. All of them but ext2fs are dynamically linked; ld-aarch64.so.1
sucessfully locates and maps the programs themselves and their required
dependencies, and Mach pages in code and data pages from ext2fs as they
are accessed, transparently to the program, just as one would expect it
to.

It turned out that Mach on i386 and x86_64 did not enforce the (lack of)
execute permission on pages, i.e. even pages mapped without
VM_PROT_EXECUTE were executable in practice. This caused a number of
bugs related to mapping executable stacks to go unnoticed, there were
issues in all of Mach, glibc, and the Hurd's exec server related to
not creating executable stacks as actually executable. As I implemented
the execute permission properly in Mach on AArch64 (indeed, I even
support execute-only pages when the hardware implements FEAT_EPAN), I
have encountered all of those oversights one by one when trying to run
progressively more code, and have hopefully fixed them all. Hopefully
we'll stop requiring executable stacks for glibc and Hurd libraries some
time, and then we'll get working non-executable stacks on AArch64.

As expected, I have done some tweaks to the AArch64-specific Mach APIs
(primarily thread state and exception code definitions) compared to the
"preliminary sketches" of them that I posted in January, but they were
actually rather small. I've got some more confidence in the APIs now
after having implemented support for them from both sides now, and
having tested that it works in practice. No more backwards-incompatible
changes to AArch64-specific Mach APIs are expected (by me anyway); we'll
definetely want to add more things later (aarch64_debug_state for GDB,
PAC RPCs, and more), but those should be purely additive.

I have added a new Mach syscall (trap), thread_set_self_state (), to
implement sigreturn () on top of. I have originally hoped that it would
be possible to use the regular thread_set_state (mach_thread_self ())
call for it (special-casing it on AArch64 to allow setting the calling
thread's state), and indeed have initially implemented that on the Mach
side. I have then realized that this would cause a number of annoying
issues (there are good reasons why Mach doesn't allow one to call
thread_set_state () on the calling thread) that can be elegantly avoided
by making it into a dedicated syscall that is explicitly not an RPC.
Please see the explanation in the gnumach patch adding the syscall for a
more detailed explanation -- once I write that explanation and post that
patch, that is.

The new syscall (same as the potential RPC version of it), however, is a
security concern, much like sigreturn is in general: it makes it
feasible for attackers to use sigreturn-oriented programming (SROP)
tricks. Linux has, allegedly, mitigated SROP by placing a magic cookie
on the user stack next to (or inside) sigcontext, and later verifying it
in sigreturn () -- although I failed to find the code responsible for
implementing that in the Linux source tree, at least not in AArch64-
specific signal implementation. In any case, this approach is not
feasible for Mach, since Mach is not involved in placing the sigcontext
(nor the thread state structure) on user's stack. Suggestions on what to
do about this (including convincing me that this is OK and we don't have
to do anything about it, given that we have executable stacks...) are
very welcome.

But security concerns notwithstanding, I managed to test the signal
handling code path in practice, and it does work! (Indeed, it just
worked on the first try; somehow I got everything right.) The signal-
raising RPC gets received by the signal thread, the target thread
aborted inside _hurd_intr_rpc_mach_msg (), its state fetched and
modified to jump to the signal trampoline, then resumed; from the
trampoline, it calls the user's handler as expected, and then calls
sigreturn (), which uses thread_set_self_state (), resetting itself to
continue right from where it got interrupted, and continues from there;
it all works!

Besides core Hurd servers, I have tested a simple Unix program running
as PID 1 on the resulting system (with a proc port and a singal thread,
as described above, and all the other state). Among the things I tested
is fork () + wait (), and (with the latest fixes on the gnumach side)
this now totally works as well.

Sergey