Hello, and happy holidays!
Every now and then, I hear someone mention potential ports of gnumach
to new architectures. I think I have heard RISC-V and (64-bit?) ARM
mentioned somewhere recently as potential new port targets. Being
involved in the x86_64 port last spring was a really fun and
interesting experience, and I learned a lot; so I, for one, have
always thought doing more ports would be a great idea, and that I
would be glad to be a part of such an effort again.
Among the architectures, AArch64 and RISC-V indeed seem most
attractive (not that I know much about either). Among those two,
RISC-V is certainly newer and more exciting, but Aarch64 is certainly
more widespread and established. (Wouldn't it be super cool if we
could run GNU/Hurd everywhere from tiny ARM boards, to Raspberry Pi's,
to common smartphones, to, now, ARM-based laptops desktops?) Also I
have had some experience with ARM in the past, so I knew a tiny bit of
ARM assembly.
So I thought, what would it take to port the Hurd to AArch64, a
completely non-x86 architecture, one that I knew very little about?
There is no AArch64 gnumach (that I know of) yet, but I could try to
hack on glibc even without one, I'd only need some headers, right?
There's also no compiler toolchain, but those patches to add the
x86_64-gnu target looked pretty understandable, so — how hard could it
be?
Well, I did more than think about it :)
I read up on AArch64 registers / assembly / architecture / calling
convention, added the aarch64-gnu target to binutils and GCC, added
basic versions of mach/aarch64/ headers to gnumach (but no actual
code), and made a mostly complete port of glibc. I haven't spent much
effort on Hurd proper, but I have tried running the build, and the
core Hurd servers (ext2fs, proc, exec, auth) do get built.
I will be posting the patches soon. For now, here's just a little teaser:
glibc/build $ file libc.so elf/ld.so
libc.so: ELF 64-bit LSB shared object, ARM aarch64, version 1
(GNU/Linux), dynamically linked, interpreter /lib/ld-aarch64.so.1, for
GNU/Hurd 0.0.0, with debug_info, not stripped
elf/ld.so: ELF 64-bit LSB shared object, ARM aarch64, version 1
(SYSV), dynamically linked, with debug_info, not stripped
hurd/build $ file ext2fs/ext2fs.static proc/proc
ext2fs/ext2fs.static: ELF 64-bit LSB executable, ARM aarch64, version
1 (GNU/Linux), statically linked, for GNU/Hurd 0.0.0, with debug_info,
not stripped
proc/proc: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV),
dynamically linked, interpreter /lib/ld-aarch64.so.1, for GNU/Hurd
0.0.0, with debug_info, not stripped
glibc/build $ aarch64-gnu-objdump --disassemble=__mig_get_reply_port libc.so
libc.so: file format elf64-littleaarch64
Disassembly of section .plt:
Disassembly of section .text:
000000000002b8e0 <__mig_get_reply_port>:
2b8e0: a9be7bfd stp x29, x30, [sp, #-32]!
2b8e4: 910003fd mov x29, sp
2b8e8: f9000bf3 str x19, [sp, #16]
2b8ec: d53bd053 mrs x19, tpidr_el0
2b8f0: b85f8260 ldur w0, [x19, #-8]
2b8f4: 34000080 cbz w0, 2b904 <__mig_get_reply_port+0x24>
2b8f8: f9400bf3 ldr x19, [sp, #16]
2b8fc: a8c27bfd ldp x29, x30, [sp], #32
2b900: d65f03c0 ret
2b904: 97fffbef bl 2a8c0 <__mach_reply_port>
2b908: b81f8260 stur w0, [x19, #-8]
2b90c: f9400bf3 ldr x19, [sp, #16]
2b910: a8c27bfd ldp x29, x30, [sp], #32
2b914: d65f03c0 ret
So it compiles and links, but does it work? — well, we can't know
that, not until someone ports gnumach, right?
Well actually we can :) I've done the same thing as last time, when
working on the x86_64 port: run a statically linked hello world
executable on Linux, under GDB, carefully skipping over and emulating
syscalls and RPCs. This did uncover a number of bugs, both in my port
of glibc and in how the toolchain was set up (the first issue was that
static-init.S was not even getting linked in, the second issue was
that static-init.S was crashing even prior to the _hurd_stack_setup
call, and so on). But, I fixed all of those, and got the test
executable working! — as in, successfully running all the glibc
initialization (no small feat; this includes TLS setup, hwcaps /
cpu-features, and ifuncs), reaching main (), successfully doing puts
(), and shutting down. So it totally works, and is only missing an
AArch64 gnumach to run on.
The really unexpected part is how easy this actually was: it took me
like 3 days from "ok, guess I'm doing this, let's add a new target to
binutils and gcc" to glibc building successfully, and a couple more
days to get hello world to work (single-stepping under GDB is just
that time-consuming). Either I'm getting good at this..., or (perhaps
more realistically) maybe it was just easy all along, and it was my
inexperience with glibc internals that slowed me down the last time.
Also, we have worked out a lot of 64-bit issues with the x86_64 port,
so this is something I didn't have to deal with this time.
Now to some of the more technical things:
* The TLS implementation is basically complete and working. We're using
tpidr_el0 for the thread pointer (as can be seen in the listing above),
like GNU/Linux and unlike Windows (which uses x18, apparently) and
macOS (which uses tpidrro_el0). We're using "Variant I" layout, as
described in "ELF Handling for Thread-Local Storage", again same as
GNU/Linux, and unlike what we do on both x86 targets. This actually
ends up being simpler than what we had for x86! The other cool thing is
that we can do "msr tpidr_el0, x0" from userspace without any gnumach
involvement, so that part of the implementation is quite a bit simpler
too.
* Conversely, while on x86 it is possible to perform "cpuid" and identify
CPU features entirely in user space, on AArch64 this requires access
to some EL1-only registers. On Linux and the BSDs, the kernel exposes
info about the CPU features via AT_HWCAP (and more recently, AT_HWCAP2)
auxval entries. Moreover, Linux allows userland to read some otherwise
EL1-only registers (notably for us, midr_el1) by catching the trap that
results from the EL0 code trying to do that, and emulating its effect.
Also, Linux exposes midr_el1 and revidr_el1 values through procfs.
The Hurd does not use auxval, nor is gnumach involved in execve anyway.
So I thought the natural way to expose this info would be with an RPC,
and so in mach_aarch64.defs I have an aarch64_get_hwcaps routine that
returns the two hwcaps values (using the same bits as AT_HWCAP{,2}) and
the values of midr_el1/revidr_el1. This is hooked to init_cpu_features
in glibc, and used to initialize GLRO(dl_hwcap) / GLRO(dl_hwcap2) and
eventually to pick the appropriate ifunc implementations.
* The page size (or rather, paging granularity) is notoriously not
necessarily 4096 on ARM, and the best practice is for userland not to
assume any specific page size and always query it dynamically. GNU Mach
will (probably) have to be built support for some specific page size,
but I've cleaned up a few places in glibc where things were relying on
a statically defined page size.
* There are a number of hardware hardening features available on AArch64
(PAC, BTI, MTE — why do people keep adding more and more workarounds,
including hardware ones, instead of rewriting software in a properly
memory-safe language...). Those are not really supported right now; all
of them would require some support form gnumach side; we'll probably
need new protection flags (VM_PROT_BTI, VM_PROT_MTE), for one thing.
We would need to come up with a design for how we want these to work
Hurd-wide. For example I imagine it's the userland that will be
generating PAC keys (and settings them for a newly exec'ed task), since
gnumach does not contain the functionality to generate random values
(nor should it); but this leaves open question of what should happen to
early bootstrap tasks and whether they can start using PAC after
initial startup.
* Unlike on x86, I believe it is not possible to fully restore execution
context (the values of all registers, including pc and cpsr) purely in
userland; one of the reasons for that being that we can apparently no
longer do a load from memory straight into pc, like it was possible in
previous ARM revisions. So the way sigreturn () works on Linux is of
course they have it as a syscall that takes a struct sigcontext, and
writes it over the saved thread state. Sounds familiar to you? — of
course, that's almost exactly like thread_set_state () in Mach-speak.
The difference being that thread_set_state () explicitly disallows you
to set the calling thread's state, which makes it impossible to use for
implementing sigreturn (). So I'm thinking we should lift that
restriction; there's no reason why thread_set_state () cannot be made
to work on the calling thread; it only requires some careful coding to
make sure the return register (%eax/%rax/x0) is *not* rewritten with
mach_msg_trap's return code, unlike normally.
But other than that, I do have AArch64 versions of trampoline.c and
intr-msg.h (complete with SYSCALL_EXAMINE & MSG_EXAMINE). Whether they
work, we'll only learn once we have enough of the Hurd running to have
the proc server.