GDB breakpoints are broken in new threads -- here's why

[Sergey Bugaev]

Hello -- and pardon the clickbait-y subject line (a large part of this
was written yesterday, on April 1st :)

While debugging my multi-threaded translator [0], I ran into an
annoying issue: when I set a breakpoint on a server-side MIG routine
(S_dir_readdir in my specific case) and send the matching RPC, GDB
insists that the *main* thread has received SIGTRAP, and refuses to
let the program run any further (no matter how much I ask it with
'next', 'continue', 'signal 0', etc). If I look into the threads'
states ('info thread', 'thread 6', 'bt'), I can see that it's actually
a libports worker thread that has hit the breakpoint, but GDB does not
understand this.

[0]: 9pfs, which is still being rewritten not to use netfs, and the
no-netfs version is now finally getting on par with the original
netfs-using version -- but this all needs lots of debugging naturally.

The same bug has been reported some 13 years ago at [1]; that page
also contains a simple reproducer. I can not figure out how to use
that bug-tracking system, nor do I have an account there, so I'm just
writing this as an email to bug-hurd. What I can figure out, however,
is why this bug is happening :)

[1] https://savannah.gnu.org/bugs/?29642

What follows is a braindump (only appropriate given we're talking
about signals...) of how signals and exceptions work (or don't work).

So, Unix signals. Signals sent explicitly (with kill), other than a
few special ones like SIGKILL, are delivered to a process as
msg_sig_post RPCs on its message port. The server side of this RPC is
implemented in glibc, which has a lot of logic to pick a thread to
deliver the signal to, decide what to do with the signal (ignore, die,
dump core, run a handler...), and have the selected thread stop doing
what it's doing, run the handler, and then go back to doing what it
was doing, hopefully without ever noticing. Note that externally
posted signals (as opposed to ones sent in-process with pthread_kill
or raise) are always directed to the process as a whole, not a
specific thread, and it's up to glibc to pick what thread to deliver a
signal to; in other words there's no mention of any specific thread in
the msg_sig_post API.

But when a process is traced (debugged), we want the tracer (GDB) to
receive the signals sent to the tracee before the tracee gets them;
and we also want GDB to be able to be able to alter these signals
before passing them on to the tracee (or discard them completely,
which is what 'signal 0' and 'handle SIGFOO ignore' do); in other
words, the tracer needs to be an active interceptor.

While I guess this could be implemented by GDB replacing the tracee's
message port with its own proxy port, this approach would quickly run
into complications, for instance it would have to somehow reimplement
the refport checking logic (you are only allowed to send me this
signal if you provide a port to my controlling tty...), and in-process
signals don't have to involve any actual msgport traffic. So this is
implemented differently: the tracee *knows* it's being traced (see
_hurdsig_traced), and upon receiving a signal, it doesn't actually act
on it, instead it just suspends itself and tells the proc server about
it. The tracer then learns about this from its wait () call returning
(see WIFSTOPPED & WSTOPSIG). If the tracer then wants to deliver this
(or other) signal to the tracee for real, it can use a special
msg_sig_post_untraced RPC. Once again, note that just like the
msg_sig_post API, the wait () API has no way to talk about a specific
thread, it only works on a whole-process granularity, so the tracer
learns that the tracee has received a signal, but not what thread it
was / would have been directed to.

So this would be one thing to fix: there should be some way for GDB to
figure out, once its proc_wait () call has returned, which thread in
the tracee it was that has received the signal. This is important for
pthread_kill and raise and abort, but also for exception-induced
signals (more on these in a second). This could either be some
(backwards compatible?) extension to the proc_wait* RPC, or GDB could
somehow inspect the state of the tracee *after* it learns of the
signal; but the complication with the latter is that glibc tries not
to leave any trace (pun unintended) of the signal having been caught
and delivered to the tracer, since it wasn't received by the process
"for real"; the process just gets stopped and the signal discarded.
Another bit of info that gets "lost in translation" is the signal
detail's subcode -- the code is sent to the proc server and later
relayed to the tracer, but the subcode is not. This may be problematic
if GDB then tries to re-send the same signal (which it normally does).

Now, onto Mach exceptions! If your thread faults on an invalid memory
access or something like that, Mach sends an exception_raise message
to the thread's/task's exception port, and you can then handle that
however you like. glibc installs its own _hurd_msgport (the same port
it receives signals on) as the task's exception port; upon receiving
exception_raise () on it, it maps the Mach exception to a Unix signal,
and then does the regular signal handling logic. (There's a separate
code path, and a separate port, for handling faults in the signal
thread itself.) Unlike externally-sent signals, Mach exceptions are
always directed to the specific thread that has faulted, and glibc
does the right thing to deliver the signal to the faulting thread and
not to some random unrelated thread.

However, as you might imagine, this breaks down if the tracing
mechanisms described above are in use; this information about the
faulting thread is lost, and GDB is led to believe it was the
previously active (e.g. main) thread that has faulted. This is bad
already, but it's bearable if the signal is fatal, like if the program
segfaults or aborts: you can just look around and figure which thread
is to blame. However, SIGTRAP is used by GDB internally to implement
breakpoints. When you 'b some_addr', GDB writes an int3 at that
address, waits for the program to get a SIGTRAP, and, should its %eip
match, tells you that the program has reached the breakpoint (not that
it has received SIGTRAP), and _does not_ re-send SIGTRAP to be
actually handled by the tracee. GDB then knows to restore the original
instruction when you want to continue the program. (I don't know how
this works if you want to keep the breakpoint installed.) So for
SIGTRAP, it is really important that GDB has the right idea about
which thread has caught SIGTRAP; otherwise we get the behavior
described in the bug report.

In addition to this all, GDB also has a better mechanism for dealing
with Mach exceptions than having glibc turn them into Unix signals and
notifying the tracer via proc_mark_stop. Namely, GDB will "steal" the
task's exception port, installing its own port instead. This way, GDB
will get all the Mach exceptions sent to the task first, with the full
info that comes in the exception_raise message (including the faulting
thread), and can later forward the exception to the task's own
exception port if desired, i.e. unless the exception is EXC_BREAKPOINT
(which is the Mach exception version of SIGTRAP). This is neat, and
not only it works around the issue described above (if it actually
worked...), but importantly it avoids relying on the tracee itself to
do the right thing with received exceptions (i.e. stop itself and tell
the proc server about it). This makes it possible both to debug very
early program startup (before the signal thread is set up), and to
debug programs that don't link to glibc at all and don't do any
signal/exception/msgport handling. Isn't that just great?

Unfortunately that just doesn't work for the normal case of a program
that does use glibc and is already not in early setup, for the simple
reason that glibc does task_set_exception_port (mach_task_self (),
_hurd_msgport), and this resets the port that was put there by GDB. In
other words: GDB thinks that it is stealing the exception port from
the task (and plans to forward exceptions to the original port), while
in reality what happens is the task steals the exception port from GDB
(and does not think about forwarding at all).

It looks like this was actually supposed to work though. When you
spawn the process, GDB forks off a child and makes it exec the target
executable (or first a shell and then the executable). After
completing the exec, the process is expected to catch SIGTRAP all by
itself (i.e. without running into an int3); GDB waits for that and
then does its exception port stealing, signal thread detection, etc.

The issue is that this "initial SIGTRAP" logic is implemented twice,
once in glibc and once in the exec server. The exec server, if
EXEC_SIGTRAP flag is set, makes the newly exec'd task get a SIGTRAP by
simply suspending (not resuming) it and doing proc_mark_stop on it.
This makes it seem as if the task has received SIGTRAP on the very
first instruction (i.e. in _start), which makes a lot of sense. The
implementation in glibc checks that EXEC_SIGTRAP flag is *not* set,
and in that case raises a SIGTRAP once it has set up the signal
handling.

In this second case, by the time the initial SIGTRAP happens, the
task's exception port is already set up by glibc (so GDB can steal
it), and the signal thread is already running (so GDB can detect it).
So if we were hitting this second code path for the case where the
executable is using the normal glibc, GDB would presumably work much
better. And likely at some point in the past it did.

But of course we want to have our cake and eat it too, we want to both
debug early startup and glibc-less executables, and to get the full
nice experience once the program starts doing its own signal/exception
handling. A way to get there, I think, would be for GDB to run this
logic twice: once on the first instruction of the program, and a
second time once the program sets up signals. We can't just make GDB
wait for _two_ initial SIGTRAPs, since we don't know whether the
program will ever raise the second one (and we might want to debug it
before that), so there needs to be another way for GDB to learn that
the program is done setting up signal handling.

One such way that does not require modifying glibc is watching for
no-senders notifications on the port GDB sets as the task's exception
port; since once the task replaces it with its own one, the old send
right is destroyed. That would require GDB to use a separate port for
exceptions and not reuse the event port for this as GDB does now, but
that's doable. The upside of this is it doesn't require the tracee to
cooperate, and would fire off automatically when the exception port
gets replaced. The downside is it's asynchronous, the tracee keeps
running after resetting its exception port, and can already run into
breakpoints, get exceptions. etc. while GDB is processing the
no-senders. We really need a synchronous mechanism.

I propose the following: before resetting the exception port, glibc
would fetch the previous one, and if it's non-null, it will perform a
special synchronous RPC on it, both passing the new exception port
that it would set to the tracer (so there's no need to actually set
it), and telling the tracer what its signal thread is (currently GDB
just tries to guess that the second thread is the one, except this
again doesn't work for the very same reason, there's not yet a second
thread when the task is at its very _start).

routine name_to_be_bikeshedded (
        tracer_exc_port: mach_port_move_send_t;
        my_exc_port: mach_port_make_send_t;
        signal_thread: thread_t);

Does what I'm saying make sense? What do you think? Any ideas how to
fix the first issue (thread and subcode info getting lost when
forwarding signals to the tracer)?

Sergey


P.S. By the way, I believe [2] should have been fixed, maybe it should
be closed.

[2]: https://savannah.gnu.org/bugs/?30096