On Tue, Jun 16, 2020 at 9:32 AM Peter Maydell <
peter.maydell@linaro.org> wrote:
>
> On Tue, 16 Jun 2020 at 17:08, Alex Bennée <
alex.bennee@linaro.org> wrote:
> > Apart from "a more perfect emulation" is there a particular use case
> > served by the extra functionality? AIUI up until this point we've
> > basically supported glibc's use of clone() which has generally been
> > enough. I'm assuming you've come across stuff that needs this more fine
> > grained support?
>
> There are definitely cases we don't handle that cause problems;
> notably
https://bugs.launchpad.net/qemu/+bug/1673976 reports
> that newer glibc implement posix_spawn() using CLONE_VM|CLONE_VFORK
> which we don't handle correctly (though it is now just "we don't
> report failures correctly" rather than "guest asserts").
This originally came up for us at Google in multi-threaded guest binaries using TCMalloc (
https://github.com/google/tcmalloc). TCMalloc does not have any special `at_fork` handling, so guest processes using TCMalloc spawn subprocesses using a custom bit of code based on `clone(CLONE_VM)` (notably *not* vfork()).
We've also been using this patch to work around similar issues in QEMU itself when creating subprocesses with fork()/vfork(). Since QEMU (and GLib) rely on locks to emulate multi-threaded guests that share virtual memory, QEMU itself is also vulnerable to deadlocks when a guest forks. Without this patch we've run into deadlocks with TCG region trees, and GLib's `g_malloc`, there are likely others as well, since we could still reproduce the deadlocks after fixing these specific problems.
The issues caused by using fork() in multi-threaded guests are pretty tricky to debug. They are fundamentally data races (was another thread in the critical section or not?), and they usually manifest as deadlocks, which show up as timeouts or hangs to users. I suspect this issue happens frequently in the wild, but at a low enough rate/user that nobody bothered fixing it/reporting it yet. Use of `vfork()` with `CLONE_VM` is common as mentioned by Alex. For example it is the only way to spawn subprocesses in Go on most platforms:
https://github.com/golang/go/blob/master/src/syscall/exec_linux.go#L218I tried to come up with a good reproducer for these issues, but I haven't been able to make one. The cases we have at Google that trigger this issue reliably are big and they contain lots of code I can't share. When compiling QEMU itself with TCMalloc, I can pretty reliably reproduce a deadlock with a program that just calls vfork(), but that's somewhat synthetic.
> The problem has always been that glibc implicitly assumes it
> knows what the process's threads are like, ie that it is the
> only thing doing any clone()s. (The comment in syscall.c mentions
> it "breaking mutexes" though I forget what I had in mind when
> I wrote that comment.) I haven't looked at these patches,
> but the risk of being clever is that we end up implicitly
> depending on details of glibc's internal implementation in a
> potentially fragile way.
>
>
> I forget whether QEMU can build against musl libc, but if we do
> then that might be an interesting test of whether we have
> accidental dependencies on the libc internals.
I agree it would be interesting to test against musl. I'm pretty sure it would work (this patch only relies on POSIX APIs + Platform ABIs for TLS), but it would be interesting to confirm.
--
Josh Kunz