Re: Heads up: Recent status: emacs24/25 FTBFS since a long time on GNU/Hurd

[Richard Braun]

On Thu, Dec 08, 2016 at 10:44:09AM +0100, Svante Signell wrote:
> Since a long time emacs FTBFS due to unknown reasons. The latest version
> building was Debian 24.5+1-5, from 28 Nov 2015.

As already mentioned, the real issue is in Emacs. See the relevant
LWN article [1] for details.

> Even before successful builds were by pure luck. One suspicious issue is that
> emacs use sbrk() for memory allocation, right? Notably sbrk() is not 
> fool-proof
> as implemented for Hurd in glibc: Is it or is it not?

No it's not. And this is the real point I want to make in this message.

For performance reasons, the implementation of virtual memory maps
was changed in GNU Mach [2]. Basically, VM maps maintain a red-black
tree of holes for O(log(n)) allocations of virtual memory. This is what
Linux does too, except they have "augmented" their red-black tree to
use the one that already stores entries, instead of adding one for
holes.

It also works a bit differently because it still allows
bottom-up / top-down allocations. That's the big difference. GNU Mach
used to implement a bottom-up policy prior to this change, on which
the sbrk() implementation relied. This is no longer true. Mappings can
be created anywhere in the map, and it turns out that some are created
immediately following the heap, preventing sbrk() from doing its job.

Now, we could fix this by doing the same thing Linux does, but as usual,
someone has to do it, and the benefits aren't that big. Our virtual
spaces have never been as tidy as on other systems, and it doesn't
prevent most programs from working just fine. It usually matters for
things like emulators, such as Wine, or debuggers like Valgrind, which
have strong requirements regarding address values, but we currently
don't build them because other dependencies are missing.

On the other hand, the performance improvement is really, really needed.
It is now common, even on monolithic systems, to have processes with
thousands of entries in their address space, but keep in mind the Hurd
is a multi-server system, which basically means that what would be in
the kernel for Linux is in a userspace process on the Hurd, and with
the improvements on the page cache, it's actually normal for an ext2fs
server to have between 1k and 10k map entries, sometimes more. Here
is an example, run on one of our buildd systems :

# vminfo -v 410 | wc -l
20157

And that's one file system instance. At the time, the system was using
a total of around 56k map entries.

In other words, we can't go back for now, especially not just to make
an obsolete interface used by only one piece of software that is
likely to fix the issue soon. Unless you really want to work on those
red-black trees, stop pinging about this issue.

-- 
Richard Braun

[1] https://lwn.net/Articles/673724/
[2] 
https://git.sceen.net/hurd/gnumach.git/commit/?id=1db202eb8406785500f1bc3ceef7868566e416a1