Emacs 21 mmap problem

[Robert A Basch]

In Emacs 21, there is a serious bug in the use of mmap() to manage
buffers.  I believe this bug causes the crashes described as related
to dynamic linking on IRIX 6.5.  (This type of crash is mentioned in
the emacs-21.2 etc/PROBLEMS file, with the suggested workaround of
setting the environment variable LD_BIND_NOW to 1).  I also believe
the bug is a potential problem on any operating system where
USE_MMAP_FOR_BUFFERS is defined.

The mmap_enlarge() function (in src/buffer.c), which is called to make
an existing mapped region larger, calls mmap() with the address of
the end of the existing region, the needed additional size, and the
MAP_FIXED flag, which tells the OS to interpret the given address
exactly.  Apparently this code expects that, when MAP_FIXED is specified,
the OS will choose a different address, or fail, if the specified
address range is unavailable, and deals with that appropriately.  On
IRIX, though, it appears that the use of MAP_FIXED will replace ANY
existing mapping at the given address range, including program text,
data, etc., or, in this case, a mapped shared library.  (In the crash
I was able to debug, such a call overlaid a portion of libungif.so,
which resulted in the subsequent segmentation fault in the runtime
linker's lazy symbol resolution).

>From my reading of various documents, including the Open Group's
Single UNIX Specification, I believe the IRIX behavior is correct.
With MAP_FIXED, the new mapping can replace any existing mapping for
the address range.  In fact, the use of MAP_FIXED is discouraged; it
seems that it is only intended for use by a program which manages its
own address space entirely, and/or has OS-dependent knowledge of how
to choose a valid address for MAP_FIXED.

Fortunately, the IRIX implementation also seems proper in its handling
of the case where a non-NULL address is specified, and not MAP_FIXED;
in this case, the address is taken as a "hint" to the OS for the
desired address.  On IRIX, testing revealed that the "hint" is
recognized, and the OS will use the given address if it does not
conflict with any existing mapping, otherwise returning a different
address.  Since mmap_enlarge() properly handles the return of a
different address, we can fix the bug for IRIX by simply not
specifying the MAP_FIXED flag.  This change is sufficient for the
MIT Athena environment; the only other operating systems we care about
are Linux and Solaris, and neither currently uses mmap() for buffer
allocation.  (The src/s/sol2-5.h header explicitly disables use of
mmap(), with a comment noting reported problems with it; I'm guessing
this could well be related to the use of MAP_FIXED).

Unfortunately, this change may not be sufficient on other platforms
using mmap().  The specification seems to allow an implementation
latitude in how it uses the given non-NULL address.  I did some
testing of the behavior on Linux and Solaris when MAP_FIXED is not
used.  The Linux behavior seems to match the IRIX behavior, i.e. the
address "hint" is used where possible.  But on Solaris, the hint seems
to be ignored -- the OS returns a different address every time, which
would probably result in a significant performance penalty.  In that
case, we would need to use OS-dependent code to check the suitability
of the address range, e.g. using mincore() or perhaps the /proc file
system, or disable the use of mmap.

Besides IRIX, a quick grep over the src/s directory shows that the
following systems are affected, (i.e. they #define USE_MMAP_FOR_BUFFERS):

freebsd
osf5-0
sol2 (i.e. solaris 2 prior to 2.5)
sunos413

I do not have access to these platforms, so I can neither test if
their mmap implementations would require adding a check of the
suitability of a desired address range, nor ascertain the correct
way to perform such a check.  But I imagine the continued use of
MAP_FIXED, without a prior check of the address range, is also
dangerous on these platforms.

[Can you explain why mmap is used (only) on these platforms, instead
of the "relocating allocator"?  If it is an optimization, have you
benchmarked the performance of mmap versus realloc?  Have you chosen
not to use mmap elsewhere (besides Solaris) due to buggy behavior?]

The patch below (against the original 21.2 buffer.c) will stop the use
of MAP_FIXED.  Again, this change is fine for our purposes, but may
not be sufficient for the platforms listed above (or any others which
might switch to using mmap() in the future).

Please let me know if you want more information, or if any of this
is unclear.

Bob Basch
MIT Information Systems


*** buffer.c.orig-21.2  Mon Jan 28 11:55:08 2002
--- buffer.c    Fri Apr 12 09:49:19 2002
***************
*** 4383,4389 ****
          POINTER_TYPE *p;
        
          p = mmap (region_end, nbytes, PROT_READ | PROT_WRITE,
!                   MAP_ANON | MAP_PRIVATE | MAP_FIXED, mmap_fd, 0);
          if (p == MAP_FAILED)
            ; /* fprintf (stderr, "mmap: %s\n", emacs_strerror (errno)); */
          else if (p != (POINTER_TYPE *) region_end)
--- 4383,4389 ----
          POINTER_TYPE *p;
        
          p = mmap (region_end, nbytes, PROT_READ | PROT_WRITE,
!                   MAP_ANON | MAP_PRIVATE, mmap_fd, 0);
          if (p == MAP_FAILED)
            ; /* fprintf (stderr, "mmap: %s\n", emacs_strerror (errno)); */
          else if (p != (POINTER_TYPE *) region_end)