bug#26397: [PATCH] Use vfork if possible on Darwin (bug#26397)

[YAMAMOTO Mitsuharu]

>>>>> On Sun, 9 Apr 2017 20:18:49 +0100, Alan Third <alan@idiocy.org> said:

> src/conf_post.h: Remove defines forcing use of fork.
> src/process.c (create_process): Use fork if pty_flag is set, otherwise
> vfork.

With this patch, setsid in callproc.c gets called in a vfork child
context.  It results in EPERM on Darwin.

According to ChangeLog, setsid in callproc.c seems to be used for
disconnecting from the controlling terminal (so as to avoid using
/dev/tty accidentally?).

1997-06-11  Paul Eggert  <eggert@twinsun.com>

        * callproc.c (Fcall_process): Use setsid to disconnect child
        process from controlling terminal.

If that is the case (i.e., if we don't have to make the child process
a session leader), then we don't need setsid when the Emacs process
doesn't have the controlling terminal, which is common for Mac-native
GUI sessions.  It doesn't apply for TTY or X11 sessions, but fork is
not that slow there.

                                     YAMAMOTO Mitsuharu
                                mituharu@math.s.chiba-u.ac.jp

diff --git a/src/callproc.c b/src/callproc.c
index 1e3d661eef..b3ffeb57af 100644
--- a/src/callproc.c
+++ b/src/callproc.c
@@ -591,6 +591,20 @@ call_process (ptrdiff_t nargs, Lisp_Object *args, int 
filefd,
   pid = child_setup (filefd, fd_output, fd_error, new_argv, 0, current_dir);
 #else  /* not WINDOWSNT */
 
+#ifdef DARWIN_OS
+  /* Darwin doesn't let us run setsid after a vfork, so use fork when
+     necessary. */
+  bool did_fork_p;
+  int ctfd = emacs_open ("/dev/tty", O_NOCTTY, 0);
+
+  if (ctfd >= 0)
+    {
+      emacs_close (ctfd);
+      pid = fork ();
+      did_fork_p = true;
+    }
+  else
+#endif
   /* vfork, and prevent local vars from being clobbered by the vfork.  */
   {
     Lisp_Object volatile buffer_volatile = buffer;
@@ -609,6 +623,9 @@ call_process (ptrdiff_t nargs, Lisp_Object *args, int 
filefd,
       callproc_fd_volatile[i] = callproc_fd[i];
 
     pid = vfork ();
+#ifdef DARWIN_OS
+    did_fork_p = false;
+#endif
 
     buffer = buffer_volatile;
     coding_systems = coding_systems_volatile;
@@ -631,6 +648,9 @@ call_process (ptrdiff_t nargs, Lisp_Object *args, int 
filefd,
     {
       unblock_child_signal (&oldset);
 
+#ifdef DARWIN_OS
+      if (did_fork_p)
+#endif
       setsid ();
 
       /* Emacs ignores SIGPIPE, but the child should not.  */