bug#20555: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty => not a tty"

[白井彰]

Today I found the same problem on Ruby:

| grantpt() doesn't work with CLOEXEC on Solaris 10
| http://svn.ruby-lang.org/cgi-bin/viewvc.cgi?revision=33519&view=revision

They say that Solaris grantpt() internally uses a setuid root program
/usr/lib/pt_chmod which receives an fd and calls ptsname(), chown(),
and chmod().  If FD_CLOEXEC is set for the fd, the fd passed for
/usr/lib/pt_chmod is already closed and /usr/lib/pt_chmod does not
work.

The following is their fix for the problem:

% gzcat ruby-2.2.2.tar.gz | tar tvf - | grep ruby-2.2.2/ext/pty/pty.c
-rw-r--r-- 1044/1044  22152 Nov 15 20:49 2014 ruby-2.2.2/ext/pty/pty.c

% cat -n ruby-2.2.2/ext/pty/pty.c | sed -n '221,223p;234,256p;330,341p;450p'
   221  static int
   222  get_device_once(int *master, int *slave, char SlaveName[DEVICELEN], int 
nomesg, int fail)
   223  {
   234  #if defined(__sun) || (defined(__FreeBSD__) && __FreeBSD_version < 
902000)
   235      /* workaround for Solaris 10: grantpt() doesn't work if FD_CLOEXEC 
is set.  [ruby-dev:44688] */
   236      /* FreeBSD 9.2 or later supports O_CLOEXEC
   237       * http://www.freebsd.org/cgi/query-pr.cgi?pr=162374 */
   238      if ((masterfd = posix_openpt(O_RDWR|O_NOCTTY)) == -1) goto error;
   239      if (sigaction(SIGCHLD, &dfl, &old) == -1) goto error;
   240      if (grantpt(masterfd) == -1) goto grantpt_error;
   241      rb_fd_fix_cloexec(masterfd);
   242  #else
   243      {
   244          int flags = O_RDWR|O_NOCTTY;
   245  # if defined(O_CLOEXEC)
   246          /* glibc posix_openpt() in GNU/Linux calls open("/dev/ptmx", 
flags) internally.
   247           * So version dependency on GNU/Linux is same as O_CLOEXEC with 
open().
   248           * O_CLOEXEC is available since Linux 2.6.23.  Linux 2.6.18 
silently ignore it. */
   249          flags |= O_CLOEXEC;
   250  # endif
   251          if ((masterfd = posix_openpt(flags)) == -1) goto error;
   252      }
   253      rb_fd_fix_cloexec(masterfd);
   254      if (sigaction(SIGCHLD, &dfl, &old) == -1) goto error;
   255      if (grantpt(masterfd) == -1) goto grantpt_error;
   256  #endif
   330  #if defined(__sun)
   331      /* workaround for Solaris 10: grantpt() doesn't work if FD_CLOEXEC 
is set.  [ruby-dev:44688] */
   332      if((masterfd = open("/dev/ptmx", O_RDWR, 0)) == -1) goto error;
   333      s = signal(SIGCHLD, SIG_DFL);
   334      if(grantpt(masterfd) == -1) goto error;
   335      rb_fd_fix_cloexec(masterfd);
   336  #else
   337      if((masterfd = rb_cloexec_open("/dev/ptmx", O_RDWR, 0)) == -1) goto 
error;
   338      rb_update_max_fd(masterfd);
   339      s = signal(SIGCHLD, SIG_DFL);
   340      if(grantpt(masterfd) == -1) goto error;
   341  #endif
   450  }