closed file descriptors on HP-UX

[Bruno Haible]

This is a note about a portability pitfall regarding closed file
descriptors. Spotted while investigating a diffutils test failure [1].

Such closed file descriptors can be "created" through the shell syntax
<&- and >&- , as specified by POSIX (section 2.7.5 of [2]).

The issue
=========

On most platforms, a file descriptor closed in a parent process is also
closed in the child process, and can be recognized by the fact that
fstat(fd) or fcntl(fd,F_GETFL) fail with error EBADF.

On HP-UX 11.31, however, the exec() call transforms a closed file
descriptor to a file descriptor that behaves identically to /dev/null,
regarding fstat and fcntl. To distinguish such a file descriptor from
a real open("/dev/null",O_RDONLY), you need to read() from it: On the
substitute for the closed file descriptor, read() will fail with EBADF,
whereas on the real open("/dev/null",O_RDONLY), read() will succeed
with 0 bytes read.

For the details, see below.

Conclusion
==========

There are two reasonable ways to treat closed file descriptors:

  (A) Signal an error. To do this portably, it is not sufficient
      to test the file descriptor with fstat() or fcntl(). You also
      need to read() from it.

  (B) Treat it like /dev/null. To do this portably, you need to
      - Map EBADF in fstat() or fcntl() calls to success,
      - Map EBADF in read() calls to a success with 0 bytes.

Details
=======

Here are two programs:
------------------------------- child.c -------------------------------
#include <stdio.h>
#include <sys/stat.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>

int main () {
  struct stat buf;
  int ret = fstat (0, &buf);
  fprintf(stderr, "In child: fstat() -> ret=%d dev=%u ino=%u\n", ret, (unsigned 
int) buf.st_dev, (unsigned int) buf.st_ino);
  ret = fcntl (0, F_GETFL, NULL);
  fprintf(stderr, "In child: fcntl() -> %d %d\n", ret, errno);
  char bbb[3];
  ret = read (0, bbb, 3);
  fprintf(stderr, "In child: read() -> %d %d\n", ret, errno);
  return 0;
}
------------------------------ parent.c ------------------------------
#include <stdio.h>
#include <sys/stat.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>

int main () {
  close (0);
 {
  struct stat buf;
  int ret = fstat (0, &buf);
  fprintf(stderr, "In parent: fstat() -> ret=%d dev=%u ino=%u\n", ret, 
(unsigned int) buf.st_dev, (unsigned int) buf.st_ino);
  ret = fcntl (0, F_GETFL, NULL);
  fprintf(stderr, "In parent: fcntl() -> %d %d\n", ret, errno);
 }
  execl ("child", "child", NULL);
}
-----------------------------------------------------------------------

Results on HP-UX:
$ ./parent
In parent: fstat() -> ret=-1 dev=0 ino=1
In parent: fcntl() -> -1 9
In child: fstat() -> ret=0 dev=1073741827 ino=450
In child: fcntl() -> 5 0
In child: read() -> -1 9
$ ./child </dev/null
In child: fstat() -> ret=0 dev=1073741827 ino=450
In child: fcntl() -> 0 0
In child: read() -> 0 0

Results on Linux/glibc:
$ ./parent 
In parent: fstat() -> ret=-1 dev=0 ino=0
In parent: fcntl() -> -1 9
In child: fstat() -> ret=-1 dev=0 ino=0
In child: fcntl() -> -1 9
In child: read() -> -1 9
$ ./child </dev/null
In child: fstat() -> ret=0 dev=6 ino=6
In child: fcntl() -> 32768 0
In child: read() -> 0 0
-----------------------------------------------------------------------


[1] https://lists.gnu.org/archive/html/diffutils-devel/2018-12/msg00022.html
[2] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html