Re: moving $(mkdir_p) from automake to autoconf

[Paul Eggert]

Ralf Wildenhues <address@hidden> writes:

> http://lists.gnu.org/archive/html/bug-autoconf/2006-04/msg00041.html

I like that suggestion, found a few issues with it, and came up with
the following further patch, which I installed.  Hmm, but I now see
that my version may cause clashes between AC_PROG_MKDIR_P and
AM_PROG_MKDIR_P.  I'll look into that now.  Perhaps AC_PROG_MKDIR_P
should set a new output variable MKDIR_P rather than Automake's
mkdir_p?

Also, automake should be updated to use the new install-sh.  I'll send
off a message to automake-patches after I've sorted out the name space
collision.

2006-05-10  Paul Eggert  <address@hidden>

        * NEWS: New macro AC_PROG__MKDIR_P.  AS_MKDIR_P is now more robust.
        * config/install-sh: Don't use 'path' to talk about file names,
        as per GNU coding standards.  Close a race condition reported by Ralf
        Wildenhues and Stepan Kasal.  There is still a race condition
        on hosts that predate POSIX 1003.1-1992, but we can't help this.
        Don't mishandle weird characters like space on pre-POSIX hosts.
        Invoke mkdir at most once per dir arg on pre-POSIX hosts.
        * doc/autoconf.texi (Programming in M4sh): Cross-reference to
        AC_PROG_MKDIR_P from AS_MKDIR_P.
        (Limitations of Usual Tools): Cross-reference to AC_PROG_MKDIR_P
        from mkdir.  Mention that Autoconf 2.60 install-sh is safe but
        earlier editions are not (including Automake 1.8.3).
        Do not suggest mkinstalldirs for thread-safety.
        * lib/autoconf/programs.m4 (AC_PROG_INSTALL): Insist on an 'install'
        that understands -d, so that AC_PROG_MKDIR_P can fall back on $INSTALL.
        * lib/m4sugar/m4sh.m4 (AS_MKDIR_P): Make it more robust in the
        presence of special characters and race conditions.
        * tests/local.at (AT_CHECK_ENV): Add mkdir_p to the list of variables
        in Autoconf's name space.

2006-05-10  Bruno Haible  <address@hidden>
        and Paul Eggert  <address@hidden>

        * lib/autoconf/programs.m4 (AC_PROG_MKDIR_P): New macro, taken
        from Automake with minor changes.
        * doc/autoconf.texi (Particular Programs): Document AC_PROG_MKDIR_P.

2006-05-10  Paul Eggert  <address@hidden>

        * config/install-sh: Update to Automake CVS version, as follows:
        2006-04-25  Stepan Kasal  <address@hidden>
        * lib/install-sh: Simplify the expr implementation of dirname.
        2006-04-24  Paul Eggert  <address@hidden>
        * lib/install-sh: Handle --, and diagnose unknown options.


Index: NEWS
===================================================================
RCS file: /cvsroot/autoconf/autoconf/NEWS,v
retrieving revision 1.368
diff -p -u -r1.368 NEWS
--- NEWS        7 May 2006 20:36:01 -0000       1.368
+++ NEWS        10 May 2006 22:58:30 -0000
@@ -5,6 +5,13 @@
   and ac_cv_prog_cc_c89 to 'no' as well, for backward compatibility with
   obsolete K&R tests in the Automake test suite.
 
+** AC_PROG_MKDIR_P
+  New macro.
+
+** AS_MKDIR_P
+  Now more robust with special characters in file names, or when
+  multiple processes create the same directory at the same time.
+
 * Major changes in Autoconf 2.59c
 
   Released 2006-04-12, by Ralf Wildenhues.
Index: config/install-sh
===================================================================
RCS file: /cvsroot/autoconf/autoconf/config/install-sh,v
retrieving revision 1.12
diff -p -u -r1.12 install-sh
--- config/install-sh   10 May 2006 19:17:56 -0000      1.12
+++ config/install-sh   10 May 2006 22:58:30 -0000
@@ -1,7 +1,7 @@
 #!/bin/sh
 # install - install a program, script, or datafile
 
-scriptversion=2006-04-25.22
+scriptversion=2006-05-10.12
 
 # This originates from X11R5 (mit/util/scripts/install.sh), which was
 # later released in X11R6 (xc/config/util/install.sh) with the
@@ -47,7 +47,8 @@ scriptversion=2006-04-25.22
 # Don't use :- since 4.3BSD and earlier shells don't like it.
 doit="${DOITPROG-}"
 
-# put in absolute paths if you don't have them in your path; or use env. vars.
+# Put in absolute file names if you don't have them in your path;
+# or use environment vars.
 
 mvprog="${MVPROG-mv}"
 cpprog="${CPPROG-cp}"
@@ -311,9 +312,9 @@ do
       # step, checking for races as we go.
 
       case $dstdir in
-       /*) pathcomp=/ ;;
-       -*) pathcomp=./ ;;
-       *)  pathcomp= ;;
+       /*) prefix=/ ;;
+       -*) prefix=./ ;;
+       *)  prefix= ;;
       esac
 
       case $posix_glob in
@@ -333,19 +334,36 @@ do
       $posix_glob && set +f
       IFS=$oIFS
 
+      prefixes=
+
       for d
       do
-       test "x$d" = x && continue
+       test -z "$d" && continue
 
-       pathcomp=$pathcomp$d
-       if test ! -d "$pathcomp"; then
-         $mkdirprog "$pathcomp"
-         # Don't fail if two instances are running concurrently.
-         test -d "$pathcomp" || exit 1
+       prefix=$prefix$d
+       if test -d "$prefix"; then
+         prefixes=
+       else
+         if $posix_mkdir; then
+           $mkdirprog -m "$mkdir_mode" -p -- "$dstdir" && break
+           # Don't fail if two instances are running concurrently.
+           test -d "$prefix" || exit 1
+         else
+           case $prefix in
+             *\'*) qprefix=`echo "$prefix" | sed "s/'/'\\\\\\\\''/g"`;;
+             *) qprefix=$prefix;;
+           esac
+           prefixes="$prefixes '$qprefix'"
+         fi
        fi
-       pathcomp=$pathcomp/
+       prefix=$prefix/
       done
-      obsolete_mkdir_used=true
+
+      if test -n "$prefixes"; then
+       # Don't fail if two instances are running concurrently.
+       eval "\$mkdirprog $prefixes" || test -d "$dstdir" || exit 1
+       obsolete_mkdir_used=true
+      fi
     fi
   fi
 
Index: doc/autoconf.texi
===================================================================
RCS file: /cvsroot/autoconf/autoconf/doc/autoconf.texi,v
retrieving revision 1.1014
retrieving revision 1.1016
diff -p -u -r1.1014 -r1.1016
--- doc/autoconf.texi   4 May 2006 07:08:19 -0000       1.1014
+++ doc/autoconf.texi   10 May 2006 23:42:59 -0000      1.1016
@@ -3521,6 +3521,29 @@ not found in standard @code{install} pro
 @file{Makefile.in} files.
 @end defmac
 
address@hidden AC_PROG_MKDIR_P
address@hidden
address@hidden mkdir_p
+Set output variable @code{mkdir_p} to a command that ensures that for
+each argument, a directory named by this argument exists, creating it
+and its parent directories if needed.  The command is checked to make
+sure that it is thread-safe (@pxref{Limitations of Usual Tools}).
+
+This macro uses the @samp{mkdir -p} command if possible.  Otherwise, it
+falls back on @code{$INSTALL}, so your package should
+contain @file{install-sh} as described under @code{AC_PROG_INSTALL}.
+
+This macro is related to the @code{AS_MKDIR_P} macro (@pxref{Programming
+in M4sh}), but it sets an output variable intended for use in other
+files, whereas @code{AS_MKDIR_P} is intended for use in scripts like
address@hidden  Also, @code{AS_MKDIR_P} does not accept options,
+but if you are willing to assume Posix 1003.2-1992 or later, a
address@hidden can use options, e.g., a makefile might invoke
address@hidden(mkdir_p) -m 0 dir}.
address@hidden defmac
+
+This macro differs from
+
 @defmac AC_PROG_LEX
 @acindex{PROG_LEX}
 @ovindex LEX
@@ -9726,6 +9749,9 @@ succeeds if @var{file-name} is a symboli
 even though Posix is unclear whether @samp{mkdir -p} should
 succeed in that case.  If creation of @var{file-name} fails, exit the
 script.
+
+Also see the @code{AC_PROG_MKDIR_P} macro (@pxref{Limitations of Usual
+Tools}).
 @end defmac
 
 @defmac AS_SHELL_SANITIZE
@@ -13035,7 +13061,8 @@ found"} if there are no @samp{.c} files.
 @cindex Making directories
 None of @command{mkdir}'s options are portable to older systems.  Instead of
 @samp{mkdir -p @var{file-name}}, you should use use
address@hidden(@var{file-name})} (@pxref{Programming in M4sh}).
address@hidden(@var{file-name})} (@pxref{Programming in M4sh})
+or @code{AC_PROG_MKDIR_P} (@pxref{Particular Programs}).
 
 Posix does not clearly specify whether @samp{mkdir -p foo}
 should succeed when @file{foo} is a symbolic link to an already-existing
@@ -13053,9 +13080,9 @@ version 4.0c), address@hidden 5.0, an
 known to have a
 race-free @code{mkdir -p}.  This possible race is harmful in parallel
 builds when several @file{Makefile} rules call @code{mkdir -p} to
-construct directories.  You may use @command{mkinstalldirs} or
address@hidden -d} as a safe replacement, provided these scripts are
-recent enough (the copies shipped with Automake 1.8.3 are OK, those from
+construct directories.  You may use
address@hidden -d} as a safe replacement, provided this script is
+recent enough (the copy shipped with Autoconf 2.60 is OK, those from
 older versions are not thread-safe either).
 
 
Index: lib/autoconf/programs.m4
===================================================================
RCS file: /cvsroot/autoconf/autoconf/lib/autoconf/programs.m4,v
retrieving revision 1.50
retrieving revision 1.52
diff -p -u -r1.50 -r1.52
--- lib/autoconf/programs.m4    17 Mar 2006 20:37:26 -0000      1.50
+++ lib/autoconf/programs.m4    10 May 2006 23:43:36 -0000      1.52
@@ -2,7 +2,7 @@
 # Checking for programs.
 
 # Copyright (C) 1992, 1993, 1994, 1995, 1996, 1998, 1999, 2000, 2001,
-# 2002, 2004, 2005 Free Software Foundation, Inc.
+# 2002, 2003, 2004, 2005, 2006 Free Software Foundation, Inc.
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -568,13 +568,16 @@ case $as_dir/ in
            grep pwplus "$as_dir/$ac_prog$ac_exec_ext" >/dev/null 2>&1; then
            # program-specific install script used by HP pwplus--don't use.
            :
-         else
+         elif rm -fr conftest.dir &&
+           $as_dir/$ac_prog$ac_exec_ext -c -d conftest.dir/d >/dev/null 2>&1 &&
+           rmdir conftest.dir/d conftest.dir >/dev/null 2>&1; then
            ac_cv_path_install="$as_dir/$ac_prog$ac_exec_ext -c"
            break 3
          fi
        fi
       done
     done
+    rm -fr conftest.dir
     ;;
 esac])
 ])dnl
@@ -605,6 +608,65 @@ AC_SUBST(INSTALL_DATA)dnl
 ])# AC_PROG_INSTALL
 
 
+# AC_PROG_MKDIR_P
+# ---------------
+# Check whether `mkdir -p' is supported, fallback to mkinstalldirs otherwise.
+#
+# Automake 1.8 used `mkdir -m 0755 -p --' to ensure that directories
+# created by `make install' are always world readable, even if the
+# installer happens to have an overly restrictive umask (e.g. 077).
+# This was a mistake.  There are at least two reasons why we must not
+# use `-m 0755':
+#   - it causes special bits like SGID to be ignored,
+#   - it may be too restrictive (some setups expect 775 directories).
+#
+# Do not use -m 0755 and let people choose whatever they expect by
+# setting umask.
+#
+# We cannot accept any implementation of `mkdir' that recognizes `-p'.
+# Some implementations (such as Solaris 8's) are not thread-safe: if a
+# parallel make tries to run `mkdir -p a/b' and `mkdir -p a/c'
+# concurrently, both version can detect that a/ is missing, but only
+# one can create it and the other will error out.  Consequently we
+# restrict ourselves to GNU mkdir (using the --version option ensures
+# this.)
+AC_DEFUN([AC_PROG_MKDIR_P],
+[AC_REQUIRE([AC_PROG_INSTALL])dnl
+if mkdir -p --version . >/dev/null 2>&1 && test ! -d ./--version; then
+  # We used to define $(mkdir_p) as `mkdir -p .', in order to
+  # allow $(mkdir_p) to be used without argument.  As in
+  #   $(mkdir_p) $(somedir)
+  # where $(somedir) is conditionally defined.  However we don't do
+  # that anymore.
+  #  1. before we restricted the check to GNU mkdir, `mkdir -p .' was
+  #     reported to fail in read-only directories.  The system where this
+  #     happened has been forgotten.
+  #  2. in practice we call $(mkdir_p) on directories such as
+  #       $(mkdir_p) "$(DESTDIR)$(somedir)"
+  #     and we don't want to create $(DESTDIR) if $(somedir) is empty.
+  #     To support the latter case, we have to write
+  #       test -z "$(somedir)" || $(mkdir_p) "$(DESTDIR)$(somedir)"
+  #     so $(mkdir_p) always has an argument.
+  #     We will have better chances of detecting a missing test if
+  #     $(mkdir_p) complains about missing arguments.
+  #  3. $(mkdir_p) is named after `mkdir -p' and we don't expect this
+  #     to accept no argument.
+  #  4. having something like `mkdir .' in the output is unsightly.
+  mkdir_p='mkdir -p'
+else
+  # On NextStep and OpenStep, the `mkdir' command does not
+  # recognize any option.  It will interpret all options as
+  # directories to create.
+  for d in ./-p ./--version
+  do
+    test -d $d && rmdir $d
+  done
+  mkdir_p="$INSTALL -d"
+fi
+AC_SUBST([mkdir_p])
+])# AC_PROG_MKDIR_P
+
+
 # AC_PROG_LEX
 # -----------
 # Look for flex or lex.  Set its associated library to LEXLIB.
Index: lib/m4sugar/m4sh.m4
===================================================================
RCS file: /cvsroot/autoconf/autoconf/lib/m4sugar/m4sh.m4,v
retrieving revision 1.184
diff -p -u -r1.184 m4sh.m4
--- lib/m4sugar/m4sh.m4 25 Apr 2006 12:27:45 -0000      1.184
+++ lib/m4sugar/m4sh.m4 10 May 2006 22:58:31 -0000
@@ -901,17 +901,24 @@ $as_ln_s $1 $2
 # Emulate `mkdir -p' with plain `mkdir'.
 m4_define([AS_MKDIR_P],
 [AS_REQUIRE([_$0_PREPARE])dnl
-{ if $as_mkdir_p; then
-    test -d $1 || mkdir -p $1
-  else
-    as_dir=$1
+{ as_dir=$1
+  case $as_dir in #(
+  -*) as_dir=./$as_dir;;
+  esac
+  test -d "$as_dir" || { $as_mkdir_p && mkdir -p "$as_dir"; } || {
     as_dirs=
-    while test ! -d "$as_dir"; do
-      as_dirs="$as_dir $as_dirs"
+    while :; do
+      case $as_dir in #(
+      *\'*) as_qdir=`echo "$as_dir" | sed "s/'/'\\\\\\\\''/g"`;; #(
+      *) as_qdir=$as_dir;;
+      esac
+      as_dirs="'$as_qdir' $as_dirs"
       as_dir=`AS_DIRNAME("$as_dir")`
+      test -d "$as_dir" && break
     done
-    test ! -n "$as_dirs" || mkdir $as_dirs
-  fi || AS_ERROR([cannot create directory $1]); }dnl
+    test -z "$as_dirs" || eval "mkdir $as_dirs"
+  } || test -d "$as_dir" || AS_ERROR([cannot create directory $as_dir]); }dnl
+  # The last "test -d" is in case some other process made the directory.
 ])# AS_MKDIR_P
 
 
Index: tests/local.at
===================================================================
RCS file: /cvsroot/autoconf/autoconf/tests/local.at,v
retrieving revision 1.30
diff -p -u -r1.30 local.at
--- tests/local.at      28 Apr 2006 04:17:51 -0000      1.30
+++ tests/local.at      10 May 2006 22:58:31 -0000
@@ -292,7 +292,7 @@ if test -f state-env.before && test -f s
       [F77_DUMMY_MAIN|f77_(case|underscore)],
       [FC(_DUMMY_MAIN|FLAGS|LIBS|FLAGS_f)?],
       [ALLOCA|GETLOADAVG_LIBS|KMEM_GROUP|NEED_SETGID|POW_LIB],
-      [AWK|LEX|LEXLIB|LEX_OUTPUT_ROOT|LN_S|M4|RANLIB|SET_MAKE|YACC],
+      [AWK|LEX|LEXLIB|LEX_OUTPUT_ROOT|LN_S|M4|mkdir_p|RANLIB|SET_MAKE|YACC],
       [GREP|[EF]GREP|SED],
       address@hidden|.[*#?$].|LINENO|OLDPWD|PIPESTATUS|RANDOM|SECONDS]))=' \
      $act_file 2>/dev/null |