bug#13588: Pax hangs in case big UID

[Stefano Lattarini]

On 04/24/2013 02:54 PM, Pavel Raiskup wrote:
> Hi, thanks a lot for working on this!  I would like to point some notes,
> sorry for doing it so late..
>
No worries, I hadn't merged the patch yet.

>> diff --git a/m4/tar.m4 b/m4/tar.m4
>> index ec8c83e..61c1206 100644
>> --- a/m4/tar.m4
>> +++ b/m4/tar.m4
>> @@ -81,7 +81,31 @@ do
>>    AM_RUN_LOG([tardir=conftest.dir && eval $am__tar_ >conftest.tar])
>>    rm -rf conftest.dir
>>    if test -s conftest.tar; then
>> -    AM_RUN_LOG([$am__untar <conftest.tar])
> 
> We are dropping here the testing of unpack phase completely, I am not sure
> whether this is intentional?
>
Not intentional, I messed up.  Thanks for catching it!

>>    rm -rf conftest.dir
>>    if test -s conftest.tar; then
>> -    AM_RUN_LOG([$am__untar <conftest.tar])
>> +    m4_if([$1], [ustar], [
>> +      if test "$_am_tool" = pax; then
> 
> The problem here is not about pax utility, it is rather general problem;
> there is not worth to check whether even the GNU tar works if we are 
> requesting
> 'ustar' format of archive && $UID/$GID >= 2^21 (it will _always_ fail).
> 
>> [..skip..]
> 
> I would go rather for the approach of (a) check whether the
> $UID/$GID > bigger than 2^21 and if yes, write some error output to *.log file
> (don't try any archivers as they all should fail).
>
Seems sensible.  Done something similar in the updated patch below.  The logic
and order of the checks are quite different from those in the previous version,
so reviews and testing are appreciated.

> I would also add the advice to use 'pax' format (which is not currently
> supported by 'pax' utility - it is quite funny - but GNU tar is OK here).
>
I don't know, I'd rather not mess with historical Automake behaviour ...
Certainly not in a bug-fixing patch.  Still, it might be something
worth considering for Automake 2.0, in case someone is willing to write
the patches and report all the necessary explanations ;-)

> Pavel
> 

---- 8< ---- 8< ---- 8< ---- 8< ---- 8< ---- 8< ---- 8< ---- 8< ---- 8< ----

>From d657168e8b7b8c166f022867aa3adc9024aabc77 Mon Sep 17 00:00:00 2001
Message-Id: <address@hidden>
From: Stefano Lattarini <address@hidden>
Date: Sun, 17 Feb 2013 16:42:46 +0100
Subject: [PATCH] tar: pax could hang configure when big UID are involved

See automake bug#8343 and bug#13588.

Tom Rini address@hidden says (in bug#8343):

    When the user has a UID or GID that is larger than the ustar format
    supports, pax does not error out gracefully in some cases (FC13).

Marc Herbert <address@hidden> adds (in bug#8343):

    When "configure" is run by a user with an UID bigger than 21 bits,
    BSD pax 3.4 aborts when trying to create the 'conftest.tar' test
    archive and leaves an empty or corrupted conftest.tar file behind.
    In the next step, pax tries to extract this incomplete or corrupted
    archive and this *** hangs the whole ./configure script ***.

    Note: GNU cpio 2.9 pretends to pass the test but it is a LIE: it
    silently truncates any big UID to its lower 21 bits. I don't know
    what can be the consequences of this lie.

Months later, Petr Hracek <address@hidden> reports a similar issue
(in bug#13588) for Fedora 17:

    I am trying to solve problem in case a user is created with big
    UID and during configuration pax hangs with message

        ATTENTION! pax archive volume change required.
        Ready for archive volume: 1
        Input archive name or "." to quit pax.
        Archive name >

    and needs user interaction.

    Reference: <https://bugzilla.redhat.com/show_bug.cgi?id=843376>

Time to fix this issue, on the line of a preliminary patch provided by
Petr Hracek in bug#13588.  The final patch ended up being remarkably
different from that original proposition, though.

* m4/tar.m4 (_AM_PROG_TAR): If the UID or GID of the current user is
too high (> 2097151), the 'ustar' format cannot work.  Adjust checks
accordingly.  Some related (minor) code reordering and clean-up.
* NEWS: Update.
* THANKS: Likewise.
* t/tar-ustar-id-too-high.sh: New test.
* t/list-of-tests.mk: Add it.
* t/tar2.sh: While at it, tweak and enhance a little.
* t/tar3.sh: Likewise.
* t/tar-override.sh: Likewise.

Helped-by: Pavel Raiskup <address@hidden>
Helped-by: Petr Hracek <address@hidden>
Helped-by: Marc Herbert <address@hidden>
Helped-by: Tom Rini <address@hidden>
Signed-off-by: Stefano Lattarini <address@hidden>
---
 NEWS                       |  6 ++++
 THANKS                     |  3 ++
 m4/tar.m4                  | 54 +++++++++++++++++++++++-----
 t/list-of-tests.mk         |  1 +
 t/tar-override.sh          | 12 ++++---
 t/tar-ustar-id-too-high.sh | 88 ++++++++++++++++++++++++++++++++++++++++++++++
 t/tar2.sh                  | 11 +++---
 t/tar3.sh                  | 10 +++---
 8 files changed, 163 insertions(+), 22 deletions(-)
 create mode 100755 t/tar-ustar-id-too-high.sh

diff --git a/NEWS b/NEWS
index f9a1fb1..863ffdf 100644
--- a/NEWS
+++ b/NEWS
@@ -78,6 +78,12 @@ New in 1.13.2:

 * Bugs fixed:

+  - When the 'ustar' option is used, the generated configure script no
+    longer risks hanging during the tests for the availability of the
+    'pax' utility, even if the user running configure has a UID or GID
+    that requires more than 21 bits to be represented.
+    See automake bug#8343 and bug#13588.
+
   - The obsolete macros AM_CONFIG_HEADER or AM_PROG_CC_STDC work once
     again, as they did in Automake 1.12.x (albeit printing runtime
     warnings in the 'obsolete' category).  Removing them has turned
diff --git a/THANKS b/THANKS
index 66498d4..a574909 100644
--- a/THANKS
+++ b/THANKS
@@ -224,6 +224,7 @@ Luo Yi                          address@hidden
 Maciej Stachowiak               address@hidden
 Maciej W. Rozycki               address@hidden
 Manu Rouat                      address@hidden
+Marc Herbert                    address@hidden
 Marcus Brinkmann                address@hidden
 Marcus G. Daniels               address@hidden
 Marius Vollmer                  address@hidden
@@ -311,6 +312,7 @@ Peter Muir                      address@hidden
 Peter O'Gorman                  address@hidden
 Peter Rosin                     address@hidden
 Peter Seiderer                  address@hidden
+Petr Hracek                     address@hidden
 Petter Reinholdtsen             address@hidden
 Petteri Räty                    address@hidden
 Phil Edwards                    address@hidden
@@ -391,6 +393,7 @@ Tim Rice                        address@hidden
 Tim Van Holder                  address@hidden
 Toshio Kuratomi                 address@hidden
 Tom Epperly                     address@hidden
+Tom Rini                        address@hidden
 Ulrich Drepper                  address@hidden
 Ulrich Eckhardt                 address@hidden
 Václav Haisman                  address@hidden
diff --git a/m4/tar.m4 b/m4/tar.m4
index ec8c83e..4b728e6 100644
--- a/m4/tar.m4
+++ b/m4/tar.m4
@@ -23,16 +23,53 @@ AC_DEFUN([_AM_PROG_TAR],
 [# Always define AMTAR for backward compatibility.  Yes, it's still used
 # in the wild :-(  We should find a proper way to deprecate it ...
 AC_SUBST([AMTAR], ['$${TAR-tar}'])
-m4_if([$1], [v7],
-     [am__tar='$${TAR-tar} chof - "$$tardir"' am__untar='$${TAR-tar} xf -'],
-     [m4_case([$1], [ustar],, [pax],,
-              [m4_fatal([Unknown tar format])])
-AC_MSG_CHECKING([how to create a $1 tar archive])
-# Loop over all known methods to create a tar archive until one works.
+
+# We'll loop over all known methods to create a tar archive until one works.
 _am_tools='gnutar m4_if([$1], [ustar], [plaintar]) pax cpio none'
+
+m4_case([$1],
+
+  [v7],
+    [am__tar='$${TAR-tar} chof - "$$tardir"' am__untar='$${TAR-tar} xf -'],
+
+  [ustar],
+    [# Automake bugs #8343 and #13588: the ustar format doesn't support
+    # an UID or a GID that require more than 21 bits to be stored.
+    # In fact, the 'pax' utility can hang when such IDs are involved.
+    am_max_uid=2097151 # 2^21 - 1
+    am_max_gid=$am_max_uid
+    # The $UID and $GID variables are not portable, so we need to resort
+    # to the POSIX-mandated id(1) utility.  Errors in the 'id' calls below
+    # are definitely unexpected, so allow the users to see them (that is,
+    # avoid stderr redirection).
+    am_uid=`id -u || echo unknown`
+    am_gid=`id -g || echo unknown`
+    AC_MSG_CHECKING([whether UID '$am_uid' is supported by ustar format])
+    if test $am_uid -le $am_max_uid; then
+       AC_MSG_RESULT([yes])
+    else
+       AC_MSG_RESULT([no])
+       _am_tools=none
+    fi
+    AC_MSG_CHECKING([whether GID '$am_gid' is supported by ustar format])
+    if test $am_gid -le $am_max_gid; then
+       AC_MSG_RESULT([yes])
+    else
+       AC_MSG_RESULT([no])
+       _am_tools=none
+    fi],
+
+  [pax],
+    [],
+
+  [m4_fatal([Unknown tar format])])
+
+AC_MSG_CHECKING([how to create a $1 tar archive])
+
+# Go ahead even if we have the value already cached.  We do so because we
+# need to set the values for the 'am__tar' and 'am__untar' variables.
 _am_tools=${am_cv_prog_tar_$1-$_am_tools}
-# Do not fold the above two line into one, because Tru64 sh and
-# Solaris sh will not grok spaces in the rhs of '-'.
+
 for _am_tool in $_am_tools
 do
   case $_am_tool in
@@ -82,6 +119,7 @@ do
   rm -rf conftest.dir
   if test -s conftest.tar; then
     AM_RUN_LOG([$am__untar <conftest.tar])
+    AM_RUN_LOG([cat conftest.dir/file])
     grep GrepMe conftest.dir/file >/dev/null 2>&1 && break
   fi
 done
diff --git a/t/list-of-tests.mk b/t/list-of-tests.mk
index f1e3dca..664a902 100644
--- a/t/list-of-tests.mk
+++ b/t/list-of-tests.mk
@@ -1151,6 +1151,7 @@ t/tags-pr12372.sh \
 t/tar.sh \
 t/tar2.sh \
 t/tar3.sh \
+t/tar-ustar-id-too-high.sh \
 t/tar-override.sh \
 t/target-cflags.sh \
 t/targetclash.sh \
diff --git a/t/tar-override.sh b/t/tar-override.sh
index 863b9ab..e3e82ff 100755
--- a/t/tar-override.sh
+++ b/t/tar-override.sh
@@ -44,22 +44,24 @@ $AUTOCONF
 $AUTOMAKE
 ./configure

+clean_temp () { rm -f *.tar.* *has-run*; }
+
 $MAKE dist
-test -f $me-1.0.tar.gz
+test -f $distdir.tar.gz
 ls | grep has-run && exit 1

-rm -f *.tar.* *has-run*
+clean_temp

 TAR="$cwd/am--tar foo" $MAKE distcheck
-test -f $me-1.0.tar.gz
+test -f $distdir.tar.gz
 test "$(cat am--tar-has-run)" = foo

-rm -f *.tar.* *has-run*
+clean_temp

 TAR=; unset TAR
 # Creative use of eval to pacify maintainer checks.
 eval \$'MAKE dist "TAR=./am--tar mu"'
-test -f $me-1.0.tar.gz
+test -f $distdir.tar.gz
 test "$(cat am--tar-has-run)" = mu

 :
diff --git a/t/tar-ustar-id-too-high.sh b/t/tar-ustar-id-too-high.sh
new file mode 100755
index 0000000..79ae89d
--- /dev/null
+++ b/t/tar-ustar-id-too-high.sh
@@ -0,0 +1,88 @@
+#! /bin/sh
+# Copyright (C) 2013 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Check that UID or GID too high for the ustar format are correctly
+# rwcognized and diagnosed by configure.  See bug#8343 and bug#13588.
+
+. test-init.sh
+
+cat > configure.ac <<END
+AC_INIT([$me], [1.0])
+AM_INIT_AUTOMAKE([tar-ustar])
+AC_CONFIG_FILES([Makefile])
+AC_OUTPUT
+END
+
+: > Makefile.am
+
+run_configure()
+{
+  st=0; ./configure ${1+"$@"}  >stdout || st=$?
+  cat stdout || exit 1
+  test $st -eq 0 || exit 1
+}
+
+checked ()
+{
+  grep "^checking $1\.\.\. $2$" stdout
+}
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+mkdir bin
+cat > bin/id <<'END'
+#!/bin/sh -e
+case "$*" in
+  -u) echo "${am_uid-1000}";;
+  -g) echo "${am_gid-1000}";;
+   *) echo "id: bad/unexpected usage" >&2; exit 1;;
+esac
+END
+chmod a+x bin/id
+
+PATH=$(pwd)/bin$PATH_SEPARATOR$PATH
+
+# Problematic ID reported in
+# <https://bugzilla.redhat.com/show_bug.cgi?id=843376>.
+am_uid=16777216; export am_uid
+am_gid=1000;     export am_gid
+run_configure
+checked "whether UID '$am_uid' is supported by ustar format" "no"
+checked "whether GID '1000' is supported by ustar format" "yes"
+checked "how to create a ustar tar archive" "none"
+
+# Another problematic ID reported in
+# <https://bugzilla.redhat.com/show_bug.cgi?id=843376>.
+am_uid=1000;     export am_uid
+am_gid=17000000; export am_gid
+run_configure
+checked "whether UID '1000' is supported by ustar format" "yes"
+checked "whether GID '$am_gid' is supported by ustar format" "no"
+checked "how to create a ustar tar archive" "none"
+
+# The minimal ID that is too big.
+two_to_twentyone=$((32 * 32 * 32 * 32 * 2))
+# <https://bugzilla.redhat.com/show_bug.cgi?id=843376>.
+am_uid=$two_to_twentyone; export am_uid
+am_gid=$two_to_twentyone; export am_gid
+run_configure
+checked "whether UID '$two_to_twentyone' is supported by ustar format" "no"
+checked "whether GID '$two_to_twentyone' is supported by ustar format" "no"
+checked "how to create a ustar tar archive" "none"
+
+:
diff --git a/t/tar2.sh b/t/tar2.sh
index 5a9d4d7..758d89a 100755
--- a/t/tar2.sh
+++ b/t/tar2.sh
@@ -18,8 +18,8 @@

 . test-init.sh

-cat > configure.ac << 'END'
-AC_INIT([tar2], [1.0])
+cat > configure.ac <<END
+AC_INIT([$me], [1.0])
 AM_INIT_AUTOMAKE([tar-pax])
 AC_CONFIG_FILES([Makefile])
 AC_OUTPUT
@@ -32,9 +32,12 @@ $AUTOCONF
 $AUTOMAKE
 ./configure

-if grep 'am__tar.*false' Makefile; then
+grep 'am__tar' Makefile # For debugging.
+if grep '^am__tar = false' Makefile; then
   skip_ "cannot find proper archiver program"
 fi

 $MAKE distcheck
-test -f tar2-1.0.tar.gz
+test -f "$distdir.tar.gz"
+
+:
diff --git a/t/tar3.sh b/t/tar3.sh
index 040d7b4..befc23f 100755
--- a/t/tar3.sh
+++ b/t/tar3.sh
@@ -18,8 +18,8 @@

 . test-init.sh

-cat > configure.ac << 'END'
-AC_INIT([tar2], [1.0])
+cat > configure.ac <<END
+AC_INIT([$me], [1.0])
 AM_INIT_AUTOMAKE([tar-pax tar-v7])
 AC_CONFIG_FILES([Makefile])
 AC_OUTPUT
@@ -37,8 +37,8 @@ grep "'tar-v7'"  tar-err

 rm -rf autom4te.cache

-cat > configure.ac << 'END'
-AC_INIT([tar2], [1.0])
+cat > configure.ac <<END
+AC_INIT([$me], [1.0])
 AM_INIT_AUTOMAKE
 AC_CONFIG_FILES([Makefile])
 AC_OUTPUT
@@ -47,6 +47,6 @@ END
 echo 'AUTOMAKE_OPTIONS = tar-pax' > Makefile.am

 AUTOMAKE_fails
-grep '^Makefile\.am:1:.*tar-pax.*AM_INIT_AUTOMAKE' stderr
+grep "^Makefile\.am:1:.*'tar-pax'.*AM_INIT_AUTOMAKE" stderr

 :
-- 
1.8.2.1.610.g562af5b