Evading the command line length limit (on Linux)

[Ralf Wildenhues]

Hello bug-make readers,

Xan Lopez brought up an interesting Automake issue about long commands
that led us to an issue GNU make could be more helpful with[1][2]:

Linux >= 2.6.23 has removed the in-kernel hardwired command line length
limit[3].  So, once stack size has been increased sufficiently with

  ulimit -s 32768

or so, one could hope to use longer commands in makefiles.  However,
there is also a new limit on the length of each individual command line
argument[4], MAX_ARG_STRLEN (131072).  This kills the advantage when GNU
make uses

  sh -c 'very long shell command line now all in one arg'

for the command, i.e., iff the command contains any shell active stuff.
I don't like this inconsistency, I would like complex commands to have
no smaller limit than simple ones.  :-)


1) One possibility would be to drop -c and pipe the command line to sh
on standard input.  Just like parallel make, this would prevent using
the standard input of the make process within the child command process.

2) Another possibility would be a hack like the following: split the
long line into N substrings, each below the argument limit, and invoke

  sh -c 'eval "${1}${2}...${10}...${N}"' sh quoted-substring1 ...

This changes the following user-observable semantics:

- the positional parameters have different settings; this can be
  fixed with a 'set x; shift;' at the start of the eval'ed string,
- LINENO is not reliable in eval with some shells.
- ${10} and up are not portable to pre-Posix shells like Solaris sh and
  the heirloom shell under Linux
  (See autoconf.info 'Limitations of Builtins' for more details on these
  issues.)

I don't think these cause big problems in practice, more so if we go
this way only with command line lengths that would otherwise have failed
anyway, and only on Linux systems.

3) Yet another possibility would be to ask the kernel developers to
increase MAX_ARG_STRLEN.  However, since there seems to be some safety
issue involved in its value (see links referenced in [3]), I doubt that
would be the most successful approach.


What do you think?  Below is an initial patch to do (2); the index
computations probably still need fixes.  There is the problem that the
MAX_ARG_STRLEN value is obtained at compile time, and only if those
headers are present, but 'make' may be deployed on a system with a
different setting later.  And I've probably failed to follow some coding
guidelines for GNU make, and broken some unrelated platform.  I've
blissfully assumed the number of substrings would have at most 5 digits,
and failed to produce test suite additions so far.  The evaled string
contains many more parameter expansions than are normally needed, this
is just supposed to be an upper bound on them (any further parameters
are empty anyway).

Tested on GNU/Linux only, with the test suite and the example from [2],
and several shells.  Also tested with 'make SHELL=/bin/sh\ -x' to ensure
that, when we set $0 the '-x' does not end up munged in $1 (this code
probably has a bug if SHELL contains quotes or backslashes or so).

For my system and with a stack size of 32M, the patch allows 2^6 times
longer command lines in the makefile (roughly 7M), which should buy
WebKitGTK+ some time until hitting the next limit.

Thanks,
Ralf

[1] http://lists.gnu.org/archive/html/bug-autoconf/2009-07/msg00038.html
[2] http://lists.gnu.org/archive/html/bug-autoconf/2009-07/msg00046.html
[3] http://www.in-ulm.de/~mascheck/various/argmax/#linux
[4] http://www.in-ulm.de/~mascheck/various/argmax/#maximum_number


2009-07-29  Ralf Wildenhues  <address@hidden>

        * configure.in: Check for sys/user.h and linux/binfmts.h
        headers.
        * job.c: Include them if available.
        (construct_command_argv_internal): When constructing the command
        line with 'sh -c', use multiple arguments together with eval
        expansion to evade the Linux per-argument length limit
        MAX_ARG_STRLEN if it is defined.
        Problem reported against Automake by Xan Lopez <address@hidden>.

Index: configure.in
===================================================================
RCS file: /cvsroot/make/make/configure.in,v
retrieving revision 1.150
diff -u -r1.150 configure.in
--- configure.in        4 Jun 2009 06:30:27 -0000       1.150
+++ configure.in        29 Jul 2009 19:08:55 -0000
@@ -64,7 +64,8 @@
 AC_HEADER_STAT
 AC_HEADER_TIME
 AC_CHECK_HEADERS(stdlib.h locale.h unistd.h limits.h fcntl.h string.h \
-                memory.h sys/param.h sys/resource.h sys/time.h sys/timeb.h)
+                memory.h sys/param.h sys/resource.h sys/time.h sys/timeb.h \
+                sys/user.h linux/binfmts.h)
 
 # Set a flag if we have an ANSI C compiler
 if test "$ac_cv_prog_cc_stdc" != no; then
Index: job.c
===================================================================
RCS file: /cvsroot/make/make/job.c,v
retrieving revision 1.193
diff -u -r1.193 job.c
--- job.c       9 Jun 2009 15:35:38 -0000       1.193
+++ job.c       29 Jul 2009 20:44:04 -0000
@@ -29,6 +29,11 @@
 
 #include <string.h>
 
+#if defined (HAVE_LINUX_BINFMTS_H) && defined (HAVE_SYS_USER_H)
+#include <sys/user.h>
+#include <linux/binfmts.h>
+#endif
+
 /* Default shell to use.  */
 #ifdef WINDOWS32
 #include <windows.h>
@@ -2717,9 +2722,18 @@
 #endif
     unsigned int line_len = strlen (line);
 
+#ifdef MAX_ARG_STRLEN
+    static char eval_line[] = "eval\\ \\\"set\\ x\\;\\ shift\\;\\ ";
+#define ARG_NUMBER_DIGITS 5
+#define EVAL_LEN (sizeof(eval_line)-1 + shell_len + 4 \
+                 + (7 + ARG_NUMBER_DIGITS) * 2 * line_len / (MAX_ARG_STRLEN - 
2))
+#else
+#define EVAL_LEN 0
+#endif
     char *new_line = alloca (shell_len + (sizeof (minus_c)-1)
-                             + (line_len*2) + 1);
+                             + (line_len*2) + 1 + EVAL_LEN);
     char *command_ptr = NULL; /* used for batch_mode_shell mode */
+    char *args_ptr;
 
 # ifdef __EMX__ /* is this necessary? */
     if (!unixy_shell)
@@ -2732,6 +2746,30 @@
     memcpy (ap, minus_c, sizeof (minus_c) - 1);
     ap += sizeof (minus_c) - 1;
     command_ptr = ap;
+
+#if !defined (WINDOWS32) && defined (MAX_ARG_STRLEN)
+    if (unixy_shell && line_len > MAX_ARG_STRLEN)
+      {
+       unsigned j;
+       memcpy (ap, eval_line, sizeof (eval_line) - 1);
+       ap += sizeof (eval_line) - 1;
+       for (j = 1; j <= 2 * line_len / (MAX_ARG_STRLEN - 2); j++)
+         ap += sprintf (ap, "\\$\\{%u\\}", j);
+       *ap++ = '\\';
+       *ap++ = '"';
+       *ap++ = ' ';
+       /* Copy only the first word of SHELL to $0.  */
+       for (p = shell; *p != '\0'; ++p)
+         {
+           if (isspace ((unsigned char)*p))
+             break;
+           *ap++ = *p;
+         }
+       *ap++ = ' ';
+      }
+#endif
+    args_ptr = ap;
+
     for (p = line; *p != '\0'; ++p)
       {
        if (restp != NULL && *p == '\n')
@@ -2779,6 +2817,14 @@
           }
 #endif
        *ap++ = *p;
+
+#if !defined (WINDOWS32) && defined (MAX_ARG_STRLEN)
+       if (unixy_shell && line_len > MAX_ARG_STRLEN && (ap - args_ptr > 
MAX_ARG_STRLEN - 2))
+         {
+           *ap++ = ' ';
+           args_ptr = ap;
+         }
+#endif
       }
     if (ap == new_line + shell_len + sizeof (minus_c) - 1)
       /* Line was empty.  */