Re: [MLB] Fix for arg list too long (another final! attempt)

[Robert Boehne]

Alexandre Oliva wrote:
> 
> On Feb 20, 2001, Robert Boehne <address@hidden> wrote:
> 
> > I think the problem may be that $CXX is getting a command from
> > libtool that isn't too long, but that cxx is passing a
> > command line to ld that IS too long.  Anyone agree?
> 
> Yep, that's the most likely reason.  We should probably reserve some
> bytes for extra arguments passed by the compiler.  I can't think of
> any good rule, though...  Maybe set usable_cmd_len to
> max(max_cmd_len-1024,max_cmd_len/2)?
> 
> > I do intend to get all these quirks fixed, and to make sure that
> > goofy characters will still work, but I want to get the main
> > framework out there so that more eyes will see it.
> 

Here is a refactored patch including a mention of piecewise linking
in the documentation.  This version caches the maximum command length,
after bumping it down to 3/4 of what it initially calculates.  I used
this because 1/2 seemed too drastic, while -1024 was not enough to
prevent overrunning the limit under Tru64 4.0f.  I also renamed the
scheme 'piecewise linking' and cleaned up the implementation a bit.
I'm satisfied with it, and I've tested it on an hp10.20, Tru64 alpha,
Linux dual i686 (w/ make -j2), Sun, IRIX 6.5 and AIX 4.3.
:)

Special thanks to Alexandre, you helped out a lot!

Robert

-- 
Robert Boehne             Software Engineer
Ricardo Software   Chicago Technical Center
TEL: (630)789-0003 x. 238
FAX: (630)789-0127
email:  address@hidden

 ChangeLog entry:

2001-02-22  Robert Boehne  <address@hidden>

        * ltconfig.in: Add a test to find the approximate limit
        to the length of command line arguments.  The number
        calculated here should always be lower than the actual
        limit.
        * ltmain.in: Test the length of the command line to be
        executed and use an incremtnal linking scheme if the
        command is too long to be interpreted without error.
        * doc: Test the length of the command line to be
        executed and use an incremtnal linking scheme if the
        command is too long to be interpreted without error.
        * doc/libtool.texi (Reloadable Objects): Added a few
        sentences to describe how piecewise linking is done
        for shared objects by creating reloadable object files.


Index: ltconfig.in
===================================================================
RCS file: /home/cvs/libtool/Attic/ltconfig.in,v
retrieving revision 1.246.2.44
diff -u -r1.246.2.44 ltconfig.in
--- ltconfig.in 2001/02/15 21:55:40     1.246.2.44
+++ ltconfig.in 2001/02/22 23:49:33
@@ -216,6 +216,7 @@
 ac_ext=c
 libext=a
 cache_file=
+max_cmd_len=
 
 ## Dependencies to place before and after the object being linked:
 predep_objects=
@@ -774,6 +775,37 @@
 no_builtin_flag="$ac_cv_prog_cc_no_builtin"
 can_build_shared="$ac_cv_prog_cc_can_build_shared"
 
+# find the maximum length of command line arguments
+echo "$progname:@LINENO@: finding the maximum length of command line 
arguments" 1>&5
+echo $ac_n "finding the maximum length of command line arguments... $ac_c" 1>&6
+if test "${lt_cv_sys_max_cmd_len+set}" = set; then
+  echo $ac_n "(cached) $ac_c" 1>&6
+else
+  i=0
+  testring="ABCDEF"
+  while test `$CONFIG_SHELL $0 --fallback-echo "X$testring" >/dev/null 2>&1` 
== `echo "X$testring" >/dev/null 2>&1` &&
+          new_result=`expr "X$testring" : ".*" 2>&1` &&
+          lt_cv_sys_max_cmd_len=$new_result &&
+          test $i != 32 # 1 MB should be enough
+  do
+    i=`expr $i + 1`
+    testring=$testring$testring
+  done
+  testring=
+  # add a significant safety factor because C++ compilers can tack on massive 
amounts
+  # of additional arguments before passing them to the linker.  1/4 should be 
good.
+  len=`expr $lt_cv_sys_max_cmd_len \/ 4`
+  lt_cv_sys_max_cmd_len=`expr $lt_cv_sys_max_cmd_len - $len`
+fi
+echo "$progname:@lineno@: result: $lt_cv_sys_max_cmd_len" 1>&5
+echo "${ac_t}$lt_cv_sys_max_cmd_len" 1>&6
+
+if test -n $lt_cv_sys_max_cmd_len ; then
+  max_cmd_len=$lt_cv_sys_max_cmd_len
+else
+  max_cmd_len=none
+fi
+
 # Check to see if options -o and -c are simultaneously supported by compiler
 echo $ac_n "checking if $compiler supports -c -o file.$objext... $ac_c" 1>&6
 $rm -r conftest 2>/dev/null
@@ -2221,6 +2253,9 @@
 # Additional compiler flags for building library objects.
 pic_flag=$pic_flag
 pic_mode=$pic_mode
+
+# What is the maximum length of a command?
+max_cmd_len=$max_cmd_len
 
 # Does compiler simultaneously support -c and -o options?
 compiler_c_o=$compiler_c_o
Index: ltmain.in
===================================================================
RCS file: /home/cvs/libtool/ltmain.in,v
retrieving revision 1.200.2.44
diff -u -r1.200.2.44 ltmain.in
--- ltmain.in   2001/01/24 20:08:34     1.200.2.44
+++ ltmain.in   2001/02/22 23:49:33
@@ -3007,13 +3007,90 @@
        else
          eval cmds=\"$archive_cmds\"
        fi
-       IFS="${IFS=     }"; save_ifs="$IFS"; IFS='~'
-       for cmd in $cmds; do
-         IFS="$save_ifs"
-         $show "$cmd"
-         $run eval "$cmd" || exit $?
-       done
-       IFS="$save_ifs"
+        if len=`expr "X$cmds" : ".*"` &&
+           test $len -le $max_cmd_len; then
+          :
+        else
+          # the command line is too long to link in one step, link piecewise
+          $echo "creating reloadable object files..."
+          # save the value of $output and $libobjs because we want to use them 
later
+          save_libobjs=$libobjs
+          save_output=$output
+          # clear the reloadable object creation command queue and initialize 
k to one.
+          test_cmds=
+          concat_cmds=
+          objlist=
+          delfiles=
+          last_robj=
+          k=1
+          output=$output_objdir/$save_output-${k}.$objext
+          # loop over the list of objects to be linked
+          for obj in $save_libobjs
+          do
+            eval test_cmds=\"$reload_cmds $objlist $last_robj\"
+            if len=`expr "X$test_cmds" : ".*"` &&
+                  test $len -le $max_cmd_len; then
+              objlist="$objlist $obj"
+            else
+              # the command $test_cmds is almost too long
+              if test $k -eq 1 ; then
+                # the first file doesn't have a previous command to add
+                eval concat_cmds=\"$reload_cmds $objlist $last_robj\"
+              else
+                # all subsequent reloadable object files will link in the last 
one created
+                eval concat_cmds=\"\$concat_cmds~$reload_cmds $objlist 
$last_robj\"
+              fi
+              last_robj=$output_objdir/$save_output-${k}.$objext
+              k=`expr $k + 1`
+              output=$output_objdir/$save_output-${k}.$objext
+              objlist=$obj
+              len=1
+            fi
+          done
+          # handle the remaining objects by creating one last reloadable 
object file
+          # all subsequent reloadable object files will link in the last one 
created
+          eval concat_cmds=\"\$concat_cmds~$reload_cmds $objlist $last_robj\"
+
+          # set up a command to remove the reloadale object files
+          i=0
+          while test $i -lt $k
+          do
+            i=`expr $i + 1`
+            delfiles="$delfiles $output_objdir/$save_output-${i}.$objext"
+          done
+
+          $echo "creating a temporary reloadable object file: $output"
+
+          # loop through the commands generated above and execute them
+          IFS="${IFS=  }"; save_ifs="$IFS"; IFS='~'
+          for cmd in $concat_cmds; do
+            IFS="$save_ifs"
+            $show "$cmd"
+            $run eval "$cmd" || exit $?
+          done
+          IFS="$save_ifs"
+
+          libobjs=$output
+          # restore the value of output
+          output=$save_output
+
+          # expand the library linking commands again to reset the value of 
$libobjs for piecewise linking
+          # Do each of the archive commands.
+          if test -n "$export_symbols" && test -n "$archive_expsym_cmds"; then
+            eval cmds=\"$archive_expsym_cmds\"
+          else
+            eval cmds=\"$archive_cmds\"
+          fi
+          # append the command to remove the reloadable object files to the 
just-reset $cmds
+          eval cmds=\"\$cmds~$rm $delfiles\"
+        fi
+        IFS="${IFS=    }"; save_ifs="$IFS"; IFS='~'
+        for cmd in $cmds; do
+          IFS="$save_ifs"
+          $show "$cmd"
+          $run eval "$cmd" || exit $?
+        done
+        IFS="$save_ifs"
 
        # Restore the uninstalled library and exit
        if test "$mode" = relink; then
@@ -3912,8 +3989,37 @@
 #          $run eval '(cd $xdir && ${LN_S} $obj $baseobj)' || exit $?
 #        fi
 #      done
+
+        eval cmds=\"$old_archive_cmds\"
 
-       eval cmds=\"$old_archive_cmds\"
+        if len=`expr "X$cmds" : ".*"` &&
+             test $len -le $max_cmd_len; then
+          :
+        else
+          # the command line is too long to link in one step, link in parts
+          $echo "using piecewise archive linking..."
+          objlist=
+          concat_cmds=
+          save_oldobjs=$oldobjs
+          for obj in $save_oldobjs
+          do
+            oldobjs="$objlist $obj"
+            objlist="$objlist $obj"
+            eval test_cmds=\"$old_archive_cmds\"
+            if len=`expr "X$test_cmds" : ".*"` &&
+               test $len -le $max_cmd_len; then
+              :
+            else
+              # the above command should be used before it gets too long
+              oldobjs=$objlist
+              eval concat_cmds=\"\$old_archive_cmds~$concat_cmds\"
+              objlist=
+            fi
+          done
+          oldobjs=$objlist
+          eval concat_cmds=\"\$old_archive_cmds~$concat_cmds\"
+          eval cmds=\"$concat_cmds\"
+        fi
       fi
       IFS="${IFS=      }"; save_ifs="$IFS"; IFS='~'
       for cmd in $cmds; do
Index: doc/libtool.texi
===================================================================
RCS file: /home/cvs/libtool/doc/libtool.texi,v
retrieving revision 1.86.2.16
diff -u -r1.86.2.16 libtool.texi
--- doc/libtool.texi    2001/01/08 01:59:20     1.86.2.16
+++ doc/libtool.texi    2001/02/22 23:53:01
@@ -4367,7 +4367,11 @@
 On all known systems, a reloadable object can be created by running
 @kbd{ld -r -o @var{output}.o @var{input1}.o @var{input2}.o}.  This
 reloadable object may be treated as exactly equivalent to other
-objects.
+objects.  This is the piecewise method that libtool uses to get around
+shell limitations when building large libraries.  The object files
+created by libtool are linked together in as few steps as necessary
+into a single reloadable object file.  This temporary object file
+is then used to generate a shared library.
 
 @node Archivers
 @subsection Archivers