Re: execvp: /bin/sh: Argument list too long

[Ralf Wildenhues]

* Pippijn van Steenhoven wrote on Mon, Nov 08, 2010 at 09:01:12PM CET:
> On Mon, Nov 08, 2010 at 08:03:06PM +0100, Ralf Wildenhues wrote:
> > Also, we wrote a short section in the manual about this (in recent
> > versions):
> >   info Automake "Length Limitations"
> 
> I hadn't seen that, before. Now I read it, I noticed something:
> 
>   Automake itself employs a couple of strategies to avoid long command
>   lines. For example, when ‘${srcdir}/’ is prepended to file names, as
>   can happen with above $(data_DATA) lists, it limits the amount of
>   arguments passed to external commands.
> 
> Why doesn't it do that, always? Is the performance hit too large?

That's not the point.

With code like this:

  foo = some long list of files
  target: $(foo)
        $do_something_with $(foo)

the fork for the shell to $do_something_with may fail already if $(foo)
is too long[1].  The above strategy is only eligible for issues when
$(foo) itself is short enough to be expanded, but prepending $(srcdir)/
everywhere *will* make it go over the limit.

[1] On some systems this is not true; e.g., GNU make evades the limit on
MSYS for this particular point, IIRC using a response file.

> > > What are the plans on fixing this?
> > 
> > We'll fix what we can fix.  Without GNU make-specifics, that is hard.
> > One thing I'd definitely like to see done is allowing multiple
> > parallel-tests test suites in one Makefile.am, so you can still have a
> > nonrecursive setup.  That may mean that 'make dist' still fails, but
> > my hope is that is not needed to work on every platform.
> > 
> > Would that be sufficient for your needs?
> 
> I'd be happy if make dist worked on my system, but it doesn't and I
> didn't find a way to make it work, yet (any suggestions here?).

You need to specify exactly where it fails if my previous suggestions
were not enough.  Showing the Makefile.am in question would help too.

> > > but since by that time, Makefile.in is
> > > already generated, it can't help. Maybe storing the list of dist files
> > > and check files in a different file and reading this with xargs < $file
> > > would help.
> > 
> > One problem is that with many constructs, automake would like to know
> > the exact set of files it deals with.  We can try to deviate from that
> > in some places but my guess is that will always look like a hack that
> > avoids some but not all issues.
> 
> Automake can know the maximum set of files it deals with. Wouldn't that
> help, already?

Well, that's not the hard part.  automake would need to rewrite long
lists of files into separate variables, and use them in separate
recipes.  Generating sets of such recipes for several targets is
probably doable, but not straight-forward.  Throw conditionals and
@substed@ parts into the mix, and it can easily get a mess.

* Paul Smith wrote on Mon, Nov 08, 2010 at 09:03:59PM CET:
> On Mon, 2010-11-08 at 21:01 +0100, Pippijn van Steenhoven wrote:
> > I am root on my (Linux) system and I set the stack size to unlimited.
> > The libtool macro reported a few billion (or something other really
> > large) for maximum argument list length, bash also agreed (it easily
> > executed the "distdir" target when copied into a bash script), but
> > make doesn't.
> 
> What version of GNU make are you using?  There was an issue in GNU make
> 3.81 related to setting stack limits, that I believe is resolved in GNU
> make 3.82.  I can't recall the details.

I don't think it's that.  It's that GNU make also requires the
per-argument limit to be large, and IIRC that is still 128K on Linux
(yes, speaking about the kernel only).  I wrote about this before:
http://thread.gmane.org/gmane.comp.gnu.make.bugs/4219
but my proposed hack back then was (understandably) not too popular,
and I haven't gotten back to anything better yet.  ;-)

Cheers,
Ralf