Re: Resend: Work-around for Small Kernel Environment Size

[Paul Smith]

On Sat, 2009-02-28 at 17:21 -0600, Tom Browder wrote:
> On Sat, Feb 28, 2009 at 5:03 PM, Paul Smith <address@hidden> wrote:
> ...
> >> The MAKE_LONG_LIST function generates a file with all of the contents
> >> of the long MAKE variable, one element per line.
> >
> > So, I think the behavior you're seeing is correct (unless I just don't
> > understand the problem you're describing).
> >
> > If you read the original email you'll see he then uses xargs to group
> > the generated file into multiple files at a time.
> >
> > The "500" in the original message, as I understand it, is NOT the number
> > of elements printed to the same line of the output file.  Rather, it's
> > the number of elements that can appear in a single instance of the
> > printf function, internally.
> 
> That's what I don't understand.  Why does that matter since only one
> element is used per line?
> 
> My problem started when I had a list of over 4000 (4067, to be exact)
> object files to be deleted, and "-rm $(OBJ)" failed.

It would be very helpful to see what exactly the new command line you're
using is.

> The I used the solution which worked great.  However, the generated
> file is also 4067 lines long so I can't see that the iteration is
> doing anything.  If it isn't, then there might be a little
> simplification possible (if I could get my brain around the macros and
> functions).

The file is 4067 _lines_ long.  When you pass that to xargs, it will
combine as many lines together as possible while still fitting into the
system's environment restrictions.  That's what xargs does.

In other words, the original way you ran this:

        rm $(OBJ)

constructed a single command line invoking rm with 4067 arguments.  That
was too large for the kernel to invoke.

What the code you're using does is create FILE containing 4067 lines
(files can, obviously, be more than 4067 lines long!!), then use that as
input to the xargs program, like this:

        xargs rm < FILE

xargs is READING A FILE, not obtaining the arguments from the command
line, and obviously the limits on environment size don't apply to files.
Then xargs combines some number of lines of the file, such that it is
not too large for the kernel, and generates a command line.  Then it
takes the next batch and does the same thing.

What all the trickery in MAKE_LONG_LIST, etc. is doing is writing all
those elements out to a file, without ITSELF running into the
environment size restriction.

It's true that another way to go would be to forget the output file and
generate the same command multiple times directly, rather than writing a
file and using xargs.  The advantage to the xargs method is that you
don't have to worry about how large the environment is on the computer
you're using; you let xargs worry about it for you.

If you're still not seeing what's going on let me know.

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <address@hidden>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.mad-scientist.us
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist