Proposal for Long Command-Line Remedy

[William C. Cox]

Hi Folks,

This is a RFC on a proposed new feature for GNU make.  Call it a here-
document for make-controlled commands, analogous to the here-document
implemented in several Unix shells.  The difference is that the
proposed here-document is intended to supply lengthy strings of
options for a single shell command line in a Makefile.

Motivation:
        The motivation for the feature is that it's still easy, on
        some Windows machines, to exceed the command-line length that
        the shell will accept.  The problem is not only on older
        versions of Windows.  We first saw it on Windows 98, but we're
        now seeing it on Windows 2000.

        You'd think that the failing command line would be a linker
        command, with hundreds of object files.  That's exactly what
        we noticed first, and we've been able to work around it with
        the use of "@foreach" and a generated target for each object
        file.  If you're interested, I can write it up.

        Lately, we have also exceeded the shell's length limits with
        compiler commands with dozens of '-I' options with long
        pathnames to tell the overworked preprocessor where to look
        for header files.  The "@foreach" technique seems not to
        apply.

        It's a more common problem than I expected, when software
        projects get big enough.

Requirements:

        No existing Makefile shall have its meaning changed, except as
        the new feature requires - the "non-interference" requirement.

        The feature shall be robust in the face of errors, both by the
        user and by the underlying software - the "unbreakable"
        requirement.  This implies thorough automated tests of the
        feature.

        The feature shall be easy to read and comprehend, with good
        documentation - the "transparency" requirement.

        Since the feature is most useful in really large Makefile
        structures, it must be easy to migrate them to the new feature
        - the "transition" requirement.
 
Functional Sketch:

        An associate is adding a feature to GCC which will allow a
        file on the command line to hold the (some of) the options.
        You will soon be able to code (1):

            gcc -opt1 -opt2 -@ optionfile -opt400 -opt401 filenames...

        where optionfile is the name of a file which contains:
            -opt3 -opt4 \
            -opt5 \
            ... \
            -opt399
        and where newlines and whitespace in the option file are
        considered as whitespace would be on the make command line.
        There may be several optionfiles per command line.

        What I propose here is a minimally-invasive way to modify
        existing Makefiles to enable the new feature in GCC.  Make
        should handle the naming of the options file, its creation,
        passing to the command, and its deletion.

        Here's a simple example.  Given a command line like this (2):

                gcc -opt1 -opt2 <@<
               -opt3 -opt4 \
               -opt5 \
               ... \
               -opt399 < -opt400 -opt401 filenames...

        And make generates (1) for you, writing the options to a
        temporary file and filling its name in command line (1).  Make
        also takes responsibility for deleting the temporary file when
        the command is finished.  There are make facilities to control
        the directory to which the options file is written, and its
        default name template.  Make can also print out the options
        files that it has created, each uniquely named, as debug aids,
        and NOT delete them in that case.

        The syntax starting with "<@<" and ending with "<" as the
        first character of a line will often be added to just one of
        the included Makefiles, where the list of options is defined
        for the whole lot.  So, this proposal seems to meet the
        transition criterion.

Questions 1:

        Does this seem a useful addition to GNU make?  Does the
        proposal so far meet the requirements?  Are the requirements
        appropriate?  Complete?  Does the proposed syntax actually
        avoid existing useful command syntax that might occur in
        practice?

Detailed Functionality:

        Make Options:

            Ignore the here-document feature by discarding the syntax
            and passing the options in place on the command line, in
            the style of (1).

            Process here-documents in debug mode, by adding them to
            the command-printing output and not deleting the files
            when the command has finished.

        Command Processing:

            Make processes the here-document control characters ('<'
            in this RFC) *after* the variable-expansion phase of
            command processing, so that expanded variables can be
            written to the here-document.  This means that the control
            characters may be the result of a variable's expansion,
            which is the most likely way that a here-document will be
            used.
 
            The characters between the starting '<' control characters
            form the option string, which is interposed onto the
            generated command line. If the option string has no
            leading hyphen, one is supplied.  So the example "<@<" is
            interposed as "-@" on the command line.

            A hyphen following the "<@<" causes leading whitespace on
            each line of the here-document, and on the terminating
            line (with the '<' first on it) to be suppressed.  An
            optional equal sign following the "<@<" places an equal
            sign between the option string and the option file name;
            otherwise a single space is the separator.
            Backslash-newline sequences are preserved.  The body of
            options is correctly terminated by an un-backslashed '<',
            and is incorrectly terminated by an unbackslashed newline.

            Any error (malloc failure, input/output failure) in
            processing a correctly spelled here-document causes its
            handling to go to the "ignore mode" described under the
            Make Options section, and a warning is printed.  Incorrect
            here-document syntax will cause a parsing error to be
            printed, and make will terminate.

        Automatic Variables:

            $(HERE_DIR), $(HERE_FILE) are pre-defined, but changeable,
            variables containing the directory (defaulting to $TMPDIR)
            and filename template (defaulting to HERE_DOCcc-xxxx-yy)
            used in the style of the tmpfile(3) library call.  The cc
            field of the name is a copy of the characters between the
            first two '<' control characters which mark the start of
            the here-document.  The xxxx and yy fields, which can be
            any number of characters, are expanded into a 
            command-group number and a here-document number within the
            command-group.

Questions 2:

        Are the implementation descriptions clear enough to 
           a) document the feature
           b) write tests
           c) implement from?