Re: Parallel make not working the way I expect...

[Philip Guenther]

On 2/9/07, David Wuertele <address@hidden> wrote:
> Here is my makefile:
...
> LOTS_OF_FILES := one two three four five six
...
> COMMAND_WHICH_BUILDS_LOTS_OF_FILES := for file in $(LOTS_OF_FILES); do echo
> hello >> $$file; done
>
> $(LOTS_OF_FILES): Makefile
>         $(COMMAND_WHICH_BUILDS_LOTS_OF_FILES)
...
> Here is the output:
>
> $ make clean; make -j2
> rm -rf one two three four five six
> for file in one two three four five six; do echo hello >> $file; done
> for file in one two three four five six; do echo hello >> $file; done
>
> Why does the "COMMAND_WHICH_BUILDS_LOTS_OF_FILES" get run twice?

Because make doesn't know that "COMMAND_WHICH_BUILDS_LOTS_OF_FILES"
creates more than just the target that make invokes it for.  The key
problem is this rule:

$(LOTS_OF_FILES): Makefile
        $(COMMAND_WHICH_BUILDS_LOTS_OF_FILES)

It might look like it tells make that to build $(LOTS_OF_FILES),
invoke $(CWBLOF) just once, but that's not correct.  A (non-pattern)
rule with multiple targets is *exactly* the same as if each target was
listed separately, ala:

one: Makefile
     $(COMMAND_WHICH_BUILDS_LOTS_OF_FILES)
two: Makefile:
     $(COMMAND_WHICH_BUILDS_LOTS_OF_FILES)
...etc

That is, that rule tells make that to build any one of the files
listed in $(LOTS_OF_FILES), invoke $(CWBLOF).  That's not exactly
true: the command builds more than what make runs it to build, so
things don't behave the way you want them to.

If you want to make the makefile work correctly with parallel make,
you need to do one of two things:

Choice 1) don't build more than one file with any given command.  The
commands for a rule should build just $@ and nothing else.


Choice 2) use a pattern rule with multiple targets so that make knows
that one command builds all of them.  You should arrange for the files
to have names that follow some sort of pattern (all have the same
prefix, of suffix, or *something* in common) so that the patterns you
give use to match them make some sense.


> The strange thing is that this is not predctable:  sometimes, the "COMMAND..."
> only gets run once.  I see it get run twice about 70% of the time on my system.

If the command completes before the second make process can start up
and check whether the second target is up-to-date, then the second
process will decide there's nothing to do.


Philip Guenther