Re: Parallel build dependency locking

[Paul D. Smith]

%% Christopher Sean Morrison <address@hidden> writes:

  csm> On Jun 9, 2005, at 8:01 PM, John Graham-Cumming wrote:
  >> On Thu, 2005-06-09 at 19:11 -0400, Christopher Sean Morrison wrote:

  >>> There is already support for processing multiple targets
  >>> simultaneously albeit through some form of locking to ensure that
  >>> each dependency/target is being handled by only one of the -j
  >>> workers (in that process).  That is the simple nature of a -j
  >>> parallel build.  All I'm suggesting is a means to extend that same
  >>> locking mechanism on dependancies being processed by child,
  >>> sub-process, or otherwise concurrent makes also.

I agree with Boris and John here.  What you're asking for is not only
very difficult, but not necessary.  A well-formed makefile environment
will have exactly one makefile controlling any given target, and that
one makefile will be invoked once by one instance of make (or at least,
one at a time), never more than once at the same time.

If your makefile environment is well-formed, then the kind of locking
etc. you are looking for is completely unnecessary as far as I can
tell.  I don't see the point in expending huge amounts of time and
effort to better support what is, basically, bad makefile design.

  csm> That is indeed what I'd be interested in having.  Another
  csm> alternative might be to consider the sub-process makes as not
  csm> being independent, sharing the exec knowledge of their parent via
  csm> some mechanism (environment vars, process id arguments, etc).  It
  csm> would seem more simple to just handle the more general
  csm> independent process case, but considering only sub-process makes
  csm> and making them dependent or otherwise state sharing would even
  csm> be beneficial.

If you want this model you should just use inclusive (non-recursive)
makefile structures and be done with it, IMO.  As above, I don't see the
point in creating a very complex way for multiple instances of make to
share dependency graph information, detailed runtime information,
etc. when all you need to do is have one make instance read in both
makefiles to get the same effect.

  csm> The 'problem' is perhaps just a limitation of the make logic
  csm> being exported by automake and presumptions on possible target
  csm> dependancies (e.g. SUBDIRS, bin_PROGRAMS, lib_LTPROGRAMS, etc are
  csm> always traversed in order one at a time).

As far as I can see this is true of SUBDIRS (which is, of course,
unfortunate), but it isn't true of programs, libraries, etc.  They are
all listed as prerequisites of the all-am: target, and as such they are
eligible to be built in parallel, unless there are further prerequisite
statements made.

  csm> Since it processes each program/library in order, if there are a
  csm> lot of 'em (hundreds as there are in my case), parallel build
  csm> performance is crippled.  It ends up compiling an object, linking
  csm> a binary, compiling an object, linking a binary over and over
  csm> effectively serializing the build process.

I don't know what you mean by this.  Make does NOT require a sibling
prerequisite to be completely built before it moves to the next one
(again, assuming no other prerequisite relationships).  I must be
misunderstanding the problem you're seeing.  Consider this makefile:

  .PHONY: all
  all: one.x two.x three.x four.x five.x six.x

  %.x : %.y ; @echo cp $< $@; sleep 1

  one.y two.y three.y four.y five.y six.y: ; @echo touch $@; sleep 1

This simulates a build such as you describe, where lots of little
programs are built each from one object file.  If I run this without -j,
I get the serial behavior you'd expect:

    $ make
    touch one.y
    cp one.y one.x
    touch two.y
    cp two.y two.x
    touch three.y
    cp three.y three.x
    touch four.y
    cp four.y four.x
    touch five.y
    cp five.y five.x
    touch six.y
    cp six.y six.x

And, if I run it with -j6 I get the parallel behavior you'd expect:

    $ make -j6
    touch one.y
    touch two.y
    touch four.y
    touch five.y
    touch six.y
    touch three.y
    cp one.y one.x
    cp two.y two.x
    cp four.y four.x
    cp five.y five.x
    cp six.y six.x
    cp three.y three.x

Soooo... I'm missing something :-).

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <address@hidden>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.paulandlesley.org
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist