[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Parallel build dependency locking

From: Christopher Sean Morrison
Subject: Re: Parallel build dependency locking
Date: Thu, 9 Jun 2005 21:39:30 -0400

On Jun 9, 2005, at 8:01 PM, John Graham-Cumming wrote:

On Thu, 2005-06-09 at 19:11 -0400, Christopher Sean Morrison wrote:

There is already support for processing multiple targets simultaneously
albeit through some form of locking to ensure that each
dependency/target is being handled by only one of the -j workers (in
that process).  That is the simple nature of a -j parallel build.  All
I'm suggesting is a means to extend that same locking mechanism on
dependancies being processed by child, sub-process, or otherwise
concurrent makes also.
What you are suggesting is, no doubt, possible but you are asking for
locking on a target across independent make processes.  Clearly, one
could implement locking of that form (although doing it on every
platform that GNU Make supports might be tricky).

That is indeed what I'd be interested in having. Another alternative might be to consider the sub-process makes as not being independent, sharing the exec knowledge of their parent via some mechanism (environment vars, process id arguments, etc). It would seem more simple to just handle the more general independent process case, but considering only sub-process makes and making them dependent or otherwise state sharing would even be beneficial.

However, I think it's the wrong solution to your problem.  It appears
that what you want to do is speed up your build, and you've chosen to do
that by running a parallel build.   If you want to do that then there
are a couple of necessities: you need to make sure that there are no
missing dependencies and that nothing is built twice.

In the cases I have been involved with, those two concerns are generally not a problem to say the least. Most (including the one in question) are 'gnu build system' projects where the dependencies are already in place being described via automake and libtool declarations. With those declarations in place, the build proceeds just fine with proper dependency ordering and nothing builds twice. The double-compile problem that I was working through was clearly undesirable of course and I'd probably even go so far as to say building twice is simply 'wrong'. Mind you, I'm referring to the actual compilation of the source to an object file, not reaching the Makefile rule and realizing there's nothing to do (e.g. making target binary foo depend on binary bar's objects and bar itself -- when processing foo's dependencies, bar's objects are listed explicitly and indirectly through bar's rule).

The 'problem' is perhaps just a limitation of the make logic being exported by automake and presumptions on possible target dependancies (e.g. SUBDIRS, bin_PROGRAMS, lib_LTPROGRAMS, etc are always traversed in order one at a time). Since it processes each program/library in order, if there are a lot of 'em (hundreds as there are in my case), parallel build performance is crippled. It ends up compiling an object, linking a binary, compiling an object, linking a binary over and over effectively serializing the build process. All of the objects are actually independent (and specified as such via the automake rules) so they can go in parallel, as are all the binaries so they could all link in parallel so long as the corresponding objects are done. Minimally, I now know that it 'can' be done, since I just worked in my own make traversal logic that does just that -- it now compiles all object files in parallel and then links all binaries together in parallel.

That said, I was only able to do that by avoiding that separate make process that automake or autoconf injected on the 'all' target. While I do heavily concede that it's probably more a design limitation of the automake/autoconf-generated Makefiles since they could just as easily provide a similar 'fast' make path like I wrote, I believe that the cross-process dependency locking would allow much greater potential for more effective parallel builds without requiring too much hoop-jumping and customization on the Makefile writer. It would also produce the exact same end result with no detriment to other parallel or non-parallel builds (that come to mind).

It should indeed only 'build things once'. I'm suggesting extending the same logic that exists with the -j option across make instances. So if make -j2 works, so would running make& twice and it should be possible to make it execute with practically identical behavior.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]