[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Parallel build dependency locking

From: Christopher Sean Morrison
Subject: Re: Parallel build dependency locking
Date: Thu, 9 Jun 2005 19:11:37 -0400

Sounds like
it's fairly safe to presume that there's no concurrent make file/job
locking across to submakes (e.g. some file-based semaphore locking,
shared memory locking, etc).  Is this a bug/feature request worth

No. ;-)

Seriously, if you are building the same target more than once then
it's a bug in your makefile and you normally wouldn't want make to
hide it from you.

That may very well be true, but in this particular instance, the workaround/solution to use the all-am did nothing to change which targets were reached. It merely avoided a secondary invocation of make (that did the exact same thing, reached the same targets/rules in parallel, etc).

There is already support for processing multiple targets simultaneously albeit through some form of locking to ensure that each dependency/target is being handled by only one of the -j workers (in that process). That is the simple nature of a -j parallel build. All I'm suggesting is a means to extend that same locking mechanism on dependancies being processed by child, sub-process, or otherwise concurrent makes also.

Basically, that would be a means to share the current processing environment across concurrent makes. It should only impact the -j parallel builds if done correctly, allowing the workers that have completed with one dependency to progress to the next set of rules while remaining coherent with respect to each other's progress. This would be a significant boost on my larger SMP systems where a -j32 or -j200 or larger is not unheard of -- with proper locking, you could even start to think about structuring together makefiles that perform simultaneous multi-directory builds (e.g. build all your libraries in parallel before building your binaries (also in parallel)) or more complicated rule resolutions. Heck, one could even invoke make directly N times (without -j and with proper external locking) and have it successfully build in parallel instead of having the various processes stomp on each other's results (quite useless).

It's a feature that I've heard from others as well that work on larger SMP systems with make-based projects. In general, it's just requests for better support for parallel builds in general and the aforementioned suggestion would be an overall improvement. For now, I've hacked around it enough so that almost our entire package will now compile and link in parallel, significantly reducing our compile times -- so I'm quite happy. ;-) The only thing that I didn't attempt was processing multiple independent directories simultaneously.. I'll save that for another rainy day.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]