[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Parallel build dependency locking

From: Christopher Sean Morrison
Subject: Re: Parallel build dependency locking
Date: Wed, 8 Jun 2005 17:57:00 -0400


Thanks for the reply! One of your comments pushed me into the right direction; I believe I have it working now and maybe even understand what was happening that caused problems. I simplified the fast target rule to just simply build without recursing since the recursion was not strictly relevant to that particular library example (since I was already fully recursed with no remaining SUBDIRS). That said, it does seem like a possible contention bug or perhaps just undesirable behavior. I simplified the 'fast' target rule to just:

fast: $(FAST_OBJECTS) all

and the problem still remained. Yet when I removed the 'all' dependency, the problem went away. Looking at the automake-generated 'all' dependency, the problem becomes more clear:

        $(MAKE) $(AM_MAKEFLAGS) all-am

So there is the second invocation of make coming into play that hits up the 'all-am' rule, which eventually includes the same objects included in FAST_OBJECTS. So when the first process is working on the last couple FAST_OBJECTS objects, the second one continues to the 'all' rule, causes make to reinvoke, and ends up picking up where the build seemingly left off at which ultimately clobbers the first. Sounds like it's fairly safe to presume that there's no concurrent make file/job locking across to submakes (e.g. some file-based semaphore locking, shared memory locking, etc). Is this a bug/feature request worth reporting?

The fix/workaround for this particular example was rather simple. The second invocation can be avoided by using 'all-am' as the 'fast' rule dependency instead of 'all'. Once I did that, the sub-make is skipped. The entire recursion and fast build subsequently works great. For our particular software suite which includes over 400 binaries and a couple dozen libraries, this saves quite a bit of time (wish the automake folks provided such a facility automatically). Thanks again.


On Jun 8, 2005, at 12:15 PM, Boris Kolpackov wrote:

First off, there are no threads; GNU make issues (certain) commands in
parallel when run with -j flag.

Christopher Sean Morrison <address@hidden> writes:

fast: fast-local $(FAST_OBJECTS) fast-recursive all

        @list='$(SUBDIRS)'; for subdir in $$list; do \
test "$$subdir" = . || (cd $$subdir && $(MAKE) $(AM_MAKEFLAGS) fast);

Perhaps you are building the same object file from two concurrent make
invocations. One is current and the other is the result of the
fast-recursive rule?

Compared to wavelet.c, vers.c is very trivial so I can see how a thread
working on vers.lo would finish before the wavelet.lo target's thread
gets a chance to even start writing.

That would only make sense,
though, if the OBJECTS are not being properly locked by make on a
parallel build

GNU make doesn't do any locking, AFAICT.

-- and it doesn't seem to explain why the
target with the same target dependancies doesn't exhibit the same

Perhaps it's because fast-recursive rule is not built?


reply via email to

[Prev in Thread] Current Thread [Next in Thread]