[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: make internals and parallelization

From: Michael Muratet
Subject: Re: make internals and parallelization
Date: Wed, 12 Aug 2009 12:10:16 -0500

On Aug 12, 2009, at 11:13 AM, Paul Smith wrote:

On Wed, 2009-08-12 at 10:23 -0500, Michael Muratet wrote:
I have been searching the list archives looking for information on how make implements parallelization with make -j. From a most global view, it appears that there is a data structure of dependancies and multiple
branches from the root are candidates for parallel processing. I
haven't seen any mention of a developer's manual, is there one? Can
anyone point me to some documentation or explain the process?


There is no developers manual.  I'm not aware of too many open source
projects that provide such a thing: there's the source code and the
comments therein after all :-).
Me either, but it never hurts to ask. I will check out the latest version and start reading.

It's not clear what exactly you're trying to understand so it's hard to
explain the process.
I'm looking at a very complicated make file that runs data analysis tools as opposed to building code and I'm trying to understand if it's possible for racing to occur, i.e., a dependancy that's obscured and so one branch of the process crashes because it needed something in another branch that hasn't completed.

There's a discussion of GNU make's jobserver algorithm on my page (see
the signature), but that talks only about how make ensures that no more
than N jobs are run for "make -jN".
I read the discussion and found it very interesting. The answers to my questions may lie within, let me digest it for awhile.

Make doesn't implement any sort of threading. Since make spends most of
its time just waiting for child processes to complete, when you're
running in parallel it simply uses that "down time" to determine the
next job that could be run.  When it finds one, it will start that job
and begin looking through its data structures for the next one, etc.
If and how it threads were among my questions. So when it starts the nth job, how does it do that? We've had some problems with vfork in the past and I had some posts about it a few weeks ago. I'm 99% sure it's a memory problem and I'm chasing that, but is make using vfork to start jobs? How does it start processes.

Thanks for the help


Paul D. Smith <address@hidden>          Find some GNU make tips at:            
"Please remain calm...I may be mad, but I am a professional." --Mad Scientist

reply via email to

[Prev in Thread] Current Thread [Next in Thread]