Re: make internals and parallelization

[Paul Smith]

On Wed, 2009-08-12 at 12:10 -0500, Michael Muratet wrote:
> > It's not clear what exactly you're trying to understand so it's hard  
> > to explain the process.
> I'm looking at a very complicated make file that runs data analysis  
> tools as opposed to building code and I'm trying to understand if it's  
> possible for racing to occur, i.e., a dependancy that's obscured and  
> so one branch of the process crashes because it needed something in  
> another branch that hasn't completed.

Well, there's nothing in learning the algorithm of make that will help
you with that.  If your makefile doesn't declare all prerequisites for
each target then parallel builds are potentially going to fail,
regardless of the algorithm used.

> > Make doesn't implement any sort of threading.  Since make spends  
> > most of its time just waiting for child processes to complete, when you're
> > running in parallel it simply uses that "down time" to determine the
> > next job that could be run.  When it finds one, it will start that job
> > and begin looking through its data structures for the next one, etc.
> If and how it threads were among my questions. So when it starts the  
> nth job, how does it do that? We've had some problems with vfork in  
> the past and I had some posts about it a few weeks ago. I'm 99% sure  
> it's a memory problem and I'm chasing that, but is make using vfork to  
> start jobs? How does it start processes.

You don't specify what operating system you're using, so I can't tell.
GNU make uses the autoconf AC_FUNC_FORK macro (see the autoconf manual
for details), which checks that vfork exists and verifies that it
doesn't have any of several well-known issues.  If it doesn't, then make
will use vfork().  If any of those issues are detected, make will use
fork().  You can check the symbols in your executable, or, if you built
GNU make yourself, look in the config.h file to see what configure
determined, and/or the config.log file.

Make starts a job just as you'd expect: it forks/execs the command.  If
the command contains shell operators/builtins, then make will fork/exec
a shell and pass the command line to that (called "the slow path"); if
not it will fork/exec the command directly (called "the fast path").

There can be no way that fork vs. vfork use in make can cause any
applications started by make to fail: once the exec happens there are no
remnants of the old process left, even using vfork.  vfork is only
potentially dangerous to the parent program (make, in this case), so
unless you're seeing _make_ fail this is not an issue.

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <address@hidden>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.mad-scientist.net
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist