RE: Parallel builds across makefiles

[Gardell, Steven]

OK. Thanks! I will try the latest version. 

FWIW, I am measuring is total wall clock time to complete the build. (date; 
gmake... ; date) This goes up substantially with all sync modes other than 
"none" if gmake has parallel invocations of sub-makes going (e.g. "gmake  -f 
SomeOtherMakefile").  When I am only parallelizing individual makefiles (with 
ACE, these tend to be simple lists of files to compile), then the behavior is 
as promised by the documentation. 

My expectation for enhanced parallelism is of course a bit more nebulous. At 
some point we are clearly going to run out of resources. But monitoring the 
state of the machine (HP G9 with gobs of memory and 56 hyper-threads available) 
suggests it is not breathing hard through most of the build.

-----Original Message-----
From: Paul Smith [mailto:address@hidden 
Sent: Sunday, July 17, 2016 11:33 AM
To: Gardell, Steven <address@hidden>; address@hidden
Subject: Re: Parallel builds across makefiles

On Thu, 2016-07-14 at 15:40 -0700, sgardell wrote:
> But when I try to build across makefiles in parallel then I see two 
> different things:
>    1) If I turn on any sort of output sync it gets dramatically slower.
> Sometimes slower than our -j1 time!

I recommend trying with the latest release, GNU make 4.2.1 to see if that helps.

If not then some debugging is needed as I can't reproduce this here.
It's true that, depending on the sync output mode you've selected (which you 
didn't specify), you may not see individual updates as you originally did 
(there may be long pauses in the output followed by a massive dump of output) 
but the overall build time shouldn't be significantly impacted.

>    2) Even with output sync turned completely off there is no 
> meaningful improvement over a parallel build of just the leaves.

I'm not sure I understand this comment, but this isn't too surprising to me.  
The only improvement of complete parallelism across the entire tree over 
parallelism of only the leaves would be at the edges, where right as one leaf 
makefile is finishing up it could hand off some extra jobs to another leaf 
makefile which wasn't done yet.  That may, depending on the way the makefiles 
work and the amount of parallelism you have enabled, not be very noticeable.