Re: Parallel Build with GNU make

[Howard Chu]

Martin Dorey wrote:
> -----Original Message-----
> From: address@hidden [mailto:address@hidden On Behalf Of Roland Schwingel
> Sent: Monday, February 01, 2016 23:40
> To: Eli Zaretskii; address@hidden
> Subject: Re: Parallel Build with GNU make
>
> Sorry for following up on my own post,but there has been a "bug" in the
> last sentence... See for correction below
>
> Am 02.02.2016 um 08:35 schrieb Roland Schwingel:
> > Hi...
> >
> > Eli Zaretskii wrote on 01.02.2016 20:14:17:
> >
> >   > Add "--debug=j" to the make command-line switches and see if it
> >   > launches more than one command at a time.
> >   >
> >   > Anyway, your makefile.mak file can be written in such a way as to
> >   > effectively prohibit parallelism, due to how it lays out dependencies.
> > This is something I am struggeling on. I tried this a couple of times
> > but could not make it working in a way that is satisfying for me, as I
> > could not find the correct way to control parallelism. I am aware of
> > .NOTPARALLEL but this might not be enough for me.
> >
> > I have this examplarily layout:
> >
> > subfolder1 - subfolder4 - sub4_1.c
> >                sub1_1.c     sub4_2.c
> >                sub1_2.c
> >                ...
> > subfolder2 - sub2_1.c
> >                sub2_2.c
> > subfolder3 - sub3_1.c
> >                sub3_2.c
> > main1.c
> > main2.c
> > main3.c
> >
> > All subfolders shall be visited recursively first. No parallelism is
> > allowed in visiting folders. But when in a folder and it comes to
> > concrete compiling parallelism should be allowed.
> >
> > This should result in this compileorder for the above example
> > Visit subfolder1 (not in parallel)
> > Visit subfolder4 (not in parallel)
> > Compile sub4_1.c and sub4_2.c in parallel
> > Return to subfolder1
> > Compile sub1_2.c and sub1_2.c in parallel
> > Return to main
> > Visit subfolder2 (not in parallel)
> > Compile sub2_1.c and sub2_2.c in parallel
> > Return to main
> > Vist subfolder3 (not in parallel)
> > Compile sub3_1.c and sub3_2.c in parallel
> > Return to main
> > Compile main1.c,main2.c and main3.c in parallel
> >
> > When doing a make -j 4 for instance make is visiting all subfolders in
> > parallel which is not desired. When I am using .NOTPARALLEL the whole
> > subfolder is not compiled in parallel which is also not desired. Each
> > folder visit is a concrete submake in my case. Each folder has its own
> > makefile
> >
> > Any clue on how to achieve the flow I want to have? Visiting folders non
> > recursive while compiling itself inside of the folders is recursive?
> Should read:
> Visiting folders non PARALLEL while compiling itself inside of the
> folders is IN PARALLEL.

Use a for-loop for anything that must be explicitly serial.

Generally, to me this is a sign of a broken Makefiles - when you define your 
dependencies correctly, everything should naturally run in the correct order, 
regardless of degree of parallelism. The fact that you want subfolder 2 to be 
built after subfolder 1 means sub1 *is* a dependency of sub2 and if you 
haven't stated that in your Makefile then you've made an error in your Makefile.

Read this:
http://highlandsun.com/hyc/#Make
in particular this
http://highlandsun.com/hyc/GNUYou.htm

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/