Re: feature request: parallel builds feature

[Howard Chu]

Jim Michaels wrote:
> I wasn't digressing.  I was explaining the point.  the concept I am trying to
> present as a solution to the problem of making parallel stdin for --jobs in
> gnu make (which currenty doesn't work and is I guess single-threaded) is to
> make a separate terminal or command shell for each job, such as via a
> generated batch file or shell script.
>
> this is as simple as I can make it.

Who said stdin was a problem? Fundamentally the jobs spawned by make are batch 
jobs - they should not be requesting input from stdin in the first place.

Semantically the effect of running parallel make must be identical to running 
serial make. You cannot guarantee this to be true if jobs are reading from 
stdin because stdin's supply of data is inherently serial but the order it 
gets read is non-deterministic in a parallel build.

If the jobs you're spawning from make require input from stdin you need to 
rewrite those jobs.

> at the end of the shell script, you can put in whatever you like, such as
> synchronization stuff saying "I am done" by creating  a semaphore file, a flag
> file.
>
> the problem would then be porting BASH to windows and other platforms in order
> to handle --jobs.

bash has already been ported to Windows.
> I have learned that on a machine with 12 threads and 64GB of memory, you can
> have 50+ jobs running.
>
>
>     
> ------------------------------------------------------------------------------
>     *From:* Paul Smith <address@hidden>
>     *To:* Jim Michaels <address@hidden>
>     *Cc:* address@hidden
>     *Sent:* Monday, April 22, 2013 10:56 AM
>     *Subject:* Re: feature request: parallel builds feature
>
>     On Mon, 2013-04-22 at 00:42 -0700, Jim Michaels wrote:
>      > it currently has a problem with stdin, because at this point there is
>      > only one of those, only 1 of them gets it, and the others starve. so
>      > if your build needs stdin or creates files from the commandline using
>      > heredocs, you can't use it (check first!). you will get an error. gnu
>      > has not yet figured out a solution yet (I have, multiple shells IF you
>      > can control them... probably can't without some work creating batch
>      > files for the jobs). so there is a solution. even with a batch file,
>      > make would need some sort of way of reporting back error conditions. I
>      > think there are ways of managing that with files via presence-detect,
>      > sort of like semaphores. they should get cleared when the job ends, or
>      > when a switch is given to clear the state for that session if the
>      > session was broken with ctrl-c. well, I suppose a ctrl-c handler
>      > should still kill those terminals or cmd shells and clear up those
>      > files.
>      > what do you think?
>      > if a terminal is opened, it should be created without a window. some
>      > OS's have that option. some don't, like freeDOS, which would not have
>      > the ability to do terminals, parallel shell windows, or even the
>      > --jobs feature (but that's implementation-dependent).
>
>     Please keep the mailing list CC'd.  Thanks.
>
>     I'm afraid I still don't understand what you're asking for here.  You'll
>     need to back up and provide a description of your needs in a clear,
>     orderly way without digressions.
>
>     Yes, it's true that GNU make only provides its stdin to one job at a
>     time and which job gets it is essentially random.  In order to address
>     this we'd need to see a specific use-case or requirement, but my
>     suspicion is that all such possible use-cases are better solved by a
>     change of process at a level above what make can provide.
>
>
>
>
>
>
> _______________________________________________
> Bug-make mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/bug-make


--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/