Re: Cooperation with distributed job processing systems

[SF Markus Elfring]

> Jobserver is a method of communicating how many jobs make thinks are
> running between different instances (parent/child) of the make
> program itself, so it knows that no more than N jobs are invoked
> between all instances.

How do you think about to delegate such an implementation detail
to a configurable job submission system?


> What you seem to be suggesting is something different:
> a facility for make to start jobs on remote systems.

Yes. - My suggestion goes into this direction.

There are programming interfaces available which support transparent
data exchange between local and remote information systems.


> There are various methods for this: the most common is to use distcc in
> conjunction with make: https://code.google.com/p/distcc/ although this
> is really restricted to building C/C++ programs

It is nice when remote compilation can be used with dedicated tools already.


> so not generally applicable.

I would appreciate when this limitation can be dropped eventually.


> There is a nascent facility in the GNU make code that provides for make
> invoking jobs through a separate facility: the "remote" capability.

I am curious to know a bit more about such a possibility.


> Currently there is a remote-stub.c which defines the interface, and a
> remote-cstms.c which attempts to use the Customs facility for starting
> remote jobs.  However, this is obsolete and I'm not sure if anyone has
> tried to use it for years.

Interesting background information ...


> Nevertheless if one were to try to add native facilities for starting
> jobs on remote systems into GNU make itself, rather than doing it
> through use of a special SHELL variable setting, etc. then remote would
> likely be the best way to do it, not jobserver.

Does the software use a clean internal API already?


> I urge you to explore these other methods first, though, as they're a
> lot less effort all the way around.  If you can write a small program
> that will forward a command to a remote system, then you can just
> replace SHELL in your makefile with that command:
> 
>     SHELL = /bin/submit-job
> 
>     all:
>             run-a-job

I find that this approach would apply one step too late because the make
program manages corresponding process parallelisation by the
parameter "-j".

Is an alternative job submission needed earlier?

Regards,
Markus