Re: Cooperation with distributed job processing systems

[Paul Smith]

On Sat, 2015-01-03 at 23:30 +0100, SF Markus Elfring wrote:
> There are programming interfaces available which
> provide support for submitting jobs to bigger and
> more powerful computer systems.

> I imagine that the software "jobserver" could
> be extended for the convenient reuse of such APIs.
> http://make.mad-scientist.net/papers/jobserver-implementation/

I find it highly unlikely that jobserver will be helpful directly.
Jobserver is a method of communicating how many jobs make thinks are
running between different instances (parent/child) of the make program
itself, so it knows that no more than N jobs are invoked between all
instances.

What you seem to be suggesting is something different: a facility for
make to start jobs on remote systems.

There are various methods for this: the most common is to use distcc in
conjunction with make: https://code.google.com/p/distcc/ although this
is really restricted to building C/C++ programs so not generally
applicable.

There is a nascent facility in the GNU make code that provides for make
invoking jobs through a separate facility: the "remote" capability.
Currently there is a remote-stub.c which defines the interface, and a
remote-cstms.c which attempts to use the Customs facility for starting
remote jobs.  However, this is obsolete and I'm not sure if anyone has
tried to use it for years.

Nevertheless if one were to try to add native facilities for starting
jobs on remote systems into GNU make itself, rather than doing it
through use of a special SHELL variable setting, etc. then remote would
likely be the best way to do it, not jobserver.

I urge you to explore these other methods first, though, as they're a
lot less effort all the way around.  If you can write a small program
that will forward a command to a remote system, then you can just
replace SHELL in your makefile with that command:

    SHELL = /bin/submit-job

    all:
            run-a-job

now make will invoke: "/bin/submit-job -c run-a-job" (you can control
the flags with the .SHELLFLAGS variable).