Currently I work on basic distributed build system and found that scaling local and external resource is little tricky. I base on basic idea to run preprocessor locally and push his output to remote node for compilation.
My problem is that currently gmake could control only one job group (via -jN) and sometimes it overload main node with preprocessing jobs. Based on theory that preprocessing should be "much faster" that remote compilation I got idea of second jobserver for externals works.
#new keyword which should mark rules for second/external jobserver
%.i: %.c $(CC) -E $< -o $@
$(REMOTE_CC) $< -o $@
#gmake -j2 -x4 all
Which should run 2 local preprocessing work in parallel and up to 4 external compilations if any preprocess job finish. IMHO this may good contol local and external resources. Since I'm not guru in this matter then I want ask before start coding: Have such solution any sense?
If yest then could anyone give me some tips where start hacks?