[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: How to restrict parallelism in make? [PATCH]
|
From: |
Alexey Neyman |
|
Subject: |
Re: How to restrict parallelism in make? [PATCH] |
|
Date: |
Fri, 29 Apr 2005 18:45:42 +0400 |
|
User-agent: |
KMail/1.6.2 |
Hi,
an updated (an more thouroughly tested) patch to support .WAIT
attached (includes documentation in make.texi and printing .WAIT
dependencies).
Regards,
Alexey.
On Friday 29 April 2005 13:28, Alexey Neyman wrote:
> Hi,
>
> I have come across the following problem with parallel jobs in make.
I
> have a project that uses recursive build. I'd like to have
> subdirectories being built before the local targets. In a sequential
> Makefile, one would do the following:
>
> <<<<
> .PHONY: all all-subdir all-local
> all: all-subdir all-local
> all-subdir:
> # visit the subdirectories and run "make all" there
>
> all-local: $(whatever local targets are)
> >>>>
>
> In parallel make, however, the 'all-local' and 'all-subdir' targets
> will be made in parallel. Note that making 'all-local' depend on
> 'all-subdir' won't help: the prerequisites of 'all-local' will still
> be made in parallel with 'all-subdir'.
>
> Well, this could be worked around by making 'all-subdir' an
order-only
> prerequisite to each and every target that 'all-local' depends upon,
> and their prerequisites, and so on... Though I believe this is
rather
> counternatural, it helps.
>
> And the real problems arise when I try to add the 'install' target.
> Similarly, I'd like the 'install' target to walk into the
> subdirectories first and then perform local activities:
>
> <<<<
> install: install-subdir install-local
> >>>>
>
> But if I add the install-subdir as an order-only prerequisite to all
> targets, as described above, this will cause "make install" executed
> in the subdirectories even in case of "make all" in the top-level
> directory. Well, this could be worked around by having
> 'install-subdir' target depend on 'all', but this way the
> subdirectories will be traversed twice: once for "make all" and once
> again for "make install". Alternatively, one could serialize the
> targets artificially (at a cost of extra instance of make running):
>
> <<<<
> all:
> $(MAKE) all-subdir
> $(MAKE) all-local
>
> install:
> $(MAKE) install-subdir
> $(MAKE) install-local
> >>>>
>
> An elegant solution to this problem has long been available in BSD
> version of make (pmake):
>
>
http://www.freebsd.org/cgi/man.cgi?query=make&apropos=0&sektion=0&manpath=FreeBSD+5.3-stable&format=html
>
> There, the special '.WAIT' target when listed as a dependency, acts
as
> a "serializer": all dependencies _before_ it in the list are
finished
> before any of those _after_ .WAIT are even started to be updated.
> With this, the above requirements could be supported easily:
>
> <<<<
> all: all-subdir .WAIT all-local
> install: install-subdir .WAIT install-local
> >>>>
>
> I therefore decided to implement it in GNU make. The suggested patch
> is attached. I also added the ability to specify the whole target as
> requiring sequential update by specifying that target as a
dependency
> for the .NOTPARALLEL target (without prerequisites, it disables
> parallelism at all).
>
> Also attached is a test Makefile and the results of the tests. Run
> with "make -j WAIT=.WAIT <target>", where target is one of the
"all",
> "all2", "all3" (they test the .WAIT behavior for implicit
> dependencies, explicit ones and .NOTPARALLEL, respectively).
>
> There is an alternative in the SCO version of make available
(.MUTEX):
>
> http://uw713doc.sco.com/en/man/html.1/make.1.html
>
> However, the BSD variant is more flexible, so I chose that.
>
> Regards,
> Alexey.
>
> --
> For mundane material concerns, seek the help of the one-eyed beings.
> -- Pkunks, SC2
>
--
But wait!
This doesn't have to be the end!
-- Pkunks, SC2
Index: default.c
===================================================================
RCS file: /cvsroot/make/make/default.c,v
retrieving revision 1.44
diff -u -r1.44 default.c
--- default.c 16 May 2004 19:16:53 -0000 1.44
+++ default.c 29 Apr 2005 14:40:28 -0000
@@ -532,7 +532,7 @@
char *p = default_suffixes;
suffix_file->deps = (struct dep *)
multi_glob (parse_file_seq (&p, '\0', sizeof (struct dep), 1),
- sizeof (struct dep));
+ sizeof (struct dep), NULL);
(void) define_variable ("SUFFIXES", 8, default_suffixes, o_default, 0);
}
}
Index: dep.h
===================================================================
RCS file: /cvsroot/make/make/dep.h,v
retrieving revision 1.19
diff -u -r1.19 dep.h
--- dep.h 13 Apr 2005 03:16:33 -0000 1.19
+++ dep.h 29 Apr 2005 14:40:28 -0000
@@ -41,6 +41,7 @@
unsigned int changed : 8;
unsigned int ignore_mtime : 1;
unsigned int need_2nd_expansion : 1;
+ unsigned int wait : 1;
};
@@ -53,7 +54,7 @@
};
-extern struct nameseq *multi_glob PARAMS ((struct nameseq *chain, unsigned int
size));
+extern struct nameseq *multi_glob PARAMS ((struct nameseq *chain, unsigned int
size, int *seen_special));
#ifdef VMS
extern struct nameseq *parse_file_seq ();
#else
Index: file.c
===================================================================
RCS file: /cvsroot/make/make/file.c,v
retrieving revision 1.78
diff -u -r1.78 file.c
--- file.c 13 Apr 2005 03:16:33 -0000 1.78
+++ file.c 29 Apr 2005 14:40:29 -0000
@@ -414,6 +414,28 @@
f->intermediate = 1;
}
+/* Notice special dependencies. */
+static void
+notice_special_deps(struct dep **d_ptr)
+{
+ struct dep *d;
+
+restart_loop:
+ for (; *d_ptr; d_ptr = &(*d_ptr)->next)
+ {
+ d = *d_ptr;
+ if (!strcmp(d->name, ".WAIT"))
+ {
+ free(d->name);
+ if (d->next)
+ d->next->wait = 1;
+ *d_ptr = d->next;
+ free(d);
+ goto restart_loop;
+ }
+ }
+}
+
/* Expand and parse each dependency line. */
static void
expand_deps (struct file *f)
@@ -422,6 +444,7 @@
struct dep *new = 0;
struct dep *old = f->deps;
unsigned int last_dep_has_cmds = f->updating;
+ int seen_special = 0;
int initialized = 0;
f->updating = 0;
@@ -457,7 +480,7 @@
new = (struct dep *)
multi_glob (
parse_file_seq (&p, '|', sizeof (struct dep), 1),
- sizeof (struct dep));
+ sizeof (struct dep), &seen_special);
if (*p)
{
@@ -473,12 +496,18 @@
*d_ptr = (struct dep *)
multi_glob (
parse_file_seq (&p, '\0', sizeof (struct dep), 1),
- sizeof (struct dep));
+ sizeof (struct dep), &seen_special);
for (d1 = *d_ptr; d1 != 0; d1 = d1->next)
d1->ignore_mtime = 1;
}
+ /* Before dependencies are entered as files, treat special
+ dependencies. Need to do this before entering them as files
+ as treatment might involve deletion of a dependency. */
+ if (seen_special)
+ notice_special_deps(&new);
+
/* Enter them as files. */
for (d1 = new; d1 != 0; d1 = d1->next)
{
@@ -538,7 +567,7 @@
{
struct file *f;
struct file *f2;
- struct dep *d;
+ struct dep *d, *d2;
struct file **file_slot_0;
struct file **file_slot;
struct file **file_end;
@@ -646,9 +675,20 @@
if (f != 0 && f->is_target)
posix_pedantic = 1;
+ /* .NOTPARALLEL with deps makes these targets non-parallel (i.e. all
+ their dependencies are serialized). Without deps, it forbids
+ parallel execution at all. */
f = lookup_file (".NOTPARALLEL");
if (f != 0 && f->is_target)
- not_parallel = 1;
+ {
+ if (f->deps == 0)
+ not_parallel = 1;
+ else
+ for (d = f->deps; d != 0; d = d->next)
+ for (f2 = d->file; f2 != 0; f2 = f2->prev)
+ for (d2 = f2->deps; d2 != 0; d2 = d2->next)
+ d2->wait = 1;
+ }
/* Remember that we've done this. */
snapped_deps = 1;
@@ -788,17 +828,17 @@
/* Print all normal dependencies; note any order-only deps. */
for (d = f->deps; d != 0; d = d->next)
if (! d->ignore_mtime)
- printf (" %s", dep_name (d));
+ printf (" %s%s", d->wait ? ".WAIT " : "", dep_name (d));
else if (! ood)
ood = d;
/* Print order-only deps, if we have any. */
if (ood)
{
- printf (" | %s", dep_name (ood));
+ printf (" | %s%s", d->wait ? ".WAIT " : "", dep_name (ood));
for (d = ood->next; d != 0; d = d->next)
if (d->ignore_mtime)
- printf (" %s", dep_name (d));
+ printf (" %s%s", d->wait ? ".WAIT " : "", dep_name (d));
}
putchar ('\n');
Index: function.c
===================================================================
RCS file: /cvsroot/make/make/function.c,v
retrieving revision 1.87
diff -u -r1.87 function.c
--- function.c 4 Mar 2005 12:52:32 -0000 1.87
+++ function.c 29 Apr 2005 14:40:29 -0000
@@ -353,7 +353,7 @@
That would break examples like:
$(patsubst ./%.c,obj/%.o,$(wildcard ./?*.c)). */
0),
- sizeof (struct nameseq));
+ sizeof (struct nameseq), NULL);
if (result == 0)
{
Index: implicit.c
===================================================================
RCS file: /cvsroot/make/make/implicit.c,v
retrieving revision 1.47
diff -u -r1.47 implicit.c
--- implicit.c 13 Apr 2005 03:16:33 -0000 1.47
+++ implicit.c 29 Apr 2005 14:40:29 -0000
@@ -75,6 +75,7 @@
char *intermediate_pattern; /* pattern for intermediate file */
unsigned char had_stem; /* had % substituted with stem */
unsigned char ignore_mtime; /* ignore_mtime flag */
+ unsigned char wait;
};
static void
@@ -185,6 +186,28 @@
return beg;
}
+/* Handle special dependencies in implicit rules. */
+static void
+notice_special_ideps(struct idep **id_ptr)
+{
+ struct idep *id;
+
+restart_loop:
+ for (; *id_ptr; id_ptr = &(*id_ptr)->next)
+ {
+ id = *id_ptr;
+ if (!strcmp(id->name, ".WAIT"))
+ {
+ free(id->name);
+ if (id->next)
+ id->next->wait = 1;
+ *id_ptr = id->next;
+ free(id);
+ goto restart_loop;
+ }
+ }
+}
+
/* Search the pattern rules for a rule with an existing dependency to make
FILE. If a rule is found, the appropriate commands and deps are put in FILE
and 1 is returned. If not, 0 is returned.
@@ -261,6 +284,10 @@
that is not just `%'. */
int specific_rule_matched = 0;
+ /* Nonzero if we saw possibly special dependencies (e.g. .WAIT) while
+ parsing the dependencies for that rule. */
+ int seen_special;
+
register unsigned int i = 0; /* uninit checks OK */
register struct rule *rule;
register struct dep *dep, *expl_d;
@@ -453,6 +480,9 @@
pattern_search won't try to use it. */
rule->in_use = 1;
+ /* This rule hasn't encountered special dependencies (so far). */
+ seen_special = 0;
+
/* From the lengths of the filename and the matching pattern parts,
find the stem: the part of the filename that matches the %. */
stem = filename
@@ -557,14 +587,14 @@
multi_glob (
parse_file_seq (&p2,
order_only ? '\0' : '|',
- sizeof (struct idep),
- 1), sizeof (struct idep));
+ sizeof (struct idep), 1),
+ sizeof (struct idep), &seen_special);
/* @@ It would be nice to teach parse_file_seq or
multi_glob to add prefix. This would save us
some reallocations. */
- if (order_only || add_dir || had_stem)
+ if (order_only || had_stem)
{
unsigned long l = lastslash - filename + 1;
@@ -609,6 +639,10 @@
file->stem = 0;
+ /* If there were special dependencies, handle them. */
+ if (seen_special)
+ notice_special_ideps(&deps);
+
/* @@ This loop can be combined with the previous one. I do
it separately for now for transparency.*/
@@ -836,6 +870,7 @@
dep = (struct dep *) xmalloc (sizeof (struct dep));
dep->ignore_mtime = d->ignore_mtime;
+ dep->wait = d->wait;
dep->need_2nd_expansion = 0;
s = d->name; /* Hijacking the name. */
d->name = 0;
@@ -908,7 +943,7 @@
{
struct dep *new = (struct dep *) xmalloc (sizeof (struct dep));
/* GKM FIMXE: handle '|' here too */
- new->ignore_mtime = 0;
+ new->ignore_mtime = new->wait = 0;
new->need_2nd_expansion = 0;
new->name = p = (char *) xmalloc (rule->lens[i] + fullstemlen + 1);
bcopy (rule->targets[i], p,
Index: main.c
===================================================================
RCS file: /cvsroot/make/make/main.c,v
retrieving revision 1.202
diff -u -r1.202 main.c
--- main.c 13 Apr 2005 03:16:33 -0000 1.202
+++ main.c 29 Apr 2005 14:40:29 -0000
@@ -2108,7 +2108,7 @@
ns = multi_glob (
parse_file_seq (&p, '\0', sizeof (struct nameseq), 1),
- sizeof (struct nameseq));
+ sizeof (struct nameseq), NULL);
/* .DEFAULT_TARGET should contain one target. */
if (ns->next != 0)
@@ -2124,7 +2124,7 @@
goals = (struct dep *) xmalloc (sizeof (struct dep));
goals->next = 0;
goals->name = 0;
- goals->ignore_mtime = 0;
+ goals->ignore_mtime = goals->wait = 0;
goals->need_2nd_expansion = 0;
goals->file = default_goal_file;
}
@@ -2290,7 +2290,7 @@
}
lastgoal->name = 0;
lastgoal->file = f;
- lastgoal->ignore_mtime = 0;
+ lastgoal->ignore_mtime = lastgoal->wait = 0;
lastgoal->need_2nd_expansion = 0;
{
Index: read.c
===================================================================
RCS file: /cvsroot/make/make/read.c,v
retrieving revision 1.143
diff -u -r1.143 read.c
--- read.c 13 Apr 2005 03:16:33 -0000 1.143
+++ read.c 29 Apr 2005 14:40:29 -0000
@@ -252,7 +252,7 @@
d->name = 0;
d->file = enter_file (*p);
d->file->dontcare = 1;
- d->ignore_mtime = 0;
+ d->ignore_mtime = d->wait = 0;
d->need_2nd_expansion = 0;
/* Tell update_goal_chain to bail out as soon as this file is
made, and main not to die if we can't make this file. */
@@ -366,6 +366,7 @@
read_makefiles = deps;
deps->name = 0;
deps->file = lookup_file (filename);
+ deps->ignore_mtime = deps->wait = 0;
if (deps->file == 0)
deps->file = enter_file (xstrdup (filename));
if (filename != ebuf.floc.filenm)
@@ -373,6 +374,7 @@
filename = deps->file->name;
deps->changed = flags;
deps->ignore_mtime = 0;
+ deps->wait = 0;
deps->need_2nd_expansion = 0;
if (flags & RM_DONTCARE)
deps->file->dontcare = 1;
@@ -807,7 +809,7 @@
files = multi_glob (parse_file_seq (&p2, '\0',
sizeof (struct nameseq),
1),
- sizeof (struct nameseq));
+ sizeof (struct nameseq), NULL);
free (p);
/* Save the state of conditionals and start
@@ -1002,7 +1004,7 @@
filenames = multi_glob (parse_file_seq (&p2, '\0',
sizeof (struct nameseq),
1),
- sizeof (struct nameseq));
+ sizeof (struct nameseq), NULL);
*p2 = ':';
if (!filenames)
@@ -2955,7 +2957,7 @@
that have room for additional info. */
struct nameseq *
-multi_glob (struct nameseq *chain, unsigned int size)
+multi_glob (struct nameseq *chain, unsigned int size, int *seen_special)
{
extern void dir_setup_glob ();
register struct nameseq *new = 0;
@@ -2983,6 +2985,10 @@
}
}
+ if (seen_special && old->name[0] == '.'
+ && isalpha (old->name[1]) && isupper (old->name[1]))
+ *seen_special = 1;
+
#ifndef NO_ARCHIVES
if (ar_name (old->name))
{
Index: remake.c
===================================================================
RCS file: /cvsroot/make/make/remake.c,v
retrieving revision 1.116
diff -u -r1.116 remake.c
--- remake.c 8 Apr 2005 12:51:20 -0000 1.116
+++ remake.c 29 Apr 2005 14:40:30 -0000
@@ -361,6 +361,19 @@
error (NILF, msg_parent, "*** ", file->name, file->parent->name, ".");
}
+/* Mark the dependencies as considered in this pass (to avoid skipping them
+ on the next pass). */
+static void
+consider_deps (struct dep *d)
+{
+ for (; d; d = d->next)
+ {
+ if (d->file->considered != considered)
+ consider_deps (d->file->deps);
+ d->file->considered = considered;
+ }
+}
+
/* Consider a single `struct file' and update it as appropriate. */
static int
@@ -471,6 +484,13 @@
int maybe_make;
int dontcare = 0;
+ /* The code below will set the state to cs_deps_running. */
+ if (d->wait && running)
+ {
+ consider_deps (d);
+ break;
+ }
+
check_renamed (d->file);
mtime = file_mtime (d->file);
@@ -542,20 +562,28 @@
if (must_make || always_make_flag)
{
for (d = file->deps; d != 0; d = d->next)
- if (d->file->intermediate)
- {
- int dontcare = 0;
+ {
+ /* The code below will set the state to cs_deps_running. */
+ if (d->wait && running)
+ {
+ consider_deps(d);
+ break;
+ }
- FILE_TIMESTAMP mtime = file_mtime (d->file);
- check_renamed (d->file);
- d->file->parent = file;
+ if (d->file->intermediate)
+ {
+ int dontcare = 0;
- /* Inherit dontcare flag from our parent. */
- if (rebuilding_makefiles)
- {
- dontcare = d->file->dontcare;
- d->file->dontcare = file->dontcare;
- }
+ FILE_TIMESTAMP mtime = file_mtime (d->file);
+ check_renamed (d->file);
+ d->file->parent = file;
+
+ /* Inherit dontcare flag from our parent. */
+ if (rebuilding_makefiles)
+ {
+ dontcare = d->file->dontcare;
+ d->file->dontcare = file->dontcare;
+ }
dep_status |= update_file (d->file, depth);
@@ -585,7 +613,8 @@
if (!running)
d->changed = ((file->phony && file->cmds != 0)
|| file_mtime (d->file) != mtime);
- }
+ }
+ }
}
finish_updating (file);
Index: rule.c
===================================================================
RCS file: /cvsroot/make/make/rule.c,v
retrieving revision 1.37
diff -u -r1.37 rule.c
--- rule.c 13 Apr 2005 03:16:33 -0000 1.37
+++ rule.c 29 Apr 2005 14:40:30 -0000
@@ -205,7 +205,7 @@
deps = (struct dep *) xmalloc (sizeof (struct dep));
deps->next = 0;
deps->name = depname;
- deps->ignore_mtime = 0;
+ deps->ignore_mtime = deps->wait = 0;
deps->need_2nd_expansion = 0;
}
@@ -390,7 +390,7 @@
ptr = p->dep;
r->deps = (struct dep *) multi_glob (parse_file_seq (&ptr, '\0',
sizeof (struct dep), 1),
- sizeof (struct dep));
+ sizeof (struct dep), NULL);
if (new_pattern_rule (r, 0))
{
Index: doc/make.texi
===================================================================
RCS file: /cvsroot/make/make/doc/make.texi,v
retrieving revision 1.27
diff -u -r1.27 make.texi
--- doc/make.texi 8 Apr 2005 12:51:20 -0000 1.27
+++ doc/make.texi 29 Apr 2005 14:40:30 -0000
@@ -2707,15 +2707,28 @@
@xref{Variables/Recursion, ,Communicating Variables to a
address@hidden
address@hidden .WAIT
address@hidden .WAIT
address@hidden parallel execution, serializing dependencies
+
+This has no special meaning when specified as a target. When specified as
+a prerequisite, @code{.WAIT} acts as a sequence point: all dependencies
+preceding @code{.WAIT} are completed before those following @code{.WAIT}
+ are started.
+
@findex .NOTPARALLEL
@item .NOTPARALLEL
@cindex parallel execution, overriding
-If @code{.NOTPARALLEL} is mentioned as a target, then this invocation of
address@hidden will be run serially, even if the @samp{-j} option is
-given. Any recursively invoked @code{make} command will still be run in
-parallel (unless its makefile contains this target). Any prerequisites
-on this target are ignored.
+If @code{.NOTPARALLEL} is mentioned as a target without prerequisites,
+then this invocation of @code{make} will be run serially, even if the
address@hidden option is given. Any recursively invoked @code{make} command
+will still be run in parallel (unless its makefile contains this
+target). Any prerequisites on this target are ignored.
+
+If you specify prerequisites for @code{.NOTPARALLEL}, then @code{make}
+will remake those files sequentially, as if @code{.WAIT} was specified
+before each dependency.
@end table
Any defined implicit rule suffix also counts as a special target if it