[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Prioritizing non-dependent targets in parallel make

From: Eric Melski
Subject: Re: Prioritizing non-dependent targets in parallel make
Date: Mon, 04 Jan 2010 14:19:48 -0800
User-agent: Thunderbird (X11/20090817)

Tim Murphy wrote:

I'm nobody official.  I have seen Electric Accelerator using history
files which may be doing something a bit like this.

Personally I don't like the idea of priorities very much.  Large
builds that I have done do have big targets but since we are building
5000 of the same kind of target (using a macro to define a generic
template and $eval to instantiate it for each specific target) it
becomes impossible for us to provide hints.  In the end we'd have to
force the users of our build system say  "webkit.dll" is pretty huge
or  "euser.dll" isn't.  Since this is all relative information, how
can we expect a build made up of components that various people
produce to have the right "relative" weights?

So I, personally, want something that learns.  I would like to be able
to store the learned information into source control and use it as the
basis for the next attempt.  One would have to check for patents on
this, I think.

Hi Tim!

ElectricAccelerator doesn't factor runtimes into scheduling decisions, although we have talked about doing so in the past. I spent some time investigating the possibility, most recently a couple years ago. What I did was tweak the Simulator report in ElectricInsight to simulate this kind of scheduling, so each time it chooses the next job to "run", it picks the longest job of those that are available to start. I ran the simulation on a variety of real-world builds.

Unfortunately, the results were disappointing. In my recollection, the best case gave a savings of about 5% of the total build time, and in general it was barely measurable -- maybe 1%. It seems that in practice, most of the jobs in a build are about the same length. The notable exception is the final link, which is of course much longer. But then -- of course -- that can't start until all the other jobs are done anyway, so the fancy scheduling doesn't help.

I'll see if I can dig up the patch for ElectricInsight and maybe write it up in a blog. up that code, and maybe write it up in a blog.


Eric Melski
Electric Cloud, Inc.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]