[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: EAGAIN in "read jobs pipe"

From: Howard Chu
Subject: Re: EAGAIN in "read jobs pipe"
Date: Wed, 10 Jan 2007 01:53:11 -0800
User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20061216 Netscape/7.2 (ax) Firefox/1.5 SeaMonkey/1.5a

address@hidden wrote:
Noticed something else.

Something else completely pointless!

<ridiculous> must not be > 16385

True, if you have 16 KB pipes...

 gmake -j 16386; # or above hang
 gmake -j 16385; # or below fire along rapidly

Because you're trying to write more into the job tokens pipe than it can

You appearantly have 16K pipes (ulimit -a should tell you)... I get the
same on x86 Linux if I go beyond 4K jobs...

This code (from main.c) assumes that the pipe will be big enough to hold
all tokens:

      while (--job_slots)
          int r;

          EINTRLOOP (r, write (job_fds[1], &c, 1));
          if (r != 1)
            pfatal_with_name (_("init jobserver pipe"));

A bug?

An essential design choice. This stuff relies on reads and writes of the job_fd being atomic and the writes never blocking. POSIX guarantees a 4K buffer for pipes. Perhaps the code should check the resource limit and complain if the -j argument exceeds the resource limit: "Error: why don't you just use '-j' ??"

 Perhaps... On the other hand, if you're using -j 65K, why not
just -j? Does you build even have 65K jobs?

Very good question.
I could be wrong of course, but I in my experience you don't gain any
real benefit from going beyond 3-4 jobs per (virtual) core... What's the
difference in build time from, say, -j 128 and -j 65385 for you?

I usually count on 1.5 jobs per core, but obviously the right balance depends on your disk speeds relative to your CPUs...
The fact that your machine stays responsive during a build does not
necessarily mean that you will gain any benefit from increasing the -j

When I see that total CPU idle time is about 3% or less I figure that's enough jobs.

 -- Howard Chu
 Chief Architect, Symas Corp.  http://www.symas.com
 Director, Highland Sun        http://highlandsun.com/hyc
 OpenLDAP Core Team            http://www.openldap.org/project/

reply via email to

[Prev in Thread] Current Thread [Next in Thread]