Re: Problem with thousands of small jobs.

[Felipe Alvarez]

Use the cores you have : 32

Do you still get those issues?

Cheers,
Felipe

On Wed, Feb 11, 2015 at 12:55 PM, xmoon 2000 <xmoon2000@googlemail.com> wrote:

On 7 February 2015 at 14:23, Ole Tange <tange@gnu.org> wrote:
> On Sat, Feb 7, 2015 at 10:11 AM, xmoon 2000 <xmoon2000@googlemail.com> wrote:
>> On 6 February 2015 at 20:42, Ole Tange <ole@tange.dk> wrote:
>>> On Fri, Feb 6, 2015 at 3:56 PM, xmoon 2000 <xmoon2000@googlemail.com> wrote:
>>>
>>> My experience with Cygwin is limited, but I seem to remember that
>>> spawning is expensive (in the order of 0.1 second), which is one of
>>> the reasons for recommending MSYS. So if your jobs all finish at the
>>> same time, GNU Parallel will need 3.2 seconds to start 32 new jobs.
>>> Maybe you will have a better experience if you delay starting jobs by
>>> 0.1 seconds:
>>>
>>>   cat /tmp/parList | parallel -j 28 --delay 0.1 --eta;
>>>
>>> What is your reason for not using all 32 cores?
>>
>> 1. Reason for not using 32 cores is then I can't do anything else. I
>> need to do emails, check web, simple stuff whilst waiting for jobs to
>> finish.
>
> I tested this on a 4 core Microsoft Windows 8 machine and it is not
> true there: The machine will be slower, but not useless. On a 32 core
> machine I would expect this to be even less of a problem.
>
> On GNU/Linux you would use --nice which would mean that all other
> processes would get more CPU time than the jobs. In practice the jobs
> will only use the spare CPU time. I have not tested if --nice gives
> the same behaviour on Microsoft Windows.
>
> If you simply assumed your machine would be slower, then I will
> encourage you to try it out. You might be pleasantly surprised.
>
> If it is possible for you to use MSYS instead of Cygwin it would be
> interesting to see if that makes the problem go away.
>
>
> /Ole

Ole,

On my machine, I get 2 issues:

1. If allowing 50 core to be used - I get very slow response times for
anything else I do. Slw enough to be not workable.

2. I aslo get "Resource busy" & "fork failed" message. I think this is
because I am starting too many forked processes. Each jobs I start
might have 3 or 4 forks. I suspect cygwin/windows is running out of
handles or something