Re: Problem with thousands of small jobs.

[xmoon 2000]

Do you think the arguments to the parallel command are OK?

On 6 February 2015 at 14:58, Felipe Alvarez <felipe.alvarez@gmail.com> wrote:
> echo $SHELL
>
> Bash is default in cygwin
>
>
> On Sat, 7 Feb 2015 00:56 xmoon 2000 <xmoon2000@googlemail.com> wrote:
>>
>> I am running this in cygwin. I call parallel from the standard
>> window/shell supplied by cygwin.
>>
>> What makes you think it is bash? (You may well be right)
>>
>> On 6 February 2015 at 14:43, Felipe Alvarez <felipe.alvarez@gmail.com>
>> wrote:
>> > Is there a"lighter" shell that you can use other than bash?
>> >
>> >
>> > On Fri, 6 Feb 2015 23:20 xmoon 2000 <xmoon2000@googlemail.com> wrote:
>> >>
>> >> Hi,
>> >>
>> >> I need to run about 4,000 jobs that each take around 20 seconds to
>> >> complete
>> >>
>> >> using:
>> >>
>> >>  cat /tmp/parList | parallel -j 28 --eta;
>> >>
>> >> On my 32 core machine works OK,   BUT there is a "lull" in processing
>> >> every few seconds as new jobs are started, once the current crop have
>> >> completed. I assume this is due to an overhead in starting jobs that
>> >> is only noticable because my jobs are so short.
>> >>
>> >> Is there any way I could make this more efficient, so my cores are
>> >> fully utilised and getting through the whole process is faster?
>> >>
>> >> Moon
>> >>
>> >