parallel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: GNU parallel - resumable jobs


From: rambach
Subject: Re: GNU parallel - resumable jobs
Date: Mon, 09 Jan 2012 17:47:18 +0100
User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7

On 1/7/2012 4:31 AM, Ole Tange wrote:
On Fri, Dec 16, 2011 at 12:45 PM, Ole Tange<address@hidden>  wrote:
On Fri, Dec 16, 2011 at 9:01 AM, rambach<address@hidden>  wrote:
On 12/15/2011 11:35 PM, Ole Tange wrote:
On Wed, Dec 14, 2011 at 2:35 PM, rambach<address@hidden>    wrote:
On 12/12/2011 11:07 PM, Ole Tange wrote:
I am thinking of re-using --joblog and adding --resume. I see at least
2 approaches:

* Only look for the job-number.
This is now implemented. You can do:

   timeout -k 1 1 parallel -j2 --resume --joblog /tmp/joblog2 sleep {}
::: 1.1 2.2 3.3 4.4
   parallel -j2 --resume --joblog /tmp/joblog2 sleep {} ::: 1.1 2.2 3.3 4.4;

Please test it.

/Ole
thanks, very good job.
the functionality works nice and smooth.

i'm sure others will benefit from this feature as well.

however, what i found during testing is that GNU Parallel has some sort of memleak:
the following command
seq 100000 | parallel -j200 "echo {}; sleep 1"
starts with a virtual mem usage of about 38 MB, and reaches 50 MB at around 25000 finished jobs. the size of used memory increases steadily, so at 12MB per 25000 jobs, you'd run out of mem on a 128 MB sys pretty quick.
the leak is independent of the --resume option and even --joblog.

best regards,
roland rambach





reply via email to

[Prev in Thread] Current Thread [Next in Thread]