[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: limits on stdout

From: Rob Sargent
Subject: Re: limits on stdout
Date: Thu, 27 Feb 2014 15:06:38 -0700
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0

Using parallel -u --results -j 3 <dir> myApp -a {}> /dev/null 2>&1 ::: a1 a2 a3 a4 appears to be working. All three stdouts are being written to equally. Thanks. (All three are still running so I haven't confirmed the redirect yet...)

It appears to me that one cannot use parameters in the redirect e.g "> path/{}.foo". Is that correct?

Ever grateful,

On 02/26/2014 07:44 PM, Ole Tange wrote:
On Wed, Feb 26, 2014 at 8:07 PM, Rob Sargent <> wrote:

I'm running GNU parallel 20131222 with --results outdir --jobs 4 and getting
4 processors running at capacity but only getting the stdout from two of the
jobs.  It looks as if the other two are buffering the output since they are
gaining memory while the other two jobs are at constant memory.  The stdout
for each will be ~2Gbytes.

Is this expected?
GNU Parallel should not increase memory usage: Buffering is done on
disk. But what you see is two jobs that choose for themselves to
buffer in memory, then GNU Parallel cannot force them to empty that
buffer. As part of the release procedure every version of GNU Parallel
is tested with output > 4GB, so GNU Parallel does not have a 2 GB

Can you reproduce the problem?

        ยท A complete example that others can run that shows the
          problem. This should preferably be small and simple. A
          combination of yes, seq, cat, echo, and sleep can
          reproduce most errors. If your example requires large
          files, see if you can make them by something like seq
          1000000 > file or yes | head -n 10000000 > file. If your
          example requires remote execution, see if you can use
          localhost - maybe using another login.

        If you suspect the error is dependent on your environment
        or distribution, please see if you can reproduce the error
        on one of these VirtualBox images:


reply via email to

[Prev in Thread] Current Thread [Next in Thread]