Re: Too much memory being used with "find" and parallel

[Ole Tange]

On Thu, Jul 24, 2014 at 4:51 AM, Nelson A. de Oliveira <naoliv@gmail.com> wrote:

> find $TILEDIR -type f -name "*.png" | \
>         parallel --env optimize -j +0 --eta optimize
>
> But there is one problem: parallel starts to eat all my RAM and I just
> can't run it.

>From the man page:

       --eta    Show the estimated number of seconds before finishing. This
                forces GNU parallel to read all jobs before starting to find
                the number of jobs. GNU parallel normally only reads the
                next job to run.  Implies --progress.

So if you use --eta you need to have enough RAM to keep all jobs in
memory. --eta (and --bar but not --progress) forces GNU Parallel to
first read all jobs before executing them. Remove --eta and GNU
Parallel should eat around 20 MB RAM.


/Ole