[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Too much memory being used with "find" and parallel

From: Ole Tange
Subject: Re: Too much memory being used with "find" and parallel
Date: Fri, 25 Jul 2014 14:11:38 +0200

On Thu, Jul 24, 2014 at 4:51 AM, Nelson A. de Oliveira <> wrote:

> find $TILEDIR -type f -name "*.png" | \
>         parallel --env optimize -j +0 --eta optimize
> But there is one problem: parallel starts to eat all my RAM and I just
> can't run it.

>From the man page:

       --eta    Show the estimated number of seconds before finishing. This
                forces GNU parallel to read all jobs before starting to find
                the number of jobs. GNU parallel normally only reads the
                next job to run.  Implies --progress.

So if you use --eta you need to have enough RAM to keep all jobs in
memory. --eta (and --bar but not --progress) forces GNU Parallel to
first read all jobs before executing them. Remove --eta and GNU
Parallel should eat around 20 MB RAM.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]