Re: RAM memory consumption

[Ole Tange]

On Wed, May 10, 2017 at 8:08 AM, Samdani A <samdani1593@gmail.com> wrote:
> Dear all,
>
> I have been tried using parallel combining with another executable file to
> run jobs parallel. For example
>
> ls -1U | parallel -j 24 a.exe {}
>
> This work fine when there are 1500 files. But when I tried with 65,000
> files. After the process has started sometime later the RAM memory usage
> started to raise slowly and reaching out consuming all the RAM memory usage.
> Is there any solution to maintain the RAM memory usage consumption while
> running the process with GNU parallel. Kindly provide me your suggestions.

So I tested this:

    $ seq 65000 | time -v parallel -j24 true 2>&1 | grep 'Maximum resident'
        Maximum resident set size (kbytes): 16940
    $ seq 1500 | time -v parallel -j24 true 2>&1 | grep 'Maximum resident'
       Maximum resident set size (kbytes): 16884

This is the expected memory usage. So I cannot reproduce your error.
Please follow:

    LESS=+/REPORTING man parallel

       Report bugs to <bug-parallel@gnu.org> or
      https://savannah.gnu.org/bugs/?func=additem&group=parallel

      See a perfect bug report on
      https://lists.gnu.org/archive/html/bug-parallel/2015-01/msg00000.html

      Your bug report should always include:

      · The error message you get (if any).

      · The complete output of parallel --version. If you are
        not running the latest released version (see
        http://ftp.gnu.org/gnu/parallel/) you should specify
        why you believe the problem is not fixed in that
        version.

      · A minimal, complete, and verifiable example (See
        description on http://stackoverflow.com/help/mcve).

        It should be a complete example that others can run
        that shows the problem including all files needed to
        run the example. This should preferably be small and
        simple, so try to remove as many options as possible.
        A combination of yes, seq, cat, echo, and sleep can
        reproduce most errors. If your example requires large
        files, see if you can make them by something like seq
        1000000 > file or yes | head -n 10000000 > file.

        If your example requires remote execution, see if you
        can use localhost - maybe using another login.

      · The output of your example. If your problem is not
        easily reproduced by others, the output might help
        them figure out the problem.

      · Whether you have watched the intro videos
        (http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1),
        walked through the tutorial (man parallel_tutorial),
        and read the EXAMPLE section in the man page (man
        parallel - search for EXAMPLE:).

      If you suspect the error is dependent on your
      environment or distribution, please see if you can
      reproduce the error on one of these VirtualBox images:
      http://sourceforge.net/projects/virtualboximage/files/
      http://www.osboxes.org/virtualbox-images/

      Specifying the name of your distribution is not enough
      as you may have installed software that is not in the
      VirtualBox images.

       If you cannot reproduce the error on any of the
      VirtualBox images above, see if you can build a
      VirtualBox image on which you can reproduce the error.
      If not you should assume the debugging will be done
      through you. That will put more burden on you and it is
      extra important you give any information that help. In
      general the problem will be fixed faster and with less
      work for you if you can reproduce the error on a
      VirtualBox.

/Ole