[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Out of memory when doing --pipe --block ?

From: Hubert Kowalski
Subject: Re: Out of memory when doing --pipe --block ?
Date: Fri, 9 Feb 2018 12:56:31 +0100 (CET)

If I may ask while we're at it: can we make some option to have GNU Parallel NOT buffer things and/or not output things? Some operations are heavy on I/O and not computationally intensive, so there's no need for either buffering or output from them :)

> Dnia 9 luty 2018 o 09:45 hubert depesz lubaczewski <address@hidden> napisał(a):
> On Thu, Feb 08, 2018 at 07:50:22PM +0100, Ole Tange wrote:
> > In practice this means you need 2.3 GB of free RAM and 3.8 GB of free
> > virtual memory (RAM+swap) to run this.
> >
> > Given your output from 'free' I think that if you add 2 GB of swap,
> > then that will solve your issue. Just because Perl loves swap, it does
> > not mean that it will start swapping - it just wants to feel safe that
> > it could if it wanted to. I have seen the same behaviour of other
> > programs, so my guess is that you might get more performance from your
> > system simply by adding a little swap space.
> >
> > The other option is to use --cat as described on
> >
> Yes, but --cat creates temp files, which I can't have as it slows down
> things considerably.
> Anyway - looks like I simply can't use parallel. It's memory
> requirements are (from my perspective) impossible to handle, and I still
> don't understand what exactly is the benefit of buffering stuff in
> memory.
> depesz

reply via email to

[Prev in Thread] Current Thread [Next in Thread]