parallel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Huge consumption of tmpdir while running parallel


From: Antoine Drochon (perso)
Subject: Huge consumption of tmpdir while running parallel
Date: Fri, 13 Jun 2014 17:13:43 -0700

Hi Parallel gurus,

I am running into an disk space issue when I run a parallel command (GNU 
parallel 20140322).

The pseudo code is as defined below:

---
echo 300k lines <pipe> parallel "script.bash {}"
--—

The Bash script perform a dig command, some pure Bash instructions and write a 
single line of 50 to 100 characters to stdout.

Nothing crash except the /tmp partition is reported to use several GB of data 
while the script is running.
I first tried --tmpdir to a larger partition, but after a couple of hours, 15 
GB were used.

I interrupted the execution and I assume Parallel trapped properly the signal 
to cleanup the temporary directory. I got back the 15 GB.

Note: I was unable to see any temporary file in the tmpdir directory.

Any idea what could cause such a big temporary buffer output usage?

Thanks,
Antoine


reply via email to

[Prev in Thread] Current Thread [Next in Thread]