[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Huge consumption of tmpdir while running parallel
From: |
Antoine Drochon (perso) |
Subject: |
Huge consumption of tmpdir while running parallel |
Date: |
Fri, 13 Jun 2014 17:13:43 -0700 |
Hi Parallel gurus,
I am running into an disk space issue when I run a parallel command (GNU
parallel 20140322).
The pseudo code is as defined below:
---
echo 300k lines <pipe> parallel "script.bash {}"
--—
The Bash script perform a dig command, some pure Bash instructions and write a
single line of 50 to 100 characters to stdout.
Nothing crash except the /tmp partition is reported to use several GB of data
while the script is running.
I first tried --tmpdir to a larger partition, but after a couple of hours, 15
GB were used.
I interrupted the execution and I assume Parallel trapped properly the signal
to cleanup the temporary directory. I got back the 15 GB.
Note: I was unable to see any temporary file in the tmpdir directory.
Any idea what could cause such a big temporary buffer output usage?
Thanks,
Antoine
- Huge consumption of tmpdir while running parallel,
Antoine Drochon (perso) <=