Re: What to do if $TMPDIR runs full?

[Ole Tange]

On Mon, Dec 2, 2013 at 12:53 PM, Andrew Clegg
<andrew.clegg+parallel@gmail.com> wrote:
> On 2 December 2013 11:34, Ole Tange <ole@tange.dk> wrote:
>> On Mon, Dec 2, 2013 at 12:01 PM, Andrew Clegg
>> <andrew.clegg+parallel@gmail.com> wrote:
>>>
>>> Warning: 3 jobs returned non-zero exit status
>>> zcat 2013-01-01-01.gz returned 1
>>> zcat 2013-01-01-02.gz returned 1
>>> zcat 2013-01-01-03.gz returned 1
>>
>> That I do not like. If the user is interested in the number of failing
>> jobs, he should look at $? which contains this information. See
>> http://www.gnu.org/software/parallel/man.html#exit_status
>
> Fair enough. You might consider applying the -v and -t options to the
> job output as well as the job input, so the user could see not only
> how many jobs failed, but also which ones, and with what return codes.
>
> I think that would be in the spirit of "verbose".

I like that much better. I know that people use the output from -v for
further processing, so I imagine the error could be printed on stderr.

<-v on stdout>
<stdout from job>
<stderr from job>
parallel: warning: Job failed (status: X): <-v output>

Example (the 2 last lines are printed on stderr):

$ parallel -v ls existing_file ::: non-existing_file
ls existing_file non-existing_file
existing_file
ls: cannot access non-existing_file: No such file or directory
parallel: Warning: Job failed (status: 1): ls existing_file non-existing_file

I am more reluctant to changing the -t behavior as I would like to
keep that compatible with xargs' -t.

> By the way -- I tested using compress, with my original temp dir which
> was too small for the output, and it failed, but not with the error I
> was expecting.
>
> [davteam@lo3uppaldbs001 parallel_bug_data]$ ls *.gz |
> /home/aclegan/parallel_bug/parallel/src/parallel --gnu -j 4 --compress
> zcat > test4.txt
> <citation message removed>
> gzip: stdin: not in gzip format

When using compression I cannot do the
append-8kb-to-test-for-disk-full trick. So the behaviour of that is
unchanged.

I believe I could open a new file in $TMPDIR, write 8kb to that, and
see if that fails. The time that will cost will be minimal compared to
spawning the compress program, and it will only cost a single file
handle, so all in all that ought to be doable.


/Ole