Re: Enhancement request for tee - please add the option to not quit on SIGPIPE when someother files are still opened

[Jirka Hladky]

Hi,

thanks for the script, it definitely provides more control than the plain tee command. I have done some modifications so that progress can be watched live 

I have tried two versions of the main command (one with >/dev/full, other one with >&-). 

cat /dev/zero | head -c500M | (/dev/shm/AAA/coreutils-8.24/src/tee -p $d/fifo1 $d/fifo2 $d/fifo3 $d/fifo4 >/dev/full ) 2>&1 | tee $d/run.log &

and

cat /dev/zero | head -c500M | (/dev/shm/AAA/coreutils-8.24/src/tee -p $d/fifo1 $d/fifo2 $d/fifo3 $d/fifo4 >&- ) 2>&1 | tee $d/run.log &

Both of them are working fine, except that following messages are emitted:

tee: standard output: No space left on device

tee: standard output: Bad file descriptor

I still think that "--no-stdout" option would be helpful for this usecase. Any feedback on that? Should I open the enhancement request ticket and post implementation of "--no-stdout" option there?

Thanks

Jirka

PS: Modified script is attached. 

On Fri, Nov 20, 2015 at 12:02 PM, Pádraig Brady <address@hidden> wrote:

On 20/11/15 04:33, Assaf Gordon wrote:
> Hello Jirka,
>
> Regarding this:
>
> On 11/19/2015 08:58 PM, Jirka Hladky wrote:
>>> The general problem I have with >(process substitutions) are that
>>> they are completely asynchronous.  There is no way to tell if they
>>> are done.
>>
>> Yes, I agree with you on this one. However, I don't see the other way
>> how to send the output of one process to multiple sub-processes in
>> shell.
>
> If I may suggest this slightly verbose shell script (attached) - it should do what you want (sending output to multiple processes)
> while still allowing tight control over each background process, and also collecting their results in an organized fashion
> (ie keeping stdout,stderr,exitcode in a file for each test) - making further diagnosis much easier.
>
> if there's a need to combine the outputs from all the tests (e.g. to find the smallest p-value from all tests) -  it's just a matter of "cat *.out" once
> all the tests are done.
>
> Note that this does not solve the "--no-stdout" issue - just the ">()" part. It should also make the shell script portable (except using GNU tee's "-p" parameter).

Note there is no async issue with >() once the output is piped further,
as then the background processes are waited for.
Though yes, using fifos give more fine grained control over processes and exit status etc.

> The output should be:
>
>      tee: standard output: Bad file descriptor
>      == Test 1 exited with code 0 ==
>      == Test 1 STDOUT ==
>      104857600
>      == Test 2 exited with code 0 ==
>      == Test 2 STDOUT ==
>      1
>      == Test 3 exited with code 1 ==
>      == Test 3 STDOUT ==
>      == Test 3 STDERR ==
>      wc: unrecognized option '--foo'
>      Try 'wc --help' for more information.
>      == Test 4 exited with code 0 ==
>      == Test 4 STDOUT ==
>      32768
>      ==
>      Test results stored in /tmp/tmp.esLAoUxeLQ
>
>
> Comments and corrections welcomed.

Yes this is a useful pattern.
I noted something similar for use with split(1) at:
http://lists.gnu.org/archive/html/coreutils/2011-05/msg00012.html
with the number of parallel processes potentially determined with nproc(1).

Minor comments on the script. I'd proably `rm -f fifo*` before creating them
to allow clean rerun after Ctrl-C. Also the eval can be simplified to:
  eval TEST_PID=\$TEST${i}_PID

cheers,
Pádraig.

5.1.sh
Description: Bourne shell script