Re: Pipe status codes

[Ole Tange]

On Thu, Mar 29, 2012 at 9:23 PM, David Erickson <halcyon1981@gmail.com> wrote:
> Hi there-
> I just discovered parallel and it looks great.

Good to hear. How did you discover it? What could I have done to make
you discover it earlier?

> One existing use case
> I have for commands I would like to run is something like:
>
> parallel 'cmd1 {} 2>&1 | tee somewhere.log'
>
> I have a couple questions regarding this use case:
>
> 1) Will parallel correctly honor my request to redirect stderr to stdout?

This is easy for you to test, so I will leave that for you as an exercise.

> 2) Is there some way to use --halt with the error code returned from
> cmd1?  IE in bash I can do this with ${PIPESTATUS[0]}, because tee
> will always return 0.

GNU Parallel looks at the exit code, so you can do:

parallel 'cmd1 {} 2>&1 | tee somewhere.log; exit ${PIPESTATUS[0]}'

You can combine this with --halt if you need GNU Parallel not to spawn
more jobs. If you just need to know how many jobs failed, the default
(--halt 0) will work for you.

> Alternatively, can parallel log both stdout and
> stderr for each command to a unique file with naming of my choice?

You probably want one of these:

  ... | parallel cmd1 {} > all_stdout 2> all_stderr
  ... | parallel --tag cmd1 {} > all_stdout 2> all_stderr
  ... | parallel cmd1 {} \> {}_stdout 2\> {}_stderr
  ... | parallel cmd1 {} \> {#}_stdout 2\> {#}_stderr

But more creative ways exists. If every 3rd argument is the
stdout-name and the next is the stderr-name:

  ... | parallel -N3 cmd1 {1} \> {2} 2\> {3}

If you have a file with the names of the stdout-names and another file
with the stderr-filenames:

  ... | parallel --xapply cmd1 {1} \> {2} 2\> {3} :::: -
stdoutfilenames stderrfilenames

Also you may find --joblog useful.


/Ole