parallel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Difference between single and double quotes for parallel?


From: Ole Tange
Subject: Re: Difference between single and double quotes for parallel?
Date: Thu, 16 Dec 2010 00:26:07 +0100

On Wed, Dec 15, 2010 at 11:25 AM, Maciej Pilichowski
<pilichowski.maciej@gmail.com> wrote:

>  In order to preserve the space within some argument, normally you would
> enclose it in double quotes.

Start by reading the man page on QUOTING.

>  However, when you use composed commands for parallel you have to put those
> in some quotes -- if your command has already double quotes, you cannot
> enclose the entire line with them. But if you use single quotes, the
> commands will be passed as-is, and I only guess parallel handles parsing
> such argument.

GNU Parallel is very liberal in quoting. You only need to quote
characters that have special meaning in shell:

( ) $ ` ' " < > ; | \

and depending on context these needs to be quoted, too:

* ~ & # ! ? space * {

When you want to use a shell variable you need to quote the $-sign.
Here is an example using $PARALLEL_SEQ. This variable is set by GNU
Parallel itself, so it the evaluation of the $ must be done by the sub
shell started by GNU Parallel:

seq 1 10 | parallel -N2 echo seq:\$PARALLEL_SEQ arg1:{1} arg2:{2}

If the variable is set before GNU Parallel starts you can do this:

$ VAR=this_is_set_before_starting
$ echo test | parallel echo {} $VAR
Prints: test this_is_set_before_starting

If the variable should not be evaluated by the shell starting GNU
Parallel but be evaluated by the sub shell started by GNU Parallel,
then you need to quote it:

$ echo test | parallel VAR=this_is_set_after_starting \; echo {} \$VAR
Prints: test this_is_set_after_starting

$$ is the shell variable containing the process id of the shell. This
will print the process id of the shell running GNU Parallel:

seq 1 10 | parallel echo $$

And this will print the process ids of the sub shells started by GNU Parallel.

seq 1 10 | parallel echo \$\$

If the special characters should not be evaluated by the sub shell
then you need to protect it against evaluation from both the shell
starting GNU Parallel and the sub shell:

$ echo test | parallel echo {} \\\$VAR
Prints: test $VAR

GNU Parallel can protect against evaluation by the sub shell by using -q:

$ echo test | parallel -q echo {} \$VAR
Prints: test $VAR

This is particularly useful if you have lots of quoting. If you want
to run a perl script like this:

perl -ne '/^\S+\s+\S+$/ and print $ARGV,"\n"' file

It needs to be quoted like this:

ls | parallel  perl -ne '/^\\S+\\s+\\S+\$/\ and\ print\ \$ARGV,\"\\n\"'

Notice how spaces, \'s, "'s, and $'s need to be quoted. GNU parallel
can do the quoting by using option -q:

ls | parallel -q  perl -ne '/^\S+\s+\S+$/ and print $ARGV,"\n"'

However, this means you cannot make the sub shell interpret special
characters. For example this WILL NOT WORK:

       ls *.gz | parallel -q "zcat {} >{.}"

       ls *.gz | parallel -q "zcat {} | bzip2 >{.}.bz2"

because > and | need to be interpreted by the sub shell.

>  Ok, I tried single quotes and some of my variables were not resolved, let's
> say:
>
> FN=something
> '.... "${FN}" .... '

Please provide a full example and explanation of what it does and what
you expected it to do. It is unclear what you mean by the above. This
works as expected:

$ FN=a
$ echo 1 | parallel echo $FN
# prints a 1

$ echo 1 | parallel FOO=\$\$ \; echo \$FOO
# prints the processid of the subshell followed by 1

This shows the difference between getting the shell variable evaluated
by the shell calling GNU Parallel and getting it evaluated by the
subshell started by GNU Parallel:

seq 1 10 | parallel echo $$
seq 1 10 | parallel echo \$\$


/Ole



reply via email to

[Prev in Thread] Current Thread [Next in Thread]