Re: Parallel a script

[Ole Tange]

On Sat, Jun 25, 2011 at 8:34 PM, yacob sen <yacob_123@yahoo.com> wrote:
>
> Dear Ole,
> Thank you so much. I am learning more and more from your responses.

Most of this is actually covered in the intro videos. Did you watch those?

http://www.youtube.com/watch?v=OpaiGYxkSuQ
http://www.youtube.com/watch?v=P40akGWJ_gY
http://www.youtube.com/watch?v=1ntxT-47VPA
http://www.youtube.com/watch?v=fOX1EyHkQwc

> I am not sure why "-N"  is not correct to use.

It was never designed for the situation you are in. If it works, it is
shear luck.

> The way I want the program should take the entries from files "name_list" and 
> "date_list" is as follows :
> say "name_list" has entries in raw form
> a00101.11
> a00102.11
> while   "date_list" has entries in raw form
> 2011-01-01
> 2011-01-02
>
> If I use
> parallel   prog -i  {1}  -type {2}  ::::  name_list   date_list
> the program will run 4 times like the for loop:
> parallel   prog -i a00101.11    -type 2011-01-01
> parallel   prog -i a00101.11    -type 2011-01-02
> parallel   prog -i a00102.11    -type 2011-01-01
> parallel   prog -i a00102.11    -type 2011-01-02
>
> But I want the program to run only twice,
> parallel   prog -i a00101.11    -type 2011-01-01
> parallel   prog -i a00102.11    -type 2011-01-02

Yep.

> So the -N option some how helps me to pick the above options. I will be happy 
> if there is another way to get the same result, i,e,
> parallel   prog -i a00101.11    -type 2011-01-01
> parallel   prog -i a00102.11    -type 2011-01-02

The supported way to do that is:

parallel --xapply  prog -i {1} -type {2} :::: name_list date_list

> The "prog" program does output to the same file (several files for that 
> matter) . But when I run it as you have suggested
> parallel 'mkdir {1}_{2}.dir; cd {1}_{2}.dir; prog -i ../{1} -type {2};
> mv file.out ../{1}_{2}.out; cd ..; rmdir {1}_{2}.dir'
> I got an error:
> parallel: Input is tty. Press CTRL-D to exit.
>
> Any idea why ?

You need to append the argument files. So the full line will be:

parallel --xapply 'mkdir {1}_{2}.dir; cd {1}_{2}.dir; prog -i ../{1}
-type {2}; mv file.out ../{1}_{2}.out; cd ..; rmdir {1}_{2}.dir' ::::
name_list date_list

Also you need to change file.out to the output file that prog creates.

If you want more help, sign up for the mailing list
http://lists.gnu.org/mailman/listinfo/parallel and explain in more
details what a single job looks like.

/Ole