Re: -X doesn't quite work as documented or perhaps expected

[Ole Tange]

Thanks for your testing. It makes the tool more stable.

On Wed, Jun 22, 2011 at 9:37 PM, Jay Hacker <jayqhacker@gmail.com> wrote:
> Hello,
>
> I'm using the new version 20110622.  I'd like to run one job, with all
> arguments appended, and context replace.  I thought that's what -X
> would do, from the manual:
>
> -X
>    Multiple arguments with context replace. Insert as many arguments
> as the command line length permits.
>
> However, what parallel actually seems to do is run one job for each
> CPU core, and try to spread the arguments evenly.  On a machine with
> four cores:

That is correct. The documentation lacks behind here. It used to be
the case, but the documentation should clearly be updated.

Will you create a bug-report or should I?

> $ seq 1 5 | parallel -X echo file{}.jpg
> file1.jpg file2.jpg
> file3.jpg file4.jpg
> file5.jpg
>
> $ seq 1 5 | parallel -j2 -X echo file{}.jpg
> file1.jpg file2.jpg file3.jpg
> file4.jpg file5.jpg
>
> This is arguably the right thing to do for a parallel processing tool,
> but if that's the intent it would be nice to have it documented.  -m
> and --xargs work similarly; however, this seems misleading if you're
> trying to reproduce xargs' behavior exactly.

Maybe --xargs should do the "fill the line" where as -m/-X do the
"distribute args". Do people have any opinions on that?

> A few other surprises: The manual says -L and -N "implies -X unless -m
> is set," but they do not do context replace as -X does:
>
> $ seq 1 5 | parallel -L10 echo file{}.jpg
> file1 2 3 4 5.jpg
>
> $ seq 1 5 | parallel -N10 echo file{}.jpg
> file1 2 3 4 5.jpg

Clearly a bug. Thanks. Will you create a bug-report or should I?

> It would be nice to have context replace with -L and -N.  Given these,
> I would expect -s to operate similarly, but I'm not sure what -s is
> doing:

-s without reading multiple args make very little sense. Combine it
with -X/-m/-L/-l/-n.

> And as a symptom of this, -x seems to be the default; that is, -s will
> always exit if the limit is exceeded:
>
> $ seq 1 5 | parallel -s1 echo
> Command line too long (6 >= 1) at number 1: 1...

In this case it does the same as xargs:

$ seq 1 5 | xargs -s1 echo
xargs: can not fit single argument within argument list size limit

> Finally, if you forget the argument to -s or -N or -L, it would be
> nice to get a message indicating this, instead of crashing Perl:
>
> $ seq 1 5 | parallel -s echo
> Can't exec "-s": No such file or directory at
> /usr/lib/perl5/5.8.8/IPC/Open3.pm line 246.
> open3: exec of -s echo 1 failed at /usr/bin/parallel line 3182

I use the GetOptions for getting the options. And that function will
simply leave -s hanging if it is not followed by a valid string.

Would it make sense to give a warning if you try to execute a command
starting with '-'? What do others say? Should it even be an error?

I try to make GNU Parallel work sensible even in extreme situations.
And I must admit that I have never had a valid UNIX command whose name
started with '-'. But what if people actually _had_ a command starting
with '-'? Should we then just give an error saying: "Your are trying
to run the command '-XXX'. If that is really what you want, give full
path to the command.'

> So, I think I can solve my original problem with -Xj1,

That is a fairly common construct, so it sounds plausible that it can
solve your problem.

> but seem to
> have discovered a few others along the way.  Let me know if any of
> these should go in a bug report.

Done above. The last issue needs discussion on the mailing list before
I will act on it.

/Ole