Re: [Help-bash] Commands run via xargs and bash returns different results

[Bob Proulx]

Richard Taubo wrote:
> Bob Proulx wrote:
> > Richard Taubo wrote:
> >> I am running the following command via xargs and bash with no problems;
> >>    /usr/bin/mdfind text -0 -onlyin /This/Path | /usr/bin/xargs -0 -I {} 
> >> bash -c 'echo -e "{}";
> > 
> > It is better if you can create a small test case using standard
> > utilities like 'find' or echo/printf/seq or bash so that the rest of
> > us can follow along.  Being a free(dom) software list most of us don't
> > have Apple Mac computers but are using GNU system software.  So we
> > don't have mdfind on most of our systems.  I'm just say'in...
> 
> Yes, the test case was not the best, sorry about that.
> mdfind should be available via Darwin and Darwin derivatives,
> so I guess the freedom part could be questioned, but I get
> your point :-)

<chuckle/>

When I searched the web for mdfind I found people asking if there was
a free(dom) version of mdfind and the answer I found there was no that
it was unique to Mac OS X.  That makes it impossible to run on a
free(dom) software system.  (I always try to be clear that free
software is about freedom and not about no cost.)  Since this is a GNU
list after all it would be nice if the examples were available to run
on such a system. :-)  I would do the same in reverse if I were asking
on an Apple list.

> > Hmm...  Why are you using a full hard coded path to /usr/bin?  Why not
> > just "mdfind" and "xargs"?  Is /usr/bin not in your PATH?  If not then
> > it should be.  Hard coded paths are Evil.
> 
> Do you mean Evil as in “Eval is evil” or due transportability issues?

Evil due to portability issues.  Two of the worse offenders are grep
and basename.  Different systems put them in different directories.
Some in /bin and some in /usr/bin.  But the same could apply to
needing a locally compiled version from /usr/local/bin of some a
program such as to acquire a newer version of it than offered by the
system.  Or maybe it is one of the systems that uses /opt/foo/bin for
various software (possibly with "foo" being "gnu").

Hard coding paths causes software to be very rigid.  Basically it says
that it was written for exactly one system and should never work on
any other system.  And often that means not on future versions of the
same system as things change from version to version.  Rigid software
is fragile software.

And usually when someone capitalizes something like Evil you should
think of a voice like Vincent Price's saying it and then laughing
maniacally.  (mwahahaha)

> By setting: my_path="{}” I was trying receive each of the paths
> handed over from mdfind via xargs, and then use this variable to
> create html list based output, e.g something like:  
>       "<li>$my_path</li>”
> (it was a little more complicated than that, e.g. splitting based on
> forward-slashes, conditional expressions etc, but the output was 
> meant for html list output).

Ah...  So you were (probably) wanting the script to emit the same
string multiple times as in doing a template generator or some such
and that was beyond the string substitution features of xargs.
Gotcha!

> Based on your description above, I understand there is a
> correct/safer/better way to accomplish this, but you would humour me
> if you would point out the "bizzarity" part of the command. Clunky
> and error-prone, yes, but what makes it bizarre? I had to somehow
> grab the output from mdfind, and it seemed to me at the time, using
> this clunky method, that I could only grab the output using "{}"
> ...  I am sure I am wrong :-)

Sorry if I had originally come off as caustic.  I am trying to soften
it up with the tone in this message.

I found it bizarre because it wasn't using xargs in the main target
design of what xargs would be used for.  By design xargs would be used
to supply a command with arguments.  Normally you would have a
program, utility, or script and it would be invoked with arguments.
But here you were trying to inject arguments into the middle of a
script.  I didn't like that because it runs into trouble with quoting.

  xargs -0 -I {} bash -c 'my_path="{}";echo -e "$my_path";'

The part of concern is the my_path="{}" part.  I see that you have
single quotes around the large argument and therefore are using double
quotes for the variable assignment.  But xargs is going to do the
replacement into the variable.  Whereas before you had protected it
with -0 for null terminated strings now the variable will contain a
lot of file names all smashed together without that protection.
Basically completely throwing away the -0 protection entirely.  And
now you will need to parse the filenames again.

You can see it by using echo and some synthetic input.

  $ printf "%s\0" one two three | xargs -0 -I {} echo bash -c 
'my_path="{}";echo -e "$my_path";'
  bash -c my_path="one";echo -e "$my_path";
  bash -c my_path="two";echo -e "$my_path";
  bash -c my_path="three";echo -e "$my_path";

Okay.  Now let's mess with it a little bit.

  $ printf "%s\0" '"one"' "\"two" '$(date -R)' | xargs -0 -I {} echo bash -c 
'my_path="{}";echo -e "$my_path";'
  bash -c my_path=""one"";echo -e "$my_path";
  bash -c my_path=""two";echo -e "$my_path";
  bash -c my_path="$(date -R)";echo -e "$my_path";

Now these are contrived but should be a little scary.  Now that things
echoed out plainly they show the problems.  Or maybe features because
I would be the type who would use the $(...) as a feature.  But in any
case it probably isn't as expected.  What if there were a ';' injected
in there?  Or parens?  I considered that bizarre because it was
injecting it straight into a scriptlet.  Let's run it.

  $ printf "%s\0" '"one"' "\"two" '$(date -R)' | xargs -0 -I {} bash -c 
'my_path="{}";echo -e "$my_path";'
  one
  bash: -c: line 0: unexpected EOF while looking for matching `"'
  bash: -c: line 1: syntax error: unexpected end of file
  Mon, 03 Feb 2014 17:04:41 -0700

Better would be to create a script.  I will do something too short to
be correct but small for email just for the idea of it.

  #!/bin/sh
  test $# -gt 0 || { echo "Error" ; exit 1 ;}
  file=$1
  printf "%s\n" "$file"
  exit 0

Then calling it with xargs avoids the problems.  Because now the
arguments are all sufficiently quoted.

  $ printf "%s\0" '"one"' "\"two" '$(date -R)' | xargs -0 -L1 /tmp/trial
  "one"
  "two
  $(date -R)

Of course you will need to "safe" arguments for emitting into html to
avoid html entities.  But now that you have the script that is much
easier to do than trying to put the entire thing on the command line.

And btw the "/tmp/trial" part is just for the testing.  I wouldn't leave
that in place for production.  Hard coded path and all of that.  :-)

I used -L1 so I wouldn't need to loop in the shell script to keep it
small.  That is perhaps fine for a text generator.  If the script is
large and if passing in several hundred thousand arguments then it
will be slow to call it several hundred thousand times.  In which case
it justifies a loop in the script to process all of the arguments in
one invocation.  But, "Premature optimization is the root of all
evil." -- Donald Knuth.  Take performance measurements first.  But I
think it would be fine to put a loop in the script just for the
convenience of it.

For this rather than use xargs at all I would simply use a script as I
had previously suggested.  It just feels better.  But using xargs will
also work fine too.  It all depends upon the circumstance.

I will also say that for the most part xargs is now obsolete.  The
xargs command was created before the find "{} +" syntax was created.
Without it there was only "{} \;" which ran one argument per
invocation.  Not very efficient with a large number of arguments.
But now that "{} +" has been added and propagated there is little
reason not to use it instead of xargs.  However I know you were using
mdfind and I have no idea if mdfind has such capability.  I assume not
or you wouldn't have been using xargs.  Which is why it is only "for
the most part" and not completely obsolete.  I still use xargs out of
convenience.

Hope this discussion was useful/interesting,
Bob