Re: Parallel script with variable name

[Ernst, Kevin]

Hi Lindsey,

If I’m understanding you correctly, then {} (a pair of curly braces) is what you want. It’s the “variable” that holds the names of the things coming in on parallel‘s standard input, and it gets a different value, one for each input line, for each invocation of lots_of_stuff.

The “parallel way” would look like this:

find *filepattern* | parallel lots_of_stuff

which implicitly means

find *filepattern* | parallel lots_of_stuff {}

which can be simplified to:

parallel lots_of_stuff ::: *filepattern*

…since the shell will expand the wildcards—also called “globs”—for you. This also saves a process (because there is no pipe), which is not a super-huge deal in this simple example, but it’s, like, points for style. A complete reference for how this “filename expansion” happens is here.

It most cases, it’s fine, but in general it’s more conservative to let the shell itself do wildcard expansion, or to use find rather than ls, as discussed here (tl;dr: “word splitting”). Parsing the output of ls can be problematic… for reasons you are sure to run into someday, if you haven’t already.

If you haven’t checked them out already, the videos the author posted to YouTube are really excellent, and not too long. You can get lots of insight about GNU Parallel (from basic use cases that you can employ immediately, all the way up to some mind-bending possibilities) with just about 30 minutes of your time.

If you ever lose the URL to the YouTube playlist, it’s also mentioned in the man page.

Hope this helps.

—Kevin

​