[bug #29512] [feature request] xargs parallel processing env-val improvement

[anonymous]

URL:
  <http://savannah.gnu.org/bugs/?29512>

                 Summary: [feature request] xargs parallel processing env-val
improvement
                 Project: findutils
            Submitted by: None
            Submitted on: Sat Apr 10 15:45:43 2010
                Category: xargs
                Severity: 3 - Normal
              Item Group: None
                  Status: None
                 Privacy: Public
             Assigned to: None
         Originator Name: Tomas Mudrunka
        Originator Email: address@hidden
             Open/Closed: Open
         Discussion Lock: Any
                 Release: None
           Fixed Release: None

    _______________________________________________________

Details:

Hello.
It would be usefull to have ability to launch xargs -P with different
envinronment value for each N processes. (eg. 2 with blank, 4 with something
in this env and 4 more with something different).

Nowadays i can run some command parralely on multiple CPU cores like this:

cpus=2;
command | xargs -n 1 -P "$cpus" some_command --

but i am thinking about processing on whole cluster of computers with sshds,
imagine that xxd command is VERY VERY CPU intensive and i will do something
like this:


0 ;) address@hidden Shared $ DPREFIX=''
0 ;) address@hidden Shared $ echo lol | $DPREFIX xxd
0000000: 6c6f 6c0a                                lol.

0 ;) address@hidden Shared $ DPREFIX='ssh harvie.cz'
0 ;) address@hidden Shared $ echo lol | $DPREFIX xxd
Host key fingerprint is 41:72:28:3d:f5:f5:d3:a2:0b:f6:e5:c7:a2:c4:b0:d3

0000000: 6c6f 6c0a                                lol.

so i can simply trigger if the process will be launched localy, or remotely.
now imagine bit more complex example:


#!/bin/bash
cpus='2;4=ssh node1.harvie.cz;4=ssh node2.harvie.cz';
#now xargs will run 10 processes total.
#2 with blank DPREFIX env value
#+4 with DPREFIX='ssh node1.harvie.cz'
#+4 with DPREFIX='ssh node2.harvie.cz'
#you need to have public key copied on those nodes or something...

process() {
        cat $1 | $DPREFIX xxd > $2
        #handle failed ssh connection here
}
export -f process;
command | xargs -0 -n 1 -P "$cpus" bash -c 'process "$@"' --


So you will be able to run xargs on top of network cluster... Maybe this does
not fit all of possible situations, but i think sometimes it can be really
usefull.

this is just IMHO most cool of them, but there are many other ways how to
make use of such feature...
eg.: you can use 4 cores, but run with bigger nice value on one of them, so
there will be some CPU power left if you will want to work on something else.




    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?29512>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/