parallel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

GNU Parallel as job queue processor


From: Ole Tange
Subject: GNU Parallel as job queue processor
Date: Tue, 5 Oct 2010 23:12:07 +0200

In connection with this
http://groups.google.com/group/comp.unix.shell/browse_thread/thread/b1eb90038930b58d/a1cbc5178fb54215
I started thinking: How can GNU Parallel be used as a job queue?

I have now added the following to the manpage. Please help me rephrase
it so it is easier to understand.

EXAMPLE: GNU Parallel as queue system/batch manager
       GNU Parallel can work as a simple job queue system or batch manager.
       The idea is to put the jobs into a file and have GNU Parallel read from
       that continuously. As GNU Parallel will stop at end of file we use tail
       to continue reading:

       echo >jobqueue; tail -f jobqueue | parallel

       To submit your jobs to the queue:

       echo my_command my_arg >> jobqueue

       You can of course use -S to distribute the jobs to remote computers:

       echo >jobqueue; tail -f jobqueue | parallel -S ..

EXAMPLE: GNU Parallel as dir processor
       If you have a dir in which users drop files that needs to be processed
       you can do this on GNU/Linux (If you know what inotifywait is called on
       other platforms file a bug report):

       inotifywait -q -m -r -e CLOSE_WRITE --format %w%f my_dir | parallel -u
       echo

       This will run the command echo on each file put into my_dir or
subdirs of my_dir.

       The -u is needed because of a small bug in GNU parallel. If that proves
       to be a problem, file a bug report.

       You can of course use -S to distribute the jobs to remote computers:

       inotifywait -q -m -r -e CLOSE_WRITE --format %w%f my_dir | parallel -S
       ..  -u echo

/Ole
PS: In case you did not notice: GNU Parallel is in issue 120 of Linux Magazine
http://www.linux-magazine.com/Issues/2010



reply via email to

[Prev in Thread] Current Thread [Next in Thread]