[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
GNU Parallel as job queue processor
From: |
Ole Tange |
Subject: |
GNU Parallel as job queue processor |
Date: |
Tue, 5 Oct 2010 23:12:07 +0200 |
In connection with this
http://groups.google.com/group/comp.unix.shell/browse_thread/thread/b1eb90038930b58d/a1cbc5178fb54215
I started thinking: How can GNU Parallel be used as a job queue?
I have now added the following to the manpage. Please help me rephrase
it so it is easier to understand.
EXAMPLE: GNU Parallel as queue system/batch manager
GNU Parallel can work as a simple job queue system or batch manager.
The idea is to put the jobs into a file and have GNU Parallel read from
that continuously. As GNU Parallel will stop at end of file we use tail
to continue reading:
echo >jobqueue; tail -f jobqueue | parallel
To submit your jobs to the queue:
echo my_command my_arg >> jobqueue
You can of course use -S to distribute the jobs to remote computers:
echo >jobqueue; tail -f jobqueue | parallel -S ..
EXAMPLE: GNU Parallel as dir processor
If you have a dir in which users drop files that needs to be processed
you can do this on GNU/Linux (If you know what inotifywait is called on
other platforms file a bug report):
inotifywait -q -m -r -e CLOSE_WRITE --format %w%f my_dir | parallel -u
echo
This will run the command echo on each file put into my_dir or
subdirs of my_dir.
The -u is needed because of a small bug in GNU parallel. If that proves
to be a problem, file a bug report.
You can of course use -S to distribute the jobs to remote computers:
inotifywait -q -m -r -e CLOSE_WRITE --format %w%f my_dir | parallel -S
.. -u echo
/Ole
PS: In case you did not notice: GNU Parallel is in issue 120 of Linux Magazine
http://www.linux-magazine.com/Issues/2010
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- GNU Parallel as job queue processor,
Ole Tange <=