[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Persistent queue system with remote execution using parallel?
From: |
Åsmund Skjæveland |
Subject: |
Persistent queue system with remote execution using parallel? |
Date: |
Thu, 31 May 2018 09:16:49 +0000 |
Hello,
Is it possible to use GNU Parallel to implement a persistent queue system with
remote execution?
My setup is: two compute servers, identical configuration, no shared network
drive. The job in question downloads its input data over FTP, and --return is
used for the processed data.
Using 'parallel -S 4/server1 -S 4/server2 command' works fine as long as I only
have one set of tasks at a time. However I'd like to run both hourly jobs and
daily jobs.
If I just run them as two separate simultaneous Parallel jobs the compute
servers will run out of memory and swap themselves to a snail's pace. So I'd
like to run both sets of jobs in a shared queue.
Will a named semaphore do what I want, if I use the same semaphore name for
both ses of jobs?
Will it use the correct number of CPUs (4) on each server if I specify '-S
4/server1 -S 4/server2', or do I also need to specify --jobs?
The hourly job takes around 15-20 minutes to complete. Ideally I'd want the
hourly job to have higher priority so that the daily jobs are only running when
there are no hourly jobs queued, but it seems this is not possible with
Parallel.
Åsmund Steen Skjæveland, PhD
Senior Engineer
Norwegian Mapping Authority, Geodetic Insititute
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- Persistent queue system with remote execution using parallel?,
Åsmund Skjæveland <=