[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: file permissions on joblog

From: Ole Tange
Subject: Re: file permissions on joblog
Date: Sat, 30 Jul 2022 07:52:48 +0200

On Thu, Jul 28, 2022 at 5:45 PM Rob Sargent <> wrote:
> On 7/28/22 09:28, Christian Meesters wrote:
> On 7/28/22 14:56, Rob Sargent wrote:
> On Jul 28, 2022, at 1:10 AM, Christian Meesters <> wrote:
> Hi,
> not quite. Under SLURM the jobstep starter (SLURM lingo) is "srun". You do 
> not do ssh from job host to job host, but rather use "parallel" as a 
> semaphore avoiding over subscription of job steps with "srun". I summarized 
> this approach here:
>  (uh-oh - I need to clean up that site, many outdated sections there, but 
> this one should still be ok)
> One advantage: you can safely utilize the resources of both (or more) hosts - 
> the master hosts and all secondaries. How much resources you require depends 
> on your application and the work it does. Be sure to consider I/O (e.g. 
> stage-in file to avoid random I/O with too many concurrent applications, 
> etc.), if this is an issue for your application.
> Cheers
> Christian
> Christian,
> My use of GNU parallel does not include ssh. Rather I simply fill the slurm  
> node with —jobs=ncores
> That would require to have an interactive job and having 
> ncores_per_node/threads_per_application ssh-connections, and you have to 
> manually trigger the script. My solution is to use parallel in a SLURM-job 
> context and avoid the synchronization step by a human, whilst offering a 
> potential multi-node job with smp applications. It's your choice, of course.
> if I follow correctly that is what I am doing.  Here's my slurm job

Would this work:

chmod a+x $LOGDIR/*
touch "$logfile"
chmod -R a+rw "$LOGDIR"

. /uufs/

parallel \
    --joblog +"$joblog" \
     --verbose \
     --jobs 50% \
     --delay 1 \
83a9a2ad-fe16-4872-b629-b9ba70ed5bbb $endtime $JOBDIR ::: {1..750}

The idea is the same as my original: Make the file and set the
permissions before starting GNU Parallel.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]