[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Is there any advantage to turning the following rsync cmd into a paralle

From: Alex Muir
Subject: Is there any advantage to turning the following rsync cmd into a parallel cmd?
Date: Thu, 19 Jul 2012 15:33:55 -0400


I've created a nice script with parallel and then I use rsync to
return output files which are various files, html and zip in several
different directories. My client is wondering if the rsync aspect at
the end can be done with parallel and if there would be an benefit to
doing that and insisted I ask about it.

Looking at the documentation I saw the one can --return files that
were used as input files as well as create another command to return
files generally. It was not clear to me though if one could return
directories recursively as I do with the below rsync commands.

I launch the script with the following command in these case it runs
up to 1000 listed files on three servers
./ SEC/10k-GHU/2004/,, 1000

ls /mnt/xslt_volume/i4ContentSource/$1*.zip | head -$3 |  parallel
-j-0 --eta --progress --sshlogin $2 --timeout 1500 --transfer --joblog
/mnt/jobs.txt "sh /mnt/config/
/mnt/config/xproc.xpl /mnt/{/.}Log.txt {}" /mnt/output/$1

#I wonder can the below which loops through N CSV severs rsyncing data
from each server be converted into a few parallel statements and if
there is any advantage to doing that?


for v in $var
do if test $v != ':'; then
      rsync -r -v -e ssh $v:/mnt/result/$1 /mnt/result/$1;
      rsync -r -v -e ssh $v:/mnt/logs/ /mnt/logs/;



Alex G. Muir
Software Engineering Consultant
Linkedin Profile :

reply via email to

[Prev in Thread] Current Thread [Next in Thread]