[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: how can I not have to spend 40 minutes _waiting_ in themorningfor ea

From: eric.berg
Subject: RE: how can I not have to spend 40 minutes _waiting_ in themorningfor each of 40 updates?
Date: Fri, 10 Oct 2008 10:13:26 -0400

Hi, Chaitanya,
If the primary goal of this exercize is to get a copy of the same files onto each machine, then rync ( may well be the right tool for the job.  I've used rsync for many years now to address a number of issues from syncing source trees on multiple machines to deploying static content and even for backups to remote machines.  It supports ssh and compression.  I consider it to be fundamental to my toolkit, right behind ssh and perl.
I recommend that you do some research into how long it takes to rsync on a few days with varying degrees of change in your code.  The first thing that rsync does it to assess the files on both sides in order to determine what needs to be copied.  This can be an expensive operation -- in fact, the most expensive part for minor changes.  It may
While it can do partial file diffs, it may be a good ide to set it to transfer entire files ( -W param).  Compression is an option to be explored as well.  There are a number of options that might work to optimize this.
Also, and I've never done this, but it may be helpful to add a level to the distribution by first distributing to a set of machines that then push to the others to spread out the load.  Consider geography here to minimize bandwidth.
And one more point that I'll make:  I've run into some barriers when attempting to rsync directory structures with many files ( > 60,000).  Rsync croaked, so I wrote a short script to break it out by doing an separate rsync for individual top-level directories.  This may or may not be an issue, and it may actually reduce the overall time it takes to do the rysnc. YMMV.
Helpful rsync development params:
-n - dry run.  no files are changed.
-stats - as you would expect....

From: S Chaitanya [mailto:address@hidden
Sent: Friday, October 10, 2008 9:50 AM
To: Berg, Eric
Cc: address@hidden; address@hidden
Subject: Re: how can I not have to spend 40 minutes _waiting_ in themorningfor each of 40 updates?

Hi Eric,

I have the code on 40 different machines from the trunk or t-o-t or head or whatever you call it. So, no different branches involved here. I will have a go at understanding rsync and using it for syncing the changed files from 1 machine to the 39 others.

Btw, did you ever try this or run into similar problems?


On Fri, Oct 10, 2008 at 7:06 PM, <address@hidden> wrote:
Has the suggestion to use rsync been raised here?  I'm not sure if
you're updating to different branches for the various platforms, but
exporting to one or a few locations and then running rsync to transfer
just the diffs without having to repeatedly go back to CVS might be a
workable solution.

As for time servers, you'll want to sync to several servers.  NTP's
algorithm takes travel time into account, though closer is better.




This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: By messaging with Barclays you consent to the foregoing.  Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP.  This email may relate to or be sent from other members of the Barclays Group.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]