[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: how can I not have to spend 40 minutes _waiting_ in the morning for

From: Todd Denniston
Subject: Re: how can I not have to spend 40 minutes _waiting_ in the morning for each of 40 updates?
Date: Thu, 09 Oct 2008 10:03:33 -0400
User-agent: Thunderbird (X11/20080707)

Please read Arthur's comments, as they all apply, including the comment about the subject.

chaitu wrote, On 10/08/2008 09:57 PM:
Hi Todd,Mark, everyone,

I've another tricky situation to beat. Got this huge 3rd party code
repository (+2G of code) that I need on each of my 40 build machines
(for different platforms of course). This repository is compiled
everytime there's a code change and the resulting libraries are used
for compiling my product's code.

Is this 2GB of very small text files or 2GB of large (>2MB) binary files?
Are you pulling in the source code for this library, or the resulting compiled library, or both by using CVS?

The problem is that I need to fire cvs update on each of the machines
early in the morning before I start building my product. And just the
damn update takes over 40 minutes. And since all machines start this
update process at around the same time, there's some bandwidth crunch
being recorded.

Ever hear of cron?
I would have the Unix hosts issuing the update calls for me, probably having each machine give the previous ~30 minutes to run before starting it's own, early in the AM such that they should be finishing around the time I normally arrive. I think even windows has a time of day command runner, but IIRC some user has to be logged into the box for it to work.

Also are you using either the -z option with cvs or the -C option of ssh (note: use one or the other, not both)? I think -C on ssh is the better one as it compresses all of the cvs traffic not just the file transfers. Note: IIRC you would have to put the -C in ~/.ssh/config, so the easy test is -z.

when one machine is running update by itself, i.e., the other 39 are not updating while it is, how long does an update take? how long does that update take if the sandbox is already up to date?
try running:
cd my/sandbox/ ; \
/usr/bin/time -p -o updatetime cvs -z update ; \
/usr/bin/time -ap -o updatetime cvs -z update ; \
cat updatetime

note: some people need reminded that my/sandbox/ needs replaced with the location of their sandbox in the file system.

I was thinking of this solution around the problem and would
appreciate if you can chip in with your criticism.

Update on only 1 machine, and from the update log, get all the U and P
entries and create a tarball which I can send over LAN to the other 39
machines and unzip them.

if you are going to trap the [UP] entries, I would then take that data and issue cvs commands on the other machines instead. using cvs commands will keep the metadata correct and would stop the server from looking through it's tree to see if OTHER files had changed, i.e., you would run the risk with this or your method that if someone checked in a change between your first machine running and the last, you could get old and inconsistent data.

Do you have 40 different architectures or different versions of the OS for which you are building? if it is more of a thing where you are building 5 different released versions of your code for windows XP i386 on 5 different machines but against one version of the library, this is one place where I would build the lib on one XP machine and SMB share the _resulting_LIB_ to the other 4. with the caveat that I don't expect the versions of any other libs on those machines to affect the build of the lib you are building.

Todd Denniston
Crane Division, Naval Surface Warfare Center (NSWC Crane)
Harnessing the Power of Technology for the Warfighter

reply via email to

[Prev in Thread] Current Thread [Next in Thread]