Re: [Monotone-devel] newbie question - SHA1 vs serials

[Richard Levitte - VMS Whacker]

In message <address@hidden> on Wed, 20 Apr 2005 10:20:53 -0700, "K. Richard 
Pixley" <address@hidden> said:

rich> Jon Bright wrote:
rich> 
rich> > I'm pretty sure the case of 128 developers working on the same
rich> > branch at the same time would be a rare one.  Since branches
rich> > are such light-weight things, I think it's much more likely
rich> > you end up with (as we have now) net.venge.monotone and
rich> > net.venge.monotone.cvssync - the person working on the cvssync
rich> > feature works away in their own branch and will eventually
rich> > merge back to head.
rich> 
rich> That is one instance of what I mean by partitioning.  I'm pretty
rich> sure that all software development concurrency problems can be
rich> solved by partitioning.  That's really just a fancy way of
rich> saying we can solve the problem manually if the tool doesn't
rich> offer us sufficient support.
rich> 
rich> My point is that currently available tools offer differing
rich> levels of support before solving the problem manually is
rich> required.  128 concurrent developers is only rare because
rich> existing tools have maxed out in this range forcing people to
rich> think in terms of solving the problem manually, (ie,
rich> partitioning), much sooner.  Partitioning has a high overhead so
rich> we'd really prefer to avoid it for as long as possible although
rich> partitioning is better than failing to solve the problem.

Honestly, I think this is a social problem rather than a technological
one.

rich> > So don't count code as produced until it's merged from the
rich> > branch where it was being produced into mainline (or the
rich> > mainline for this subproject, or whichever).  One ready-made
rich> > motivation for getting branches merged closer to mainline (and
rich> > therefore for getting heads in your branch merged, since
rich> > otherwise you can't merge to mainline).
rich> 
rich> This is not how programming is currently taught.  We've had
rich> source code control for at least 30 years and yet source code
rich> control is still a new concept to many developers.  I agree that
rich> a change task isn't complete until it's delivered to the
rich> appropriate branch, perhaps not until it's suitably tested, and
rich> maybe not even until it's in the customer's hands. However, as a
rich> CSM professional, I'm well aware that that's not currently how
rich> most coders think.  I can educate.  And I do.  But it's not
rich> currently a reasonable expectation.

I disagree.  I've taught coders in this area, and they usually have no
difficulties grasping the ideas of SCM (VCS, CSM, whatever).  Most
often, they're quite enthusiastic about the new way to do
development.  Cooperation is a strong insentive.

rich> > You can do this today.  Run a second copy of monotone on the
rich> > central machine, pull with it from the first.  Stop it.  Back
rich> > up its DB.  Repeat.  Presto, continual backups, as often as
rich> > you want, of the central repository.
rich> 
rich> No key distribution.  No backups of system software or
rich> configurations.  What I really want is the ability to reload the
rich> original machine's (replacement) hard disk with a verbatim copy
rich> of whatever it had yesterday.  I'm talking about disaster
rich> recovery, not necessarily archival, court records, or data
rich> redundancy.

That's actually incredibly easy.  On the server, place the live
database (the one being served to the rest of the network) in a
separate partition that isn't backed up, for example:

  /no-backup/db

Have the rest of the file used by monotone (the server at least) in a
different structure, for example /etc/monotone (typically, you would
want to have a file with appropriate lua hooks, for example
/etc/monotone/monotonerc).  Your server command line would probably be
something se simple as this:

  /usr/bin/monotone --db=/no-backup/db serve local.host.name collection

(collection is the base of all your branch names.  This is the only
place where something like that is enforced.  Oh, and you can have
more than one collection)

Then have a cron job like this:

  0,10,20,30,40,50 * * * * /usr/local/bin/backup-monotone-db.sh

That cron job would basically be like this:

  #! /bin/sh

  # creating a directory is the cheapest lock ever...
  mkdir /tmp/backup-monotone-db.lock && \
  ( mkdir -p /var/backups/monotone
    # pull a copy of the current database
    /usr/bin/monotone --db=/var/backups/monotone/db \
        pull local.host.name collection
    # because private keys aren't transfered with netsync, we need to
    # copy them manually.  This will be a non-issue when private keys
    # are moved off the database (to happen pretty soon).
    # We speak directly with the live database, but those bursts are
    # so short it should be harmless.  Private keys do not change very
    # often either.
    keyIDs="`/usr/bin/monotone --db=/no-backup/monotone list keys | sed -e 
'1,/^\[private/d' -e '/^$/,$d' | cut -f2 -d' '`"
    echo "$keyIDs" | \
        while read keyID; do \
            /usr/bin/monotone --db=/no-backup/db privkey $keyID \
                > /var/backups/monotone/privkey_$keyID
        done
    rmdir /tmp/backup-monotone-db.lock )

Note that the private key thing will disappear soon, since there's a
concensus they don't really belong in the database.

Now, all that's needed is for a backup script to have all of this
machine backed up, except for /no-backup.  It should probably make
sure to use the same locking mechanism as the above script.

Note that the above is a 10 minute hack, and could be evolved further,
and integrated with whatever backup system you have.

rich> >> CVS doesn't handle branching very well, and merging is even
rich> >> worse.
rich> >
rich> > monotone handles branching very, very well and merging.. well,
rich> > certainly much better than CVS.  I think the relative ease of
rich> > branches in monotone might be something you're overlooking.
rich> 
rich> I don't think so.  I take ease of branching for granted.  It's
rich> management of the branches that seems to be an issue in most
rich> free systems today.  Contrast to clearcase branches, which,
rich> while nominally more user overhead, are significantly and
rich> seriously easier to track and manage.  In a distributed system,
rich> I can't even ask what branches exist, since no perspective is
rich> necessarily even complete.

This is a matter of policy.  Just tell the developers that unless
their work is pushed to the central server, they can't count on
whatever they do to be considered appropriate work.  It's exactly the
same with developers who are afraid of committing unless their
(sometimes HUGE) changes work as expected.  Coders should be
encouraged to pull, update, commit and push fairly often.

Cheers,
Richard

-----
Please consider sponsoring my work on free software.
See http://www.free.lp.se/sponsoring.html for details.

-- 
Richard Levitte                         address@hidden
                                        http://richard.levitte.org/

"When I became a man I put away childish things, including
 the fear of childishness and the desire to be very grown up."
                                                -- C.S. Lewis