[Gnu-arch-users] RFC: arch protocol, smart server, and tla implementation prototypes

[Colin Walters]

Hi,

I've been working on a "smart server" over the past few days.  There's a
lot to talk about, so I'll try to break it up into cleanly separated
parts.

First, "why a smart server?".  This has been discussed on the mailing
list in the past, but here's a far-from-exhaustive list of reasons:

o Ability to have a much more efficient protocol, avoiding server 
  roundtrips
o Execute post-commit hooks on the server when a revision is committed
o Ability to have very flexible authentication and access control
o Removing the need for a shared Unix account for multiple committers
o Ability to record which revisions are accessed frequently, and
  dynamically adjusting cached revisions to match

Ok, now that that's out of the way, let's discuss the three major parts.

o Definition of an Arch Protocol - the wire by format which clients
  and servers communicate
o Prototype implementation of a server
o Prototype implementation of client functionality for tla

The status is that I am pretty close to being done with everything you
need for read-only functionality in all three of these, and writing is
coming along nicely as well.

Here's a quick sample of where I am now:

address@hidden> ./tla/tla whereis-archive address@hidden
arch://localhost
address@hidden> ./tla/tla abrowse address@hidden
address@hidden
  archd
    archd--main
      archd--main--0
        base-0 .. patch-21
address@hidden> ./tla/tla cat-archive-log address@hidden/archd--main--0--base-0
Revision: archd--main--0--base-0
Archive: address@hidden
Creator: Colin Walters <address@hidden>
Date: Tue Jan 27 14:21:17 EST 2004
Standard-date: 2004-01-27 19:21:17 GMT
Summary: initial import
Keywords:
New-files: GOALS Makefile.am archd archives.conf.sample
    autogen.sh configure.ac manual.xml users.conf.sample
New-patches: address@hidden/archd--main--0--base-0
 
address@hidden> ./tla/tla make-category -A address@hidden foo
address@hidden> ./tla/tla abrowse address@hidden
address@hidden
  archd
    archd--main
      archd--main--0
        base-0 .. patch-21
 
  foo
address@hidden>


Now, in more detail, let's go over these three components.

o The Arch Protocol

It's pretty strongly influenced by both HTTP and IMAP, with some
personal touches.  It's fairly flexible.  For more detail than this,
see the PROTOCOL document I've attached.

o A sample smart server, called "archd"

This is written in Python.  It has both reading and the start of writing
functionality.  The major issue now is that it's single-threaded.  This
makes atomic operations easier, but I do plan to thread it in the near
future.  

The server is *extremely* pluggable.  You can plug in both
authentication modules and archive backends just by tossing together 
a bit of Python code and dropping it in one of the module directories,
then adding its name to the configuration file.

For authentication, there's an IP-based authentication module and a
DIGEST-SHA1 module.  More can be easily added.  

There is only one backend right now, it's called "patchdir".  It's very
simple, and similar in broad outline to how arch stores patches on a
filesystem.  However it is different in many details to prevent people
from just pointing an arch client at it directly and expecting that to
work.  I am planning to provide a nice way to import a filesystem
archive into archd - however I want to do it "right", and that means
finishing the "commit" operation so you can just use arch to mirror it,
and load it that way.

o Implementation of the protocol in tla

This was helped quite a bit by Tom's work on abstracting out the
filesystem-specific stuff into archive-pfs.c.  We can make things more
efficient (avoiding server roundtrips) with some more work in here with
the data the server sends back already.

My code's kind of ugly - it aborts on almost any error.  That will need
to be fixed eventually.

o Comments, etc

The tla implementation is here:
address@hidden/tla--archd--1.2

The archd server is here:
address@hidden/tla--main--0

And again I've attached the PROTOCOL document.  I'll be keeping it
up-to-date in the archd source tree.

So, at this point I am very interested in comments on (and changesets
for) all three of things.  

Incidentally, here's how you can run archd:

./archd -d -v -v --config=archd.conf.sample 
--archive-config=archives.conf.sample --user-config=users.conf.sample

PROTOCOL
Description: Text document

signature.asc
Description: This is a digitally signed message part