[Gnu-arch-users] a challenge: smart server, anyone?

[Tom Lord]

Arch currently provides archive access, as we know, using a "least
common denominator" set of features found in file-system-like
transports.  Aside from implementing a few basic file system
operations, no server-side computation is required or presumed.

Users have identified two areas where additional server-side
computation would traditionally be used:

*) domain-specific access controls on archives

*) triggering of server-side "hooks" for purposes such as 
   sending post-commit email or performing pre-commit 
   testing

Arch has traditionally shunned moving to a "smart server" for such
features.  To a large degree, both features can provided by ordinary
configuration tricks and there are strong arguments that this simpler
approach is the best one.

Nevertheless, both features are quite open-ended:  there are huge
spaces of both access control mechanisms and hook applications.   It
has always seemed likely that in that huge space there are likely to
exist points with two simultaneous properties:

a) they're very useful
b) they're awkward to perform without additional, arch-specific, 
   server-side mechanisms

and so, since the days of larch, there has been occaisional idle
discussion about the possibility of implementing a "smart server".  

When I wrote tla, I kept that possibility in mind and changed the
structure of the implementation to accomodate.   I paved the way to
providing smart-server functionality and now the only trick left is to
actually write one (or two or N+1).

The biggest change I made in tla from larch was to impose a carefully
designed abstraction barrier between the details of archive access and
the rest of the implementation.  In larch, even some very high-level
code would sometimes treat an archive as a file system, for example by
asking for a directory listing of some part of an archive.  In tla,
only a very few source files regard an archive as a file system -- the
rest of the code sees only an archive-specific abstraction.

The interface to the client-half of archive access is defined by 30
functions which must be separately implemented for each transport.
The prototype for 27 of those is provided in

        src/tla/libarch/archive.h(struct arch_archive_vtable)

The remaining three are used to:

        purpose                         example:
        -------                         --------

   identify which URIs to               arch_pfs_supported_protocol
   support


   conect to an archive                 arch_pfs_archive_connect

   create a new archive                 arch_pfs_make_archive


With that background:

                     How to Build a Smart Server
                     ===========================


Step 1:  Build a "proxy archive class"

  All archive access in a client is via a `struct arch_archive'.
  Let's call that an "archive handle".

  Note that it would be quite easy to "stack" archive handles: 
  to have a handle A which does most of its real work by calling 
  an (arbitrary) handle B.

  As a simple example, A's implementation of "make_category" might
  take the following steps:

        1) Verify that the proposed category name matches a regexp
           used to enforce a site-specific naming convention.

        2) Invoke B->make_category

        3) Execute a "post-category-creation" hook.

   Implement a type of archive handle (a sibling to archive-pfs.[ch] and
   archive-fs.[ch] which is a proxy handle.    It should do most of
   its work by calling a second archive handle, created in the usual
   way, from a substring of the uri.

   It should not be possible directly "instantiate" this new type of 
   archive hook.   The new proxy class should define a new type of
   vtable for the hook functions it knows how to call -- we'll fill
   those in with several different subclasses.


Step 2: Build a config-file subclass of the proxy archive class

   Build an (instantiable) subclass of the proxy archive class whose
   purpose is to execute arbitrary code selected by a config file.

   That is, at `connect' time or `make_archive' time, this class of 
   archive should parse a config file that says what actions to take
   when various archive proxy hooks are called.   It would be in this
   config file, for example, that you could add a line to generate 
   post-commit email.


Step 3: Build one or more remote-client subclasses of the proxy
        archive class (and their servers)

   Build one or more proxy archive subclasses whose vtable contains
   functions which simply martial the arguments to archive handle
   vtable functions, transmit them according to a protocol of your
   choice, and read a reply.

   Implement new stand-alone tla programs (siblings to src/tla/tla.c)
   which provide the server-half:  reading the arguments to archive
   functions, constructing and invoking functions on a local archive
   handle, and returning replies.

   Two likely to be useful kinds of client:

        1) a client that invokes an arbitrary subprocess, writes 
           calls to that process' stdin, and reads replies from that
           processes stdout.    This should be useful in combination 
           with programs like ssh or even telnet.

        2) a client that converts calls into HTTP POST requests.
           This should be useful in situations where the server
           is installed as a CGI script.

Optional additional steps:

Step *: Build an authentication subclass of the proxy archive class

  Build a (highly configurable, presumably) proxy archive class which
  implements a stand-alone form of "arch user accounts" using, for
  example, public-key techniques and long URIs to authenticate users
  at connect or make_archive time.

  Add as much "fine-grained" access control as you like (e.g. perhaps
  some users are permitted to commit `patch-N' revisions to a
  development line but not `version-0' or `versionfix-N' revisions).


Step *: Build a lazy, caching mirror subclass of the proxy archive
  class

  Build a proxy archive subclass which does much of its work by 
  passing requests to some other archive handle, but which also
  locally caches those results which are invariant over time (e.g.,
  `get_patch').    Such a proxy is an alternative to the current
  system of mirrors:  one that fetches data only on-demand and that
  can locally cache an arbitrary subset of a remote archive (i.e, 
  cache patch-145...patch-187 but not base-0...patch-145).



The resulting system will be quite flexible.   The amount of work
proposed here is relatively small (measured in weeks to get initial
useful results).

-t