[Gnu-arch-users] Archive metadata

[David Allouche]

I would like to start a thread of it own for the design of the mythical
archive-metadata category, so I have intentionally broken the discussion
thread with previous postings.

On Fri, Jan 23, 2004 at 09:46:53AM -0500, Miles Bader wrote:
> On Fri, Jan 23, 2004 at 03:28:28PM +0100, David Allouche wrote:
> > The conclusion was:
> > 
> >   Metadata is to be stored in a special "archive-metadata" category.
> >   This fixes mirroring and atomicity issues and preserve compatibility.
> > 
> > I hinted toward this solution earlier in the thread. It has a number of
> > nice properties, like preserving good performance with abrowse and the
> > like if the last revision is cached. But then we hit the "mirror update
> > of cachedrevs" issue; ironically, that is an issue which can be fixed
> > with the archive-metada category.
> > 
> > The main problem with that approach is that it would require a new
> > internal support infrastructure which is not present yet.
> 
> I'm not really sure what you're referring too -- are you suggesting that for
> every operation that wants to look at some meta-data (which is quite a few at
> least for the functionality I'm talking about), tla _check out a branch_?
> That seems, um, absurd (absurdly slow, anyway)...

That's an interesting design discussion, and I think it is indeed time
to start it.

== Design ==

So far, I know of two approaches to metadata storage:

  1. archive-metadata--<feature> branches, in the usual archive
     namespace
  
  2. =metadata/category/branch/version directories

The latter was mentionend by jblack on #arch. I have not yet thought out
the respective pros and cons of these storage solution. My intuitively
preferred solution is the former.

One advantage with that solution is that you can get the whole archive
metadata for a given feature with a single checkout. Since doing such a
checkout for each individual metadata read would be indeed absurdly
slow, the metada would have have to be cached in ~/.arch-params. For
example there could be one directory
  ~/.arch-params/cache/metadata/<archive>/<feature>
for each archive-feature metadata pair ever accessed.

Some issues appear immediately:

=== Growing logs ===

If we keep patchlogs for all previous revisions of metadata trees, we
could end up with quite big a cache. Since anyway we are never going to
branch or merge metadata trees, we can just remove the log for revision
N-1 when setting up revision N+1.

=== Updates and locking ===

We do not want to update the cache every time we access it. A reasonable
update frequence may be "each time a user command first need to access a
metadata directory". Since a command will probably expect the cache not
to move under its feet (either for reading or writing) we need a locking
mechanism.

The lock applies to a local tree, so there is little contention. A given
metadata cache is only intended to be used by a single user. If the user
runs robots which may acquire the lock, contention can be avoided by
making robots use distincts .arch-params directories by setting the HOME
environment variable.


=== Needed commands ===

  metadata-find [--init] <archive> <feature> <version>
  
    Give the absolute path to the cached metada directory.
    If not already present in the cache, get it first.

    If --init is set, the metadata version must not already exist, an
    empty dir is locked, created in the cache, and is set-up with
    init-tree.
    
  metadata-lock [--unlock|--break] <archive> <feature> <version>
  
    Acquire the lock for a cached metadata directory.  Only the lock
    owner is allowed to modify the directory.

Everything else can be done with the usual commands, "update", "commit",
etc.

== Application ==

=== Archive descriptions ===

Descripitions for categories, branches and versions are stored in
archive-metadata--readme--1.

This tree is a subset of the archive tree, and contains "-readme" files
where appropriate. Note the naming designed to avoid clashes. The
tagging-method is "names".

For example:

  frob/
    -readme
    frob/
      -readme
      frob--0.9/
        -readme
      frob--1.0/
        -readme
    frob--devo/
      -readme

Getting a description is (almost) trivial:

  % D = `tla metadata-find $ARCHIVE readme 1`
  % if test $? != 0 ; then
  >   tla metadata-lock $ARCHIVE readme 1
  >   tla update --dir "$D"
  >   if test -e "$D/frob/frob--devo/-readme" ; then
  >     cat "$D/frob/frob--devo/-readme"
  >   fi;
  >   tla metadata-lock --unlock $ARCHIVE readme 1
  > fi

Setting a description is a bit more involved:

  % D = `tla metadata-find $ARCHIVE readme 1`
  % if test $? = 0 ; then
  >   D = `tla metadata-find --init $ARCHIVE readme 1`
  >   echo "names" > "$D/{arch}/=tagging-method"
  >   if tla import --dir "$D" --summary "initial import" ; then
  >     : # sucessful
  >   else
  >     tla join-branch --dir "$D" archive-metadata--readme--1
  >     tla undo --dir "$D" --no-output
  >     tla metadata-lock --unlock $ARCHIVE readme 1
  >     echo "concurrent change in metadata" >&2 ; exit 1
  >   fi
  > else
  >   tla metadata-lock $ARCHIVE readme 1
  >   tla update --dir "$D"
  > fi
  % mkdir -p "$D/frob/frob--devo"
  % cat description > "$D/frob/frob--devo/-readme"
  % # [ remove logs for all but the last revision ]
  % if tla commit --dir "$D" --summary "set frob--devo" ; then
  >   tla metadata-lock --unlock $ARCHIVE readme 1
  > else
  >   tla undo --dir "$D" --no-output
  >   tla metadata-lock --unlock $ARCHIVE readme 1
  >   echo "concurrent change in metadata" >&2 ; exit 1
  > fi

If someone concurrently tries modify the branch, the import or the
commit will fail. We can do little about it but put the cache in a valid
state, release the lock and complain to the user.

=== Closed versions ===

This is left as an exercice to the reader...

=== Merged and rejected patches ===

Ideas?

-- 
                                                            -- ddaa