[Gnu-arch-users] Re: in-tree pristines fatally wounded (merge-fest etc)

[Pau Aliagas]

On Mon, 1 Dec 2003, Tom Lord wrote:

> I'm caught up on Rob's integegration branch now.  Among the
> interesting new features was the --hard-links hack which I've revised
> and extended.  You can get by, now, without in-tree pristine copies
> at all and the only thing left is to make it as easy as possible.

This is great news!

> ** get --library 
> ** get --hard-links
> ** get --sparse
> 
> ** build-config --library
>    build-config --hard-links
>    build-config --sparse

I agree with Miles that we should match the --hard-links option name with 
the gnu standard that can be found in cp. It's already done in my tree:

address@hidden/tla--devo--1.1
patch-3
    Rename --hard-links option to --link and -l in commands get and build-cfg

Another thing that I would do is to have --sparse set by default.  
Usually, when you have a revision in the library but lack older ones, that 
is because you have not needed them, so I find it impractical to add them 
recursively by default, a lose of disk space and time. I'd instead add the 
--no-sparse option to add all the revisions. Good defaults are what make 
users happy.

> * New hooks:
> 
> In addition, there are three new hook functions that are called when,
> otherwise, arch might build an in-tree pristine tree or scratch tree.  
> The hooks are:
> 
>    ~ find-pristine  -- called when arch wants but can do without
>                        a given pristine copy
> 
>    ~ make-pristine  -- called when arch definately needs a given 
>                        pristine copy and will fall-back by making
>                        an in-tree pristine
> 
>    ~ make-tmp-pristine -- called when arch definately needs a given
>                           pristine copy but will fall back by building
>                           and later deleting a scratch tree

As you know, I thought very much about these issues and fought with 
them... been there, done that, blah blah blah.

I had the same initial idea of implementing three hooks (patch-9), but 
quickly realised that only one was really needed (patch-10). This is the 
comment I added to the patchset:

-it only makes sense to have one hook: find-cached-revision
 * when you look for a revision, if you don't have it and need it,
   use the hook to add it.
 * it makes no sense to have hook to add a revision and a tmp revision
 * moreover, as it was implemented, it was useless :)

I think that once we decide we don't need pristine trees anymore, we
should always have a revision library and add the "cached revisions"  
there, forgetting completely about pristine trees.

That means that we should have a default library path (I suggest 
~/.arch-cache) that we should be able to change via:
-my-default-library
-hook scripts

The spirit of the hooks you have added is exactly the same, but I went a 
little further to implement what I called:
* Add revision library path option to library commands

Not only I used the hook to add a revision, but I implemented multiple 
library locations so that you could choose where to store the 
cached revision, as per Miles suggestions. This is how my hook looks like:

find-cached-revision)
        tla library-add -d ~/.arch-cache $ARCH_ARCHIVE/$ARCH_REVISION
        echo -n "~/.arch-cache" 1>&3
        break
        ;;

patch-6:
Summary: Add revision library path option to library commands
Keywords: library -d --dir alternate libraries

-add library path to library functions:
 * allows multiple revision libraries in different paths
 * backwards compatible change
 * modify all calls to library functions adding a path parametre

-arch_library_add:
 * correct small bug in sparse revision libraries adding always
   a ,,patchset directory

patch-8-10:
-add 3 hooks, then move to only one
-correct bugs

In summary:
-add -d, --dir options to library commands to specify the library path
 * this is a must if you work in projects stored in different partitions, 
   disks or even computers.
 * you may want to have your cached versions "near" the working trees.
 * if you want hard-linked trees, you need to store the libraries in the 
   same partition.

But this is useless unless you can choose every time where to look for the 
revisions and where to store them. Two possibilitites arose:
-create per-project configurarion files (too cumbersome for my taste)
-decide in the hook where the library is added and return that path; if 
 you don't return a path, my-default-library is used.

I've been using it for a few weeks without a single glitch, so I think 
it's pretty easy to integrate it. The "path return mechanism" is via file 
descriptor 3 as I thought that it should be easy to use in scripts, but if 
you come up with something cleverer and simple as this one, I wouldn't 
mind. I folllowed the KISS principle.

> By using --hard-links and/or --library and by defining at least the 
> hook `make-pristine', you can now operate entirely without in-tree
> pristine copies.
> 
> By using --hard-links, you can operate on huge trees without excessive
> disk-space consumption even if you have many working copies.
> 
> --hard-links should, in theory, make some operations (e.g. `get')
> faster in many common situations on huge trees -- whether or not the
> theory is true remains to be seen.  The code is currently pretty
> liberal about doing inventory traversals and that may or may not need
> some optimization work to truly realize the time savings.
> 
> Regardless, the disk-space savings should be there: people working in
> very tight (disk-space-wise) environments would do well to spend some
> effort implementing the three hooks mentioned above to manage their
> resources.
> 
> All the raw materials are now in place to get decent performance for,
> oh, say, lk-sized trees -- it's just a matter of configuration and
> tuning.

I completely agree that now we are ready to use these huge trees and be 
superior to other non-free alternatives: less space and more speed, what 
else can you ask for?

> * Heros of the Arch Revolution
> 
> Special thanks to:
> 
>         Robert Collins -- integration branch work and the inode-sig
>                           sanity-check
> 
>         Pau Aliagas -- the --hard-links option to get
> 
>         Me :-) -- extending the above to mortally wound in-tree
>                   pristines and to fix upward-compatability problems

I feel really happy today :) Hard work has been worth it!

Tom, thanks for the huge work done!
Pau