[Gnu-arch-users] Re: give us a hand with arch

[Pau Aliagas]

On Thu, 25 Sep 2003, Andrea Arcangeli wrote:

> On Thu, Sep 25, 2003 at 06:29:13PM +0200, Pau Aliagas wrote:

Hi Andrea, welcome to the list.
I'll post my answer here to have more eyballing.

> actually I learn using arch just last night. Not absolutely everything,
> but in enough detail to undertsand the design and how it works
> internally.
> 
> the major problem at the moment seems speed

You should use a fairly recent version to have the inode signature 
optimization. The src.rpm I sent you already had it, try building it and 
the speed improvement will be 10:1

> (I noticed the cache possibility)

There are several ways to speed up gets and commits: pristine-trees, 
revision libraries andcached revisions, more on this later.

> and lack of user-friendly code, I mean, why shouldn't
> make-log + vi + commit be packed into a single command like in CVS? It's
> fine to have it granular but a single command that most people would use
> would fit best IMHO.

Well, we have "express commits": tla commit -L "your explanations".
Combining both like CVS is just unimportant because you could have an 
alias in your shell to do it. Once you get used it's evenbetter to have it 
this way because you can fill in the log as you do the changes. See, it 
makes you work better.

> especially I believe the directories are too long, that slowsdown the
> dcache code a lot since they gets all dynamically alocated names with
> kmalloc, and especially the compression probably hurts too much the
> checkout.

This has been discussed this week because of a problem with cygwin path 
length. Some people think tht shorter paths, avoiding redundancy, would be 
enough, so better. Now you give us a good reason to consider moving the 
archive format to shorter paths. I'm all for it and I don't think Tom is 
hardly against, but his worry is about existing archives. Recycling them 
is trivial, but anyway we prefer not to do unnecessary changes. I vote for 
it, nt only for your comment on the spped trade, but also for the 
shortening of the dir names.

> I feel like the whole database in the archive would better be rewritten
> to boost performance and to compact it. But the design looks promising,
> much better than cvs, so as a starting point looks definitely
> interesting as you suggested.

Well, it's simply a filesystem properly organized and that's part of its 
beauty :) Maybe in a reiserfs would be faster than on ext2/ext3 or, as you 
sugges, we could even have specialised file systems.

> I've quite lots of comments, I will post them separately to the
> arch-user list (is that the best place?).

Yes, ther's lots of people that know much more than I do about arch and
its internals.

> > If only you and a few others would use it and see how damn easy everything 
> > becomes!! And more people using it would mean also more hacking and a 
> > better arch. more platforms. Hey, it's up to you!
> 
> I believe it's realistic to use it.

Reading your initial impediments, I'm almost sure that the difficulties 
you are finding are more due to being unfamiliar with arch than to real 
problems.

> However I've a number of problems for using it even in 2.4-aa, that is
> simply it's fundamental to me to maintain the changesets intact. If I
> want to make a change to a changeset, I've to change it, not to make the
> modification at the end like in cvs. I mean, I've to change the
> patch-2001. And ideally I would like to name each patchset with a more
> meaningful name (I know there's the summary so that may not be too bad).

That's damn easy with arch! I'll forward you privately an email posted by
Tom called "prims-merging technique" (you can look for it in the archives 
too). To make it short, you can create sevral branches to work on several 
features, merge them selectively to the trunk (this would be you linux-aa) 
and from here export your consolidated changeset, made of patches from all 
your private-feature branches. That way you develop randomly n your 
private features but export "complete changesets". Read the complete email 
I'll send.

Naming patches is not possible, but you have summary + keywords + log 
associated to each one. Much more expressive than a simple name.

> But exactly because it's purerely driven by changesets, it sounds very
> reasonable to use it even for 2.4-aa (with hundred patches that will
> never get merged).

I keep several GPL projects synchonised with my private changes and keep 
my unmerged pathes in my private branch. It's just getting used on ow to 
do it with arch. My technique is as follows:
-create a project of the original source (in this case linux--linus)
-create a project of your branch tagged from the original:
 tla tag linux--linus--2.4 linux--aa--2.4
-apply your changes to your branch
-keep linus tree updated mergin in changes (from cvs, release patches, 
 etc...)
-merge its changes into your tree and solve conflicts

It would be easier if Linux picked changes from your tree as arch would 
detect it and no conflicts would arise.

-In your branch youl¡ll always have the pending patches ready to export to 
 the current kernel, effortlessly ;)
 
> The checkout isn't really different from my script
> tha applyes each patch in order. However arch compared to my script
> sounds a bit overcomplex, just for this purpose.

Not at all. Read the prism-merging development technique mail and see.
It's only a matter of disciplined development adapted to the beauty of
arch power.

> Also I can't upload 200m every time, so I need to tell arch to tag
> against an unpacked source somewhere in the tree and I must be able to
> change the tag at the top of the tree without affecting the changesets.
> This isn't possible yet if I understood well.

I don't quite get it. You should never download or upload 200 Mb but the 
first time. The steps should be more or less the following:

-create your own archive (repository)

-create original linux project (name it linux-kernel--linus--2.4)
-import source into it (preferibly from the CVS tree if you want fine 
 grained detail, using of the existing tools, like Miles' arch-tools)

This original linux tree on arch is CRUCIAL as evreybody will keep on
getting changesets from it. We now have on linux archive, but made from
tars and release patches, not from the CVS, so it lacks the level of
detail you may need. This tree should ideally be mirrored in kernel.org,
for the availability and bandwith, and people who'd like to work on linux
using arch would ideally mirror it locally. Instead of downloading patches 
they'd mirror again.

-create your branch tagged from a revision of linux--linus tree.

-patch the tree carefully, feature by feature, so that you can export each 
 one separately

-as time goes by, you'll synchronise with the original CVS (the bk->cvs) 
 using Miles' arch tools, so that everything is automated.

-you'll replay those new patches in the original tree in your linux--aa, 
 probably with little intervention. Today Tom has posted a mail explaining 
 how to avoid conflicts when merging. There are several techniques and 
 each one is useful depending on the level of interference among patches.

> We'll see, but it certainly sounds just very reasonable, with multiple
> people working on the project and merging *regularly* like bitkeeper.
> For the *almost never merge* case it maybe more problematic.

I don't see any problem. If people want patches from your tree, they can 
replay your changes one by one, synchronise using star-merge... it feels 
natural.

If people wnat read-only access, they can mirror to have fast access. This 
is even less problematic.

> I also feel it needs some sort of naming enforcment, basically I would
> enforce the inventory explicit model, by simply avoiding arch to look at
> any file that isn't explicitly invetoried (showing '?' like CVS). that
> sounds much safer when there's lots of junk in the tree and there's no
> risk to forget an explicit rename. The automatic inventory embedded in
> the files is too deep.

Miles' arch-tools already do this automagically from CVS ;) I'm sure he
can give you a hand. We have everything almost ready, we only need someone 
motivated enough to import the kernel CVS.

I prefer tagline naming method, but as you say that's too deep by now. 
Explicit tagging is by far the best choice.

> Oh, another way is to write a LD_PRELOAD library
> to trap mv/rm/create, but that sounds very dirty. The ideal would be to
> have streams in the fs.

If you mean it for importing from the CVS, arch-tools already does its 
beest ot detect renaming and mv. No need for anything special.

> Also I would need to convert all the changesets into regular patches
> with their own meaningful name and in order, to be able to export all
> the stuff in the SuSE tree (I doubt the SuSE tree would switch
> immediatly to arch too)

You can export changsets to patches directly doing tla get-patch. That'd 
do the trick :)

> > Again, if you need any help just tell me.
> 
> I will send an email to arch-users.
> thanks

I've seen how Tom has already jumped to the wagon.

> Andrea - If you prefer relying on open source software, check these links:
>           rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
>           http://www.cobite.com/cvsps/
>           svn://svn.kernel.org/linux-2.[46]/trunk

This signature will grow RSN.

Pau