Re: [Gnu-arch-users] Arch Roadmap Draft (the anticipated part 3)

[James Blackwell]

Oh, Tom Lord wrote all of these:
>   1. Arch is underutilized.   No, I don't mean that there
>      aren't yet enough users (which sure, is likely to be true
>      for a very long time).   Rather, the existing users (including 
>      me) aren't making as good a use of arch as we could be.
>      There's a fairly large gap there, I think.
>


>      One example is the very ad hoc way that we exchange
>      patches, even when we're using arch for that exchange.
>      It would currently be impossible (just because we're
>      sloppy and inconsistent about patch submission) to 
>      do something obvious and useful like set up a browser
>      that can show all of the changes that are in the 
>      Bug Goo queue.

I've been thinking about this a bit too. As far as I can tell, the merge
requests are freeform.

I think this is rather easily addressed. All that needs to be done is to
put up a page on the wiki that says "Merge requests should have the
following format:". Make a public announcement that that's how merge
requests should be done. Once a couple people do it, everybody else will
fall right in line. 

With a well defined format, somebody can write a script that can
automate a good deal of the work --  statistics, a human readable diff,
make test results, so forth and so on.

I suggest the following format for a merge request: 

archive-location: (optional) http://bitmovers.com/archives/subvert-thyself
archive-name: address@hidden
contact: (optional) address@hidden
merge-patches: patch-10, patch-15..patch-20

(whitespace comments)


Andrew Suffield has done a bit more thinking on this issue, but here's a
simple algorithm to handle the work:


On merge submission: 

1. if necessary, register the submitted archive
2. get the stated changeset
2. generate a diff report from the changeset
3. record the patchlogs
4. get the development head (ex. address@hidden/tla--devo)
5. apply the changeset against the development head
6. attempt to compile the now modified dev. head and record results
7. perform make test and record results
8. email all recorded results to the development team
9. store the results on a website somewhere


>      Another example is the fairly ad hoc (and inconsistent, over
>      time) way that I tag releases.   If you want to go back
>      and get various historic releases out of the archive, you
>      have to do some poking around to figure out how the contents
>      of each release were recorded.

Aren't the stored configs working for you? If not, perhaps a tag-only
releases branch? Retroactively making a tla--releases, and tagging a
bunch of versions shouldn't take you much more than an hour to do.


>   2. Too many conflicting demands on my time suggest the need for
>      some process improvements.
>
>      There's been grousing, lately, about my backlog of merges.
>
>      The actual arch work of processing these changes can be pretty
>      fast but I'm hampered by my slow modem.   I have to wait for
>      changes to be downloaded locally, I have to push my merges
>      back upstream.   I must cut releases on my local machine yet
>      the subsequent uploads, announcement posting, mirror updating,
>      and so forth takes a substantial part of a work day for each 
>      release:  a day mostly spent just waiting and checking
>      periodically that modem lights are still blinking.   As
>      streamlined as even something like FreshMeat is it's _still_ 
>      a disruptive pain-in-the-ass to go through all that for each
>      release.

Much of this can probably be automated with three scripts: 

1. You review the aforementioned merge request that was emailed to you.
The make test looks clean, the diff output looks promising. So you pipe
the email through | mergetest (somedirname). Mergetest performs the
following: 

      1. grabs your latest revision (with a get --link) and places it in
         (somedirname)
      2. applies the patches that are listed, warning appropriately
         about any rejects.
      3. dumps a copy of the email in (somedirname)/,,merge-request


You then flip to another tty, cd into (somedirname). You resolve any
rejects, inspect more closely, etc. When satisfied, you run
merge-accept. merge-accept performs the following activities:

   1. commits the applied changesets
   2. sends an email to buggoo (via the list) that closes the merge
   3. touches ~/did-archw-ork


At 2 am, a cronned script looks to see if ~/didwork exists. If so, 
then the following occurs: 

   1. runs pon (or whatever command it is that gets you on the internet)
   2. does a su lord -c "push-mirror address@hidden"
   3. removes /home/lord/did-arch-work
   4. runs poff (or whatever else gets you off the internet)



The third script would be a release-arch (someversion). This script
would perform the following:

  1. check out dists--devo
  2. copy dists.tla.template or whatnot into emf.net/tla with a short little
  sed command that replaces VERSION with (someversion)
  3. tags tla--releases--VERSION to tla--devo
  4. lynxes you to the freshmeat url for submitting a new release of
  arch. You're already logged in because you've set up lynx to accept cookies.


In a less verbose way, "Write some scripts dude!"

>   3. The market for arch-related labor exists, but there aren't 
>      enough customers.  Therefore, there is a danger of corruption
>      of the project.
>
>      Of course we want the market to grow in dollary terms, that's 
>      obvious.
>
>      What may be less obvious is that currently, there are only a
>      small number of employers who pay for arch work.  Consequently,
>      there is a lack of diversity of interests in the marketplace and,
>      not to be chicken little, but if we aren't careful, the interests
>      of those few employers might come to dominate the future of arch,
>      at the expense of our larger community interests.
>
>      We need to make arch easier to deploy in order to attract 
>      new commercial interests to it.   We need to make sure that no
>      one or two employers is in a position to "take over" arch.

Marketing, marketing, marketing. The better that arch works for the
common user, the more bulletpoints we can put on promotional material
intended for pointy heads, the larger the potential arch market.

>    4. I share some of the user complaints or complaints close them.
>      Specifically:
>
>      User complaint:  There are too many commands.
>
>      Tom's take: There aren't enough commands.   What's missing are
>       a few higher-level commands that capture the most common and
>       most recommended use patterns of arch.   Once those are in
>       place, most users can focus on just those few commands and
>       ignore the rest.   An improved help system can focus attention
>       on just those few commands.

I agree with you in spirit, but not to the letter. 

>       In other words: Arch doesn't have too many commands but,
>       currently, average users have to use too many of the commands
>       to do their work.   Consequently, they tend to be a little 
>       daunted at the number of commands they have to learn to
>       get started, and the help system listing of commands has
>       to list an intimidating number of commands.

Hey! I just had a *SILLY* idea. How about "tla help-me". tla help-me can
examine the user's current state of affairs by linting the user's arch
state. For example: 

$ tla help-me
tla: I think the first thing you should do is introduce yourself to me
     by typing tla my-id "Your Name <address@hidden>". I use my-id
     to record your commits

$ tla my-id "James Blackwell <address@hidden>"
$ tla help-me
tla: There are a few things you could do here. You might want to set up 
     a location by making a new archive (tla help-me make-archive), if 
     you don't need your own archive, then you may want to get somebody
     else's archive (see that with tla help-me get)
$ tla help-me get
 .....

This help-me command could get quite extravagant -- for example, if
help-me sees that you're in a project tree, it may check to see if there
are changes (thereby suggeting commit), or checking to see if its a get
from a remote archive (suggest replay or update)

I know. Its kind of screwy, but it just might work! 


>      User complaint: Some of the error messages are still bogus.
>
>      Tom's Take: A valid complaint that continues to be answered 
>        by incremental improvements.
>

Yeah. I don't think this is a significant issue any more. Most of the
more annoying errors have already been taken care of. The ones left are
generally of the sort when we *want* people to come to us.

>      User complaint: Tagging methods are hard to understand and use.
>
>      Tom's Take: Again, some higher-level help is probably needed
>        to capture common patterns.

I think the users don't know what they're doing, and rather than admit
they can't figure it out, they're screaming that its broken. So lets
make it easier by telling them what to do.

How about we package with tla some example =tagging-method files that
match certain profiles... This one for handling things this way, another
one for another way...  Perhaps even one for building in tree! 

>      User complaint: Performance on large trees is not up to snuff yet
>        for some operations.
>
>      Tom's Take: The largest improvements needed in this area are:
>        summary deltas; fast, general, selective commit; and improved
>        ease-of-use for (a generalization of) configs.

I share the complaint (I spent some time dillying with the linux kernel
in arch). However, that was *quite* a long while ago -- in fact, my
experience predates libraries its so old. 

One thing that really hits hard is when I try and get the latest version
of something (for the first time) that has 500 older revisions. I think
we could do a lot of good here by gently prodding users to cacherev more
often. 

What about a gentle advisement on patch-(n / 50 ==0) along the lines of
"tla: now may be a good time to cacherev *before* you perform any
miroring"

>      User complaint: There isn't a bundled graphical merge tool.
>      User complaint: There isn't a bundled GUI.
>
>      Tom's Take: The bottlenecks here are my programming environment
>        at home and my time.  The roadmap aims to improve the use of my
>        time, but the programming environment is a harder problem.  In
>        particular, I use a somewhat (by today's standards) slow
>        and low-memory machine with limited disk space running an old
>        operating system and X11 release.   I simply do not have the
>        facilities on hand to _build_ most of the candidate merge tools
>        let alone contemplate integrating them with arch.   Even if
>        someone else were to do that work, I lack the facilities for 
>        examining and testing their results!

I haven't been part of this group, but I'll openly admit that I don't
know how to merge rejects. I have no idea how everybody else learns to
do it. I can't find any howtos, books, or ectetra. This leads me to
believe that conflict resolution is some sort of closely held dark art. 

What if the arch documentation contained some sort of mini-howto that
explained in clear steps how to perform conflict resolution?

>      User complaint: It's hard, in some environments, to properly
>                      configure a remote archive which can be written
>                      by more than one person.

Without a superserver (which I oppose), I think this is just the cost of
doing business. For everyone else out there, I share your pain! 


>      User complaint: There isn't a built-in way to prune patch logs
>                      and keep them at a managable size.

To resolve this, we can simply point people at the many scripts that are
out there. For example, in tlacontrib there's a dumb prune-libraries
script that removes all but the latest revision in all versions.


>      User complaint: In some situations, it is hard to know how best
>                      to manage archive cached revisions and the like.

Do you mean the problem with cached revisions not being mirrored
optimally?


>      User complaint: The documentation needs some improvement.
>
>      Tom's Take: Yes, it does.

Sigh. I'll take the fall for this.

>   Two broad themes emerge from the lists of complaints:
>
>   1. The arch project itself needs some process streamlining.
>
>   2. We need new features (higher level commands, mostly) to help
>      "share the love": to make it easier for new users to implement
>      good processes from the start.
>
>   Those are complementary.   We can use the arch project itself as
>   a "sandbox" in which to develop new features that will answer 
>   most of the common complaints from users.   If we do that, we'll
>   also be more efficient and hopefully put to rest the contributor
>   complaint.

I suppose we could go about these first.

> ** Project Process Changes/Stable Branch
>
>   Each release attracts new users.
>
>   Frequent releases with lots of merged in contributions keeps
>   contributors happy.
>
>   Clearly, even though it is important that I be able to spend more
>   time working on new things (within arch and without), it is also
>   important that my going away to think and work on stuff not disrupt
>   the flow of merges and new releases.
>
>   At the same time, without wishing to denegrate any of the
>   contributors, none of the contributors are "good enough" yet 
>   to have authority over what should be merged.   Some of the 
>   key contributors are very close, but I still find that I have
>   to edit contributions from just about everyone.

That's why you're in charge! 

Getting patches accepted in arch is like getting laid. If I go more than
a week without getting laid, I get grumpy. :)

>   So:
>
>     (a) we need a separate "stable" mainline into which 
>         small changes can be merged and from which frequent
>         releases can be cut
>
>     (b) we need to construct the process for the stable 
>         mainline in such a way that I can review and approve
>         (or kick back for revision) contributions at a 
>         very high rate, in spite of my slow modem.

That's what gnuarch.org is for. Your effective bandwidth would go
up considerably if you worked from there. I even mirrored every
archive that I could find so that it would be there within easy
reach. :)

>   So there is some work to do to set up a stable mainline with the
>   desired properties and, if it's to happen anytime soon, I think it's
>   likely I'll need some volunteer help to create the infrastructure.
>
>   I therefore think it's reasonable for the GNU Arch project to
>   shift focus a bit and become a bit more "introspective" -- to focus 
>   for now on its own internal needs and on solving those needs, using
>   arch.

I can march in that direction. Count me in. 

>   As we do that work, at every step, we should be asking ourselves:
>   How can we implement the infrastructure we're building for the 
>   arch project itself in such a way that users can easily emulate
>   that infrastructure in their own projects?
>
>   In other words, I'd like a closed solution and a generalized
>   solution: I'd like the entire infrastructure that we build to be
>   bundled with arch ("closed") and I'd like that infrastructure to be
>   general enough for people to instantiate for other projects
>   ("generalized").  I stated above, under maintainer complaints, my
>   opinion that many user complaints are best answered by the addition
>   of higher level commands.  Development of an improved infrastructure
>   for the arch project itself is precisely the testbed in which such
>   new commands should be thought out and then implemented.

In my humble opinion, that means changing direction entirely, and
devoting our time to distributing an arch centric BTS that is easy to
deploy. 

I absolutely hate the idea of an arch superserver. But if its really
necesary, what about a standalone bug server, the client of which would
be built directly into arch?


>   I think that we should spend some period of time, let's say two
>   weeks, during which the new infrastructure is discussed and roughed
>   out.  I invite everyone but, primarily the main contributors, to
>   help with this planning.  What are _your_ ideas about what is
>   practical and desirable in the project infrastructre?
>
>   A few of my thoughts, particularly some that characterize the
>   "maintainer's perspective":

[long list of things that should be handled]

I think the process we have, with the accomodations you listed above,
would go a long way towards solving most of these.

>       % tla fork
>
>         [hack hack hack]
>
>         % tla submit
>
>      and have their quick fix show up in the merge queue.   The closer
>      we can get to that ideal the better, especially if it is easy for
>      other projects to provide a similar interface.

I suppose this is doable, though there are some catches:

1. This idea implies having to get a copy of address@hidden and
performing a whats-missing. 

2. We'll have to track what patches have already been submitted so as to
avoid duplicates

3. There will need to be an --exclude option for revisions (hack,
commit, aw shit, commit that undoes previous commit)

4. We'd have to build a dumbed down mail client into arch (or something
that can carry equivilant information). This probably means sockets,
which I imagine would mean that we'd take a hit on portability.


I suppose we could go this way, and I'll go this way if you want, but I
really do think that a small handful of scripts for the top maintainer
is really the better solution. Once you've built a small handful of
scripts, you document the process you use, release the scripts, and then
everyone knows how to handle the problems that you do.



> * Release Map
>
>   Release Arch 1.3 should be made soon, prior to any infrastruture 
>   changes.  It should include the security patch from 1.2.1 plus
>   all trivial fixes and improvements in the current merge queue.

I agree. Actually, throw 1.2.1 away. There was *another* exploit for
neon, so just pull in neon-current into your tree.

>   I wonder if one of the established contributors would be interested
>   in preparing release 1.3, but subject to a particular requirement.
>   In particular, would any of you (who know who you are) be willing
>   to prepare a 1.3 branch with the changes for the release divided
>   into clean, separate changesets?   In other words, to make it easy
>   for me to just read the changesets, one by one, veting each before
>   moving on to the next?   This would be close to a "by hand"
>   simulation of the hoped-for stable branch.

I nominate Aaron Bentley, though I personally think he'll decline. 

So failing that.... 

I can gather together patches and sort them into three groups: 

1. No good patches - (I record them and throw them away)

2. obviously good patches - I merge these into
address@hidden/tla--good--1.3

3. policy changing patches - I merge these into 
address@hidden/tla--noidea--1.3

4. patches I can't evaluate - I merge these into
address@hidden/tla--questionable--1.3

I'm more than happy to talk with people about their patches about
cleanups etc. Also, when there's related patches, I'll conflate them
for ya


>
>   Thereafter, we can maintain:
>
>       1.3.X --- the stable branch
>
>         1.4 -- the development branch for the advanced features
>                needed to create our infrastructure
>
>
>
> * A Reminder
>
>   Just because I spend some time working on things that don't
>   immediately produce a new release or carry out a new merge, that
>   doesn't mean I'm not busy doing things that contribute to arch and
>   my other projects.  Really, it's almost the opposite:  mindlessly
>   merging and cranking releases would not be much of a help to the
>   arch project;  working out good plans and then getting them started
>   is a big help, I hope, for the project.   I try to keep my work at
>   least minimally visible via the public mirrors of my archive.

I looked back at your commit dates, and I've got to admit that I'm
flummoxed. I could have sworn that things went something like "Tom pulls
patches, disapears for three weeks, Tom pulls some more patches". But
according to your commit dates, you were doing one or two a day.

Maybe the perception would be different if you set up a cronjob to push
your archive every morning at 3am?

>   GNU Arch is a community supported project, at the moment, both in
>   terms of labor and in terms of money.  The project exists today
>   because of contributions from users like you and I (and satisfied
>   arch users everywhere) are grateful for your support.  Your support
>   allows me and the community generally to keep arch going and
>   improving as we continue down our path of simply and unambiguously
>   _taking_over_ the space of revision control tools on the strength of
>   our fundamentally superior technology.  It is also no small pleasure
>   to work on a project that promises to correct the intrusion on the
>   free software world by BitMover.

Come on you cheap bastards! I'm running out of cash! Send Tom some
money. If two hundred arch users gave up taking their family out to
the movies one night each month, Tom wouldn't have money problems.

Heck. Then maybe he'd have enough money to send *me* some, so that I
could afford to keep the supermirror going!


-- 
James Blackwell          Try something fun: For the next 24 hours, give
Smile more!              each person you meet a compliment!

GnuPG (ID 06357400) AAE4 8C76 58DA 5902 761D  247A 8A55 DA73 0635 7400