[Gnu-arch-users] Arch Roadmap Draft (the anticipated part 3)

[Tom Lord]

                         Arch Project Roadmap


* Background

  For a few weeks, some community members have quietly pressured me to
  make a roadmap for future arch development and for some of that time
  I've promised to do so and started work on it.
  
  As gnu-arch-users readers are aware, I started stating the plans in a
  "bottom up" way: listing a large number of feature plans and then some
  low-level details of furth.  This makes a lot of sense to _me_, at
  least, because before I can be comfortable summing up the "big
  picture" I have to be confident that the big picture sums up a set of
  definately doable, real ideas rather than being a hopelessly vague set
  of "goals" like "improve the user interface" or "simplify
  deployment".  In some sense, the "big picture" is just a set of
  smaller steps, each of which can clearly be done, and which relate
  to one another in synergistic ways.
  
  While working on the plans and on enough of the code to convince
  myself the plans could be done, I paused merging and release making.
  It seems to me that between 1.2 and 1.2.1 we have stable and
  reasonable releases out there for people to use.  It seems to me that
  while the set of pending submissions has some nice things in it, none
  of them are especially time-critical and, meanwhile, I'll have a better
  idea of which and how to merge those once I have the long-term plan in
  place.
  
  The combination of those two actions, posting lower level feature
  plans and pausing merging and release making, drew an _interesting_
  response from the user community.   Basically, it pissed a number of
  people off and they started complaining at me.
  
  It's not really desirable to piss people off but sometimes, as in this
  case, it couldn't really be avoided while at the same time doing my
  actual job of designing and building out arch.
  
  However, while it wasn't my intent, it turns out that it was at least
  a little bit _useful_ to piss people off because among their
  complaints are some insights into what priorities will, in the longer
  term, keep the user and contributor communities happy.

  Here's a roadmap for future Arch development.  I suppose that this
  counts as a "draft": revisions may or may not be needed depending on
  the feedback from various quarters.



* Overview

  The roadmap comes in two parts:

    ~ Issues to Address
    ~ Plans of Action

  In outline form, these expand to:


    ~ Issues to Address
    ~~ user complaints
    ~~ contributor complaints
    ~~ maintainer complaints

    ~ Plans of Action
    ~~ Strategic Overview
    ~~ Project Process Changes/Stable Branch
    ~~ Feature Plans and Plans of Attack
    ~~ Contingencies
    ~~ Release Map



* Issues to Address

  The roadmap is intended to address _real_ needs, concerns, and
  (most importantly) opportunities.   It would easy to just 
  "make up" a direction to go, but it seems to be far too late
  in the lifecycle of the project to do that.   If the real needs
  aren't met, there are these days enough people who will complain
  about that the plan won't fly.

  I've classified each of the issues to address by an (idealized)
  classification of who is most directly concerned with 
  the issue: users, contributors, or me, the maintainer.

  The category "Users" represents issues that arise in day
  to day _use_ of arch.

  "Contributors" represents issues that pertain to the patch-flow
  and release-making within the GNU Arch project from the 
  perspective of people other than me who contribute code.

  "Maintainer" issues are those that express my concerns about
  the design of Arch, based on my observations of how it is being
  used and my knowledge and beliefs about what is possible.


** User Complaints

  A simple, short grocery list, in no particular order:

  1. Revlibs glitch in some NFS environments in a way that
     requires them to be rebuilt frequently.  (This is fixed
     in mainline but not yet in a release.)

  2. There are too many commands.

  3. Some of the error messages are still bogus.

  4. Tagging methods are hard to understand and use.

  5. Performance on large trees is not up to snuff yet
     for some operations.

  6. It's hard, in some environments, to properly configure
     a remote archive which can be written by more than
     one person.

  7. There isn't a bundled graphical merge tool.

  8. There isn't a bundled GUI.

  9. There isn't a built-in way to prune patch logs and 
     keep them at a managable size.

  10. In some situations, it is hard to know how best to 
      manage archive cached revisions and the like.

  11. The documentation needs some improvement.


** Contributor Complaints

  1. Since April, Tom has been much too slow (essentially halted) at
     processing submissions.


** Maintainer Complaints

  1. Arch is underutilized.   No, I don't mean that there
     aren't yet enough users (which sure, is likely to be true
     for a very long time).   Rather, the existing users (including 
     me) aren't making as good a use of arch as we could be.
     There's a fairly large gap there, I think.

     One example is the very ad hoc way that we exchange
     patches, even when we're using arch for that exchange.
     It would currently be impossible (just because we're
     sloppy and inconsistent about patch submission) to 
     do something obvious and useful like set up a browser
     that can show all of the changes that are in the 
     Bug Goo queue.

     Another example is the fairly ad hoc (and inconsistent, over
     time) way that I tag releases.   If you want to go back
     and get various historic releases out of the archive, you
     have to do some poking around to figure out how the contents
     of each release were recorded.


  2. Too many conflicting demands on my time suggest the need for
     some process improvements.

     There's been grousing, lately, about my backlog of merges.

     The actual arch work of processing these changes can be pretty
     fast but I'm hampered by my slow modem.   I have to wait for
     changes to be downloaded locally, I have to push my merges
     back upstream.   I must cut releases on my local machine yet
     the subsequent uploads, announcement posting, mirror updating,
     and so forth takes a substantial part of a work day for each 
     release:  a day mostly spent just waiting and checking
     periodically that modem lights are still blinking.   As
     streamlined as even something like FreshMeat is it's _still_ 
     a disruptive pain-in-the-ass to go through all that for each
     release.

     Big deal, right?   Except that conflicting with the need for 
     me to do that busy work is the need for me to _think_ about
     where to take arch from here and to actually do that work.
     And, there's demands on my time that are not _directly_ arch 
     related such as hackerlab and pika hacking.


  3. The market for arch-related labor exists, but there aren't 
     enough customers.  Therefore, there is a danger of corruption
     of the project.

     Of course we want the market to grow in dollary terms, that's 
     obvious.

     What may be less obvious is that currently, there are only a
     small number of employers who pay for arch work.  Consequently,
     there is a lack of diversity of interests in the marketplace and,
     not to be chicken little, but if we aren't careful, the interests
     of those few employers might come to dominate the future of arch,
     at the expense of our larger community interests.

     We need to make arch easier to deploy in order to attract 
     new commercial interests to it.   We need to make sure that no
     one or two employers is in a position to "take over" arch.


   4. I share some of the user complaints or complaints close them.
     Specifically:

     User complaint:  There are too many commands.

     Tom's take: There aren't enough commands.   What's missing are
      a few higher-level commands that capture the most common and
      most recommended use patterns of arch.   Once those are in
      place, most users can focus on just those few commands and
      ignore the rest.   An improved help system can focus attention
      on just those few commands.

      In other words: Arch doesn't have too many commands but,
      currently, average users have to use too many of the commands
      to do their work.   Consequently, they tend to be a little 
      daunted at the number of commands they have to learn to
      get started, and the help system listing of commands has
      to list an intimidating number of commands.



     User complaint: Some of the error messages are still bogus.

     Tom's Take: A valid complaint that continues to be answered 
       by incremental improvements.


     User complaint: Tagging methods are hard to understand and use.

     Tom's Take: Again, some higher-level help is probably needed
       to capture common patterns.


     User complaint: Performance on large trees is not up to snuff yet
       for some operations.

     Tom's Take: The largest improvements needed in this area are:
       summary deltas; fast, general, selective commit; and improved
       ease-of-use for (a generalization of) configs.


     User complaint: There isn't a bundled graphical merge tool.
     User complaint: There isn't a bundled GUI.

     Tom's Take: The bottlenecks here are my programming environment
       at home and my time.  The roadmap aims to improve the use of my
       time, but the programming environment is a harder problem.  In
       particular, I use a somewhat (by today's standards) slow
       and low-memory machine with limited disk space running an old
       operating system and X11 release.   I simply do not have the
       facilities on hand to _build_ most of the candidate merge tools
       let alone contemplate integrating them with arch.   Even if
       someone else were to do that work, I lack the facilities for 
       examining and testing their results!


     User complaint: It's hard, in some environments, to properly
                     configure a remote archive which can be written
                     by more than one person.

     User complaint: There isn't a built-in way to prune patch logs
                     and keep them at a managable size.

     User complaint: In some situations, it is hard to know how best
                     to manage archive cached revisions and the like.


     Tom's Take: These are all aspects of the more general problem
                 that it isn't easy enough yet to deploy arch in
                 some slightly complex situations.   I think that 
                 the solution is the addition of some additional
                 high-level commands (and support for those) 
                 to further automate some "best practices" for 
                 deployment.


     User complaint: The documentation needs some improvement.

     Tom's Take: Yes, it does.




* Plans of Action

  So, given all those complaints, what's to do?


** Strategic Overview

  Two broad themes emerge from the lists of complaints:

  1. The arch project itself needs some process streamlining.

  2. We need new features (higher level commands, mostly) to help
     "share the love": to make it easier for new users to implement
     good processes from the start.

  Those are complementary.   We can use the arch project itself as
  a "sandbox" in which to develop new features that will answer 
  most of the common complaints from users.   If we do that, we'll
  also be more efficient and hopefully put to rest the contributor
  complaint.




** Project Process Changes/Stable Branch

  Each release attracts new users.

  Frequent releases with lots of merged in contributions keeps
  contributors happy.

  Clearly, even though it is important that I be able to spend more
  time working on new things (within arch and without), it is also
  important that my going away to think and work on stuff not disrupt
  the flow of merges and new releases.

  At the same time, without wishing to denegrate any of the
  contributors, none of the contributors are "good enough" yet 
  to have authority over what should be merged.   Some of the 
  key contributors are very close, but I still find that I have
  to edit contributions from just about everyone.

  So:

    (a) we need a separate "stable" mainline into which 
        small changes can be merged and from which frequent
        releases can be cut

    (b) we need to construct the process for the stable 
        mainline in such a way that I can review and approve
        (or kick back for revision) contributions at a 
        very high rate, in spite of my slow modem.

  In _theory_, this is a piece of cake with arch and setting
  it up will make for a great model of "best practices".

  In _practice_, what we have right now is most of the 
  infrastructure _parts_ for such a process, but scattered
  around the room, in parts, awaiting consolidation and 
  assembly.

  So there is some work to do to set up a stable mainline with the
  desired properties and, if it's to happen anytime soon, I think it's
  likely I'll need some volunteer help to create the infrastructure.

  I therefore think it's reasonable for the GNU Arch project to
  shift focus a bit and become a bit more "introspective" -- to focus 
  for now on its own internal needs and on solving those needs, using
  arch.

  As we do that work, at every step, we should be asking ourselves:
  How can we implement the infrastructure we're building for the 
  arch project itself in such a way that users can easily emulate
  that infrastructure in their own projects?

  In other words, I'd like a closed solution and a generalized
  solution: I'd like the entire infrastructure that we build to be
  bundled with arch ("closed") and I'd like that infrastructure to be
  general enough for people to instantiate for other projects
  ("generalized").  I stated above, under maintainer complaints, my
  opinion that many user complaints are best answered by the addition
  of higher level commands.  Development of an improved infrastructure
  for the arch project itself is precisely the testbed in which such
  new commands should be thought out and then implemented.

  Note that this approach makes building the new infrastructure harder
  than it might otherwise be.  If we didn't care about turning the
  infrastructure project into new features, we could simply hack up ad
  hoc solutions, make sure they run well on gnuarch.org, and leave it
  at that.  We could say to others "just duplicate the kind of work we
  did".    But since I do care about turning the infrastructure
  project into the way new features are added, then we can't hack up
  ad hoc solutions: we have to hack up clean, portable, and broadly
  applicable solutions.

  That ambition _may_ prove to be a problem.   The risk is that the 
  infrastructure project goes too slow and so the solution to the
  contributor complaint takes too long to arrive.   I'll tentatively
  assume that this problem won't occur: but let's also be wary that it
  might and, if it does, we'll need to adjust the strategy a little
  bit by first creating a _provisional_ infrastructure for the arch
  project and then working on turning that provisional infrastructure
  into new, polished features.

  The bottom line: who (especially among established contributors) is
  game for this?  Can we muster more effort than I can provide alone
  for persuing this strategy?



** Feature Plans and Plans of Attack

  I shouldn't be the only one who "stops to think".

  Remember, please, that one of my criticisms of the Subversion 
  project is that, when they should have, they failed to pause
  and take stock before moving forward.   The need to stop and 
  think for a bit is intrinsic to complex projects like Arch.
  It's just plain healthy and should be a no brainer, from time
  to time.

  I have my own opinions about what the improved infrastructure for
  the Arch project should look like.   I'll briefly share some of
  those here but:

  I think that we should spend some period of time, let's say two
  weeks, during which the new infrastructure is discussed and roughed
  out.  I invite everyone but, primarily the main contributors, to
  help with this planning.  What are _your_ ideas about what is
  practical and desirable in the project infrastructre?

  A few of my thoughts, particularly some that characterize the
  "maintainer's perspective":

  1. We need a standard for how to submit changes.   Yes, yes,
     if you tell me to cherry-pick this or that, or star-merge
     from here or there, sure, I can do that.   But having to 
     parse and implement every merge request by hand is a big
     bottleneck.   It would make much more sense to ask each contributor
     to spend an extra 5 or 20 minutes preparing a submission in a
     standard form than to ask me to spend 20 minutes or an hour
     trying to figure out and implement whatever is meant by 
     an ad hoc merge request.

     It's just a simple shift of labor.  I don't want to cherry
     pick contributions anymore or have to wade through weird merge
     requests.   Fortunately, arch means that we can take that
     part of the work of merging and move it from maintainer to 
     contributor.   It's easier for the contributor to do and, while
     it means more work for contributors, overall, it's the more 
     efficient approach.


  2. I shouldn't need to download proposed changes to stable.  For
     most _simple_ changes, I shouldn't even need to install them in a
     project tree until I merge from the stable mainline into which
     they've been merged.

     Rather, I'll be very happy when I can visit a web page that
     lists the submission queue, click through the diffs each
     submission contains, click through to the results of an automated
     merge of those diffs, click through to the results of a test
     suite run of those diffs, and click to indicate my approval
     or rejection, turning approvals into automatic commits to the
     stable mainline.

     I think that we can drop the time it takes to process a
     simple submission from "most of an hour" to just a few minutes.
     That's the difference between "something I'll postpone because
     I don't want to spend most-of-an-hour on it" and "something I'll
     do immediately because it only takes a few minutes."


  3. It should be easier for casual contributors to contribute.

     A platonic ideal might be for a user who has just a tree
     that came from untarring a distribution to be able to say:

        % tla fork

        [hack hack hack]

        % tla submit

     and have their quick fix show up in the merge queue.   The closer
     we can get to that ideal the better, especially if it is easy for
     other projects to provide a similar interface.


  I think that the "FEATURE PLANS" I've posted so far, plus furth, 
  are rough specs for most of the raw functionality we need to satisfy
  the three goals above.

  The bottom line: Contributors, please stop yelling at me.  Instead,
  let's set up this infrastructure that will answer your complaints
  and provide a good sandbox for answering the most common user
  complaints.   A good place to start is to help design the
  infrastruture and then help to implement it.   What are your
  thoughts on those matters?


* Contingencies

  In some sense, this roadmap reflects a fork in the road.   It might
  flop or it might be what we actually do.

  It will flop if the only responses are more complaints.

  It will be what we do if some of the current complainers step up to 
  work on the barn raising.


* Release Map

  Release Arch 1.3 should be made soon, prior to any infrastruture 
  changes.  It should include the security patch from 1.2.1 plus
  all trivial fixes and improvements in the current merge queue.

  I wonder if one of the established contributors would be interested
  in preparing release 1.3, but subject to a particular requirement.
  In particular, would any of you (who know who you are) be willing
  to prepare a 1.3 branch with the changes for the release divided
  into clean, separate changesets?   In other words, to make it easy
  for me to just read the changesets, one by one, veting each before
  moving on to the next?   This would be close to a "by hand"
  simulation of the hoped-for stable branch.

  Thereafter, we can maintain:

        1.3.X --- the stable branch

        1.4 -- the development branch for the advanced features
               needed to create our infrastructure



* A Reminder

  Just because I spend some time working on things that don't
  immediately produce a new release or carry out a new merge, that
  doesn't mean I'm not busy doing things that contribute to arch and
  my other projects.  Really, it's almost the opposite:  mindlessly
  merging and cranking releases would not be much of a help to the
  arch project;  working out good plans and then getting them started
  is a big help, I hope, for the project.   I try to keep my work at
  least minimally visible via the public mirrors of my archive.

  GNU Arch is a community supported project, at the moment, both in
  terms of labor and in terms of money.  The project exists today
  because of contributions from users like you and I (and satisfied
  arch users everywhere) are grateful for your support.  Your support
  allows me and the community generally to keep arch going and
  improving as we continue down our path of simply and unambiguously
  _taking_over_ the space of revision control tools on the strength of
  our fundamentally superior technology.  It is also no small pleasure
  to work on a project that promises to correct the intrusion on the
  free software world by BitMover.

  One of the best ways to help the Arch project continue to progress
  (and to help my other projects) is with financial contributions:

     
https://www.paypal.com/xclick/business=lord%40emf.net&item_name=support+for+arch+and+other+free+software+efforts+by+tom+lord&no_note=1&tax=0&currency_code=USD

     and

     address@hidden for www.moneybookers.com payments.


-t