[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Gnu-arch-users] the state of the union

From: Tom Lord
Subject: [Gnu-arch-users] the state of the union
Date: Tue, 17 Aug 2004 11:25:45 -0700 (PDT)

>From my perspective....

* News: 1.2.1 release

  1.2.1 is old news now.  Which means that on Tuesday or so the
  official FSF version will get made! :-) Jblack is most excellent.
  All hail....

* News: On Begging for a Living.....

  I'm going off the public good-will dole!  I'll make this more formal
  in the coming days with updates to my website and so forth, but, I'm
  going off the good-will dole.  No more .signature's with paypal
  links.  I'll probably be repurposing the paypal links on my web site
  as pay-per-incident problem help.  More news on why will wait for
  some future message.  But "Yay".  Say, by the way, thanks, everyone,
  for keeping me not only fed and sheltered, but on-line.

  And so now, with this message, rather than pass my hat around, I
  shall doff it.  Looks very smart on me, doesn't it :-) Parting shot
  gratuities will certainly not, of course, be turned down :-), and
  should all of my grand schemes and dreams come to naught, who knows,
  I may one day pass the hat again.  But for now, this is the end of
  that.   And look: nifty: arch exists.  You all helped make it so.

  Of course, parting shot gratuities aside -- if you want to celebrate
  the jblack release, I think it safe to say that he and I both agree
  you should get out your old fashioned checkbook, write a check to
  the FSF, and pop it in the mail.  Preferably to the FSF itself, of
  course.  I think if you poke around a bit you can find
  information on becoming a card carrying member.  So to speak.  (And,
  hey, tell 'em I sent ya': I like to keep an informal score :-)

  Hmm.  Busking aside.....

* CA Perspective: On Arch Competitors Like SVK

  It's sad (in an abstract sort of way) that someone went off to
  implement an arch-like changeset-oriented, distributed system on top
  o' subversion without bothering to try to make it arch compatible.
  People should try to route around that needlessly divisive effort.

  The same goes for Monotone and DARCS and arx: anyone who looks at
  all carefully nowadays can see that arch is a nice general,
  extensible, simple, winning framework in the sense that good ideas
  for those other systems can (in some cases have been) be best
  implemented as hacks on top of arch.

  Before we (the free software community, especially around the revctl
  projects) collectively understood the revision control design space,
  having these separate efforts to best CVS made sense:
  everybody was exploring a different set of ideas and it wasn't clear
  how they could be unified or which ideas were important.

  But over the past couple of years, at least as far as I can see,
  lots and lots of people both in and around all of these separate
  projects have all developed pretty hefty understandings of the
  revision control design space.   The biggest threat to Bitmover now
  isn't arch or any other project: it's the fact that so many more
  people in the free software world who care now _understand_ revision
  control in general, some of them very well.

  So what do we need?

  For one thing, we need changeset orientation.  The Subversion users
  are visibly feeling the pain of its absense.  All of the other
  systems are changeset oriented.  It seems to be essential to smart
  merging.  It's also damn handy for project management and code
  review: look at how arch has evolved so that *all* changes to the
  mainline are now patch-flowing through an issue tracker and several
  layers arch branching and merging.

  In the past, efforts to unify the changeset mechanisms of the
  various projects failed.  The primary sticking point seems to have
  been disagreement about the necessity and desirability of inventory
  tags.  Yet in the months since that debate stalled the unified
  changeset effort, new data has arrived.   For example, here is
  what the Monotone guy says on this topic, followed by my

        FAQ: Why not use more "sophisticated" merge algorithms, as in
             ... Arch

        Answer [....]

               Arch's merges are based on applying patches in various
               orders. While better suited to cherry-picking, only one
               of them (star-merge) seems to provide a 3-way merge;
               the other two are strictly weaker.

  In other words: arch provides all the interesting functionality of
  Monotone's merges (star-merge) and *in addition* supports
  cherry-picking.  (How he managed to interpret this as a weakness of
  Arch is beyond me.)

  Indeed, cherrypicking has proven very useful in Arch.  And 
  the reason cherrypicking works in arch is because of our changeset
  format.  And the reason our changeset format works is because of
  inventory tags.

  And arch is a flexible framework in this area: If we really *want*
  monotone style inventorying (checksum-based manifests) and monotone
  style 3-way tree deltas, THEY ARE EASY AND NATURAL TO ADD TO ARCH.
  The other way around though, if we want to add Arch-style changeset
  functionality to Monotone, is a much more awkward path.

  So I say it's time for the community as a whole to get serious about
  standardizing a changeset format and for the arch community to take
  the lead by insisting on and facilitating a portable mechanism for
  inventory tags.  Anything less and the FOSS world has left the
  valley of "healthy competition" among free software revision control
  projects and entered the realm of "just dicking around and
  infighting for economic and political power".

  What else do we need besides changesets?  Well, other strengths of
  arch, of course: smart merging, distribution, global cooperative
  namespace(s), cryptographic signing (which we lifted out of
  monotone, essentially) .... Other systems have some good ideas
  (e.g., Darcs curious patch algebra) but, as with Monotone-style
  manifests, these can generally be mapped onto arch in a very clean

  What do we need that Arch has got wrong so far?

  Portability to windows, certainly.   But arch is catching up quickly
  in that area and will certainly be fully ported within 6 months
  unless quite a few people suddenly become unable to advance the
  cause.  With sponsorhip, completion and integration of the porting
  effort could be accelerated.   Subversion and Monotone (probably
  others too) got right that portability is a higher priority.

  And we need a system that, in spite of being changeset-oriented,
  feels (at least optionally) more "file oriented":  so you can
  `ci' files and directories or look at individual file histories
  quickly, just like with CVS.   Arch is only part way there in these
  areas, frankly, although we have penciled out the remaining steps
  to a convincing degree, imo.

  In the early days, when we collectively didn't know much, there was
  an actual debate to be had about how to implement file orientation:
  Does it require a fancy delta-compressed version-transactioned
  filesystem (Subversion)?  or can you get by with low-tech
  brute-force techniques combining compressed archives with
  client-side caches (Arch).  The debate is over.  We know the answer.
  The answer is that the Subversion approach can deliver lower
  command-line latency for some operations but that Arch delivers
  lower administration costs, better scalability, and higher
  throughput.  Together the two experiments have shown that, in the
  grand scheme of things, the user-visible performance differences
  between the approaches are noticable and sometimes distracting --
  but frankly pretty small in absolute terms (with oddball edge-case
  freakish exceptions, of course).  Arch, due to its simplicity, is
  arguably the saner foundation for many situations --- but the
  in-principle low-latency of small few-file txns in subversion plus
  the interesting set of trade-offs that enable relatively fast
  retrieval of individual file history: these virtues of Subversion
  should also be further developed, even if not so much for straight
  software-development purposes.  The outcome of the debate is
  basically "We want both" with a slight K.I.S.S.-based bias towards
  arch for many situations.

  And, again, If We'd All Just Get Along we'd *have* both: given a
  standardized "superstructure" of standard-format changesets and
  branching structures and history --- subversion and today's core
  arch would just be alternative (wildly different) implementations of
  the same thing.   Instead of splitting the efforts of volunteers
  into two incompatible projects and perpetuating a very wasteful
  marketing competition to beat out the other guys, we could be using
  our now shared knowledge of the design space to combine efforts.

  And that's where I came in: it's sad (in an abstract sort of way)
  that someone went off to implement an arch-like changeset-oriented,
  distributed system on top o' subversion without bothering to try to
  make it arch compatible.  How wasteful.

* CA Response: Do Not Be Afraid of the XL

  If Jblack's attitude towards XL is at all typical I've done a
  hopelessly bad job of explaining my intentions.

  Ok, no problem.  Forget it.  XL (and furth and pika) still figure
  mightily in my plans but there's no need (and apparently only modest
  opportunity) to explain it all, to much, in advance.   I'll retry
  the "thinking out loud" thing a little later, when there's more
  context built up.

  In the meanwhile, I (and my colleagues, who I'll trick into doing
  most of the work) can, hopefully, contribute a bunch of neat new
  features that have only the barest perceptible connection to XL.
  Stay Tuned.

** Opinion: Business Plan Idea --- Innovation + Free Software

  Some people say that free software licensing stifles innovation
  by removing financial incentives.


  Or maybe not.  From my perspective, we're awefully close to an 
  innovators paradise in free software.

  A software innovator should be someone who is able and willing to
  beg, borrow, steal, or spend enough money to self-fund for the first
  couple (or few) months of development on a new idea.  In my case
  (arch's case) it was "borrow": faced with ruin anyway, i maxed out
  some credit cards on food rent and ISP service to bootstrap arch
  (and I still have some bad debts to show for it).  But that's ok.
  Let's say that's fair.  Alternatives would include working off hours
  while holding a day job and just having some good 'ol fashioned
  savings.  Entrepreneurialism thrives on risk taking.  So, sure, a
  software innovator in the free software world must be able to lay
  down the first few bucks himself.  Put his money where his mouth is.

  But then what happened in the early days of arch?  A few people
  adopted it, brave souls.  Otherwise there'd be no project today.
  But also: about 5, maybe 10 "luminaries" --- hackers you've heard of
  --- looked it over and said (usually privately), roughly, "Hmm.
  Interesting.  You might have something there."

  Praise indeed.

  That's where what happened with arch (so far) diverged from what
  *should* have happened with arch.

  At that point, with 5 or 10 luminaries saying, "Hmmm... *maybe*",
  I *should* have been able to monitize those endorsements.   Suppose
  that those luminaries each could (and did) give me a $350/m grant
  for 12 months ... suppose that that was the tool they possessed for
  saying "Hmm... *maybe*.".   Then that would have been a cheap R&D
  line item for their host organizations.   It would have added up to
  enough to let me continue to let my debts ride (honorably) and
  continue my entrepreneurship.   I'm not saying that literally
  selecting an illuminati and making them into a micro-grant committee 
  is the only way to do this -- it's just one concrete possibility
  that points out the general idea:

       ~ innovator takes a little risk

       ~ industry R&D budgets take on a little risk to extend
         the reach of innovators with early but promising projects

  That's a "faster, cheaper, better" approach to incubating new
  projects.   Companies (IBM, RH, Novell, HP, Sun) could certainly
  afford to pony up a few million bucks over a couple of years to
  experiment with this approach.

  Why would innovators bother, though?  A program of micro-grants
  alone would make free software R&D, at best, a break-even career
  (your living expenses equal your pay) with book deals.   Hardly
  much of an inticement.

  In a proprietary software world, innovators (or more usually their
  employers) have the "leverage" licensing fees.   In theory, they
  initially own all the revenue their innovations generate and they
  can usefully sell off that right.   

  The free software world denies any direct linkage between future
  revenues and rights to use a software innovation.  We find such
  linkages unnatural, irresponsible, awkward, and even morally
  offensive.  Yet without such linkages, can their be a profit

  There can be and there is.  The large companies profit mightily from
  free software innovation.  They profit all the moreso precisely
  because the software is not awkwardly linked to revenue-collection
  rights.   Software becomes, for these companies, just (useful) bits
  (there's a thought!) and the better the bits they have, the more
  valuable these companies can make the software-*related* services
  that they offer.   So there is plenty of "future revenue" in free
  software innovation --- it's just that the chief economic virtue of
  that future revenue is that it can be "owned" only on the basis 
  providing the best services around the innovation.

  It's oversimplified, but only a little, to say that RH, Novell,
  Linspire, HP, and Sun are placing various kinds of "bets" about what
  free software innovation will bring in the coming years.  Some
  subset of these guys, we presume, is going to make money.  There's
  the future revenue and it goes to the fittest service providers.

  Where does that leave our innovators?  The logical terminal point 
  of the microgrant game I described above is:

        ~ industry dubs the project "winning" and grants the
          innovator a bouquet of stock from various 
          companies who use free software

  What is a "winning" project?  When the companies start spending
  money on an internal effort derived from the project, then the
  project is winning.  One can imagine a kind of self-imposed tax: for
  the first N hires (or reassignments) related to project X, IBM can
  chip in K dollars each to the "reward" fund that goes to the
  innovator.  (This is a good way for companies like IBM or Sun to
  dispose of stock in free software companies in which they wish to
  invest, to ensure their existence, but not particularly to control
  or acquire --- IBM isn't limited to rewarding innovators with IBM
  stock.  Perhaps ideally the various companies should just start a
  shared mutual fund for innovators and give away shares in that while
  directing the fund to reap the indirect rewards of free software

  So I propose a three step game:

    1. Innovators take some risk on the order of 3 months of 
       self-funded effort.

    2. Free software companies institute a micro-grant program that
       can, in principle, cover the month-to-month living costs of,
       say, 500 designated innovators.   Innovators whose self-funded
       projects pass muster with more than a few known-smart hackers
       can receive micro-grants to extend the reach of their
       self-funding for, say, 6m to 2y.

    3. When a grant-receiving innovator's project begins to generate
       actual product/profit-oriented spending among the free software
       companies, the grant-giving organization "pays out" to the
       innovator with shares in a free-software-based mutual fund.

  *If* the companies can get their act together to do that, the
  existence of that game gives rise to another:  Can one be a
  "professional innovator"?  Can one "game" that micro-grant game
  into a career?   I sure hope so....

  Ultimately, free software innovation is a lot like professional
  sports, especially individual competitor sports like golf.  There is
  a huge social value to having these skilled players, to granting
  them the freedom to develop their game.  Smart business folk have
  little difficulty building up rather large industries around,
  ultimately, not simply the celebrity of the players, but just the
  basic facts on the ground of tournament games that they play.
  Millions and millions of dollars are invested to produce a
  performance by Tiger Woods in which he wins a prize of less than a
  million dollars.  Yet this is economically rational for the
  investors: the profundity and impact of the game he plays is so
  intrinsincly valuable that the investors have learned to benefit
  from its mere existence --- largely by making that game visible to a
  very large and interested audience.  Software innovation is little
  different.   There is *huge* value from having software innovators 
  "do their thing".   And, no, they won't all be Tiger Woods.  Most
  won't be anything close.   Professional innovators will live mostly
  on prize money, not endorsements.   And that's as it should be: a
  sea of competent innovators, framing the occaisional stellar
  revolutionary, such revolutionary innovators being reliably produced
  by the existence of the sea of the competent.

  The economists will, no doubt, point out no firm has incentive to
  pool free software R&D investment with another because, between
  them, it is a zero-sum money-sink from which neither firm
  gains advantage.   Worse, such a self-imposed tax worsens the
  position of both competitors against "free rider" competitors
  consuming the same innovation.

  They are wrong, though.   Free software innovation creates value
  for the *customers* of these companies.   FS innovation doesn't
  exist to give one free software firm a competitive advantage over
  another;  rather, FS innovation exists to ensure and expand the
  future revenues of the free software industry as a whole!

  The (straw man) economists making the "zero-sum" argument are
  ignoring the role of enlightened customers.   What RH, IBM, Novell,
  HP, and Sun ought to do is:

        1) coordinate among themselves (or any substantial subset)
           to implement the micro-grant/prize game

        2) *brand* that and educate their customers that, as vendors,
           they are preferable because some of the fees they charge
           goes to free software innovators.  Vendors not
           participating in the micro-grant/prize game can be fairly
           described as "free riders" who are reckless and
           irresponsible in their approach to free software.  Such
           readily identifiable free riders are to be boycotted.

* Parting Shots

  The other day I said something clever but I think it is not
  original.  I can't think of who said it first, though.  
  It goes like this:

        The end-state of every substantial, successful free software 
        project is a bureaucratic power struggle.

  The phrase "power struggle" doesn't have to mean anything untoward
  or unpleasent.  It doesn't mean that there has to be losers.
  Sometimes a theoretical power struggle is a very useful tool, often
  when the power struggle is so icky that the only way to win is to
  not bother playing.  The existence of the *theoretical* power
  struggle means that there's no need for a *real* power struggle.

  The best example of that principle is the GCC project: very, very
  indirectly ruled over (nominally) by the FSF; access-rights
  ultimately the authority of a constitutional steering committee;
  day-to-day processes structured around a release manager and
  committers (with rights to specific parts of the tree....).  The GCC
  project is a freaking virtual nation!  It's a terribly
  anti-democratic republic in which new citizens are admitted by
  sponsorship or rough consensus.   This seems to work very well.  It
  seems to be a good fit for what it feels like to work with large
  projects.  People seem generally happy about it.

  Arch is entering that phase.   Soap-operas among the core developers
  aside, the bottom line is that we are erecting a bureaucratically
  driven mainline, making the game just tedious and complicated enough
  to not be worth trying to "game";  making the game hopefully
  consonant with what's best for arch.

  That's great for me, under the circumstances.  I can start to step
  back pretty damn far from the "benevolant dictator" role and the
  arch project can have a bit of a life of its own, with nobody in
  particular in charge (except as a last resort circuit breaker).

  GCC, Debian.....   now it's arch's turn.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]