[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Gnu-arch-users] the state of the union
From: |
Tom Lord |
Subject: |
[Gnu-arch-users] the state of the union |
Date: |
Tue, 17 Aug 2004 11:25:45 -0700 (PDT) |
>From my perspective....
* News: 1.2.1 release
1.2.1 is old news now. Which means that on Tuesday or so the
official FSF version will get made! :-) Jblack is most excellent.
All hail....
* News: On Begging for a Living.....
I'm going off the public good-will dole! I'll make this more formal
in the coming days with updates to my website and so forth, but, I'm
going off the good-will dole. No more .signature's with paypal
links. I'll probably be repurposing the paypal links on my web site
as pay-per-incident problem help. More news on why will wait for
some future message. But "Yay". Say, by the way, thanks, everyone,
for keeping me not only fed and sheltered, but on-line.
And so now, with this message, rather than pass my hat around, I
shall doff it. Looks very smart on me, doesn't it :-) Parting shot
gratuities will certainly not, of course, be turned down :-), and
should all of my grand schemes and dreams come to naught, who knows,
I may one day pass the hat again. But for now, this is the end of
that. And look: nifty: arch exists. You all helped make it so.
Of course, parting shot gratuities aside -- if you want to celebrate
the jblack release, I think it safe to say that he and I both agree
you should get out your old fashioned checkbook, write a check to
the FSF, and pop it in the mail. Preferably to the FSF itself, of
course. I think if you poke around gnu.org a bit you can find
information on becoming a card carrying member. So to speak. (And,
hey, tell 'em I sent ya': I like to keep an informal score :-)
Hmm. Busking aside.....
* CA Perspective: On Arch Competitors Like SVK
It's sad (in an abstract sort of way) that someone went off to
implement an arch-like changeset-oriented, distributed system on top
o' subversion without bothering to try to make it arch compatible.
People should try to route around that needlessly divisive effort.
The same goes for Monotone and DARCS and arx: anyone who looks at
all carefully nowadays can see that arch is a nice general,
extensible, simple, winning framework in the sense that good ideas
for those other systems can (in some cases have been) be best
implemented as hacks on top of arch.
Before we (the free software community, especially around the revctl
projects) collectively understood the revision control design space,
having these separate efforts to best CVS made sense:
everybody was exploring a different set of ideas and it wasn't clear
how they could be unified or which ideas were important.
But over the past couple of years, at least as far as I can see,
lots and lots of people both in and around all of these separate
projects have all developed pretty hefty understandings of the
revision control design space. The biggest threat to Bitmover now
isn't arch or any other project: it's the fact that so many more
people in the free software world who care now _understand_ revision
control in general, some of them very well.
So what do we need?
For one thing, we need changeset orientation. The Subversion users
are visibly feeling the pain of its absense. All of the other
systems are changeset oriented. It seems to be essential to smart
merging. It's also damn handy for project management and code
review: look at how arch has evolved so that *all* changes to the
mainline are now patch-flowing through an issue tracker and several
layers arch branching and merging.
In the past, efforts to unify the changeset mechanisms of the
various projects failed. The primary sticking point seems to have
been disagreement about the necessity and desirability of inventory
tags. Yet in the months since that debate stalled the unified
changeset effort, new data has arrived. For example, here is
what the Monotone guy says on this topic, followed by my
translation:
FAQ: Why not use more "sophisticated" merge algorithms, as in
... Arch
Answer [....]
Arch's merges are based on applying patches in various
orders. While better suited to cherry-picking, only one
of them (star-merge) seems to provide a 3-way merge;
the other two are strictly weaker.
In other words: arch provides all the interesting functionality of
Monotone's merges (star-merge) and *in addition* supports
cherry-picking. (How he managed to interpret this as a weakness of
Arch is beyond me.)
Indeed, cherrypicking has proven very useful in Arch. And
the reason cherrypicking works in arch is because of our changeset
format. And the reason our changeset format works is because of
inventory tags.
And arch is a flexible framework in this area: If we really *want*
monotone style inventorying (checksum-based manifests) and monotone
style 3-way tree deltas, THEY ARE EASY AND NATURAL TO ADD TO ARCH.
The other way around though, if we want to add Arch-style changeset
functionality to Monotone, is a much more awkward path.
So I say it's time for the community as a whole to get serious about
standardizing a changeset format and for the arch community to take
the lead by insisting on and facilitating a portable mechanism for
inventory tags. Anything less and the FOSS world has left the
valley of "healthy competition" among free software revision control
projects and entered the realm of "just dicking around and
infighting for economic and political power".
What else do we need besides changesets? Well, other strengths of
arch, of course: smart merging, distribution, global cooperative
namespace(s), cryptographic signing (which we lifted out of
monotone, essentially) .... Other systems have some good ideas
(e.g., Darcs curious patch algebra) but, as with Monotone-style
manifests, these can generally be mapped onto arch in a very clean
way.
What do we need that Arch has got wrong so far?
Portability to windows, certainly. But arch is catching up quickly
in that area and will certainly be fully ported within 6 months
unless quite a few people suddenly become unable to advance the
cause. With sponsorhip, completion and integration of the porting
effort could be accelerated. Subversion and Monotone (probably
others too) got right that portability is a higher priority.
And we need a system that, in spite of being changeset-oriented,
feels (at least optionally) more "file oriented": so you can
`ci' files and directories or look at individual file histories
quickly, just like with CVS. Arch is only part way there in these
areas, frankly, although we have penciled out the remaining steps
to a convincing degree, imo.
In the early days, when we collectively didn't know much, there was
an actual debate to be had about how to implement file orientation:
Does it require a fancy delta-compressed version-transactioned
filesystem (Subversion)? or can you get by with low-tech
brute-force techniques combining compressed archives with
client-side caches (Arch). The debate is over. We know the answer.
The answer is that the Subversion approach can deliver lower
command-line latency for some operations but that Arch delivers
lower administration costs, better scalability, and higher
throughput. Together the two experiments have shown that, in the
grand scheme of things, the user-visible performance differences
between the approaches are noticable and sometimes distracting --
but frankly pretty small in absolute terms (with oddball edge-case
freakish exceptions, of course). Arch, due to its simplicity, is
arguably the saner foundation for many situations --- but the
in-principle low-latency of small few-file txns in subversion plus
the interesting set of trade-offs that enable relatively fast
retrieval of individual file history: these virtues of Subversion
should also be further developed, even if not so much for straight
software-development purposes. The outcome of the debate is
basically "We want both" with a slight K.I.S.S.-based bias towards
arch for many situations.
And, again, If We'd All Just Get Along we'd *have* both: given a
standardized "superstructure" of standard-format changesets and
branching structures and history --- subversion and today's core
arch would just be alternative (wildly different) implementations of
the same thing. Instead of splitting the efforts of volunteers
into two incompatible projects and perpetuating a very wasteful
marketing competition to beat out the other guys, we could be using
our now shared knowledge of the design space to combine efforts.
And that's where I came in: it's sad (in an abstract sort of way)
that someone went off to implement an arch-like changeset-oriented,
distributed system on top o' subversion without bothering to try to
make it arch compatible. How wasteful.
* CA Response: Do Not Be Afraid of the XL
If Jblack's attitude towards XL is at all typical I've done a
hopelessly bad job of explaining my intentions.
Ok, no problem. Forget it. XL (and furth and pika) still figure
mightily in my plans but there's no need (and apparently only modest
opportunity) to explain it all, to much, in advance. I'll retry
the "thinking out loud" thing a little later, when there's more
context built up.
In the meanwhile, I (and my colleagues, who I'll trick into doing
most of the work) can, hopefully, contribute a bunch of neat new
features that have only the barest perceptible connection to XL.
Stay Tuned.
** Opinion: Business Plan Idea --- Innovation + Free Software
Some people say that free software licensing stifles innovation
by removing financial incentives.
Maybe.
Or maybe not. From my perspective, we're awefully close to an
innovators paradise in free software.
A software innovator should be someone who is able and willing to
beg, borrow, steal, or spend enough money to self-fund for the first
couple (or few) months of development on a new idea. In my case
(arch's case) it was "borrow": faced with ruin anyway, i maxed out
some credit cards on food rent and ISP service to bootstrap arch
(and I still have some bad debts to show for it). But that's ok.
Let's say that's fair. Alternatives would include working off hours
while holding a day job and just having some good 'ol fashioned
savings. Entrepreneurialism thrives on risk taking. So, sure, a
software innovator in the free software world must be able to lay
down the first few bucks himself. Put his money where his mouth is.
But then what happened in the early days of arch? A few people
adopted it, brave souls. Otherwise there'd be no project today.
But also: about 5, maybe 10 "luminaries" --- hackers you've heard of
--- looked it over and said (usually privately), roughly, "Hmm.
Interesting. You might have something there."
Praise indeed.
That's where what happened with arch (so far) diverged from what
*should* have happened with arch.
At that point, with 5 or 10 luminaries saying, "Hmmm... *maybe*",
I *should* have been able to monitize those endorsements. Suppose
that those luminaries each could (and did) give me a $350/m grant
for 12 months ... suppose that that was the tool they possessed for
saying "Hmm... *maybe*.". Then that would have been a cheap R&D
line item for their host organizations. It would have added up to
enough to let me continue to let my debts ride (honorably) and
continue my entrepreneurship. I'm not saying that literally
selecting an illuminati and making them into a micro-grant committee
is the only way to do this -- it's just one concrete possibility
that points out the general idea:
~ innovator takes a little risk
~ industry R&D budgets take on a little risk to extend
the reach of innovators with early but promising projects
That's a "faster, cheaper, better" approach to incubating new
projects. Companies (IBM, RH, Novell, HP, Sun) could certainly
afford to pony up a few million bucks over a couple of years to
experiment with this approach.
Why would innovators bother, though? A program of micro-grants
alone would make free software R&D, at best, a break-even career
(your living expenses equal your pay) with book deals. Hardly
much of an inticement.
In a proprietary software world, innovators (or more usually their
employers) have the "leverage" licensing fees. In theory, they
initially own all the revenue their innovations generate and they
can usefully sell off that right.
The free software world denies any direct linkage between future
revenues and rights to use a software innovation. We find such
linkages unnatural, irresponsible, awkward, and even morally
offensive. Yet without such linkages, can their be a profit
incentive?
There can be and there is. The large companies profit mightily from
free software innovation. They profit all the moreso precisely
because the software is not awkwardly linked to revenue-collection
rights. Software becomes, for these companies, just (useful) bits
(there's a thought!) and the better the bits they have, the more
valuable these companies can make the software-*related* services
that they offer. So there is plenty of "future revenue" in free
software innovation --- it's just that the chief economic virtue of
that future revenue is that it can be "owned" only on the basis
providing the best services around the innovation.
It's oversimplified, but only a little, to say that RH, Novell,
Linspire, HP, and Sun are placing various kinds of "bets" about what
free software innovation will bring in the coming years. Some
subset of these guys, we presume, is going to make money. There's
the future revenue and it goes to the fittest service providers.
Where does that leave our innovators? The logical terminal point
of the microgrant game I described above is:
~ industry dubs the project "winning" and grants the
innovator a bouquet of stock from various
companies who use free software
What is a "winning" project? When the companies start spending
money on an internal effort derived from the project, then the
project is winning. One can imagine a kind of self-imposed tax: for
the first N hires (or reassignments) related to project X, IBM can
chip in K dollars each to the "reward" fund that goes to the
innovator. (This is a good way for companies like IBM or Sun to
dispose of stock in free software companies in which they wish to
invest, to ensure their existence, but not particularly to control
or acquire --- IBM isn't limited to rewarding innovators with IBM
stock. Perhaps ideally the various companies should just start a
shared mutual fund for innovators and give away shares in that while
directing the fund to reap the indirect rewards of free software
innovation.)
So I propose a three step game:
1. Innovators take some risk on the order of 3 months of
self-funded effort.
2. Free software companies institute a micro-grant program that
can, in principle, cover the month-to-month living costs of,
say, 500 designated innovators. Innovators whose self-funded
projects pass muster with more than a few known-smart hackers
can receive micro-grants to extend the reach of their
self-funding for, say, 6m to 2y.
3. When a grant-receiving innovator's project begins to generate
actual product/profit-oriented spending among the free software
companies, the grant-giving organization "pays out" to the
innovator with shares in a free-software-based mutual fund.
*If* the companies can get their act together to do that, the
existence of that game gives rise to another: Can one be a
"professional innovator"? Can one "game" that micro-grant game
into a career? I sure hope so....
Ultimately, free software innovation is a lot like professional
sports, especially individual competitor sports like golf. There is
a huge social value to having these skilled players, to granting
them the freedom to develop their game. Smart business folk have
little difficulty building up rather large industries around,
ultimately, not simply the celebrity of the players, but just the
basic facts on the ground of tournament games that they play.
Millions and millions of dollars are invested to produce a
performance by Tiger Woods in which he wins a prize of less than a
million dollars. Yet this is economically rational for the
investors: the profundity and impact of the game he plays is so
intrinsincly valuable that the investors have learned to benefit
from its mere existence --- largely by making that game visible to a
very large and interested audience. Software innovation is little
different. There is *huge* value from having software innovators
"do their thing". And, no, they won't all be Tiger Woods. Most
won't be anything close. Professional innovators will live mostly
on prize money, not endorsements. And that's as it should be: a
sea of competent innovators, framing the occaisional stellar
revolutionary, such revolutionary innovators being reliably produced
by the existence of the sea of the competent.
The economists will, no doubt, point out no firm has incentive to
pool free software R&D investment with another because, between
them, it is a zero-sum money-sink from which neither firm
gains advantage. Worse, such a self-imposed tax worsens the
position of both competitors against "free rider" competitors
consuming the same innovation.
They are wrong, though. Free software innovation creates value
for the *customers* of these companies. FS innovation doesn't
exist to give one free software firm a competitive advantage over
another; rather, FS innovation exists to ensure and expand the
future revenues of the free software industry as a whole!
The (straw man) economists making the "zero-sum" argument are
ignoring the role of enlightened customers. What RH, IBM, Novell,
HP, and Sun ought to do is:
1) coordinate among themselves (or any substantial subset)
to implement the micro-grant/prize game
2) *brand* that and educate their customers that, as vendors,
they are preferable because some of the fees they charge
goes to free software innovators. Vendors not
participating in the micro-grant/prize game can be fairly
described as "free riders" who are reckless and
irresponsible in their approach to free software. Such
readily identifiable free riders are to be boycotted.
* Parting Shots
The other day I said something clever but I think it is not
original. I can't think of who said it first, though.
It goes like this:
The end-state of every substantial, successful free software
project is a bureaucratic power struggle.
The phrase "power struggle" doesn't have to mean anything untoward
or unpleasent. It doesn't mean that there has to be losers.
Sometimes a theoretical power struggle is a very useful tool, often
when the power struggle is so icky that the only way to win is to
not bother playing. The existence of the *theoretical* power
struggle means that there's no need for a *real* power struggle.
The best example of that principle is the GCC project: very, very
indirectly ruled over (nominally) by the FSF; access-rights
ultimately the authority of a constitutional steering committee;
day-to-day processes structured around a release manager and
committers (with rights to specific parts of the tree....). The GCC
project is a freaking virtual nation! It's a terribly
anti-democratic republic in which new citizens are admitted by
sponsorship or rough consensus. This seems to work very well. It
seems to be a good fit for what it feels like to work with large
projects. People seem generally happy about it.
Arch is entering that phase. Soap-operas among the core developers
aside, the bottom line is that we are erecting a bureaucratically
driven mainline, making the game just tedious and complicated enough
to not be worth trying to "game"; making the game hopefully
consonant with what's best for arch.
That's great for me, under the circumstances. I can start to step
back pretty damn far from the "benevolant dictator" role and the
arch project can have a bit of a life of its own, with nobody in
particular in charge (except as a last resort circuit breaker).
GCC, Debian..... now it's arch's turn.
-t
- [Gnu-arch-users] the state of the union,
Tom Lord <=
- Re: [Gnu-arch-users] the state of the union, Andrew Suffield, 2004/08/17
- Re: [Gnu-arch-users] the state of the union, Tom Lord, 2004/08/17
- Re: [Gnu-arch-users] the state of the union, Andrew Suffield, 2004/08/17
- Re: [Gnu-arch-users] the state of the union, Tom Lord, 2004/08/17
- Re: [Gnu-arch-users] the state of the union, Andrew Suffield, 2004/08/18
- Re: [Gnu-arch-users] the state of the union, Tom Lord, 2004/08/18
- Re: [Gnu-arch-users] the state of the union, Andrew Suffield, 2004/08/18
[Gnu-arch-users] Re: the state of the union, Miles Bader, 2004/08/17