Re: [Monotone-devel] New project: libmtn

[Nathaniel Smith]

Pulling pieces from several of your emails together to respond...

On Fri, Jun 30, 2006 at 01:51:27AM +0400, Arseny Nasokin wrote:
> Monotone is good, but buggy :(

Have you told us about any of these bugs?  I can't find any reports
from you in the tracker or on the mailing list.

> I have started project of recomposition monotone from one big
> monotone commandline-oriented executable into monotone library (it
> can be static), commandline utility, and possibly network service.
> And work plan for me and my programmers group had created also.
> First versions I plan deploy at my site next month.

This is a bit unclear... are you planning to start from the existing
code base and refactor it, or rewrite things from scratch?  What kind
of resources are you planning to put into this?

> Site and Wiki for project will be designed at monday.
> 
> Some reasons to start the project:
> 
> - Extreme programming will produse more stable code:
>       network communication, command line and database
>       communication - three logically different tasks, which
>       should be designed separatly

I'm not sure what you mean.  Monotone is, honestly, the most stable
program I have ever seen.  I also am not quite understanding how the
first line ("Extreme programming...") and the second ("...should be
designed separately") relate.  By my understanding of XP, monotone
_is_ very XP.  YAGNI, "Do the simplest thing that could possibly
work", etc.  We wrote the simplest program that is functional -- i.e.,
a single monolithic command line program -- and we've since been
growing into a more flexible system.  But we're doing it
incrementally, preserving existing function while we do it.

> - one library - many frontends (for example, GUIs or bindings for
>   other langs)

Right.  Our strategy to allow this has been the "automate" stuff.  I
understand why people want a comprehensive library instead, and are
frustrated by not having it, but, the "automate" approach has some
important benefits:
  -- because it is incremental, we don't have to stop everything, go
     off in a cave, and try to make it work.  The test suite always
     passes.  Big-bang development is _so_ hard to pull off.
  -- because it is incremental, we always are working against real use
     cases.  We don't have to guess what people need, like you have to
     when you design a whole API before using any of it.  I think of
     this as another XP-style approach; our experience has been that
     while this can be frustrating, it's the only way of designing
     APIs that actually work.
  -- It is actually more universal than a C library API -- it is hard
     to access a C API from the shell, for example.  It is _certainly_
     more universal than another other sort of API (e.g., C++ or Lua).

Then, as we improve the interface and learn what's useful, and we keep
refactoring the code to reduce duplication, we can 

The biggest thing holding this back has been lack of manpower.

> - one library - many backends:
>       what about PostgreSQL or Oracle, not only SqlLite 3 as now?

This could simply be added to the existing code base.  I'm not
convinced it's a good idea -- concurrency is very hard. it's not
just SQLite that doesn't support it; we've been reluctant to guarantee
that our code is safe in the presence of concurrent writes. In
practice, it's almost impossible for people to accomplish this sort of
thing without sacrificing robustness (and definitely sacrificing
development speed).  Also, the only case I know of where concurrency
is really useful is for things like ViewMTN, that currently have no
clean way to pull new changes.

>From what testing we've done, SQLite seems to scale just as well as
"more serious" databases when it comes to the size of the data stored;
it was performant on our old 3-gig GCC repository tests.  It's just
concurrency where it is limited.  (Present monotone may not scale that
far, of course, just because it hasn't been tested and optimized
there.)

Nonetheless, even if (in the worst case) you ended up not being able
to convince us that this was useful for mainline, you're always
welcome to maintain an enhanced fork of monotone in a
net.venge.monotone branch.  Nothing changes minds like experience and
working code, and this would certainly be easier than redoing
everything.

> - new development will be faster.

Can you elaborate?  Library-ifying, by itself, can only slow
development, because when you expose an intermediate API, that's
something that's expensive to change.  In a monolithic codebase, you
can rearrange anything at any time, as improvements are discovered.
(This requires a good test suite, of course, but we have one.)

> design lacks: I can't disapprove revision for _one_ file from it.

This is simply a missing feature, not a design problem.  Supporting
'revert -r' has been on the TODO list for a while, and should be
reasonably straightforward to implement (not entirely trivial because
there are some workspace handling refactors that probably need to be
done first, but completely trivial to anyone who can redo the
codebase, so.)  This has simply been waiting on someone to get around
to it; if you have resources to devote, we would love to accept
patches for this.

>   or it is possible create revisions, which are complex, not simple.

Could you elaborate on this?  I don't know what you mean.

> backend migrating: It's too hard write simple frontend for migrating
>   from {CVS, SVN,...} to monotone and back.
> designing:  Monotone have _inside_ basic command set, user have only
>   complex: it's too hard write new command, which bases on basic
>   command set.

These seem to simply be complaints about how rich the programmatic
interface we expose is?  By far the most expedient way to improve
these would be to extend the 'automate' interface as needed.  As
mentioned above, the bottleneck for 'automate' is simply finding
people who want to work on it, and have the use cases that you need to
design a proper API.  If you wanted to put effort into this, again, a
lot of people would be happy.

> Monotone is critical service for my buissines

Well, we're flattered!  You are, obvious, free to do what you like
with the code and design.  I suspect, though, that you'll find that
making dramatic improvements turns out to be more difficult than your
description makes it out to be -- this is just based on experience
working on these issues, and the way that some of your goals seem to
conflict (e.g., increasing robustness while adding features that are
harder to make robust).  I'd urge you to carefully consider all the
ways you could work with the existing efforts in our community to
accomplish the kinds of improvements you want.  These changes can be
implemented immediately, and immediately benefit everybody, while
eventually working towards the kind of system that you want.

Cheers,
-- Nathaniel

-- 
"Lull'd in the countless chambers of the brain,
Our thoughts are link'd by many a hidden chain:
Awake but one, and lo! what myriads rise!
Each stamps its image as the other flies"
  -- Ann Ward Radcliffe, The Mysteries of Udolpho