[Monotone-devel] A Two-Fold Proposal: On Formats And Front-Ends

[Larry Hastings]

I have for you two separate but intimately related proposals, and I'm going to describe them in reverse order of implementation because it flows better.  Keep in mind, I'm still quite wet-behind-the-ears with respect to Monotone; I typed my first monotone command less than two weeks ago.  It wouldn't surprise me if this was all a bad idea for any number of unforseen reasons.  And yet I bravely post, having read somewhere that the monotone community is "friendly".  So... be kind.  :)

Right now monotone makes life tough for front-end developers.  The main problem I see is the N different output/file formats of monotone, where N is distressingly large:

• There's the tag filename (where filename may have whitespace) of manifests.
• There's the name: [value] or sometimes name: "value" blank-lines-are-apparently-significant of revisions.
• There's the name: value of certs.
• There's the name "value" blank-lines-are-significant format of .mt-attrs and others--the "stanza" format.  (Are revisions "stanzas"?  I can't quite tell.)
  
• I count three output formats unique to monotone automate: whitespace-separated tags (newlines are significant), the wild world of automate inventory, and the l<length>:<token>e syntax of automate stdio.
• There's the "packet" format.
• There's the catch-as-catch-can output of running various monotone commands; log, status, list keys, and list certs have wildly different output formats, just to name a few.

monotone's data stream--its input, its internal storage, and its output--is a tower of babel, and I had my work cut out for me when I set out to write m7.  (Thank goodness for Python's excellent regular expressions module.)  For a project named monotone, its output is certainly polyphonic!

Additionally: while the monotone automate brace of commands is a reasonably good idea, and a step in the right direction, it is woefully incomplete.  When writing m7, I had to call lots of non-automated monotone commands directly because monotone automate doesn't offer any actions.  You can't add/delete/rename, you can't commit, you can't create certs, you can't push/pull/sync.  All but one of the automate commands are queries; the only exception is stdio.  I suspect this is because monotone doesn't "eat its own dog food": monotone doesn't implement its user-visible commands using the automate interface, it just does operations directly on the database.

Seemingly all the automate commands were added ad-hoc, to support the needs of history visualization tools I'm guessing.  But, since people weren't writing general-purpose front-ends that day, monotone automate offers no assistance to people adding files or committing changesets.  Sure, we could add a spate of new monotone automate commands, in an effort to catch it up with the interactive command-line interface.  But even if it caught up, it would likely fall back behind later, again because monotone doesn't "eat [its] own dog food".

One unfortunate side effect of this: it denies a front-end any real atomicity.  There are many operations where m7 has to run mulitple monotone commands, and since each one is a separate monotone instance, there's no way to ensure atomicity between them.  As a result, most m7 commands suffer from race conditions.  This is particularly bad for commands that change the database; if you ran two m7 commits at once, I rather suspect you'd get two revisions with the same local revision number cert.


To me the solution is clear, and it is here we arrive at my first actual proposal: separate presentation from application logic by breaking the current monotone executable into two pieces.  One piece would be the "engine" that did all the actual work.  The second piece would be the "front-end", or "driver", which drives the monotone engine (as gcc drives the front-end, back-end, and linker).  The driver would provide the command-line interface of the current monotone executable, convert internal messages to their user-friendly localized equivalents, etc.  The communication between the two would be done over pipes in some easy-to-cope-with data specification.

This has many advantages:

• New front-ends, rather than depending on the vaguaries of monotone's frantic command-line arguments and all-over-the-map output, would have a single, stable, easily-digested input-output format to deal with.  Thus making it much easier to write and maintain front-ends, and making their communication channel with monotone far more future-proof.
  
  
• This new breakup of responsibilities would put all front-ends on equal footing.  This would force all monotone operations to be available via this new "automation" interface, thus promoting all front-ends to first-class citizens, and--if the interface is generalized enough--allowing them to offer new functionality not afforded by the monotone command-line, just as m7 does, but hopefully preserving atomicity.
  
  
• If you ship file data over the pipes, and restrict the engine to so the only file it talks to is the database, bingo!  Throw ssh around it and now you have client-server monotone.  If you were just a little bit more careful, you could even have multiple computers all sharing the same database, each with just their own working copies rather than their own full-blown monotone databases.  (Though I gather this is strongly discouraged.)  The back-end could be written to accept() on a socket directly, and run as a daemon in the background, thus allowing people to run monotone in a traditional client-server mode if they so chose.  We might even be able to fold push/pull/sync into this same socket, making database synchronization even more effortless.
  
  
• If we were extra-clever, we could break out some of the logic from the canonical front-end in such a way that third-party front-ends could leverage off it.  Working up a whole front-end is a lot of work, and there are a lot of messages which would need to be localized.  It'd be great if a front-end could call into that formatting/localization code to format its messages for the user.
  

A moment ago I handwaved what "data representation" I had in mind.  I propose there are three main candidates: XML, ASN.1, and JSON.  I will immediately dispense with the first two, and show why the third is far more likely.  :)

• How can I automatically cull XML from the running?  Simple: if Mr. Hoare was inclined to use it, he would already have done so.  One cannot swing one's dead cat without hitting a six-inch-thick book about XML Schema DTD Hierarchy Displacement Manifolds or some such.  Certainly the 'net was already hip-deep in XML when he started writing monotone.  Sure, these days monotone has grown beyond the product of just Mr. Hoare.  And it's possible that the monotone community could adopt XML against his wishes.  But it doesn't seem likely or well-advised.  On top of all that, XML is cumbersome; lots of people, myself included, find XML wordy, needlessly complex, and just plain tiresome.
  
  
• What about ASN.1?  Well, have you looked at some ASN.1?  It has none of XML's effervescent charm.  It's more parsimonious than XML, but then what data encapsulation format is not?  Its biggest failing: it looks to be wholly inflexible.  We couldn't add new fields without creating a new grammar, which would break tools that didn't upgrade their grammar.  I say screw it .
  

On the other hand, I first heard of JSON by reading a monotone IRC log somewhere, wherein Mr. Hoare hisself said "[his] money's on JSON for this sort of task":
    http://www.loglibrary.com/show_page/view/106?Multiplier=3600&Interval=6&StartTime=1124773859
JSON is small, easy to parse, easy to generate, and covers all the bases.  It has explicit and well-defined quoting rules.  It's flexible; we could add new fields to a message and it wouldn't break a receiver who wasn't expecting that field.  So I'll go ahead and assume that, if something like this did come to pass, it'd use JSON.  Specifically, JSON encoded in UTF-8.  You can read more about JSON here:
    http://www.json.org/


While proposing this I realized that, while this would fix all the output of various commands, it wouldn't fix the commands that were really just dumping monotone internal files--revisions, certs, manifests, and the like.  Thus is my second proposal revealed: rework monotone's internal data structures using this data format.  This would make the monotone engine itself easier to write and maintain, as we wouldn't have N mini-libraries for reading/writing these N formats.  We'd just have one library we used for everything.  (For backwards compatibility we could have the front-end massage the data it prints out back into the old format upon request.)  Specifically, revisions/certs/manifests would be stored as JSON, indented by one tab (\t) per indention level, lines ended with \n (no \r), and children of objects stored in sorted order.

We're already breaking all the eggs when doing rosters; upgrading existing databases will already require rebuilding every relevant data structure.   If we were ever going to consider a radical move like this, it seems to me that now, before rosters ship, is the best possible time.  Breaking up the monotone monolithic executable would (will?) be nice, but it can wait.



Undertaking any of the above would be a ton of work.  And I am sadly not volunteering to do it myself, or even contribute all that much code to it.  (Though I am working up a reasonably-clever JSON library, and I would love to help define the shape of the JSON data, particularly the communications protocol between the front- and back-ends.)  I realize that waltzing into an open-source project, vaguely sketching castles in the air, then saying "now go build it guys!", easily strays into boorishness.  So I apologize if I'm stepping on any toes.


What do you think?


Cheers,


larry