Re: GSoC: collaborative editing

[Thomas Lord]

You seem to not mind "full quoting with sparse
comments" --- I actually like that too, some don't.
In any event I'll use that style to so, scroll 
way down to find my comments :-)




On Mon, 2009-04-13 at 18:04 -0400, Brian Templeton wrote:
> Thomas Lord <address@hidden> writes:
> 
> > On Mon, 2009-04-13 at 16:43 +0200, Thien-Thi Nguyen wrote:
> >> () Brian Templeton <address@hidden>
> >>    3. A collaborative editing system using a modified version of the
> >>       Jupiter algorithm, comprising:
> >> 
> >>       - A client written in Emacs Lisp.
> >>       - A modified version of an existing server written in Erlang.
> >> 
> >> I think a server written in Emacs Lisp would gain more traction.
> >
> > Please consider this third alternative which I 
> > think has advantages to both:
> >
> > 1. Do not write a custom server.
> > 2. Use a chat server (such as a Jabber implementation).
> > 3. If some server-side computation is needed other
> >    than just forwarding messages in the manner of a chat
> >    session, write that new server-side code as a 
> >    chat client.
> >
> > Reasons:
> >
> > a. "no new server" means less work
> > b. "no new server" means less work for server
> >    administrators if they are already running
> >    a chat server
> > c. "use Jabber (XMPP)" means that the same
> >    message bus can be shared with other programs
> >    such as the "collaborative whiteboard" feature
> >    of Inkscape
> > d. XMPP implementations are actively maintained
> >    and aggressively improved.  It is hard to image
> >    a new, less general-purpose server "catching up"
> >    and to catch up suggests doing a lot of redundent
> >    work
> 
> I have already written a server, so using XMPP wouldn't save any work in
> the short run.


Sure.  It would even mean a little work to switch
over.   On the scale of the whole Emacs project, 
I think that tiny blip of extra work is small
compared to what the project is set up for in the
long run if a custom server is used.


>  I evaluated XMPP when I wrote the server and wasn't able
> to determine whether using Jabber as a message bus for a Jupiter-based
> protocol would require misusing XMPP, since some server-side computation
> and modification of client messages is required.


You can do your server-side stuff either as
an XMPP extension or as a server-side client
of the XMPP session.

One suggestion I would have (just my biases) is
to not worry about performance too much up front
if you are confident it can be improved later -- 
so I would try doing any extra server side 
computation in a server-side client of vanilla 
or Jabber-ish XMPP.   If "barely usable - but usable"
can be achieved that way AND if "looks easy to 
fix the performance problems" is the outcome: that's
gold (imo).



>  But I'm not averse to
> using Jabber as a message bus if it wouldn't be considered a misuse of
> the protocol,

It's definitely not.   

Very briefly, you can make a taxonomy of 
communications busses.  There's "persistent", 
there's "store / forward", there's "lossy routing"
and there are orthogonal aspects to -- but not 
very many.

If you can reduce something to "lossy routing
network" - that suggests finding one way or another
to build on chat and if building on chat, XMPP seems
to me a good choice.

Ultimately, all those different kinds of messaging
get integrated a bit - e.g., the "store and forward"
service (such as for mail and lists) and "lossy routing"
(such as for chat) can share user identities, authentication,
and so on.  That way, we keep servers simple - they just 
(mostly) run a few very basic services, nicely integrated.



>  and it would certainly be a good fit if I implemented an
> algorithm for P2P editing, which wouldn't require server-side
> computation/message modification at all.


Don't over-reach for a summer project.  Do whichever
is easier to get working!

Even if you use XMPP but don't right away do P2P edits?
You'll still have made plausible progress on P2P edits
because the server side will already be done in the 
form of chat rooms and chat server peering - leaving you
to think about the hard problems on the client side.

(Again: don't spontaneously take on mathematically
hard problems at the last minute for a summer project.
Don't misunderstand me to suggest you should go for P2P 
edits!)



> 
> The current server is written in Erlang; it would be only moderately
> difficult to adapt it to work with ejabberd's mod_muc.

There ya go.  So, let's see what the Emacs
maintainers think, if they care to comment.


> 
> > There is a second question.  What payload goes
> > in chat messages?   How are mutually remote buffers
> > synchronized.   In that area I suggest:
> >
> > 1. Carefully evaluating and considering adopting
> >    (and helping to adapt) the "mobwrite" 
> >    system of "diff sync" (see
> >    http://code.google.com/p/google-mobwrite/
> >    )
> >
> I am planning to use operation transformation, which is also used by
> most existing collaborative editors (Gobby, SubEthaEdit, etc.).
> Operation transformation is easier to implement and more elegant than
> differential synchronization, IMO. In the context of real-time
> collaborative editing of text documents, DS does not seem to solve any
> actual problems that aren't already solved by OT.

Does "operation transformation" mean taking a 
log of edit commands and applying outcome-preserving
transformations upon it to compress it to a shorter
log of edit commands?

If so, fwiw, from my background -- that *does* sound like
a much better approach.   I could boor you for hours as to
why but ... well, why? No need. :-)



> 
> > b. Caution: "mobwrite" is new and experimental.
> >    The design needs to be scrutinized with care.
> >
> Yes, and I don't want to have to be the one to scrutinize it. 

If I understand your point about "OT vs diff-sync" above,
you get mobwrite just fine, imo.   That's care.



> There is a
> great deal of theoretical work available on OT approaches, and several
> algorithms have been proven correct; I'd rather rely on a functioning
> network (i.e., one that doesn't drop messages constantly) than attempt
> to prove the correctness of 'fuzzy' matching algorithms and
> 'best-effort' patching algorithms.


That's a confusion on your part, I think.

The network unreliability is real and unavoidable. 
You can't wish it away by insisting on a design
that assumes a loss-less network.  That doesn't
preclude OT, I think.

It's a big topic, though, so it would help if I
were more certain you meant by "OT" what I take
you to mean.   Can you point to some docs I can
read to make sure we're talking about the same thing?




> 
> > e. Reason: Since other editors are considering 
> >    or being modified to use "mobwrite", perhaps this
> >    approach can ultimately allow collaborative sessions
> >    in which some users are using "emacs" while others
> >    use "bespin" or "vim" or what have you.
> >
> Which editors? Are any editors currently using mobwrite, besides editors
> specifically written to use mobwrite?
> 


The (verbally spread) news at a recent conference
was that someone with the mobwrite project was working
on extending several editors, including Emacs, with
mobwrite support.

Both Google and the Mozilla Foundation are investing
in collaborative editing technology.

-t