Re: [DotGNU]Re: X as WOS

[Mario D. Santana]

Adam Theo wrote:

> "Mario D. Santana" wrote:
> 
> > [...]
> > What guarantees does jabberd give of a particular message's full and
> > successful propagation through the p2p network, i.e. of it's arrival at
> > all the nodes? This is basically a 2-phase commit, and though for some
> > services it may not be worth the overhead, other services (e.g., auth)
> > will probably need a strong guarantee of sane behavior in pathological
> > cases like splits.
> 
> Yes, I agree. This is a feature Jabber does quite well, and in this 
> respect Jabber is more like email than IM. A Jabber server keeps trying 
> to send a message up to a certain amount of time, like email, until 
> giving up. it would also be possible, with an advanced publish/subscribe 
> system (which jabber is working on), for messages to be permanatly 
> archived until it's announced that the destination is back online.

Also, David Nicol wrote:

> usenet is very good at propagation between peering machines and even
> has strong authentication mechanisims (pgp-signing) to protect 
> access to the control channel.

I've been thinking about this almost exclusively from the viewpoint of a
distributed authentication service. The service is made up of what
amounts to a p2p network of auth servers. In this case it's more for high
availability than performance (not to neglect performance,) so it's
important that all the servers keep their states synchronized.

The problem is that a change in state needs to be handled so that it
takes full effect before any other event that could be influenced by that
state change.

For usenet, listing a group is an event influenced by a posting, which is
a change in state. It's mostly OK that propagation takes a while, because
a poster sees her message immediately -- an event isn't influenced by
slow propagation. Even here, though, a reply posted in China might not
reach Brazil before a (redundant) Brazilian reply is posted.

Simple, robust large-scale propagation for usenet is worth a few
redundant posts. But slow propagation for an authentication service means
redundant login requests for a user, each returning a (redundant) session
that will be invalid for any other server until the propagation is
complete. The user has to either wait, or connect to an updated server.

A few hours is good enough for usenet. What's good enough for a
distributed authentication service? I don't know, exactly. For some
operations (e.g., logging out,) more delay is acceptable than for other
operations (e.g., updating session timestamps.) Some operations (e.g.,
password changes) could even propagate "soon" like usenet. But there's a
class of operations (e.g., logging in) that should propagate before
returning -- this means 2-phase commit.

Once you've figured out how to propagate, you've got to figure out how to
do it robustly, dealing sanely with net splits and the like. Again, some
operations can deal, while others can't. Questions lurk everywhere (e.g.,
should we disallow logins while the network is split?)

My goal is to have one set of rules for robust propagation that will work
for the most paranoid of operations. Then make a p2p network that follows
those rules for all operations and that is self-healing -- while keeping
the lag within usable limits! It will probably have to rearranges itself,
too, as servers come and go.

I don't know if this is feasible, as I've only just started looking
through books on network and graph designs and algorithms. I do know it's
not easy, and I would *love* to use (or at least look through) existing
source.

I've just found erlang, which might have some interesting ideas. See
http://www.erlang.org/doc/r7b/doc/index.html especially Mnesia. It's
"open source", under an MPL variant called EPL (Ericsson Public License)
so I plan to mine it for ideas. Also, I don't know how well the erlang
system lines up with DotGNU, but maybe all those "concurrent" and
"distributed" applications could be ours if we run their bytecode. 

Finally, thanks to Norbert's prodding, I found a mention of Jabber/JAM
transaction-based message passing (2-phase commit?) on
http://jam.jabber.org/docs/vision.html but there are no details.

Wow. I hope this long message doesn't overflow something for those of 
you using Outlook. ;) Cheers.

mds