Re: [GNUnet-developers] [GSoC] Question on "Rust implementation of GNUnet utils" project

[Christian Grothoff]

Hi Andrew,

Yes, we should see if we can reuse any of those components.  My issue
with NAT libraries is that coming up with a good generic API is tricky,
especially given that asking third parties to assist with NAT traversal
inherently requires coordination with higher layers.

As for sharing "core" logic, there are a few questions to answer:

1) do you really have sufficiently similar semantics?
2) do you really support multiple pluggable transports?
3) do you really support resource allocation / transport selection
   (ATS in GNUnet)?
4) can we get our "DV" routing to work in this context?

Now, I had discussions with Matthias some time ago about separating the
current transport plugins into separate processes, and if that happens
it might be possible to have a common API for processes that do things
like uTP --- or ideally even the Tor pluggable transports ---, but this
will also require some discussions.  Demos tried to organize
cross-project meetings to help us find good shared abstractions and
components to re-use. I support her efforts, and if MaidSafe doesn't
just want re-usable components but also would be happy to discuss how we
can get as much of this as possible under one hood, I'm totally happy to
support this.

That said, the crust documentation I found was VERY high level, so maybe
a first step would be for you to point me to detailed documentation on
the crust API/specs/code --- the starting point here really needs to be
a list of features, not the code itself.

Happy hacking!

Christian

On 03/21/2016 06:36 PM, Andrew Cann wrote:
> Another thing worth mentioning (and sorry for spamming the thread) is that
> everything we're writing at MaidSafe is GPL and the bosses are generally in
> favor of splitting the codebase into smaller reusable components. It'd be good
> if some of this stuff was reusable for GNUnet. Some libraries that could
> potentially be useful are the NAT traversal library and the uTP implementation
> (although these would only really be useful for a rust reimplementation of
> gnunet-core, not gnunet-utils).
> 
>  - Andrew
> 
> On Mon, Mar 21, 2016 at 01:18:20PM -0400, Andrew Cann wrote:
>> This GSoC project has come along at an interesting time actually. I'm 
>> currently
>> working at Maidsafe, who are a company building a p2p network similar to 
>> GNUnet
>> in Rust. I've been put in charge of crust, which is the low-level networking
>> component of the system similar to GNUnet core. The crust code is a mess, and
>> I've been pushing to be able to refactor/rewrite most of it since I inherited
>> it. Given a slew of recent bugs that have come out of it it looks like that
>> might be about to happen. Crust currently uses blocking IO and threads
>> everywhere and part of the refactor will be to (eventually) switch to mio. So
>> we're currently having the same discussions over at Maidsafe: what's the best
>> way to write async networking code in rust? The other maintainer is deadset 
>> on
>> using mioco. I'm also leaning towards mioco but I'm curious to see what an
>> implementation of something crust-ish using rotor would look like. If it can
>> prevent bugs, doesn't cause any of it's own problems and the code is readable
>> then it would be an option for us aswell.
>>
>> One problem with I have with all these options though is that they seem to
>> force a particular paradigm on your entire program. If you're writting an
>> application that's fine, but gnunet-rs is a library and I'd like it if the
>> users were free to write their gnunet application using state machines, green
>> threads, blocking IO or whatever. A vague solution that I'm imagining is if
>> there was some way to write a low-level mio-based library in such a way that
>> you could then use wrappers that take a generic low-level mio based library 
>> and
>> promote it to a blocking IO based library, or a coroutine-based library or
>> whatever. For now though, I don't have a better description than that.
>>
>>  - Andrew
>>
>> On Mon, Mar 21, 2016 at 12:47:48PM -0400, Andrew Cann wrote:
>>> Another option would be to use mioco. Mioco let's you write 
>>> blocking/threaded
>>> style code but underneath uses green threads and non-blocking IO. This is
>>> similar to what you'd see in Go.
>>>
>>>  - Andrew
>>>
>>> On Mon, Mar 21, 2016 at 05:44:42PM +0100, Jeff Burdges wrote:
>>>>
>>>> We first need to get your application in to Google by whatever deadlines
>>>> they have.  
>>>>
>>>> I'm seemingly listed as a mentor now and can see GNU applications when I
>>>> access their site at : https://summerofcode.withgoogle.com/dashboard/
>>>>
>>>> If you go there, or to this google-melange.com site, can you see a way
>>>> to enter the application?  If you need to identify me to them, my
>>>> account there is address@hidden  I donno if we need to provide you
>>>> with something first though.
>>>>
>>>>
>>>> On Mon, 2016-03-21 at 15:03 +0100, kc wrote:
>>>>
>>>>> Thank you for the insight, no wonder I couldn't find any Rust code!
>>>>>
>>>>> I also feel starting with the second option is better since it has a
>>>>> mucher lower entry barrier so I can start hacking ASAP.
>>>>
>>>>> I see that GNS lookup, peer info listing and identity ego lookups are
>>>>> implemented. Do you think adding async IO to those functionalities by
>>>>> mid-term evaluation is feasible? What would you like to see by the end
>>>>> of the GSoC? Perhaps finish the "Next on the list" items?
>>>>
>>>> I think one reasonable goal might be :
>>>>
>>>> Use mio via rotor/eventual_io/gj to provide the asynchronous IO
>>>> functionality of GNUNet utils scheduler and adapt the peer info protocol
>>>> from gnunet-rs to use it. 
>>>>
>>>> We can substitute peer info with whatever Christian thinks is the most
>>>> fundamental layer, but that's the one I'd expect. 
>>>>
>>>>> I've used asyc IO in a few projects before. For instance I've
>>>>> goroutines/channels (from golang) to implement a few distributed
>>>>> algorithms such as for Byzantine agreement, leader election and mutual
>>>>> exclusion. I also used some of the Haskell async libraries
>>>>> namely Control.Concurrent for a project where we had to implement the
>>>>> log-structured merge-tree. Finally, I've used pthreads when I was
>>>>> working on an enterprise backup system. I don't have any async IO
>>>>> experience in Rust but the paradigms shouldn't be to different.
>>>>
>>>> Ok great. 
>>>>
>>>>> Regarding those libraries, I can't say I'm familair with the state
>>>>> machine based ones like rotor. On the other hand, I'm familair with the
>>>>> concept behind Futures and Promises which is what gj and eventual_io
>>>>> uses. Is there a preferred async IO model for GNUnet-rs or it's still
>>>>> up for discussion?
>>>>
>>>> I've little opinion myself.  I'd just like it to be easier to read than
>>>> the current GNUnet utils stuff.  I think that'll be the case with any of
>>>> gj, eventual_io, or rotor. 
>>>>
>>>> I think the question is really : What model do you like?  And do you
>>>> have any opinions about what fits best with what situations?  I'll look
>>>> closer at the options and ask Christian for his opinion though too. 
>>>>
>>>> Jeff
>>>>
>>>>
>>>
>>>
>>
>>
>>
>>> _______________________________________________
>>> GNUnet-developers mailing list
>>> address@hidden
>>> https://lists.gnu.org/mailman/listinfo/gnunet-developers
>>
> 
> 
> 
>> _______________________________________________
>> GNUnet-developers mailing list
>> address@hidden
>> https://lists.gnu.org/mailman/listinfo/gnunet-developers
> 
> 
> 
> _______________________________________________
> GNUnet-developers mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/gnunet-developers
>

signature.asc
Description: OpenPGP digital signature