Re: [Chicken-users] Async IO (was Re: libcurl?)

[Thomas Chust]

Am 05.10.2005, 03:41 Uhr, schrieb Peter Keller <address@hidden>:

> On Tue, Oct 04, 2005 at 10:21:55PM -0000, Thomas Chust wrote:
> > CHICKEN does not have any feature that would make continuations
> > serializable; it would mean that you somehow have to transform machine
> > pointers to arbitrary memory locations where your continuations'  
> > assembler
> > code lives into stuff that can portably be transferred over a network
> > connection.
>
> I'm sure there are some ideas that could help you perform a virtual
> serialization of the continuations. For example, machine A has the
> "original" copy of the running executable, machine B has a certain form
> of a skeleton executable running in memory with a very small memory foot
> print. Now, the continuation (wholesale) is transferred to the target
> machine where it is propped up in the some executable pages in the same
> range as before and control jumps there.  Of course, the rest of the
> memory heap (known by the skeleton since it conspired with machine A)
> is protected via mprotect, and when the first segfault happens, it is
> caught, and the offending pages brought in from machine A to machine
> B. Then the continuation happily continues. After a while, a working
> set emerges, and machine B's heap gets managed independantly of machine  
> A.
>
> Machine A knows what pages got served, so it is aware of such things
> during garbage collection, and when machine B's pages are finished,
> they are merged back into Machine A's domain. I'm handwaving the merge
> process, since I haven't thought about this more than 5 minutes. Though I
> suspect the merging algorithm can move stuff out of the pages of machine
> B to better location on machine A since ultimately you're dealing with
> higher level objects and the scheme implementation isn't going to care.
>
> I'm sure there are a lot more gotchas in a method like this (like race
> condition between garbage collection on machine A and machine B's request
> for the pages), especially without deep thinking about it, but it might  
> be
> workable.
>
> > If you get this working, I honestly have to congratulate you -- I tried  
> > it
> > once (trying to cook up something like Kali Scheme on CHICKEN basis) but
> > failed miserably to come up with any decent, let alone portable or even
> > elegant solution. The day anyone has a really brilliant idea how to  
> > solve
> > the problem, I will certainly resume this dead project of mine -- it  
> > would
> > be a wonderful toy ;)
>
> Did you happen to document the methods you tried?
>
> -pete

Hello,

your scheme for doing this is fine and basically the same thing as Kali  
Scheme originally did. But Kali Scheme was running on a bytecode  
interpreter, CHICKEN is running directly on the processor, so there are  
several subtle issues involved with taking binary code from one machine  
and sending it to another to execute. Some important points are:
  * you may not be able to map everything to identical addresses on the  
target machine for various reasons, so you have to compile the full  
program as position independent code
  * linking your program to shared libraries is a nightmare
  * of course this model of communication is utterly importable across  
architecture boundaries

And apart from that you also have to change some code within CHICKEN to  
make the memory management system work with stuff that still lives on  
other machines and to handle page faults/foreign object fetches correctly.

All this sounded like too much work for me, so the thing I seriously tried  
was the following:
  * compile the same program on all the target machines
  * when the program loads, generate a two-way association between all  
continuation addresses and "names" that have been (auto-)assigned to them
  * instead of transmitting the code of continuations, just transfer its  
"name"
  * on the receiving side, map the "name" back to a code pointer and  
construct a continuation object out of it

All other data objects, except continuations, can be serialized and  
deserialized traditionally through the connection.

But as I said, I didn't manage to write some decent code that actually  
does what I described here and I didn't come up with useful documentation  
either, sorry.

What should be rather easy though, is to implement such a message passing  
system between forked processes, because they always share code segments  
-- you wouldn't even have to map code pointers to and from anything,  
because they would just be the same.

cu,
Thomas