Re: Comparing "copy" and "map/unmap"

[Matthieu Lemerre]

[This is a message that I sent to Shapiro privately because I forgot
to Cc l4-hurd in it.  I don't change the mistakes I made so that
Shapiro's reply can be reposted without any change]

So far, Shapiro showed how reference counting wasn't needed anymore,
and thus eliminates the need for a cap server, which had many issues.
Capability copy is an operation that is highly wantable.

I also note than Espen Skoglund noted that he could change L4 to add a
copy operation in a couple of lines.

So I'd like to concentrate on REVOCABLE COPY vs MAP/UNMAP, and
advocate a bit on why MAP/UNMAP could still be useful.

* First, I'd like to sum up the differences between REVOCABLE COPY in
EROS and MAP/UNMAP.

Disadvantages of EROS wrapper objects compared to the L4 UNMAP model:

1/
> The main problem with the EROS wrapper design is performance: it
> requires a number of IPCs. Fortunately, it appears to be a very rare
> design pattern in our system.

2/
>The *bad* part is that selective revocation [in EROS] requires some
>application-level planning.
This is not too much of a concern, IMO.  I'd like to (humbly) propose
a solution for this in the next few days (but it's getting late now).


Advantages:

3/
> One difference between this approach and UNMAP is that the EROS approach
> interacts well with persistence, while the UNMAP approach does not
> appear to (to me). My question about unmap is that I do not understand
> how it scales successfully when the mapping database grows large enough
> to become an out-of-memory data structure, and I have already described
> in a previous note why flushing and reconstructing the mappings doesn't
> work without severe restrictions on system architecture.

Maybe the L4 people have thought about this.  There is already a paper
about persistence on the L4ka web site.

4/
You have to allocate kernel memory for the mapping database.  But
there may be solutions to that problem ("in-kernel heaps that were
backed by an out-of-kernel storage allocator.". Is it sigma1?)

So, if points 3/ and 4/ could be solved, MAP/UNMAP has better
performances.  This is really important if it becomes a common usage
pattern, and I have some ideas that would require MAP/UNMAP
operations.


But first, here's something I wanted to ask you (I'm too lazy to find
where) and my question was not clear enough.  So let me reask:

-What happens in EROS if the client desallocate the storage for an
 object while the server is using this object (i.e. reading or writing
 to it)?  How does the server deals with the pagefault?

It seems to me that the problem is similar than if the server was
using the memory from a fpage mapped by MAP.

* I then wondered if memory provided by a client to a server for its
object allocations could just be mapped to the server.  Indeed, this
would save some RPCs between the server and the space bank (or, in our
current implementation, between the server and physmem), and thus have
better performance.

It seems to me that the problem of this design is potential pagefaults
at anytime, which are hard to recover.  But I have a proposition to
solve this problem:

In L4ng threads are mappable.  So a client could in fact map a thread
to a server so that it could serves its own requests (to avoir thread
explosion, maybe we should have one thread per principal.  But maybe
having a bunch of inactive threads isn't really harmful.).

A hostile client could then do nothing to the server, asides from
making its own thread fault.  (In fact, the thread could fault when
the client hasn't provided enough memory.  This would solve the
problem of knowing how much memory we have to allocate to a server:
the server thread would fault when needed, and the pager would be a
client thread that would install the proper mappings).

This has another advantages not only the clients pays for its memory
usage on the server, but it also pays for its CPU time usage and can
adjust it (provided that it has some control on the time slices
allocated to the thread, that it can take from its "CPU time
container")

Still there is one drawback: if the thread can be unmapped or fault at
anytime, it must not hold any global lock.  The solution is
straightforward: global operations are done by a thread that the
server owns.  So the user provided thread just has to make a call to
the server thread.  This is local IPC, so potentially very fast.

I didn't have much time to elaborate more about this design, but it
seems to me that it seems very feasible on L4ng or L4sec.

* I also noted, in a previous answer, that when you use a private
  server, you can just map memory to it : you don't have to copy it.

Thus, I am wondering if MAP/UNMAP isn't the best operation for some
kinds of pattern (with the above exemple memory usage).


* I'd like to come back on the argument of the transport chain of
 processes: you forgot that there is also a GRANT operation.  If the
 intermediary threads are meant to act only has a transport or proxy,
 either it wants to retain control and then MAPs the capability, or it
 doesn't want to and can just GRANT.  At that moment the intermediary
 processes can just die.

I'm not arguing that MAP/GRANT is a better solution than COPY for
transmitting a capability from a server to a client, I am just
wondering if there is any problem with the GRANT operations.

>> Unmapping [endpoint] objects is not that harmful: upon RPC attempt, the
>> client only has to be prepared to an error.
>
> I think that you are not thinking about this hard enough. For example,
> your design assumptios have unintentionally conceded that you will never
> need to build a shared-memory interface.

I admit that I had not thought about that yet.  In fact, I never even
conceived any shared memory protocol (but hey, I'm only 22, time will
come when I'll do that :))

> Further, you have not addressed the problem of reconstructing these
> endpoint mappings. Remember that L4 revokes in two situations:
>
>   1. When the endpoint is intended to be revoked.
>   2. When memory must be reclaimed for other use.
>
> In the second case, mapping reconstruction is required. The protocol
> needed to do this is complex, and the security and robustness
> consequences are somewhere between unpleasant and a complete disaster.

I don't understand you here.  When does L4 revokes memory (i.e. fpage
mappings?)?  And I am not talking about revocation of memory, but
about communication endpoints revocation.


> So the real question is: "Do the bindings to the storage resource
> disappear when the allocating thread disappears?" In L4sec, the answer
> appears (to me) to be "yes", but it is possible that they have some
> clever way to avoid this problem. In EROS/Coyotos, the answer is that
> the lifespan of a storage allocator is different from the lifespan of a
> thread. As long as the storage allocator survives, the object that lives
> in the storage survives.

Couldn't the allocating thread which holds memory ressource just grant
the memory to another one?  (the obvious question in this case is: to
whom?  I'd say to some thread that is using the allocated ressource.
Still, this is a very general situation, so I can't give any protocol
on when this could be possible or not)

Thanks,
Matthieu