Re: thread ids, task ids and subsystems

[Niels Möller]

Espen Skoglund <address@hidden> writes:

> > On Thu, Apr 10, 2003 at 04:39:20PM +0200, Niels Möller wrote:
> >> [...]
> >> I'd expect L4 to schedule B on cpu 1, because (1) the thread that
> >> was running there was just put to sleep, so the cpu is free to
> >> take, and (2) the RPC arguments are located in *registers* inside
> >> cpu 1.
> 
> ...and the whole cache working set of B is (perhaps) on cpu 0.
> Writing some register contents to a cache line and "transferring" that
> cache line to the destination cpu is far outweighed by the cost of
> transferring the thread's working set (stack, data, etc.) *from* the
> other cpu.  Then there's the additional cost of synchronizing
> scheduling, dealing with migration, etc.

Does this mean that if the server keeps one thread per cpu, it's best
for clients to send the rpc requests to a thread running on a
*different* cpu? You may save some working set migration, if you're
lucky enough to target the right cpu, but on the other you need
inter-cpu communication and both cpu:s are forced to do some
scheduling.

If so, it seems better to me to keep the server thread on a single
cpu: You'll use the caches better when most or all servers keeps their
own working set on one cpu.

It's some time since I read about Solaris' ports concept, but IIRC the
client thread would in some sense go on executing in the server, just
in a different address space, and that was supposed to be pretty fast.

If I understood Marcus right, the idea was to target the rpc to a
server thread on the *same* cpu as the client thread. Was that wrong?
But we were also originally talking about the thread creation rpc, and
I guess that's a kind of special case with extra scheduling
requirements.

Another subtlety with the one-thread per cpu model is that ports and
reply ports must be treated differently. Consider a Hurd rpc that is
implemented as two L4 rpc calls, one for the request and another for
the reply.

1. Thread A wants to make a request to task B, so it sends an rpc to
   one of B's threads running on a particular cpu (confused again, so
   I'm not sure now if it's the same cpu as the one A is running on,
   or a different).

2. B starts to serve the request, and while doing that it might even
   be rescheduled to a different cpu.

3. B sends the reply. It must send the reply to the thread A, no other
   thread in the same task is good enough.

So service ports will sometimes or always be served by multiple
threads, one per cpu, while a reply ports must always be a single
thread.

Regards,
/Niels