Re: thread ids, task ids and subsystems

[Espen Skoglund]

[Marcus Brinkmann]
> On Thu, Apr 10, 2003 at 04:39:20PM +0200, Niels Möller wrote:
>> [...]
>> I'd expect L4 to schedule B on cpu 1, because (1) the thread that
>> was running there was just put to sleep, so the cpu is free to
>> take, and (2) the RPC arguments are located in *registers* inside
>> cpu 1.

...and the whole cache working set of B is (perhaps) on cpu 0.
Writing some register contents to a cache line and "transferring" that
cache line to the destination cpu is far outweighed by the cost of
transferring the thread's working set (stack, data, etc.) *from* the
other cpu.  Then there's the additional cost of synchronizing
scheduling, dealing with migration, etc.

> Well, I look through the L4 X.2 specs back and forth, but as far as
> I can tell threads are not migrated from one CPU to another
> automagically.  A thread runs on a given CPU, and if you want it to
> run on a different CPU, its scheduler thread needs to migrate it
> with the schedule IPC (and this seems to be a "slow" operation in
> the sense that threads should best be created on the CPU they are
> intended to run on).

I'm not sure what you're referring to when you say "schedule IPC" (I
suppose you mean some sort of communication with the scheduler
thread), but the Schedule() system call is not slow per se.  However,
depending on the kernel implementation, scheduling a thread on another
CPU might be slower than you'd appreciate because of sending IPIs
between CPUs, acquiring locks, etc., etc.

> Also, the ThreadSwitch IPC has a destination thread parameter to
> which you can donate your remaining timeslice, and if the
> destination thread runs on a different CPU, it is ignored.  Might be
> a further indication that L4 doesn't bother to move threads between
> CPUs.

You are right in your observations that the in-kernel scheduler
doesn't automatically migrate threads between CPUs.  We once did
consider such a scheme where you could automatically migrate a thread
to the destination CPU (or vice versa), but ditched the idea because
of performance costs, ugliness of API and way too much policy within
the kernel.

        eSk