Re: vk_l4 -- CVS Setup

[Farid Hajji]

> > Why not start along the TODO list, putting only additional (new)
> > stuff to CVS, e.g. like this:
> 
> I'll start a TODO in CVS.
I've also forgot a directory:
       doc/               # project documentation!

> I'm also trying to understand what you are proposing here. It seems like
> you are trying to abastract out a base system for all operating systems.
> Isn't this what the oskit stuff is for? It seems to me that this project
> would be more of gluing together a couple of existing projects (L4
> kernel, oskit, the Hurd), rather than building a "virtual kernel" of
> sorts, but let's explore this further.
As Niels correctly pointed out: I'm trying to factor out general
infrastructure stuff that both the Hurd and any other L4-based
OS-Personality would need. The Hurd specific glue or extensions/
modifications would use this infrastructure, where it used Mach
before.

This general infrastructure that I've boldly called vk-l4 has the
purpose to bring L4 somewhat closer to Mach, by providing VM and
driver support.

Niels: I'm not trying to realize a true VK right now, just enough to
be able to boot a minimal Hurd. VK stuff is important in the long
run, but not immediately.

> >    vk-l4/             # virtual kernel, l4-backend
> > 
> >    vk-l4/oskit/       # mixins w.r.t. the _newest_ OSKit from Utah!
> 
> What things do you expect would go into here? Wouldn't it be better to
> have the full oskit sitting somewhere and just use it's headers and
> libraries from that location?
Well, consider the pager: We could use the OSKit LMM or other memory
management libraries to build it. The parts of the OSKit that we'll use
to build the pager would go here. We'll probably also need glue code
from OSKit to drop in the driver framework from Michael. This code would
also be "borrowed" from the complete OSKit sources and put here.

The reason I'd prefer that we don't import the whole OSKit is, that it
would be easier to compile only the components we need. Another reason
is that we'll clearly see what we borrowed from OSKit, so that we won't
have to worry about things breaking, when OSKit is substantially changed
by their maintainers.

I may be wrong regarding selective import of only what we need. Feedback?

> >    vk-l4/gcc/         # specs and other modifications to gcc, if needed.
> 
> I'm unfamiliar with what this would be. What is a gcc spec file? Would
> we really need to modify gcc?
This is somewhat hairy to explain. Currently, we don't need a spec file,
so you may drop gcc/ for the moment. We're not cross-compiling either
(unless some brave soul wants to work on MIPS or SPARC64 ports from
their x86 systems, or some such...).

> >    vk-l4/private/include  # includes, idls and libs used internally
> >    vk-l4/private/idl      # by the vk-l4 components
> >    vk-l4/private/lib      # not exported to OS personalities.
> 
> I understand this.
> 
> >    # Here only the public interfaces of vk-l4:
> >    vk-l4/include/     # includes needed by vk-l4 users (OS personalities)
> >    vk-l4/idl/         # IDL files (like our *.defs) used by OS personalities
> >    vk-l4/lib/         # libraries used by vk-l4 based OS personalities
> 
> I think I understand this. This would be the layer used by the OS
> personalities (hurd, bsd, linux). What would the *.def files be for
> them?
The *.defs (or more correctly in our context *.idl) files would specify
the RPC protocol between the tasks making up the vk-l4 and the OS-Personality
(and also their normal client tasks/processes). For example: the pager would
need to be queried for new pages, returning pages, mappings etc. The driver
supertask would need to be queried for opening a special driver or for
registering for interrupts etc... All this happens through IPC of some kind.

By defining the public IDL interface of the tasks making up the vk-l4,
we'll be able to provide a service to OS personalities that wish to
RPC directly those servers. For convenience, we'll also provide access
libraries in vk-l4/lib, that would provide functions that wrap those
RPCs in simple convenient calls like, say, vm_allocate(), dev_lookup(),
dev_open(), dev_write() and so on.

This is no different that the situation in the Hurd with the *.defs.
[at least as far as access to Mach IPC is concerned.]

> >    # Now for the guts of VK/L4:
> >    vk-l4/pager/       # our new general purpose L4 pager, probably UVM 
> > based.
> 
> Can we start with the sigma0 pager that comes with the various L4
> implementations? Maybe I'm misunderstanding what it is for.
sigma0 is not a true VM pager. It provides only mappings of physical
memory into requesting tasks on a first-come first-served basis.
You can't e.g. return unused pages back to sigma0 after you're finished
with them. They are lost forever. You can't support virtual memory with
sigma0, because only physical memory is mapped. Moreover, sigma0 doesn't
provide backing of dirty pages to disk etc...

A real pager like UVM would need to:
  *. request all physical memory from sigma0 it can get and manage that
     memory subsequently.
  *. provide a virtual memory system that would swap unused pages to
     disk if memory becomes tight.
  *. clients would register with the pager, so that page faults are
     redirected to a thread in that pager.
  *. clients should request vm_*() operations (see above) from this
     pager through IDL RPCs via convenience vm_*() library functions.

I've looked at /sys/uvm/* on my current NetBSD system and I would
strongly urge anyone wishing to do work on the pager to at least
study UVM very carefully beforehand. UVM is current state-of-the-art
in paging (it's more advanced than Mach's VM) and our vk-l4 pager
should probably try to use an adapter UVM system as well.

We should check with L4 developers, if they have already tried to write
a UVM-based general-purpose L4 pager. This is an area where a cooperation
would be desirable.

> >    vk-l4/drivers/     # the driver framework Michael Hohmuth talked about
> >    vk-l4/drivers/linux
> >    vk-l4/drivers/linux/net
> >    vk-l4/drivers/linux/block
> >    [...]
> >    vk-l4/drivers/freebsd
> >    vk-l4/drivers/freebsd/net
> >    vk-l4/drivers/freebsd/block
> 
> So these are l4 native drivers taken and converted from linux and
> freebsd? The oskit has the drivers from linux 2.2.something. Wouldn't it
> be better to find a way to use them in a pluggable fashion? Is that even
> possible?
I don't know what Michael Hohmuth is exactly doing w.r.t. this framework
or how it is supposed to work. We'll have to get more details on this
before.

> >    vk-l4/nameserver/  # naming service (a.k.a. rendez-vous, port-manager...)
> 
> Would this be used by all the personalities?
Why not? There is obviously a need to locate ports (not in the Hurd sense!)
for contacting the pager, the superdriver and other basic vk-l4 tasks.
One way would be to hard-code the task/thread-IDs into header files
and have the OS-personalities use them. This is obviously not desirable,
because you can't extend/modify vk-l4 this way.
Another way would be to hardcode just the nameserver's lookup task/thread-ID
in include files and have
  * the vk-l4 tasks  register with the nameserver
  * the os-personalities query the nameserver for the other vk-l4 tasks.
This is absolutely Hurd-neutral and can be used by all personalities.

Please don't confuse vk-l4/nameserver/ with l4hurd/portserver/.

> >    # Here come the OS personalities using VK/L4:
> >    l4hurd/            # Changes for the Hurd multiserver w.r.t. savannah.
> >    l4linux/           # Changes to L4Linux to adapt it to new vk-l4 
> > framework
> >    l4bsd/             # Anyone interested to port NetBSD to vk-l4? ;-)
> >    l4<whatever>/      # some other OS personality using vk-l4
> > 
> > This way, we'd incrementally contribute code to a general purpose
> > infrastructure on top of L4 upon which we could not only base the
> > Hurd, but also other single servers (L4Linux, L4BSD<vaporware right 
> > now>,...)
> > 
> > What do you think?
> 
> It sounds like a nice idea if I am understanding it correctly. My only
> question is if this is maybe biting off more than we can chew initially.
> Would this distract from the initial goal of Hurd on L4?
Well, l4bsd and l4<whatever> are certainly distinct projects. l4linux
would be nice to adapt to vk-l4, if somewhat finds the time to do it,
but its not absolutely necessary right now.

We certainly need the vk-l4 infrastructure before we can even think
to boot a minimal Hurd/L4. Without a reasonable pager (vm_*()) and
drivers, everything else will be moot.

> -- Ian

-Farid.

-- 
Farid Hajji -- Unix Systems and Network Admin | Phone: +49-2131-67-555
Broicherdorfstr. 83, D-41564 Kaarst, Germany  | address@hidden
- - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - -
One OS To Rule Them All And In The Darkness Bind Them... --Bill Gates.