Re: Part 2: System Structure

[Jonathan S. Shapiro]

On Thu, 2006-05-18 at 13:24 +0200, Bas Wijnen wrote:
> On Wed, May 17, 2006 at 11:46:21AM -0400, Jonathan S. Shapiro wrote:
> > On Wed, 2006-05-17 at 17:20 +0200, Bas Wijnen wrote:
> > 
> > > Well, perhaps it is all much easier if the program cooperates in being
> > > debugged.  But with a transparent space bank, it is at least possible to
> > > do it with a program that doesn't cooperate.  If I understand you
> > > correctly, this is not the case with a malicious program which was
> > > originally set up to be debuggable.
> > 
> > It is not. The authority to *read* a program image does not in any way
> > convey the authority to *write* the program image (e.g. to insert BPT),
> > nor to halt the process, nor to alter the process registers.
> 
> The question was: Suppose I download a binary from the internet.  I want to
> examine it.  I put it in a constructor with some glue code to set it up for
> debugging.  I start the program.  First the glue code will hand me some
> capabilities.  Then the untrusted code starts running.  Is it possible that
> the untrusted code revokes my right to debug it?

It is possible for the program to *attempt* this. If you set up your
debugger correctly, you can prevent it from succeeding.

> > The problem is that you test the code in one environment, and execute it
> > in another. If critical code is isolated in a separate address space,
> > then the *only* way it can be invoked is through its intended interface.
> > Stray pointers and/or changes to global variables that are intended to
> > be private to the implementation cannot happen.
> 
> Sure.  But does this mean every piece of critical code should be in its own
> address space?

Yes. That is *exactly* what it means. More generally, it means that
every piece of code whose robustness is pragmatically important -- even
if it is not critical -- should be in a separate address space.

> What I'm saying
> is that the parent and the child fail as a unit (this is unidirectional, the
> child and parent don't fail as a unit).

This is not necessarily true. Even when it is true in production, it is
definitely not true for purposes of fault analysis. Also, in many cases
it is possible for programs to recover sensibly from failed
sub-computations, but only if the effects of those failed computations
are known to have been properly contained.

> > A direct consequence is that no isolation of faults exists in principle.
> > If exec() screws up, you have no idea whether the problem lies in exec()
> > or in some weird behavior of the program.
> 
> When programming with untrusted helpers, the idea is of course that the parent
> is perfect.

This seems like a really bad idea. Any time you start an argument with
"assume false is true", absolutely anything looks like a good design
decision.

> I agree that this class of bugs may be
> hard to find.  I disagree that we should "protect" ourselves from it.  This
> class of bugs are still possible with a shielded exec(), and they will still
> be hard to find.

Actually, they are many orders of magnitude less frequent when critical
code is shielded, and they are several orders of magnitude easier to
find.

You are arguing contrary to empirically established fact. Isolation
boundaries have been consistently observed to make systems several
decimal orders of magnitude more robust when used appropriately. You
are, in essence, proposing to throw away the only fundamental advantage
that a microkernel offers: the ability to isolate and contain faults.

We are operating from very different philosophies. My philosophy is:
wherever you *can* design a system in a way that makes faults better
isolated and easier to analyze, you *should*. The issue goes far beyond
what "fails as a unit" in the field. It also applies to debugging and
post-failure analysis. Code that is not isolated in the field cannot be
instrumented in the field -- which is important when you are trying to
figure out what went wrong.

> The parent is isolated from errors in the child.  The child doesn't need to be
> isolated from errors in the parent, because it is useless without the parent
> (so if the parent fails, the child should die).

It appears to me that you are not considering fault analysis at all.

> > This isn't a parent/child thing. You started this discussion by saying
> > that you were planning to put the constructor functionality into a
> > library. The concern here is the complete failure of isolation between
> > the application code and the library code/data.
> 
> There are lots of things that are in the same address space even if things
> would be more robust (in a non-practical way) when they weren't.  For things
> which fail as a unit, this is not a problem.  If xmms crashes, there's no
> point at all in protecting its plugin.  However, if the plugin crashes, it is
> useful to protect xmms.  This is exactly how things work in my proposal.

Yes, this describes your proposal, but your reasoning is flawed. It is
very desirable to know that XMMS failed because it attempted an invalid
access into its plugin (invalid in the sense "contrary to interface
specification", not in the sense "disallowed for security reasons").

> > Of course it is failing. That isn't the question. The question is:
> > 
> >   You have a million line application with a stray pointer error
> >   somewhere that manifests only when the moon is full, it is raining
> >   in shanghai, and you tap the power cord with your left foot at just
> >   the right frequency.
> > 
> >   How the @)_&% do you find it?
> 
> By debugging it when you see that it seems to happen.  Which means the memory
> must have been transparent (for reading *and* writing) for you to begin with.
> ;-)

This is exactly the wrong answer. Transparency is needed at debugging
time, but the customer isn't doing the debugging. Opacity is needed at
execution time so that pointer errors will be diagnosed as quickly as
possible.

This is true for the same reason that the page frame at virtual address
0 should never be valid.

> However, this doesn't answer your question of course. :-)  I'm saying that the
> stray pointer problem doesn't go away when you isolate exec() into its own
> address space.  It will still be a problem, and it will still need to be
> solved.

Yes, it will still be a problem. The difference is that the isolation
boundary causes the problem to be more quickly noticed, and it
exponentially reduces the number of potential causes of failure that the
developer must consider.

> > >   It's just as likely that it messes up itself, or gets a segmentation
> > >   fault.
> > 
> > The empirical evidence is that this really isn't true. A very large
> > number of stray pointers result from uninitialized data, and there is a
> > substantial likelihood that the old value at that location is a pointer
> > to something valid. In consequence, many stray pointer errors do NOT
> > result in SEGV.
> 
> Ok, sorry.  It's much more likely that it messes up itself and/or gets a
> segmentation fault.  As you say, stray pointers usually point to data which
> was in use before.  The program has no reason to point its pointers to the
> library code, so this will be something that happens less than average.

Once again: be very careful about making unsupported assumptions. What
you say in the last sentence is not correct.

It is plausible that the program will not intentionally refer into
random places in the library, but we are not talking here about
intentional references. It is VERY easy for the program to make an
unintentional reference into the library as follows:

  1. Program calls something in library.
  2. Library routine places pointer on stack, later returns.
  3. Program makes uninitialized dereference through stack,
     ends up using pointer left there by library code.

> You are correct, of course.  But there's really no reason that stray pointers
> would be pointing at library code/data.  You are in fact solving a problem
> that is too rare, IMO.

I have experience that says this isn't true. In the absence of hard
data, you are speculating.

> > > If program A wants to spawn an instance of program B, it invokes a
> > > capability to the B-constructor.  The B-constructor then builds B.  When
> > > all this has happened, A and B are both children of their own constructor,
> > > which are children of the meta-constructor.  In particular, B is not a
> > > child of A.  So B has successfully guarded its runtime memory image from A
> > > (which with these definitions isn't the parent, but I think it would be in
> > > the situation you were envisioning).
> > 
> > No, because B was built using storage provided by A, so B is transparent
> > to A, so B cannot guard its content against A.
> 
> No, it wasn't.  And so it can...

Wait. You said that B was spawned by A. If A did not provide the
storage, where did the storage come from?

> > The problem here isn't the destruction of storage per se. It is the fact
> > that the destruction of storage used by the child wasn't "all or
> > nothing".
> 
> I don't see how this would work in practice.  Say we have a filesystem server.
> The user wants to read some file.  You're saying the user will instantiate the
> whole filesystem for that?  While possible, things go wrong when writing.  If
> the operation fails half-way, the disk is corrupt.  If two users both write
> the same file, there're race conditions, possibly corrupting the disk.  With
> journalling and locking this may be solvable, but I think it's much easier
> (and therefore less bug-sensitive) with a single filesystem server, taking
> requests.
> 
> Or is this a very rare example where you do want to protect against partial
> memory failure?

File servers are a rare case: programs that must manage storage on
behalf of multiple clients. This is well known to be exceptionally hard
to deal with, and file systems must be written with great care.
Realistically, they should not use client-revocable storage at all.

But the overwhelming majority of objects are single-client, or if
multi-client, all clients are in the same storage allocation domain. For
these, the right thing to do is have the subsystem fail as a complete
unit instead of having its storage be partially violated.

Note that "serves one client" does not mean "trusts that client".


shap