Re: When should ralloc.c be used?

[Daniel Colascione]

On 10/28/2016 04:40 AM, Jérémie Courrèges-Anglas wrote:
> Eli Zaretskii <address@hidden> writes:
>
> > > Cc: address@hidden, address@hidden, address@hidden
> > > From: Daniel Colascione <address@hidden>
> > > Date: Fri, 28 Oct 2016 01:11:08 -0700
> > >
> > > Say I mmap (anonymously, for simplicity) a page PROT_NONE. After the
> > > initial mapping, that address space is unavailable for other uses. But
> > > because the page protections are PROT_NONE, my program has no legal
> > > right to access that page, so the OS doesn't have to guarantee that it
>                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > can find a physical page to back that page I've mmaped. In this state,
>    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> This is what I think is a problem in your reasoning.  "Doesn't have to
> guarantee" doesn't mean that the kernel *should not* actually check the
> available memory and resource limits.

IMHO, an OS that rejects big PROT_NONE mappings merely because it might 
not be able to change them to PROT_READ|PROT_WRITE later is broken. The 
non-overcommit Linux behavior (which is identical to Windows behavior) 
is the _right _thing_ _to_ _do_. The OS is letting the process manage 
its address space and assuming that the programmer knows what he wanted 
to do.

> > > the memory is reserved.
> > >
> > > The 20GB PROT_NONE address space reservation itself requires very little
> > > memory. It's just a note in the kernel's VM interval tree that says "the
> > > addresses in range [0x20000, 0x500020000) are reserved". Virtual memory is
> > >
> > > Now imagine I change the protections to PROT_READ|PROT_WRITE --- once
> > > the PROT_READ|PROT_WRITE mprotect succeeds, my program has every right
> > > to access that page; under a strict accounting scheme (that is, without
> > > overcommit), the OS has to guarantee that it'll be able to go find a
> > > physical page to back that virtual page. In this state, the memory is
> > > committed -- the kernel has committed to finding backing storage for
> > > that page at some point when the current process tries to access it.
> >
> > I'm with you up to here.  My question is whether PROT_READ|PROT_WRITE
> > call could fail after PROT_NONE succeeded.  You seem to say it could;
> > I thought it couldn't.
>
> I wouldn't have thought that PROT_NONE vs PROT_READ|PROT_WRITE would
> have changed anything here, but on *some* OSes it does, however it is
> not portable.  At least OpenBSD doesn't behave like what you describe.

How does it behave?

> IMHO people who rely on this kind of reservations rely on
> implementation-defined behavior.

OpenBSD is a Coelacanth. It's a relic. It doesn't even a unified buffe 
cache.

> Also, sanity wise, I'd prefer having mmap(2) fail right away rather than
> having mprotect(2) fail, much later.

Then ask for PROT_READ|PROT_WRITE access right away. Ask for commit, not 
just address space.

> *If* mprotect(2) actually fails ;
> of course, you don't want to play russian roulette with your OS's
> flavor of the OOM-killer either.

That's why overcommit is an abomination.