Aloha -
I've written a short program (attached) that demonstrates how libpager's support for only a single client can be used to mount a denial of service attack against the kernel.
It works by opening a file, grabbing its associated memory object (if it can), and holding it until you hit CNTL-C. Nothing more than read access is required.
If successful, the kernel can not exec the file, because it needs a memory object to mmap() the file, and the program is already holding libpager's single memory object.
It seems like once the kernel execs a file, it continues to hold the memory object, so the attack, to be successful, needs to be against programs that have never been exec'ed. It's therefore "best" run on a cleanly booted system.
An unprivileged user can run "
grab-memory-objects /bin/*" and disrupt the whole works.
Even worse, any attempt to exec one of these files then leaves it in a state where it can never be exec'ed, even if grab-memory-objects is killed. The file remains hosed until shutdown, when we get the following sequence:
startup: notifying tmpfs none of halt...done
startup: notifying ext2fs device:hd0s1 of halt...(ipc/rcv) timed out
startup: halting Mach (flags 0x8)...It's a 60 second timeout that must terminate the ext2fs translator abnormally, because the file system is left dirty.
So, there's several problems here:
1. libpager can't handle multiple clients
2. the kernel can't recover from a failed attempt to get a file's memory object
3. ext2fs can't cleanly shutdown in this case
I'm continuing to lobby for a multi-client libpager! I can see that it's going to raise a lot of locking and concurrency issues, but this program demonstrates that we've already got problems with the current scheme. Even a simple multi-client libpager should allow shared read-only access, which would prevent an unprivileged user from mounting this attack. Root, with write access to the files in /bin, could still do it, though.
grab-memory-objects.cc