[PATCH] pfinet: check return value of mmap() in S_io_read() and S_socket_recv()

[olafBuddenhagen]

From: Olaf Buddenhagen <antrik@users.sf.net>
Date: Sun, 18 Jul 2010 02:33:40 +0200
Subject: [PATCH] pfinet: check return value of mmap() in S_io_read() and 
S_socket_recv()

---
I realized that my problems with pfinet crashing after apt-get fetching
~90 MB of packages, are due to a gaping virtual memory leak in pfinet.
(Probably the same one Samuel fixed recently; but I haven't tested his
patches yet...)

The result of the memory leak is that after all virtual memory space has
been exhausted, mmap() fails. However, the return status of mmap() isn't
checked, and pfinet tries to use the invalid pointer later on.

This patch fixes this, and makes pfinet return an error to the caller of
the respective RPC, when mmap() fails.

 pfinet/io-ops.c     |    5 +++++
 pfinet/socket-ops.c |    5 +++++
 2 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/pfinet/io-ops.c b/pfinet/io-ops.c
index 21bc3ac..ce652b9 100644
--- a/pfinet/io-ops.c
+++ b/pfinet/io-ops.c
@@ -87,6 +87,11 @@ S_io_read (struct sock_user *user,
   if (amount > *datalen)
     {
       *data = mmap (0, amount, PROT_READ|PROT_WRITE, MAP_ANON, 0, 0);
+      if (*data == MAP_FAILED)
+        /* Should check whether errno is indeed ENOMEM --
+           but this can't be done in a straightforward way,
+           because the glue headers #undef errno. */
+        return ENOMEM;
       alloced = 1;
     }
 
diff --git a/pfinet/socket-ops.c b/pfinet/socket-ops.c
index baeaad3..b9ce6c7 100644
--- a/pfinet/socket-ops.c
+++ b/pfinet/socket-ops.c
@@ -499,6 +499,11 @@ S_socket_recv (struct sock_user *user,
   if (amount > *datalen)
     {
       *data = mmap (0, amount, PROT_READ|PROT_WRITE, MAP_ANON, 0, 0);
+      if (*data == MAP_FAILED)
+        /* Should check whether errno is indeed ENOMEM --
+           but this can't be done in a straightforward way,
+           because the glue headers #undef errno. */
+        return ENOMEM;
       alloced = 1;
     }
 
-- 
1.6.6.1