Use size_t instead of int for internal variables in glob (bug 14621)

[Joseph S. Myers]

Bug 14621 is the use of int for certain internal variables in glob,
when those variables are assigned quantities derived from size_t
values.  As noted in that issue, this seems likely to be exploitable
on 64-bit systems with tens of GB of memory (I can't completely rule
out exploits with less memory or even on 32-bit systems) but I have
not attempted to construct a testcase, given the expected memory
requirements.

The code in question is used in gnulib.  gnulib has various
differences from the glibc version of this code, but they do not seem
to include anything relevant to this issue.  In fact, the gnulib
version seems to be missing some checks to avoid overflow of
expressions (newcount + 1 + 1) * sizeof (char *), where glibc has
those checks, and there is no obvious replacement in gnulib for those
checks.

As it's in gnulib it will no doubt be present in various other
packages using the gnulib code - I don't know in what circumstances
the problem gnulib code might have been statically built into binaries
on glibc systems, however.

This patch (against the glibc version, not gnulib) simply uses size_t
for the relevant variables.  Tested x86_64 and x86.

2012-09-25  Joseph Myers  <address@hidden>

        [BZ #14621]
        * posix/glob.c (next_brace_sub): Use size_t instead of unsigned
        int as type of variable DEPTH.
        (glob): Use size_t instead of int as type of variables NEWCOUNT
        and OLD_PATHC.

diff --git a/posix/glob.c b/posix/glob.c
index 68ea205..87d4f1b 100644
--- a/posix/glob.c
+++ b/posix/glob.c
@@ -217,7 +217,7 @@ static int collated_compare (const void *, const void *) 
__THROW;
 static const char *
 next_brace_sub (const char *cp, int flags)
 {
-  unsigned int depth = 0;
+  size_t depth = 0;
   while (*cp != '\0')
     if ((flags & GLOB_NOESCAPE) == 0 && *cp == '\\')
       {
@@ -960,7 +960,7 @@ glob (pattern, flags, errfunc, pglob)
                  && S_ISDIR (st.st_mode))
               : (__stat64 (dirname, &st64) == 0 && S_ISDIR (st64.st_mode)))))
        {
-         int newcount = pglob->gl_pathc + pglob->gl_offs;
+         size_t newcount = pglob->gl_pathc + pglob->gl_offs;
          char **new_gl_pathv;
 
          if (newcount > UINTPTR_MAX - (1 + 1)
@@ -1059,7 +1059,7 @@ glob (pattern, flags, errfunc, pglob)
         appending the results to PGLOB.  */
       for (i = 0; i < dirs.gl_pathc; ++i)
        {
-         int old_pathc;
+         size_t old_pathc;
 
 #ifdef SHELL
          {
@@ -1114,7 +1114,7 @@ glob (pattern, flags, errfunc, pglob)
          /* No matches.  */
          if (flags & GLOB_NOCHECK)
            {
-             int newcount = pglob->gl_pathc + pglob->gl_offs;
+             size_t newcount = pglob->gl_pathc + pglob->gl_offs;
              char **new_gl_pathv;
 
              if (newcount > UINTPTR_MAX - 2
@@ -1158,7 +1158,7 @@ glob (pattern, flags, errfunc, pglob)
     }
   else
     {
-      int old_pathc = pglob->gl_pathc;
+      size_t old_pathc = pglob->gl_pathc;
       int orig_flags = flags;
 
       if (meta & 2)

-- 
Joseph S. Myers
address@hidden