[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: mdemo ltdl failure

From: Charles Wilson
Subject: Re: mdemo ltdl failure
Date: Sat, 17 Mar 2007 14:11:00 -0500
User-agent: Thunderbird (Windows/20070221)

Charles Wilson wrote:
Charles Wilson wrote:
I'll whip up a patch and post it to the newlib list.

So, I posted the following:

However, there's no telling how long it'll be before a new cygwin kernel is released that includes the patch, and besides, libtool ought to work on current and reasonably recent cygwin kernels. So, how should libtool address this issue in the interim? Always use libltdl's argz stuff on cygwin, until <further notice: version check on uname, etc etc>?

If so, how do we induce that

Here's my attempt:

(1) if argz is found on the system, perform a runtime test to see if the system-provided argz facility actually works

(2) if so, everything is fine

(3) if not, then we need to use the libltdl-provided argz
    (a) define a new symbol SYSTEM_ARGZ_IS_BROKEN
    (b) set ARGZ_H and AC_LIBOBJ, just as if we didn't find
        argz on the system
    (c) use SYSTEM_ARGZ_IS_BROKEN in lt__glibc.h along with
        HAVE_ARGZ_H to determine whether to
        #define argz* lt__argz*

(4) if cross-compiling, be pessimistic: assume the target system's
    argz facilities are broken, and use libltdl's argz -- see (3).

The tricky part is the runtime test itself -- detecting malloc misbehavior is hard. I found a mechanism that reliably fails with current cygwin/newlib, passes with a cygwin/newlib containing this fix ( ), and passes on linux.

Even though I am pessimistic on cross-compiles, the test itself is "optimistic": if it cannot set up malloc'ed memory in the manner it expects, it returns success. The only hazard of a false failure, is if the test program cannot compile.

So, here's how the test program works:

(1) allocate a big chunk of memory. Later, we will free this memory, providing a nice target chunk for argz to realloc into.

We will try to "bracket" the memory allocated to argz initially. Most mallocs, if a specific allocation needs to realloc larger, will avoid copying by simply increasing the memory assigned to the current allocation without moving it, if possible. We want to ensure that this action is NOT possible, by allocating immovable memory immediately before and immediately after the initial argz allocation. However, arena-based mallocs can defeat this if our small "blocker" allocations are in one arena, but the larger initial argz allocation is in a different arena. We detect this situation and give up: return success (e.g. go ahead and use system argz).

(2) So, allocate a "below" blocker chunk of memory

(3) use system argz facilities a bit -- this will cause argz allocations.

(4) Allocate an "above" blocker chunk of memory. Make sure the requested size is larger than abs(argz - below). Otherwise, smart mallocs may slot this blocker in between "below" and argz. However, check that abs(argz - below) is not unreasonably large: arena mallocs may have allocated "below" and argz from completely different arenaa, in totally different locations. If abs(argz - below) is "big", assume that malloc is "weird" in this way, and give up: return success (go ahead and use system argz)

(5) make sure either
      "below" < argz < "above", or
      "above" < argz < "below"
otherwise, give up: system malloc is "wierd" -- arena based or something. return success: go ahead and use system argz

(6) free the memory allocated in step (1), to provide a nice, tempting chunk of memory for the next step to realloc into.

(7) Use argz_insert() to insert a really large chunk at the front of *argz. In the failure mode I'm trying to detect, this will realloc argz into the memory freed in step 6, just below the "below" blocker. The buggy argz_insert will then copy from this uninitialized memory into the OLD, now freed, argz memory. That's strike one. Strike two: it copies (old)argz_len bytes, but offset by len(new_entry) -- which means it overwrites past the end of the old *argz buffer by len(new_entry) bytes -- and clobbers the book-keeping info and some of the contents of one of our blockers. Strike three: freeing that blocker will coredump, since its bookkeeping info is scrogged. (Actually, with a checking-malloc, any one of the three strikes above will directly cause a coredump.)

Now, there are a bunch of ways that a broken argz could "get lucky" and pass this test. That's okay: since this test is run on all platforms if system argz is found, I wanted to be lenient: few to zero false failures. Because we've introduced a new symbol (and I use AC_CACHE), the following remedies are available:

(1) Hopefully this never happens, but a false failure report could be overridden by setting $lt_cv_sys_argz_works=yes, and the runtime test will be skipped.

(2) It's possible that on some (broken) cygwin system, the malloc might not behave as expected and the test program return success. This can be overridden by setting $lt_cv_sys_argz_works=no.

(3) Once cygwin is fixed, the libtool distributor (me) might want to ensure that the distributed libltdl is usable on systems with an older (unfixed) cygwin kernel. See #2.

(4) On cross-compile, argz.m4 is pessimistic, and assumes the target system argz is broken. You can force the use of the target system argz -- see (1). This is a little odd: even though the autoconf documentation recommends pessimism for AC_TRY_RUN's cross-compile clause, perhaps we should be optimistic instead?


One other change: you can now force the use of libltdl's argz on any system, which was not possible before when HAVE_ARGZ_H was true. BUT, doing this with HAVE_ARGZ_H true carries some risk: we've already decided whether to #define error_t ourselves, and/or set __error_t_defined, based on what we detected AFTER #including the system's argz.h -- but by forcing the use of libltdl's argz, we won't use the system's argz.h, so at compile-time we might not "see" what the error_t test "saw".

Therefore, libltdl's argz_.h needs to include most of the common places where error_t may have been "picked up" by the system argz.h. On newlib systems, this is <errno.h>. On glibc systems, this is also <errno.h>, but you must #define __need_error_t first (which glibc's argz.h does). So, I've modified libltdl's argz_.h to do that, too.


I'm still using the -export-symbols-regex ".*" fix for libmlib in tests/mdemo -- it's also included in this patch.

I've tested this under the following conditions:

(1) broken cygwin kernel
    --detects failure, builds successfully using libltdl's argz
    --resultant libraries also work after dropping in a
      fixed cygwin kernel.
(2) fixed cygwin kernel
    --detects success, builds successfully using system argz
    --resultant libraries coredump if you drop in a broken cygwin
      kernel after the fact.  This is expected: broken cygwin is
      "broken" precisely because its argz facility coredumps on
(3) fixed cygwin kernel, but with $lt_cv_sys_argz_works=no.
    --does not even try to run the test program, and successfully
      builds using libltdl's argz
    --resultant libraries works fine even after dropping in a
      broken cygwin kernel.
(4) linux (whose system argz is OK)
    --detects success, builds using system argz, works.
(5) linux, but with $lt_cv_sys_argz_works=no.
    --doesn't run the test, and builds using libltdl argz. works.
(6) mingw, which doesn't have any system argz facility at all
    --the test is not run
    --builds successfully with libltdl's argz

Full test suite results:
cygwin, broken kernel (case 1, above):
   All 115 tests passed (old test suite)
   [libtool 2.1a] testsuite: 14 16 33 34 35 49 failed
      16, 49:   known failures on this platform
      14:       java -- I dunno if gcj even works on cygwin
      33,34,35: new regressions in CVS between 20070205 and 20070316
                but I get these failures both with and without the
                attached patch

mingw: (case 6, above)
   All 115 tests passed (old test suite)
   [libtool 2.1a] testsuite: 14 16 19 23 35 39 42 49 failed
      16, 49:   known failures on this platform
      14:       java -- I dunno if gcj even works on mingw
      23,35|    anything that uses DESTDIR is broken on msys
      39,42|    because C:/DESTDIR + C:/PREFIX == C:/DESTDIRC:/PREFIX
                and that's just not right.
      19: -- I think this is pre-existing.


2007-03-17  Charles Wilson  <address@hidden>

        * libltdl/argz_.h: ensure error_t definition is obtained
        in same mechanism system argz.h would have.
        * libltdl/libltdl/lt__glibc.h: also detect if
        SYSTEM_ARGZ_IS_BROKEN when determining whether to re#def
        argz* functions.
        * libltdl/m4/argz.m4: add new runtime test to check if
        system argz facilities actually work.  Be lenient within
        test, but pessimistic on cross-compile.
        * tests/mdemo/ -export-symbols-regex fix for

Index: libltdl/argz_.h
RCS file: /cvsroot/libtool/libtool/libltdl/argz_.h,v
retrieving revision 1.6
diff -u -r1.6 argz_.h
--- libltdl/argz_.h     4 Sep 2006 17:23:30 -0000       1.6
+++ libltdl/argz_.h     17 Mar 2007 06:09:46 -0000
@@ -31,6 +31,8 @@
 #define LT__ARGZ_H 1
 #include <stdlib.h>
+#define __need_error_t
+#include <errno.h>
 #include <sys/types.h>
 #if defined(LTDL)
Index: libltdl/libltdl/lt__glibc.h
RCS file: /cvsroot/libtool/libtool/libltdl/libltdl/lt__glibc.h,v
retrieving revision 1.7
diff -u -r1.7 lt__glibc.h
--- libltdl/libltdl/lt__glibc.h 26 Oct 2006 20:39:04 -0000      1.7
+++ libltdl/libltdl/lt__glibc.h 17 Mar 2007 06:09:49 -0000
@@ -36,7 +36,7 @@
 #  include <config.h>
-#if !defined(HAVE_ARGZ_H)
+#if !defined(HAVE_ARGZ_H) || defined(SYSTEM_ARGZ_IS_BROKEN)
 /* Redefine any glibc symbols we reimplement to import the
    implementations into our lt__ namespace so we don't ever
    clash with the system library if our clients use argz_*
Index: libltdl/m4/argz.m4
RCS file: /cvsroot/libtool/libtool/libltdl/m4/argz.m4,v
retrieving revision 1.3
diff -u -r1.3 argz.m4
--- libltdl/m4/argz.m4  25 Mar 2006 11:05:02 -0000      1.3
+++ libltdl/m4/argz.m4  17 Mar 2007 06:09:50 -0000
@@ -27,6 +27,142 @@
 AC_CHECK_FUNCS([argz_append argz_create_sep argz_insert argz_next \
        argz_stringify], [], [ARGZ_H=argz.h; AC_LIBOBJ([argz])])
+dnl if success, do a runtime check to ensure it actually works
+if test -z "$ARGZ_H"; then
+       [if argz actually works],
+       [lt_cv_sys_argz_works],[dnl
+#include <stddef.h>
+#include <stdlib.h>
+#include <string.h>
+#if defined(HAVE_ARGZ_H)
+# include <argz.h>
+int main()
+    error_t e;
+    const char* in     = "a long string so that we will eventually realloc;"
+                         "and another entry that is fairly long";
+    const char* add    = "add this to the end of the argz vector";
+    const char* insert = "insert at the beginning. This should force a 
+    char* argz = 0;
+    size_t len = 0;
+    size_t cnt = 0;
+    char* some_freed_space = 0;
+    char* blocker_low = 0;
+    char* blocker_high = 0;
+    int rc = 0;
+    ptrdiff_t low_sign = 0;
+    ptrdiff_t high_sign = 0;
+    some_freed_space = (char*)malloc(1000*sizeof(char));
+    strcpy(some_freed_space, "SOME FREED SPACE");
+    blocker_low = (char*)malloc(20*sizeof(char));
+    strcpy(blocker_low, "DUMMY BLOCKER LOW");
+    e = argz_create_sep(in, ';', &argz, &len);
+    if (e)
+    {
+        free(blocker_low);
+        free(some_freed_space);
+       return 1;
+    }
+    low_sign = (blocker_low - argz > 0 ? 1 : -1);
+    cnt = argz_count(argz, len);
+    if (cnt != 2)
+    {
+        free(argz);
+        free(blocker_low);
+        free(some_freed_space);
+        return 1;
+    }
+    e = argz_add(&argz, &len, add);
+    if (e)
+    {
+        free(argz);
+        free(blocker_low);
+        free(some_freed_space);
+        return 1;
+    }
+    cnt = argz_count(argz, len);
+    if (cnt != 3)
+    {
+        free(argz);
+        free(blocker_low);
+        free(some_freed_space);
+        return 1;
+    }
+    {
+        ptrdiff_t fillup = argz - blocker_low;
+        if (fillup < 0) fillup = -fillup;
+        if (fillup > 10000)
+        {
+            /* give up and assume argz works; we don't understand
+               malloc's memory strategy */
+            free(some_freed_space);
+            free(blocker_low);
+            free(argz);
+            return 0;
+        }
+        blocker_high = (char*)malloc(2*fillup*sizeof(char));
+        strncpy(blocker_high, "DUMMY BLOCKER HIGH", 2*fillup);
+    }
+    high_sign = (blocker_high - argz > 0 ? 1 : -1);
+    /* if high_sign and low_sign are the same, the our blockers have
+       not boxed in argv.  Give up, and assume argz works */
+    if ( high_sign * low_sign > 0 )
+    {
+        free(some_freed_space);
+        free(blocker_high);
+        free(blocker_low);
+        free(argz);
+        return 0;
+    }
+    /* now free the freed space in hopes that realloc will use it */
+    free(some_freed_space);
+    e = argz_insert(&argz, &len, argz, insert);
+    if (e)
+    {
+        free(argz);
+        free(blocker_low);
+        free(blocker_high);
+        return 1;
+    }
+    cnt = argz_count(argz, len);
+    if (cnt != 4)
+    {
+        free(argz);
+        free(blocker_low);
+        free(blocker_high);
+        return 1;
+    }
+    free(argz);
+    free(blocker_low);
+    free(blocker_high);
+    return 0;
+AS_IF([test $lt_cv_sys_argz_works != yes],
+                 [This value is set to 1 to indicate that the system argz 
facility does not work])
+       ARGZ_H=argz.h
+       AC_LIBOBJ([argz])],
Index: tests/mdemo/
RCS file: /cvsroot/libtool/libtool/tests/mdemo/,v
retrieving revision 1.9
diff -u -r1.9
--- tests/mdemo/     26 Feb 2007 07:44:25 -0000      1.9
+++ tests/mdemo/     17 Mar 2007 06:09:50 -0000
@@ -37,7 +37,7 @@
 libmlib_la_SOURCES = mlib.c
 libmlib_la_LIBADD = @LIBLTDL@ "-dlopen" "-dlopen"
-libmlib_la_LDFLAGS = -no-undefined
+libmlib_la_LDFLAGS = -no-undefined -export-symbols-regex ".*"
 noinst_HEADERS = foo.h

reply via email to

[Prev in Thread] Current Thread [Next in Thread]