PATCH: --enable-pathenv option to controlSHLIB_PATH/LD_LIBRARY_PATH behavior on Linux/HP-UX

[Alexander Dupuy]

While I'm on a roll submitting libtool patches...

We want to ensure (wherever possible) that the embedded RPATH in shared 
libraries and executables can be overridden by environment variables. 
(Apparently, this is a big issue for the Debian folks, and is a reason 
that they reject use of embedded RPATH).

On some ELF platforms (Linux definitely, but not *BSD), it is possible 
to have the environment variable override the embedded runpath with the 
GNU ld flag --enable-new-dtags, which uses DT_RUNPATH instead of 
DT_RPATH.  HP-UX also provides mechanisms for controlling the priority 
of the environment variable vs. the embedded path (and even for turning 
off use of the environment variable entirely).

The attached patch (relative to vanilla libtool-1.5) adds a new 
configure option for libtool, --enable-pathenv:

--enable-pathenv=first  search environment path before hardcoded library
                        path
--enable-pathenv=last   search hardcoded library path before environment
                        path
--enable-pathenv=never
--disable-pathenv       never search environment path

This is a configure option, not a libtool option, and unlike 
--enable-shared/static, there is no ability to set it only for some 
packages.  Making it a libtool option would get more complex, since 
libtool sets $shlibpath_overrides_runpath only at configuration time, 
and inconsistent usage of the option between mode=link and mode=install 
could create problems.

Nonetheless, I think others may find the functionality useful.  I've 
cc'd some people who submitted HP-UX patches that enabled SHLIB_PATH 
searching, since I am unable to test HP-UX operation myself (although I 
read the HP-UX docs on the web pretty closely, and have every 
expectation that this should work).  I have tested the changes on Linux 
and FreeBSD 4.8 (where the --enable-new-dtags option, although supported 
by the linker, has no effect on the runtime linker behavior).

Before releasing this generally, it might be nice to have a new autoconf 
macro that actually tests whether --enable-new-dtags causes the 
environment variable to take precedence, since (as noted in the patched 
code) early versions of Linux don't support DT_RUNPATH correctly.  (It 
may be possible to key this decision off of strings embedded in 
/lib/ld-linux.so, but it seems better to have a true test of the 
functionality.)  In any case, writing such an autoconf macro is a bit 
complex (and has obvious limitations in a cross-compilation 
environment), and I don't have time to do this just yet.

However, it's worth pointing out that unless the application uses the 
AC_ENABLE_PATHENV or AC_DISABLE_PATHENV macros to set a default, or the 
user specifies --enable-pathenv=xxx or --disable-pathenv, the resulting 
libtool behavior is compatible with current behavior, so it is unlikely 
that this would bite anyone who had no idea what the potential 
problems/issues were.

@alex
--
mailto:address@hidden

Index: libtool.m4
===================================================================
RCS file: /tmp/libtool-1.5/libtool.m4,v
retrieving revision 1.1
diff -u -r1.1 libtool.m4
--- libtool.m4  5 Jun 2003 23:23:26 -0000       1.1
+++ libtool.m4  5 Jun 2003 23:21:55 -0000
@@ -97,6 +97,7 @@
 AC_REQUIRE([AC_ENABLE_SHARED])dnl
 AC_REQUIRE([AC_ENABLE_STATIC])dnl
 AC_REQUIRE([AC_ENABLE_FAST_INSTALL])dnl
+AC_REQUIRE([AC_ENABLE_PATHENV])dnl
 AC_REQUIRE([AC_CANONICAL_HOST])dnl
 AC_REQUIRE([AC_CANONICAL_BUILD])dnl
 AC_REQUIRE([AC_PROG_CC])dnl
@@ -229,6 +230,19 @@
 tagname=
 AC_LIBTOOL_LANG_C_CONFIG
 _LT_AC_TAGCONFIG
+
+if test "$enable_pathenv" != default &&
+    test "$enable_pathenv" != "$shlibpath_overrides_runpath"; then
+  case "$shlibpath_overrides_runpath-$enable_pathenv" in
+  never-no) : close enough not to care;;
+  never-*) AC_MSG_WARN([environment variable never used for search]) ;;
+  *-any) : anything other than never is good enough ;;
+  yes-*) AC_MSG_WARN([`$shlibpath_var' overrides hardcoded path]) ;;
+  no-*) AC_MSG_WARN([hardcoded path overrides environment variable]) ;;
+  *-no) AC_MSG_WARN([`$shlibpath_var' might override hardcoded path]) ;;
+  *-yes) AC_MSG_WARN([hardcoded path might override environment variable]) ;;
+  esac
+fi
 ])# AC_LIBTOOL_SETUP
 
 
@@ -1118,6 +1132,8 @@
   version_type=linux
   library_names_spec='${libname}${release}${shared_ext}$versuffix $libname.a'
   shlibpath_var=LIBPATH
+  # PORTME: what is the behavior of LIBPATH for AIX 3?
+  shlibpath_overrides_runpath=unknown
 
   # AIX 3 has no versioning support, so we append a major version to the name.
   soname_spec='${libname}${release}${shared_ext}$major'
@@ -1132,6 +1148,8 @@
     # AIX 5 supports IA64
     library_names_spec='${libname}${release}${shared_ext}$major 
${libname}${release}${shared_ext}$versuffix $libname${shared_ext}'
     shlibpath_var=LD_LIBRARY_PATH
+    # PORTME: what is the behavior of LD_LIBRARY_PATH for AIX 5L/IA64?
+    shlibpath_overrides_runpath=unknown
   else
     # With GCC up to 2.95.x, collect2 would create an import file
     # for dependence libraries.  The import file would start with
@@ -1164,6 +1182,7 @@
       soname_spec='${libname}${release}${shared_ext}$major'
     fi
     shlibpath_var=LIBPATH
+    shlibpath_overrides_runpath=yes
   fi
   ;;
 
@@ -1186,6 +1205,7 @@
   soname_spec='${libname}${release}${shared_ext}$major'
   finish_cmds='PATH="\$PATH:/sbin" ldconfig $libdir'
   shlibpath_var=LD_LIBRARY_PATH
+  shlibpath_overrides_runpath=no
   sys_lib_search_path_spec="/shlib /usr/lib /usr/X11/lib /usr/contrib/lib /lib 
/usr/local/lib"
   sys_lib_dlsearch_path_spec="/shlib /usr/lib /usr/local/lib"
   # the default ld.so.conf also contains /usr/contrib/lib and
@@ -1308,6 +1328,8 @@
     hardcode_into_libs=yes
     ;;
   *) # from 3.2 on
+    # PORTME: if $enable_pathenv=yes check if --enable-new-dtags works AND
+    # DT_RUNPATH honored by ld-elf.so (not the case as of freebsd-4.8).
     shlibpath_overrides_runpath=no
     hardcode_into_libs=yes
     ;;
@@ -1321,6 +1343,9 @@
   library_names_spec='${libname}${release}${shared_ext}$versuffix 
${libname}${release}${shared_ext}${major} ${libname}${shared_ext}'
   soname_spec='${libname}${release}${shared_ext}$major'
   shlibpath_var=LD_LIBRARY_PATH
+  # PORTME: if $enable_pathenv=yes check if --enable-new-dtags works AND
+  # DT_RUNPATH honored by runtime linker (unknown whether this is the case)
+  shlibpath_overrides_runpath=no
   hardcode_into_libs=yes
   ;;
 
@@ -1336,7 +1361,18 @@
     hardcode_into_libs=yes
     dynamic_linker="$host_os dld.so"
     shlibpath_var=LD_LIBRARY_PATH
-    shlibpath_overrides_runpath=yes # Unless +noenvvar is specified.
+    shlibpath_overrides_runpath=yes
+    # PORTME: this ought to work, but I cannot test as I have no hp-ux systems
+    case "$enable_pathenv" in
+    no) _LT_AC_TAGVAR(hardcode_libdir_flag_spec, 
$1)="$_LT_AC_TAGVAR(hardcode_libdir_flag_spec, $1) ${wl}+rpathfirst"
+       _LT_AC_TAGVAR(hardcode_libdir_flag_spec_ld, 
$1)="$_LT_AC_TAGVAR(hardcode_libdir_flag_spec_ld, $1) +rpathfirst"
+       shlibpath_overrides_runpath=no
+       ;;
+    never) _LT_AC_TAGVAR(hardcode_libdir_flag_spec, 
$1)="$_LT_AC_TAGVAR(hardcode_libdir_flag_spec, $1) ${wl}+noenvvar"
+       _LT_AC_TAGVAR(hardcode_libdir_flag_spec_ld, 
$1)="$_LT_AC_TAGVAR(hardcode_libdir_flag_spec_ld, $1) +noenvvar"
+       shlibpath_overrides_runpath=never
+       ;;
+    esac
     library_names_spec='${libname}${release}${shared_ext}$versuffix 
${libname}${release}${shared_ext}$major $libname${shared_ext}'
     soname_spec='${libname}${release}${shared_ext}$major'
     if test "X$HPUX_IA64_MODE" = X32; then
@@ -1346,22 +1382,53 @@
     fi
     sys_lib_dlsearch_path_spec=$sys_lib_search_path_spec
     ;;
-   hppa*64*)
-     shrext='.sl'
-     hardcode_into_libs=yes
-     dynamic_linker="$host_os dld.sl"
-     shlibpath_var=LD_LIBRARY_PATH # How should we handle SHLIB_PATH
-     shlibpath_overrides_runpath=yes # Unless +noenvvar is specified.
-     library_names_spec='${libname}${release}${shared_ext}$versuffix 
${libname}${release}${shared_ext}$major $libname${shared_ext}'
-     soname_spec='${libname}${release}${shared_ext}$major'
-     sys_lib_search_path_spec="/usr/lib/pa20_64 /usr/ccs/lib/pa20_64"
-     sys_lib_dlsearch_path_spec=$sys_lib_search_path_spec
-     ;;
-   *)
+  hppa*64*)
+    shrext='.sl'
+    hardcode_into_libs=yes
+    dynamic_linker="$host_os dld.sl"
+    shlibpath_var=LD_LIBRARY_PATH # How should we handle SHLIB_PATH
+    shlibpath_overrides_runpath=yes
+    # PORTME: this ought to work, but I cannot test as I have no hp-ux systems
+    case "$enable_pathenv" in
+    no) _LT_AC_TAGVAR(hardcode_libdir_flag_spec, 
$1)="$_LT_AC_TAGVAR(hardcode_libdir_flag_spec, $1) ${wl}+rpathfirst"
+       _LT_AC_TAGVAR(hardcode_libdir_flag_spec_ld, 
$1)="$_LT_AC_TAGVAR(hardcode_libdir_flag_spec_ld, $1) +rpathfirst"
+       shlibpath_overrides_runpath=no
+       ;;
+    never) _LT_AC_TAGVAR(hardcode_libdir_flag_spec, 
$1)="$_LT_AC_TAGVAR(hardcode_libdir_flag_spec, $1) ${wl}+noenvvar"
+       _LT_AC_TAGVAR(hardcode_libdir_flag_spec_ld, 
$1)="$_LT_AC_TAGVAR(hardcode_libdir_flag_spec_ld, $1) +noenvvar"
+       shlibpath_overrides_runpath=never
+       ;;
+    esac
+    library_names_spec='${libname}${release}${shared_ext}$versuffix 
${libname}${release}${shared_ext}$major $libname${shared_ext}'
+    soname_spec='${libname}${release}${shared_ext}$major'
+    sys_lib_search_path_spec="/usr/lib/pa20_64 /usr/ccs/lib/pa20_64"
+    sys_lib_dlsearch_path_spec=$sys_lib_search_path_spec
+    ;;
+  *)
     shrext='.sl'
     dynamic_linker="$host_os dld.sl"
     shlibpath_var=SHLIB_PATH
-    shlibpath_overrides_runpath=no # +s is required to enable SHLIB_PATH
+    # FIXME: Note that +s is not always sufficient to force SHLIB_PATH search;
+    # dependencies with full paths for internal names ($soname, specified with
+    # +h), or (without an internal name? and) specified on the link line with
+    # full path, are marked "static" and only the embedded path is used.  As
+    # libtool always uses +h with $soname_spec, which has no path, this should
+    # not be an issue for libraries generated with libtool; it may be an issue
+    # for 3rd-party or system libraries.  SHLIB_PATH search can be forced by
+    # using chatr -l for each dependency to switch it from static to dynamic.
+    # [see http://mail.gnu.org/archive/html/libtool/2002-05/msg00069.html]
+    shlibpath_overrides_runpath=never
+    # PORTME: this ought to work, but I cannot test as I have no hp-ux systems
+    case "$enable_pathenv" in
+    yes) _LT_AC_TAGVAR(hardcode_libdir_flag_spec, $1)="${wl}+s 
$_LT_AC_TAGVAR(hardcode_libdir_flag_spec, $1)"
+       _LT_AC_TAGVAR(hardcode_libdir_flag_spec_ld, $1)="+s 
$_LT_AC_TAGVAR(hardcode_libdir_flag_spec_ld, $1)"
+       shlibpath_overrides_runpath=yes
+       ;;
+    no) _LT_AC_TAGVAR(hardcode_libdir_flag_spec, 
$1)="$_LT_AC_TAGVAR(hardcode_libdir_flag_spec, $1) ${wl}+s"
+       _LT_AC_TAGVAR(hardcode_libdir_flag_spec_ld, 
$1)="$_LT_AC_TAGVAR(hardcode_libdir_flag_spec_ld, $1) +s"
+       shlibpath_overrides_runpath=no
+       ;;
+    esac
     library_names_spec='${libname}${release}${shared_ext}$versuffix 
${libname}${release}${shared_ext}$major $libname${shared_ext}'
     soname_spec='${libname}${release}${shared_ext}$major'
     ;;
@@ -1401,6 +1468,8 @@
     ;;
   esac
   shlibpath_var=LD_LIBRARY${shlibsuff}_PATH
+  # PORTME: if $enable_pathenv=yes check if --enable-new-dtags works AND
+  # DT_RUNPATH honored by runtime linker (unknown whether this is the case)
   shlibpath_overrides_runpath=no
   sys_lib_search_path_spec="/usr/lib${libsuff} /lib${libsuff} 
/usr/local/lib${libsuff}"
   sys_lib_dlsearch_path_spec="/usr/lib${libsuff} /lib${libsuff}"
@@ -1421,10 +1490,15 @@
   soname_spec='${libname}${release}${shared_ext}$major'
   finish_cmds='PATH="\$PATH:/sbin" ldconfig -n $libdir'
   shlibpath_var=LD_LIBRARY_PATH
-  shlibpath_overrides_runpath=no
-  # This implies no fast_install, which is unacceptable.
-  # Some rework will be needed to allow for fast_install
-  # before this can be enabled.
+  if test "$enable_pathenv" = yes; then
+    # FIXME: unpatched RedHat prior to 7.3 doesn't correctly implement
+    # DT_RUNPATH [see https://rhn.redhat.com/errata/RHSA-2001-160.html]
+    # (need new autoconf macro to test correct operation of --enable-new-dtags)
+    _LT_AC_TAGVAR(hardcode_libdir_flag_spec, 
$1)="$_LT_AC_TAGVAR(hardcode_libdir_flag_spec, $1) ${wl}--enable-new-dtags"
+    shlibpath_overrides_runpath=yes
+  else
+    shlibpath_overrides_runpath=no
+  fi
   hardcode_into_libs=yes
 
   # We used to test for /lib/ld.so.1 and disable shared libraries on
@@ -1508,6 +1582,12 @@
   soname_spec='${libname}${release}${shared_ext}$major'
   library_names_spec='${libname}${release}${shared_ext}$versuffix 
${libname}${release}${shared_ext}$major $libname${shared_ext}'
   shlibpath_var=LD_LIBRARY_PATH
+  shlibpath_overrides_runpath=no
+  # PORTME: this ought to work, but I cannot test as I have no Tru64 systems
+  if test "$enable_pathenv" = never; then
+    _LT_AC_TAGVAR(hardcode_libdir_flag_spec, 
$1)="$_LT_AC_TAGVAR(hardcode_libdir_flag_spec, $1) ${wl}-no_library_replacement"
+    shlibpath_overrides_runpath=never
+  fi
   sys_lib_search_path_spec="/usr/shlib /usr/ccs/lib /usr/lib/cmplrs/cc 
/usr/lib /usr/local/lib /var/shlib"
   sys_lib_dlsearch_path_spec="$sys_lib_search_path_spec"
   ;;
@@ -1825,6 +1905,46 @@
 ])# AC_DISABLE_FAST_INSTALL
 
 
+# AC_ENABLE_PATHENV([DEFAULT])
+# ---------------------------------
+# implement the --enable-pathenv flag
+# DEFAULT is `first', `last', `never', `any', or `default' (if omitted).
+AC_DEFUN([AC_ENABLE_PATHENV],
+[define([AC_ENABLE_PATHENV_DEFAULT], ifelse($1, [], default, $1))dnl
+AC_ARG_ENABLE([pathenv],
+    [AC_HELP_STRING([--enable-pathenv=first],
+       [search environment path before hardcoded library path])
+AC_HELP_STRING([--enable-pathenv=last],
+       [search hardcoded library path before environment path])
+AC_HELP_STRING([--enable-pathenv=never])
+AC_HELP_STRING([--disable-pathenv],
+       [never search environment path])],
+    [case $enableval in
+     first) enable_pathenv=yes ;;
+     last) enable_pathenv=no ;;
+     no|never) enable_pathenv=never ;;
+     default) enable_pathenv=default ;;
+     *) enable_pathenv=any ;;
+     esac],
+    [case ]AC_ENABLE_PATHENV_DEFAULT[ in
+     first) enable_pathenv=yes ;;
+     last) enable_pathenv=no ;;
+     no|never) enable_pathenv=never ;;
+     default) enable_pathenv=default ;;
+     *) enable_pathenv=any ;;
+     esac])
+])# AC_ENABLE_PATHENV
+
+
+# AC_DISABLE_PATHENV
+# -----------------------
+# set the default to --disable-pathenv
+AC_DEFUN([AC_DISABLE_PATHENV],
+[AC_BEFORE([$0],[AC_LIBTOOL_SETUP])dnl
+AC_ENABLE_PATHENV(never)
+])# AC_DISABLE_PATHENV
+
+
 # AC_LIBTOOL_PICMODE([MODE])
 # --------------------------
 # implement the --with-pic flag
@@ -2515,7 +2635,7 @@
 AC_LIBTOOL_SYS_DYNAMIC_LINKER($1)
 AC_LIBTOOL_DLOPEN_SELF($1)
 
-# Report which librarie types wil actually be built
+# Report which library types will actually be built
 AC_MSG_CHECKING([if libtool supports shared libraries])
 AC_MSG_RESULT([$can_build_shared])
 
@@ -5291,6 +5411,7 @@
 
     bsdi4*)
       _LT_AC_TAGVAR(export_dynamic_flag_spec, $1)=-rdynamic
+      runpath_var=LD_RUN_PATH
       ;;
 
     cygwin* | mingw* | pw32*)