[ft-devel] [PATCH] Further fix to visibility flag testing with Solaris Studio compilers

[Alan Coopersmith]

AC_COMPILE_IFELSE only tries to compile a *.c to a *.o.
The Solaris Studio 12.1 through 12.5 compilers see the -fvisibility=hidden
flag, but ignore it with a warning of:
 cc: Warning: Option -fvisibility=hidden passed to ld, if ld is invoked,
  ignored otherwise

AC_LINK_IFELSE does the compile and then tries to link the result,
at which point the Solaris linker will issue an error:
 ld: fatal: option '-fvisibility=hidden' is incompatible with building a
  dynamic executable

If we don't use AC_LINK_IFELSE to catch the error, then configure will
fail further tests which attempt to link, such as those testing
dependencies like libbz2.

Also, don't try adding -fvisibility if we've already added -xldscope,
just use one of them, since Sun Studio 12 and earlier compilers only
issue a warning, and don't try passing through to the linker to
generate an error, so AC_LINK_IFELSE doesn't catch them.

Tested on Solaris 11.4 beta with compiler versions:
 Sun Studio 8 (Sun C 5.5)
 Sun Studio 10 (Sun C 5.7)
 Sun Studio 11 (Sun C 5.8)
 Sun Studio 12 (Sun C 5.9)
 Sun Studio 12.1 (Sun C 5.10)
 Oracle Solaris Studio 12.2 (Sun C 5.11)
 Oracle Solaris Studio 12.3 (Sun C 5.12)
 Oracle Solaris Studio 12.4 (Sun C 5.13)
 Oracle Developer Studio 12.5 (Sun C 5.14)
 Oracle Developer Studio 12.6 (Sun C 5.15)
 gcc 5.5.0
 gcc 7.3.0

and verified the libfreetype.so.6 generated by each of those compilers
exported the same set of symbols.

Signed-off-by: Alan Coopersmith <address@hidden>
---
 builds/unix/configure.raw | 24 ++++++++++++++----------
 1 file changed, 14 insertions(+), 10 deletions(-)

diff --git a/builds/unix/configure.raw b/builds/unix/configure.raw
index 8eded34b4..79c93a7ca 100644
--- a/builds/unix/configure.raw
+++ b/builds/unix/configure.raw
@@ -311,22 +311,26 @@ AC_SUBST([XX_ANSIFLAGS])
 # It is recommended that shared libraries hide symbols except those with
 # explicit __attribute__((visibility("default"))).
 #
+found_visibility_flag=no
 AC_MSG_CHECKING([for -xldscope=hidden compiler flag])
 orig_CFLAGS="${CFLAGS}"
 CFLAGS="${CFLAGS} -xldscope=hidden"
-AC_COMPILE_IFELSE([AC_LANG_PROGRAM([],[])],
+AC_LINK_IFELSE([AC_LANG_PROGRAM([],[])],
+                found_visibility_flag=yes
+                AC_MSG_RESULT(yes),
+                CFLAGS="${orig_CFLAGS}"
+                AC_MSG_RESULT(no))
+
+if test "${found_visibility_flag}" = "no" ; then
+  AC_MSG_CHECKING([for -fvisibility=hidden compiler flag])
+  orig_CFLAGS="${CFLAGS}"
+  CFLAGS="${CFLAGS} -fvisibility=hidden"
+  AC_LINK_IFELSE([AC_LANG_PROGRAM([],[])],
+                  found_visibility_flag=yes
                   AC_MSG_RESULT(yes),
                   CFLAGS="${orig_CFLAGS}"
                   AC_MSG_RESULT(no))
-
-AC_MSG_CHECKING([for -fvisibility=hidden compiler flag])
-orig_CFLAGS="${CFLAGS}"
-CFLAGS="${CFLAGS} -fvisibility=hidden"
-AC_COMPILE_IFELSE([AC_LANG_PROGRAM([],[])],
-                  AC_MSG_RESULT(yes),
-                  CFLAGS="${orig_CFLAGS}"
-                  AC_MSG_RESULT(no))
-
+fi
 
 # All library tests below try `pkg-config' first.  If that fails, a function
 # from the library is tested in the traditional autoconf way (zlib, bzip2),
-- 
2.15.0