Re: [lmi] Building shared zlib

[Greg Chicares]

On 2016-07-15 21:11, Vadim Zeitlin wrote:
> On Fri, 15 Jul 2016 21:04:16 +0000 Greg Chicares <address@hidden> wrote:

[...problems building zlib for use with libxml2, which now reappear--see 
below...]

> GC> Where do we go from here?
> GC>  (1) Use './configure && make' and fix up the problems manually?
> GC>  (2) Try to fix the autotoolization problem?
> 
>  As an aside, this is how autotools undeservedly get bad reputation :-( The
> problem is that zlib has decided _not_ to use them, hence all this
> weirdness.
> 
> GC>  (3) Use the msw-specific makefile provided by zlib?
> GC>  (4) Just use the zlib.org msw binaries?
> GC> I rate those options 2 > 4 > 1 > 3 iff (2) is easy (it's not easy for me),
> GC> or 4 > 2 > 1 > 3 otherwise. If we're stuck with an msw-specific solution,
> GC> then 4 >>> 3 because it's less work and less prone to error. I think I'll
> GC> proceed with (4) provisionally; if (2) is feasible, we can switch to it.
> 
>  My personal preference is still (3) for the same reasons as stated before
> (to recap: if we build some third party libraries, let's build all of them)
> and I don't see any real drawbacks to it. (2) is easy if you agree to not
> have the version information in the generated DLL, otherwise it isn't.

Before presenting the latest problems, let me ask whether there are
other, better options:

(5) Use a zlib built by wx
(6) Use zlib binaries provided by the MinGW-w64 maintainers
(7) Drop zlib and use lzma instead

The principal motivation, as explained here:
  http://lists.nongnu.org/archive/html/lmi/2016-06/msg00046.html
is to use compression as (weak) encryption, and lzma would serve
that purpose just as well as zlib. Speed shouldn't be a problem:
  https://tukaani.org/lzma/benchmarks.html
| Decompression  In terms of speed, gzip is the winner again.
| lzma comes right behind it two to three times slower than gzip.
Because we've upgraded to libxml2-2.9.4, which was released
2016-05-23 and is still the latest:
  http://xmlsoft.org/news.html
we're able to use lzma, which has been supported since 2012-01:
  https://mail.gnome.org/archives/commits-list/2012-January/msg11021.html

Option (7) sounds most promising in light of all the difficulties
we've had with zlib--so promising that I think we should try it
before discussing the others. I won't delete what I had already
written about them below, lest that become useful someday, but
nobody ever has to read it if lzma works.

(5) I've seen this in wx build logs:
  configure: WARNING: zlib library not found or too old, will use built-in 
instead
Does that mean that wx is capable of building a zlib that actually
might work with libxml2? I see
  libwxzlib-3.1-i686-w64-mingw32.a
in my latest msw cross build, but no corresponding DLL. I guess I've
asked that question before, and you replied here:
  http://lists.nongnu.org/archive/html/lmi/2016-06/msg00047.html

(6) Use binaries provided by the MinGW-w64 project. I think that's
what's been repackaged here by debian:
  https://packages.debian.org/stretch/all/libz-mingw-w64/filelist
and it contains both 32- and 64-bit DLLs:
  /usr/i686-w64-mingw32/lib/zlib1.dll
  /usr/x86_64-w64-mingw32/lib/zlib1.dll
OTOH, a freshly installed MinGW-w64 package contains static zlib
binaries, but not the DLLs that libxml2 apparently needs:

/MinGW_[0]$ls -1 **/libz*
i686-w64-mingw32/lib/libz.a
i686-w64-mingw32/lib64/libz.a
i686-w64-mingw32/lib64/libzoneoc.a
/MinGW_[0]$ls -1 **/zlib*
i686-w64-mingw32/include/zlib.h
opt/lib/python2.7/encodings/zlib_codec.py
opt/lib/python2.7/encodings/zlib_codec.pyc
opt/lib/python2.7/encodings/zlib_codec.pyo
opt/lib/python2.7/lib-dynload/zlib.pyd
licenses/zlib:
copying.txt

and, curiously, MinGW-w64 has a separate zlib package:
  
https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/ray_linn/64bit-libraries/zlib/
but it contains only 64-bit binaries...so maybe the debian
maintainer built the binaries himself from scratch.

[Then I thought of option (7), as described above.]

Anyway, although building zlib for use with libxml2 seemed to be
working for a while, now it's not working. Excepted from the output
of 'install_msw.sh' on a machine in the office:

cd /opt/lmi/xml-scratch/zlib-1.2.8 \
  && export PATH="/MinGW_/bin/:/usr/local/bin: [...]
[hundreds of characters redacted from $PATH]
  && LDSHARED='gcc -shared -Wl,--out-implib,libz1.dll.a' LDSHAREDLIBC='' 
SHAREDLIBV='libz1.dll'  ./configure --prefix=/opt/lmi/local 
--eprefix=/opt/lmi/local  \
  && make \
  && make install \

Checking for gcc...
Checking for shared library support...
Building shared library libz1.dll with gcc.
Checking for off64_t... Yes.
Checking for fseeko... Yes.
Checking for strerror... Yes.
Checking for unistd.h... Yes.
Checking for stdarg.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
Checking for attribute(visibility) support... No.
make[1]: Entering directory '/opt/lmi/xml-scratch/zlib-1.2.8'
gcc -O3  -D_LARGEFILE64_SOURCE=1 -I. -c -o example.o test/example.c
gcc -O3  -D_LARGEFILE64_SOURCE=1   -c -o adler32.o adler32.c
gcc -O3  -D_LARGEFILE64_SOURCE=1   -c -o crc32.o crc32.c
gcc -O3  -D_LARGEFILE64_SOURCE=1   -c -o deflate.o deflate.c
gcc -O3  -D_LARGEFILE64_SOURCE=1   -c -o inffast.o inffast.c
gcc -O3  -D_LARGEFILE64_SOURCE=1   -c -o infback.o infback.c
gcc -O3  -D_LARGEFILE64_SOURCE=1   -c -o inflate.o inflate.c
gcc -O3  -D_LARGEFILE64_SOURCE=1   -c -o inftrees.o inftrees.c
gcc -O3  -D_LARGEFILE64_SOURCE=1   -c -o zutil.o zutil.c
gcc -O3  -D_LARGEFILE64_SOURCE=1   -c -o trees.o trees.c
gcc -O3  -D_LARGEFILE64_SOURCE=1   -c -o compress.o compress.c
gcc -O3  -D_LARGEFILE64_SOURCE=1   -c -o uncompr.o uncompr.c
gcc -O3  -D_LARGEFILE64_SOURCE=1   -c -o gzclose.o gzclose.c
gcc -O3  -D_LARGEFILE64_SOURCE=1   -c -o gzlib.o gzlib.c
gcc -O3  -D_LARGEFILE64_SOURCE=1   -c -o gzread.o gzread.c
gcc -O3  -D_LARGEFILE64_SOURCE=1   -c -o gzwrite.o gzwrite.c
gcc -O3  -D_LARGEFILE64_SOURCE=1 -I. -c -o minigzip.o test/minigzip.c
gcc -O3  -fPIC -D_LARGEFILE64_SOURCE=1 -DPIC -c -o objs/adler32.o adler32.c
gcc -O3  -fPIC -D_LARGEFILE64_SOURCE=1 -DPIC -c -o objs/crc32.o crc32.c
gcc -O3  -fPIC -D_LARGEFILE64_SOURCE=1 -DPIC -c -o objs/infback.o infback.c
gcc -O3  -fPIC -D_LARGEFILE64_SOURCE=1 -DPIC -c -o objs/deflate.o deflate.c
gcc -O3  -fPIC -D_LARGEFILE64_SOURCE=1 -DPIC -c -o objs/inffast.o inffast.c
gcc -O3  -fPIC -D_LARGEFILE64_SOURCE=1 -DPIC -c -o objs/inftrees.o inftrees.c
gcc -O3  -fPIC -D_LARGEFILE64_SOURCE=1 -DPIC -c -o objs/inflate.o inflate.c
gcc -O3  -fPIC -D_LARGEFILE64_SOURCE=1 -DPIC -c -o objs/trees.o trees.c
gcc -O3  -fPIC -D_LARGEFILE64_SOURCE=1 -DPIC -c -o objs/zutil.o zutil.c
gcc -O3  -fPIC -D_LARGEFILE64_SOURCE=1 -DPIC -c -o objs/compress.o compress.c
gcc -O3  -fPIC -D_LARGEFILE64_SOURCE=1 -DPIC -c -o objs/uncompr.o uncompr.c
gcc -O3  -fPIC -D_LARGEFILE64_SOURCE=1 -DPIC -c -o objs/gzclose.o gzclose.c
gcc -O3  -fPIC -D_LARGEFILE64_SOURCE=1 -DPIC -c -o objs/gzlib.o gzlib.c
gcc -O3  -D_LARGEFILE64_SOURCE=1 -I. -D_FILE_OFFSET_BITS=64 -c -o example64.o 
test/example.c
gcc -O3  -fPIC -D_LARGEFILE64_SOURCE=1 -DPIC -c -o objs/gzread.o gzread.c
gcc -O3  -fPIC -D_LARGEFILE64_SOURCE=1 -DPIC -c -o objs/gzwrite.o gzwrite.c
gcc -O3  -D_LARGEFILE64_SOURCE=1 -I. -D_FILE_OFFSET_BITS=64 -c -o minigzip64.o 
test/minigzip.c
ar rc libz.a adler32.o crc32.o deflate.o infback.o inffast.o inflate.o 
inftrees.o trees.o zutil.o compress.o uncompr.o gzclose.o gzlib.o gzread.o 
gzwrite.o 
gcc -O3  -D_LARGEFILE64_SOURCE=1 -o example.exe example.o -L. libz.a
gcc -O3  -D_LARGEFILE64_SOURCE=1 -o minigzip.exe minigzip.o -L. libz.a
gcc -O3  -D_LARGEFILE64_SOURCE=1 -o example64.exe example64.o -L. libz.a
gcc -O3  -D_LARGEFILE64_SOURCE=1 -o minigzip64.exe minigzip64.o -L. libz.a
gcc -shared -Wl,--out-implib,libz1.dll.a -O3  -fPIC -D_LARGEFILE64_SOURCE=1 -o 
libz1.dll adler32.lo crc32.lo deflate.lo infback.lo inffast.lo inflate.lo 
inftrees.lo trees.lo zutil.lo compress.lo uncompr.lo gzclose.lo gzlib.lo 
gzread.lo gzwrite.lo   
rm -f libz.so libz.so.1
ln -s libz1.dll libz.so
ln -s libz1.dll libz.so.1
gcc -O3  -D_LARGEFILE64_SOURCE=1 -o examplesh.exe example.o -L. libz1.dll
gcc -O3  -D_LARGEFILE64_SOURCE=1 -o minigzipsh.exe minigzip.o -L. libz1.dll
make[1]: Leaving directory '/opt/lmi/xml-scratch/zlib-1.2.8'
make[1]: Entering directory '/opt/lmi/xml-scratch/zlib-1.2.8'
cp libz.a /opt/lmi/local/lib
chmod 644 /opt/lmi/local/lib/libz.a
cp libz1.dll /opt/lmi/local/lib
chmod 755 /opt/lmi/local/lib/libz1.dll
cp zlib.3 /opt/lmi/local/share/man/man3
chmod 644 /opt/lmi/local/share/man/man3/zlib.3
cp zlib.pc /opt/lmi/local/lib/pkgconfig
chmod 644 /opt/lmi/local/lib/pkgconfig/zlib.pc
cp zlib.h zconf.h /opt/lmi/local/include
chmod 644 /opt/lmi/local/include/zlib.h /opt/lmi/local/include/zconf.h
make[1]: Leaving directory '/opt/lmi/xml-scratch/zlib-1.2.8'

That appears to have succeeded. And libxml2 seems to build okay until:

  CCLD     libxml2.la

*** Warning: linker path does not have real file for library -lz.
*** I have the capability to make that library automatically link in when
*** you link to this library.  But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libz and none of the candidates passed a file format test
*** using a file magic. Last file checked: 
C:/opt/lmi/MinGW-6_3_0/i686-w64-mingw32/lib/libz.a
*** The inter-library dependencies that have been dropped here will be
*** automatically added whenever a program is linked with this library
*** or is declared to -dlopen it.

...which soon leads to:

install_libxml2_libxslt.make:191: recipe for target 'libxml2-2.9.4' failed
make: *** [libxml2-2.9.4] Error 2

Oddly, everything just seems to work in my Cygwin VM, which uses
approximately the same versions of all cygwin packages. One notable
difference is that I have these files:

1d9871d4ffd71645ac3d029d8de0e03a 
*/cache_for_lmi/downloads/zlib128-dll/DLL_FAQ.txt
624cb4c112f0f393af1b182598e62272 */cache_for_lmi/downloads/zlib128-dll/foo.gz
0af2d6af45fd78e4f288583333978cce 
*/cache_for_lmi/downloads/zlib128-dll/include/zconf.h
64d8a5180bd54ff5452886e4cbb21e14 
*/cache_for_lmi/downloads/zlib128-dll/include/zlib.h
4508be33473b384e424a344b37392889 
*/cache_for_lmi/downloads/zlib128-dll/lib/zdll.lib
093438656dfa76f4f88614a0a941e57f 
*/cache_for_lmi/downloads/zlib128-dll/lib/zlib.def
51927169536b2934a200945ac55a694d 
*/cache_for_lmi/downloads/zlib128-dll/README.txt
6333538c2e1b14b27833aa4b212d9024 
*/cache_for_lmi/downloads/zlib128-dll/test/example_d.exe
d8532a804c43d9947929153334453b12 
*/cache_for_lmi/downloads/zlib128-dll/test/minigzip_d.exe
092246147f60e83bcf378167f0626149 */cache_for_lmi/downloads/zlib128-dll/USAGE.txt
092246147f60e83bcf378167f0626149 
*/cache_for_lmi/downloads/zlib128-dll/USAGE.txt.bak
007488d386fe558a18b202fc4374e5b6 
*/cache_for_lmi/downloads/zlib128-dll/USAGE.txt.diff
36885fae5e9f82de42d6f2d0edda4935 
*/cache_for_lmi/downloads/zlib128-dll/USAGE.txt.modified
b8a9e91134e7c89440a0f95470d5e47b */cache_for_lmi/downloads/zlib128-dll/zlib1.dll

which are not present on the corporate machine. They aren't in
  tar -tzvf /cache_for_lmi/downloads/zlib-1.2.8.tar.gz
so one hypothesis is that I installed some binary package that
provides them, perhaps as an early experimental alternative to
building zlib from source. Other than that, I don't see any
major differences.

But at this point I abandoned this line of research because the
lzma idea occurred to me.