RE: MSYS+MSVC for libtool branch-2-0, take 2

[Peter Ekberg]

Ralf Wildenhues wrote:
> To speed up bootstrap somewhat while you are testing:
>   reconfdirs=. ./bootstrap
> 
> skips all the test directories.

Thanks!

> > > Alternatively, if you don't like the extract_old_archive_cmds 
> > > idea, I'd
> > > still prefer something like
> > >   $AR $AR_FLAGS$AR_SEP$oldlib ..
> > > over your solution.  Have AR_SEP=' ' by default.
> > > Rationale: Less variables to keep track of.
> > 
> > Hmmm, Automake (?) seems to set up a rule to use $AR, but
> > that rule uses "$AR $ARFLAGS foo..." w/o underscore in
> > AR_FLAGS (and of course no $AR_SEP).
> > 
> > I noticed this when I added AR_FLAGS and AR_SEP with AC_SUBST
> > in an attempt to pass one of the failed tests. (I added AR_SEP
> > to AC_SUBST thinking that Automake would catch up and make use
> > of it in the future...)
> 
> Exactly.  And it'd be backward-compatible with what users expect, too.
> Hmm.  Except I don't know how to set a Make variable to SPACE.  Maybe
> Automake could use $AR $ARFLAGS$${AR_SEP-' '} or so?  Oh, chucks, we
> should ask Automake for their preferred way before we set ours in
> stone.  Will write there.

Great!

>                            The only thing I could come up so far is
> this heavyly ugly rule:
> 
>       AR_SEP="$(AR_SEP)"; if test -z "$$AR_SEP"; then 
> AR_SEP=' '; fi; \
>       $(AR) $(ARFLAGS)$${AR_SEP}$(OLDLIB) ...
> 
> AC_SUBSTing AR_SEP from Libtool's side is probably wrong, if it's used
> in Automake.  Automake need not use Libtool..

I was just poking about w/o real knowledge of how things work. So, yes,
my approach is probably not the best.

> > > By looking at `lib -link' I believe issue 1. is actually the most
> > > problematic, since `lib' encodes the path into the 
> archive by default.
> > > I have not found a switch to turn this off (which does 
> not imply there
> > > is none).  But that means we have to symlink/copy objects before
> > > archiving them.  (Could be done in the same loop as the 
> renaming, but
> > > will be expensive anyway.)
> > > 
> > > Otherwise, all use of convenience archives breaks (and running the
> > > Libtool test suite really shows that).
> > 
> > If I do
> > $ lib -OUT:foo.lib a/b/c.obj
> > $ lib -EXTRACT:a/b/c.obj foo.lib
> > I get a fresh copy of c.obj in cwd
> > 
> > If I do
> > $ ar cru bar.lib a/b/d.obj
> > $ ar x bar.lib
> > I get a fresh copy of d.obj in cwd
> > 
> > Where's the difference?
> 
> Well, first problem is that POSIX specifies that no paths be encoded
> into archives at all.  I don't like to violate this.
> (Last but not least being that I would like that installed .lib files
> be handled correctly by, e.g., cygwin ar.  Maybe that is already the
> case.)
> 
> I need to test this, though, I vaguely remember members being 
> extracted
> into subdirs.

I'll put this on hold, until there's some confirmed issue...

> > > > > > > After working around these, the next failure is when 
> > > > > linking libltdl:
> > > > > > > 
> > > > > > > | libltdl_la-ltdl.obj : error LNK2019: unresolved 
> external 
> > > > > > > symbol _lt_libltdl_LTX_preloaded_symbols referenced in 
> > > > > > > function _lt_dlinit.
> > > > > > > | .libs/ltdl-6.dll : fatal error LNK1120: 1 
> > > unresolved externals
> > > > > > 
> > > > > > Haven't tried libltdl, I do not depend on it for the project
> > > > > > I'm targeting and I have a hard time bootstrapping libtool
> > > > > > so I haven't tried the testsuite myself.
> 
> This seems to be solved, looking at your test results.  Cool!

Not really, I just hacked Makefile.am, and removed lib from
-DLTDLOPEN=libltdl... This needs a proper fix, of course.
(some time passes)
See attached version 6 of the patch for an attempt.

> > > > > > - Add paths given with -L to the LIB variable so 
> that the linker
> > > > > >   actually sees them. This should be done after 
> converting them
> > > > > >   to win32 paths, which is easy on Cygwin and 
> harder on MSYS.
> > > 
> > > > Maybe not so hard on MSYS after all, how about:
> > > > win32=$(cmd \\/C echo "$unix " | sed -e "s/\"\(.*\) \" /\1/")
> > > > 
> > > > (cmd is not directly CMD.EXE, but a shell script 
> provided by MSYS
> > > > that invokes $COMSPEC. One question is how portable /C is as an
> > > > option to $COMSPEC? I would guess that /C works for all command
> > > > interpreters from Microsoft, but I'm unsure about others...)
> > > 
> > > This /might/ be a question for the msys/mingw mailing lists.
> > > What is COMSPEC, by the way?
> > 
> > COMSPEC is an env variable that is the full path of the windows
> > command line interpreter, e.g. C:\WINDOWS\command.com or
> > C:\WINXP\System32\cmd.exe.
> 
> OK.
> 
> > > The `echo' might be problematic, if any of the shells' 
> builtin and/or
> > > external echo commands interpret backslashes.  The MinGW 
> > > shell should be
> > > fine with `printf %s ".."', I believe.
> > 
> > That echo will be fed to the builtin echo in the windows command
> > line interpreter, so it should not be a problem. Or where you
> > referring to something else?
> > 
> > The unknown is what happens if there are some non-MS command
> > line interpreter installed. I remember using DRDOS or something
> > ~10 years ago, and don't have a clue if those alternative command
> > line interpreters are still in use or if they even work in a
> > Windows environment. I'm willing to take a slight risk and
> > assume that they are compatible in this regard...
> 
> I'm rather worried about a possible cross-compiling case.  
> Not that it's
> likely someone would actually do that (although with wine or 
> similar, it
> might actually be possible already).  Then your `echo' is not 
> guaranteed
> to be fine.  (It's rather that I'd generally like to avoid 
> anything that
> hampers cross-compiling, so at least the number of issues with it does
> not increase.)

You don't seem to understand what's going on... On MSYS there
is no "official" way to convert a posix path to a win32 path
(like cygpath on Cygwin). But if you execute any executable
outside of the MSYS /bin or /usr/bin directories (and perhaps
other places as well?), any argument that looks like a posix
path is converted to a win32 path on the assumption that all
such executables expect win32 paths. The problem is that in
this case the conversion needs to be done not on an argument,
but on an environment variable that MSYS does not autoconvert.
So, it needs to be converted manually. So, since there is no
official way to convert paths, I run a program outside of MSYS
and as one of its arguments I have the posix path to be
converted. The program I have chosen to run is the command
line interpreter pointed to by $COMSPEC and I also feed some
other options to the command line interpreter (/C and echo,
since /C looks like a posix path, the slash has to escaped)
which make it print out the converted paths nicely.

Do note that the "command line interpreter" above is _not_ the
shell (which is under MSYS control and does not receive the
posix to win32 conversion). So, your complaint to not use echo
should be a complaint to not use the 'cmd' script (which runs
$COMSPEC), which really is what will potentially not work in a
cross-compile, but isn't that just like banning cygpath on
Cygwin?

If building with Wine and cl is going to work, a Wine way to
convert paths from posix to win32 has to be added, since I
believe a normal shell is used with Wine and I don't think (?)
Wine will do any autoconversions of paths.

> > > | -         $run $LTCC $LTCFLAGS -s -o $cwrapper $cwrappersource
> > > | +         if test "$with_gcc" = "yes"; then
> > > | +           $run $LTCC $LTCFLAGS -s -o $cwrapper $cwrappersource
> > > | +         else
> > > | +           $run $LTCC $LTCFLAGS -o $cwrapper $cwrappersource
> > > 
> > > Why is this necessary?
> > 
> > Because MSVC does not have a -s option. It's not strictly
> > necessary since a warning is the only evidence, but I feel
> > uncomfortable with unrecognized options...
> 
> Ah, ok.  Let's solve this differently..

You mean, detect if -s does the intended thing and store that
somewhere and make use of that instead of $with_gcc as
trigger in the "test"?

E.g $run $LTCC $LTCFLAGS $LTCSTRIP -o $cwrap... ?

> > Besides, I think MSVC creates "release" binaries unless asked
> > to do differently. Not sure if they can be "stripped" further
> > though...
> 
> and let's solve this, too.

There are 6 different "libc", and you select which one you want
with command line switches, i.e _not_ with -lc/-lcd or something
like that...

-MD  link with MSVCRT.LIB
-MDd link with MSVCRTD.LIB debug lib
-ML  link with LIBC.LIB
-MLd link with LIBCD.LIB debug lib
-MT  link with LIBCMT.LIB
-MTd link with LIBCMTD.LIB debug lib

I don't see any resonable default, maybe -MDd? And then not
add the default if any of the others appear in CFLAGS?

> > New issues:
> > 
> > In the project where I first developed the patch, functions
> > and variables are marked with the appropriate
> > __declspec(dllimport) and __declspec(dllexport), so exporting
> > and importing variables for shared libs failed miserably for
> > the general case for patches up to an including -4. In -5 this
> > is fixed. But I think I've hit a bit of a snag, importing a
> > variable from a dll requires that the "data item in the client
> > program must be declared using extern __declspec(dllimport)".
> > The snag is that it isn't known at compile time what the
> > compiled unit will be used for, so this has to be solved in
> > the project using libtool, if a clean solution is desired.
> 
> This sounds just like the issue Howard Chu has described on one of the
> libtool lists (I believe this one) a couple of weeks ago.
> I believe the bottom line is: don't do data exports, only functions
> exports.  (libltdl suffers badly from this, too.)
> 
> > I found this out when fixing problems found by the tests. This
> > version of the patch exports functions and variables correctly.
> > It also imports function correctly, but importing variables are
> > cludgy. I always say extern __declspec(dllimport), even if
> > linking statically or exporting. This works, but results in
> > various linker warnings. At least it works if I use lib (or
> > "link -lib") as archiver. There is some failure when using ar
> > as archiver that I haven't investigated, so I'm avoiding that
> > ATM...
> 
> Hmm.  I am pretty sure this will fail in various special cases.
> Cannot tell ATM, though.
> 
> > How do I get configure to not find g77 when it looks for a
> > Fortran compiler?
> 
> g77=:
> ?

I assume you mean F77=:, but with either of those I still get this
in my Makefile:

##TESTS = $(COMMON_TESTS)
#TESTS = $(COMMON_TESTS) $(F77_TESTS)
#TESTS = $(COMMON_TESTS) $(CXX_TESTS)
TESTS = $(COMMON_TESTS) $(CXX_TESTS) $(F77_TESTS)

So, whatever I do, HAVE_F77 seams to be set by configure...

> > I have made sure that the paths in the LIB variable for MSVC
> > does not contain any spaces (uses PROGRA~1 etc).
> > 
> > I configure with
> > ./configure CC=cl CFLAGS=-MD CXX=cl CXXFLAGS=-MD LD=link 
> AR=lib STRIP=:
> > 
> > Here is an annotated dump of make check results:
> 
> It'd be nice to see the output of
>   make check VERBOSE=X
> (it'd suffice for the test groups in which failures occur.  But don't
> waste time trying to sort these out if you don't want to :)

Attaching some output...

> I'll comment on your patch in another mail.  Just another 
> question: did
> you try msys/gcc or cygwin/gcc with the changes to the test suite
> applied?

I have now, and the only regression in the test suite caused by
this patch is the one in tests/sh.test. But that's not
considering the changes to libltdl/Makefile.am, of course.
Hmm, attaching version 6 of the patch with a proper (?) fix for
the libltdl/Makefile.am problem (the attached testruns are with
version 5 of the patch though, but this is the only difference
between -5 and -6).

(mdemo.exec (3 times) and mdemo.dryrun fail on MinGW, but they
 also fail without the patch. Also, some other tests pop up
 dialogs with disturbing messages on MinGW, but those tests
 PASS.)

Cheers,
Peter

demo-deplibs.gz
Description: demo-deplibs.gz

demo-hardcode.gz
Description: demo-hardcode.gz

demo-relink.gz
Description: demo-relink.gz

mdemo-exec.gz
Description: mdemo-exec.gz

pdemo-make.gz
Description: pdemo-make.gz

tagdemo-make.gz
Description: tagdemo-make.gz

msys_msvc-6.patch
Description: msys_msvc-6.patch