Re: [PATCH] Path conversion documentation

[Ralf Wildenhues]

Hi Charles,

* Charles Wilson wrote on Mon, Aug 30, 2010 at 01:50:12AM CEST:
> * doc/libtool.texi (Platform quirks): Add new subsections
> 'Cross compiling' and 'File name/path conversion'.

> OK to push?

Looks fine to me too, only one or two issues are not markup or typo
nits (but I have been very nitpicky below).

Thanks for writing this,
Ralf

> --- a/doc/libtool.texi
> +++ b/doc/libtool.texi
> @@ -223,6 +223,8 @@ Platform quirks
>  * Reloadable objects::          Binding object files together.
>  * Multiple dependencies::       Removing duplicate dependent libraries.
>  * Archivers::                   Programs that create static archives.
> +* Cross compiling::             Issues that arise when cross compiling.
> +* File name/path conversion::   Converting filenames between platforms.

FWIW, I would  s/\/path//  but only because I find the result more
readable and as informative.

> @@ -5875,6 +5879,383 @@ must be used to ``bless'' the created library before 
> linking against it,
>  with the @kbd{ranlib address@hidden command.  Some systems, like Irix,
>  use the @code{ar ts} command, instead.
>  
> address@hidden Cross compiling
> address@hidden Cross compiling
> address@hidden cross compile
> +
> +Most build systems support the ability to compile libraries and applications
> +on one platform (the @code{$build} system) for use on a different platform 
> (the

I don't much like variable references that include the $, because that
ties you to a particular language, whereas here, you are talking about
concepts rather than languages.  standards.texi just use 'the build
system'.  Here, since you're defining the term, you could use
@dfn{build system} or maybe @emph, and later on just use it without
markup.
 
> address@hidden system) -- provided a compiler capable of generating the

Likewise for `host system', and instances below.

> +appropriate output is available.  The GNU Build System
> address@hidden://www.gnu.org/software/hello/manual/automake/GNU-Build-System.html},

@uref please, and how about putting `GNU Build System' in the second
argument of @uref, and in the first argument, replacing all single
slashes with /@/ so they are line-wrapped nicely in the PDF.

Actually, why not make this a normal inter-manual reference like
@ref{GNU Build System,, The GNU Build System, automake, The Automake
Manual}

that way it will render well in all outputs?

> +of which libtool is a part, supports cross compiling via arguments passed to
> +the configure script: @option{--build=...} and @option{--host=...}. However,
> +when the @code{$build} and @code{$host} systems are very different, libtool 
> is
> +required to make certain accommodations to support these scenarios.
> +
> +In most cases, because the @code{$build} platform and @code{$host} platform
> +differ, the cross-compiled libraries and executables can't be executed or
> +tested on the @code{$build} platform where they were compiled.  The 
> testsuites
> +of most build systems will often skip any tests that involve executing such
> +foriegn executables when cross-compiling.  However, if the @code{$build} and

foreign

> address@hidden platforms are sufficiently similar, it is often possible to
> +run cross-compiled applications.  Libtool's own testsuite often attempts to
> +execute cross-compiled tests, but will mark any failures as @emph{skipped}
> +since the failure might simply be due to the differences between the two
> +platforms.
> +
> +In addition to cases where the @code{$host} and @code{$build} system are
> +extremely similar (e.g. @samp{i586-pc-linux-gnu} and 
> @samp{i686-pc-linux-gnu}),
> +there is another case in which cross-compiled @code{$host} applications may
> +be executed on the @code{$build} system.  This occurs when the @code{$build}
> +platform supports an emulation or API-enhanced environment for @code{$host}.
> +One example of this situation would be if @code{$build} were MinGW, and
> address@hidden were Cygwin (or vice versa).  Both of these platforms can 
> actually
> +operate within a single Win32 instance, so Cygwin applications can be 
> launched

Either s/Win32/w32/  globally in your patch, or use Windows, as
recommended by the GCS.

> +from a MinGW context, and vice versa -- provided certain care is taken. 
> Another
> +example would be if @code{$build} were linux on an @samp{x86} 32bit 
> processor,

Either GNU/Linux, or you write out a full triple like above.

> +and @code{$host} were MinGW.  In this situation, the WINE
> address@hidden://www.winehq.org/} environment can be used to launch "Win32"

@uref{http://www.winehq.org/, Wine}, and s/WINE/Wine/ globally

No need to double-quote Win32 here, and if there were, literal
double-quotes are almost always wrong in texinfo; that's what ``these''
are for.

> +applications from the linux operating system; again, provided certain care is

GNU/Linux

> +taken.
> +
> +One particular issue occurs when a Win32 platform such as MinGW, Cygwin, 
> MSYS,
> +or MSVC is the @code{$host} or @code{$build}, while the other platform is a 
> unix

Unix should be capitalized here and below, and I think you mean
`Unix-style' here.

> +system.  In these cases, there are often conflicts between the format of the
> +file names and paths expected within @code{$host} libraries and executables,
> +and those employed on the @code{$build} platform.
> +
> +This situation is best described using a concrete example: suppose the
> address@hidden is linux -- (specifically, @code{i686-pc-linux-gnu}), and the

Here's the first time you are /almost/ speaking of @code{build} rather
than the concept of a build system.
s/linux/GNU\/Linux/
@samp{i686-pc-linux-gnu}  for consistency with above.

> address@hidden is MinGW (specifically, @code{i586-pc-mingw32}).  On the linux

GNU/Linux

> +system, there exists a cross compiler, following the usual naming conventions
> +of such compilers, where the compiler name is prefixed by the @code{$host}
> +triple.  In this case, the C compiler is named @samp{i586-pc-mingw32-gcc}.

You are not distinguishing between the canonical triplet and between
alias names.  I'm not sure if we need to go into details; maybe cross
references to the 'Specifying Target Triplets' and 'Canonicalizing'
nodes of the Autoconf manual are sufficient.

> +As described in the @pxref{Wrapper executables} section, the MinGW 
> @code{$host}

@ref not @pxref.  The latter is for inside parentheses and prints a
"see", whereas the former doesn't.  For details see 'info texinfo
--index pxref'.  More instances below.

> +uses a wrapper executable to set various environment variables before 
> launching
> +the actual program executable.  Like the program executable, the wrapper
> +executable is cross-compiled for the @code{$host} platform (that is, for 
> MinGW).
> +As described above, ordinarily a @code{$host} executable cannot be executed 
> on
> +the @code{$build} platform, but in this case the WINE environment could be 
> used
> +to launch the MinGW application from linux.  However, the wrapper executable,

GNU/Linux

> +as a @code{$host} (MinGW) application, must set the @var{$PATH} variable so

@env{PATH}, more instances below.  @var is strictly for metasyntactic
variables (that is those which you can always replace by X or X_i for
some integer i, without losing the meaning of the text; IOW, the actual
name you chose for the variable is not relevant outside the manual).

> +that the true application's dependent libraries can be located -- but the
> +contents of the @var{$PATH} variable must be structured for MinGW. Libtool
> +must must use the WINE path mapping facilities to determined the correct

s/must must/needs to/
to determine

> +value so that the wrapper executable can set the @var{$PATH} variable to 
> point
> +to the correct location.
> +
> +For example, suppose we are compiling an application in @samp{/var/tmp} on

@file{/var/tmp}

> +linux:

GNU/Linux

> address@hidden
> +/var/tmp/foo/src/ (application code)
> +/var/tmp/foo/lib/ (library code)
> address@hidden example

What does this example code describe?  You don't state that these are
directories, and you don't state whether they belong to, say, the source
tree, the build tree, or some install tree.

> +Since the library will be built in @samp{.../foo/lib}, the wrapper executable

@file, more instances below.

> +must add that value to @var{$PATH} (actually, to @var{$objdir} under

@var{objdir} is actually acceptable since the manual explicitly defines
that variable somewhere earlier.

> address@hidden/foo/lib}, but we'll ignore that for now).  However, Win32 does 
> not
> +have a concept of paths such as @samp{/var/tmp/foo/lib}; so, WINE provides

s/paths/Unix-style file names/  ?

> +a mapping from Win32 paths such as @samp{C:/Program Files} to specific
> +unix file system locations.  WINE also provides a utility that can be used to
> +map unix file system locations to Win32 file names.
> +
> +In this case, the wrapper executable should actually add the value
> +
> address@hidden
> +Z:\\var\\tmp\\foo\\lib
> address@hidden example
> +
> +to the @var{$PATH}.  libtool contains support for path conversion of this
> +type, for a certain limited set of @code{$build}/@code{$host} combinations. 
> In

build and host combinations

> +this case, libtool will invoke WINE's @code{winepath} utility to ensure that
> +the correct @var{$PATH} value is used.  For more information,
> address@hidden name/path conversion}.
> +
> address@hidden File name/path conversion
> address@hidden File name/path conversion
> address@hidden filename conversion
> address@hidden path conversion
> +
> +In certain situations, libtool must convert file names and paths between
> +formats appropriate to different platforms. Usually this occurs when

Please two spaces after end-of-sentence.

> +cross-compiling, and affects only the ability to launch @code{$host}
> +executables on the @code{$build} platfrom using an emulation or API

platform

> +enhancement environment such as WINE.  Failure to convert paths
> +(@pxref{Path Conversion Failure}) will cause a warning to be issued, but
> +rarely causes the build to fail -- and should have no affect on the compiled

`fail---and' is the US English way to write this.
effect

> +results, once installed properly on the @code{$host} system. For more
> +information, @pxref{Cross compiling}.
> +
> +However, path conversion may also occur in another scenario: when using a

s/path/file name/

> +unix emulation system on Win32 (such as Cygwin or MSYS), combined with a
> +native win32 compiler such as MinGW or MSVC.  Only a limited set of such
> +scenarios are currently supported; in other cases (or when libtool can't
> +find the tools required for path conversion or they are misconfigured), a
> +warning is issued and path conversion is skipped. The lack of path conversion
> +usually means that uninstalled executables can't be launched, but only rarely
> +causes the build to fail (@pxref{Path Conversion Failure}).
> +
> +libtool supports path conversion in the following scenarios:
> +
> address@hidden @columnfractions .25 .25 .5
> address@hidden @code{$build} @tab @code{$host} @tab Notes
> address@hidden MinGW (MSYS) @tab MinGW (w32)
> address@hidden @pxref{Native MinGW Path Conversion}
> +
> address@hidden Cygwin @tab MinGW (w32)
> address@hidden @pxref{Cygwin/w32 Path Conversion}

That'll be interesting to see whether doc tools cope well with references
inside tables.

> address@hidden Linux + WINE @tab MinGW (w32)
> address@hidden Requires WINE. @pxref{Linux/w32 Path Conversion}
> +
> address@hidden MinGW (MSYS) @tab Cygwin
> address@hidden Requires @var{LT_CYGPATH}. @pxref{LT_CYGPATH}. Provided for 
> testing

@env{LT_CYGPATH}, multiple instances.

> +purposes only.
> +
> address@hidden Linux + WINE @tab Cygwin
> address@hidden Requires both WINE and @var{LT_CYGPATH}, but currently 
> non-functional.

I'd omit the ", but currently non-functional".  Such comments go out of
date quickly and are usually not garbage-collected even if the failure
is due to something in the same package and not in external software.
Alternatively, qualify the comment, e.g.: ..., but does not yet work
with Cygwin 27.38 and Wine 99.97.

> address@hidden/w32 Path Conversion} and @pxref{LT_CYGPATH}.
> address@hidden multitable
> +
> address@hidden
> +* Path Conversion Failure::        What happens when path conversion fails
> +* Native MinGW Path Conversion::   MSYS path conversion idiosyncracies

s/path/file name/
idiosyncrasies

> +* Cygwin/w32 Path Conversion::     Using cygpath to convert Cygwin paths

@command{cygpath} here and below

> +* Linux/w32 Path Conversion::      Using WINE to convert Linux paths
> +* LT_CYGPATH::                     Invoking cygpath from other environments
> +* Cygwin to MinGW Cross::          Other notes concerning MinGW cross
> address@hidden menu
> +
> address@hidden Path Conversion Failure
> address@hidden Path Conversion Failure
> address@hidden Path Conversion - Failure
> +
> +In most cases, path conversion is not needed or attempted.  However, when
> +libtool detects that a specific combination of @code{$build} and @code{$host}
> +do require path conversion, it is possible that the conversion may fail.
> +In these cases, you may see a warning such as the following:
> +
> address@hidden
> +Could not determine the host file name corresponding to
> +  `... a file name ...'
> +Continuing, but uninstalled executables may not work.
> address@hidden example
> +
> +or
> +
> address@hidden
> +Could not determine the host path corresponding to
> +  `... a path ...'
> +Continuing, but uninstalled executables may not work.
> address@hidden example
> +
> +This should not cause the build to fail. At worst, it means that the
> +wrapper executable will specify paths appropriate for the @code{$build}
> +platform. Since those are not appropriate for the @code{$host} environment,
> +the uninstalled executables would not operate correctly, even when the
> +wrapper executable is launched via the appropriate emulation or API
> +enhancement (e.g. WINE).  Simply install the executables on the @code{$host}
> +platform, and execute them there.
> +
> address@hidden Native MinGW Path Conversion
> address@hidden Native MinGW Path Conversion
> address@hidden Path Conversion - MinGW
> address@hidden MSYS
> +
> +MSYS is a unix emulation environment for w32, and is specifically designed
> +such that in normal usage it @emph{pretends} to be MinGW or native
> +w32, but understands unix paths and supports standard unix tools
> +and shells.  Thus, "native" MinGW builds are actually an odd sort of
> +cross-compile, from an MSYS unix emulation environment "pretending"
> +to be MinGW, to actual native w32.
> +
> +When an MSYS shell launches a native w32 executable (as opposed to other
> address@hidden executables), it uses a system of hueristics to detect any

heuristic

> +command-line arguments that contain file names or paths.  It automatically
> +converts these file names from the MSYS (unix-like) format, to the
> +corresponding w32 file name, before launching the executable.  However,
> +this auto-conversion facility is not available to libtool when it attempts
> +to write the source code for the wrapper executable; libtool must manually
> +convert MSYS paths to w32 format, so that the w32 values can be hard-coded
> +into the wrapper executable.
> +
> address@hidden Cygwin/w32 Path Conversion
> address@hidden Cygwin/w32 Path Conversion
> address@hidden Path Conversion - Cygwin to w32
> +
> +Cygwin provides a unix emulation environment for w32. As part of that
> +emulation, it provides a file system mapping that presents the w32
> +file system in a unix compatible manner.  Cygwin also provides a utility
> address@hidden that can be used to convert file names and paths between

@command{cygpath}, multiple instances

> +the two representations.  In a correctly configured Cygwin installation,
> address@hidden is always present, and is in the @var{$PATH}.
> +
> +Libtool uses @samp{cygpath} to convert from Cygwin (unix) paths to w32 format
> +when @code{$build} is Cygwin and @code{$host} is MinGW or MSVC.

For Peter's remark I suggest just using  s/MinGW or MSVC/MinGW/  here.

> +When @code{host} is Cygwin, but @code{$build} is MSYS or some unix platform,
> +libtool also uses @samp{cygpath} to convert from w32 to Cygwin format. 
> Because
> address@hidden is not Cygwin, @samp{cygpath} is not (and should not be) in the
> address@hidden  Therefore, in this configuration the shell variable
> address@hidden is required. @pxref{LT_CYGPATH}.
> +
> address@hidden Linux/w32 Path Conversion
> address@hidden Linux/w32 Path Conversion
> address@hidden Path Conversion - Linux to w32
> +
> +WINE @url{http://www.winehq.org} provides an interpretation environment for

@uref as above

> +Linux in which w32 applications can be executed. It provides a mapping 
> between the
> +Linux file system and a virtual w32 file system used by the w32 programs.
> +For this conversion to work, WINE must be installed and properly configured,
> +and the @code{winepath} application must be in the @var{$PATH} on the
> address@hidden platform.  In addition, on 32bit Linux it is usually helpful
> +if the @code{binfmt} extension is enabled.

I'd just write binfmt without @code, but not sure.

> address@hidden LT_CYGPATH
> address@hidden LT_CYGPATH
> address@hidden LT_CYGPATH
> +
> +For some cross-compile configurations (where @code{$host} is Cygwin), the
> address@hidden program is used to convert paths from the @code{$build}
> +notation to the Cygwin form.  However, because the @samp{cygpath} program
> +is not (and should not be) in the @var{$PATH} on the @code{$build} platform,
> address@hidden must specify the full @code{$build} path (that is, unix or
> +MSYS path) to the @samp{cygpath} program.
> +
> +The reason @samp{cygpath} should not be in @var{$PATH} is twofold: first,
> address@hidden is usually installed in the same directory as many other
> +Cygwin executables, such as @samp{sed}, @samp{cp}, etc.  If the @code{$build}

@command{sed}, @command{cp}

> +environment had this directory in its @var{$PATH}, then these cygwin versions

Cygwin, multiple instances

> +of common unix utilities might be used in preference to the ones provided by
> +the @code{$build} platform itself, with deleterious effects. Second, 
> especially
> +when cygwin-1.7 or later is used, multiple cygwin installations can coexist
> +within the same w32 instance. Each installation will have separate "mount
> +tables" specified in <CYGROOT-N>/etc/fstab. Each each installation's

@address@hidden/etc/fstab}   ?  More instances below.

> address@hidden utility automatically deduces the appropriate /etc/fstab file.

@file{/etc/fstab}

> +Since each <CYGROOT-N>/etc/fstab mount table may specify different mappings,
> +it matters which @samp{cygpath} is used.
> +
> +Note that @samp{cygpath} is a cygwin application; to execute this tool from
> +Linux requires a working and properly configured WINE installation, as well
> +as enabling the Linux @code{binfmt} extension.  Furthermore, the Cygwin
> address@hidden tool should have been used, via WINE, to properly install

@command{setup.exe}

> +Cygwin into the WINE file system (and registry).
> +
> +Unfortunately, WINE support for Cygwin is intermittent.  Recent releases of
> +Cygwin (1.7 and above) appear to require more Win32 API support than WINE
> +provides at present; most Cygwin applications fail to execute. This includes

s/\. /.  /    more instances below.

> address@hidden itself.  Hence, it is best @emph{not} to use the LT_CYGPATH
> +machinery in libtool when performing linux to Cygwin cross-compiles. 
> Similarly,
> +it is best @emph{not} to enable the Linux @emph{binfmt} support, because 
> while
> +WINE will fail to execute the compiled cygwin applications, it will do so
> +while returning a status code of success. This tends to confuse build systems

a bit awkward; how about "it will still exit with status zero"?

BTW, has it been reported upstream?

> +and test suites (including libtool's own testsuite, resulting in spurious
> +reported failures).  WINE support for the older cygwin-1.5 series appears
> +satisfactory, but the Cygwin team no longer supports cygwin-1.5. It is hoped
> +that WINE support will be improved such that cygwin-1.7 will again operate
> +correctly under WINE. Until then, libtool will report warnings as described
> +in @pxref{Path Conversion Failure}, in these scenarios.
> +
> +However, @var{LT_CYGPATH} is also used for the MSYS to Cygwin cross compile
> +scenario, and operates as expected.
> +
> address@hidden Cygwin to MinGW Cross
> address@hidden Cygwin to MinGW Cross
> address@hidden Cygwin to MinGW Cross
> +
> +There are actually three different scenarios that could all legitimately be
> +called a "Cygwin to MinGW" cross compile.  The current (and standard)

``Cygwin to MinGW''

> +definition is where there exists compiler that produces native w32 libraries

s/where/when/  ?
s/exists/is a/  ?

> +and applications, but which itself is a cygwin application -- just as any

s/ -- /, /

> +other cross compile setup would do.
> +
> +However, historically there were two other definitions, which we will refer
> +to as the @emph{fake} one, and the @emph{lying} one.
> +
> +In the @emph{fake} Cygwin to MinGW cross compile case, you actually use the

s/ the//

> +a native MinGW compiler, but you do so from within a Cygwin environment:
> +
> address@hidden
> +cygwin$ export PATH="/c/MinGW/bin:address@hidden@}"

$i?

I suggest @kbd instead of faking a prompt.

> +cygwin$ configure --build=i686-pc-cygwin \
> +     --host=mingw32 \
> +     NM=/c/MinGW/bin/nm.exe
> address@hidden example
> +
> +In this way, the build system "knows" that you are cross compiling, and the

``knows''

> +path conversion logic will be used.  However, because the tools
> +(@samp{mingw32-gcc}, @samp{nm}, @samp{ar}) used are actually native

@command

> +w32 applications, they will not understand any Cygwin (that is, unix-ish)
> +absolute paths passed as command line arguments (and, unlike MSYS, Cygwin

s/paths/file names/

> +does not automatically convert such arguments).  However, so long as only

Is "so long as" as right as "as long as"?  (sorry for the pun)

> +relative paths are used in the build system, and non-w32-supported unix

s/paths/file names/

> +idioms such as symlinks and mount points are avoided, this scenario should
> +work.
> +
> +In the @emph{lying} Cygwin to MinGW cross compile case, you lie to the
> +build system:
> +
> address@hidden
> +cygwin$ export PATH="/c/MinGW/bin:address@hidden@}"
> +cygwin$ configure --build=i686-pc-mingw32 \
> +     --host=i686-pc-mingw32 \
> +     --disable-dependency-tracking
> address@hidden example
> +

@noindent

> +even though you are actually running under @emph{Cygwin} and not MinGW.
> +In this case, libtool does @emph{not} know that you are performing a
> +cross compile, and thinks instead that you are performing a native MinGW
> +build.  However, as described in (@pxref{Native MinGW Path Conversion}),
> +that scenario triggers an "MSYS to w32" path conversion.  This, of course,
> +is the wrong conversion since we are actually running under Cygwin. To force
> +the correct path conversion in this situation, you should do the following
> address@hidden running configure:
> +
> address@hidden
> +cygwin$ export lt_cv_to_host_file_cmd=func_convert_file_cygwin_to_w32
> address@hidden example
> address@hidden lt_cv_to_host_file_cmd
> address@hidden func_convert_file_cygwin_to_w32
> +
> +Note that this relies on internal implementation details of libtool, and
> +is subject to change.  Also, @code{--disable-dependency-tracking} is 
> required,
> +because otherwise the MinGW gcc will generate dependency files that contain

GCC when speaking about the compiler collection, @command{gcc} when
speaking about a specific C compiler command.

> +w32 file names.  This, in turn, will confuse the Cygwin @samp{make} program,
> +which does not accept w32 file names:
> +
> address@hidden
> +Makefile:1: *** target pattern contains no `%'.  Stop.
> address@hidden example
> +
> +There have also always been a number of other details required for the
> address@hidden case to operate correctly, such as the use of so-called
> address@hidden mounts}:

@dfn or @emph, not @samp.

> address@hidden
> +# <cygwin-root>/etc/fstab

@var{cygwin-root}

> +D:/foo    /foo     some_fs binary 0 0
> +D:/bar    /bar     some_fs binary 0 0
> +E:/grill  /grill   some_fs binary 0 0
> address@hidden example
> +
> +In this way, top-level directories of each drive are available
> +using identical names within cygwin.
> +
> +Note that you also need to ensure that the standard unix directories
> +(like /bin, /lib, /usr, /etc) appear in the root of a drive. This means

@file here and below.

> +that you must install cygwin itself into the C:/ root directory (or D:/,
> +or E:/, etc) -- instead of the recommended installation into C:/cygwin/.
> +
> +In addition, all paths used in the build system must be relative, symlinks

s/paths/file names/

> +should not be used within the source or build directory trees, and all
> address@hidden options to @samp{gcc} except @code{-MMD} must be avoided.

@option{-M*}
@option{-MMD}

This is not easy to achieve with Automake, as it will happily use some
of these options.

> +
> +This is quite a fragile setup, but it has been in historical use, and so is
> +documented here.
> +
>  @node libtool script contents
>  @section @code{libtool} script contents
>  @cindex implementation of libtool