Re: [RFT PATCH v4 0/8] Sysroot series

[Charles Wilson]

On 8/5/2010 3:08 PM, Ralf Wildenhues wrote:
> * Charles Wilson wrote on Thu, Aug 05, 2010 at 08:10:10AM CEST:
>> But, that is untenable: on the one hand you want to assert that "thou
>> shalt include X: in thy prefixes when configuring for $host=mingw" even
>> when cross-compiling for any purpose,
> 
> No; I don't want to assert this.  Users may want to assert this, and I
> don't want to libtool stand in their way if that is possible.  If some
> user has only one drive, or is otherwise fine with not specifying a
> drive prefix, I'm totally fine with that.

Ooookay.  So what are we arguing about?  Current status (with Paolo's
patches) is everything appears to work as desired for $host=mingw,
native, and cross, in the following cases:
  (a) native: prefix=X:/foo
  (b) native: prefix=/foo [1]
  (c) cross: prefix=/foo [1,2]

[1] exploits windows CWD-based path resolution; opens you up to possible
security vulnerabilities

[2] sysroot support in compiler, libtool, and other elements is
    helpful here, but not *strictly* necessary. It gets really painful
    without it tho.

What you *can't* do is
  (d) cross: prefix=X:/foo
except in the very simplest of cases, or if you're really lucky.
"Fixing" that...goes beyond difficult into herculean.


But a STEP in that direction is to reduce the current pain experienced
by cross compiles for $host=mingw even with --prefix=/unixish; pain
caused by the current lack of sysroot support in libtool (and some other
utilities).

But...mingw is not the only $host in the world; the pain in (c) is
experienced right now by *all* cross users of libtool.  Other $hosts
don't care about mingw's issues with X:/foo paths.  The only situations
that matter to them are (b) and (c), and libtool's current lack of
sysroot support makes (c) very painful.

Let's fix that, THEN worry about the screwball mingw platform's
decade-festering issue with multiroot file systems.

>> The only real solution, as I've said, is:
>>
>>   1) don't cross compile for $host=mingw. Ever. Always compile
>>      natively, and use --prefix=X:/stuff WITH the drive letter.
> 
> We are at that position already, and would like to improve from there.

Well, we're a bit past THAT.  You can DO cross compiles with
--prefix=/foo, even without sysroot support.  It's ugly, and you have to
post-process the .la files and jump thru other hoops, but it can be done.

With sysroot support in the (cross) compiler, AND sysroot support in
libtool, it becomes possible to do it without jumping thru as many hoops.

Now, you are exposed to the "multiroot path resolution" issue, but
*that's not new*.  All *current* cross compile setups for $host=mingw
ALREADY have that problem (whether "sysrooted" or not).  sysroot support
in libtool just makes the whole effort less painful.  Plus it's useful
for non-mingw-$hosts.


Honestly, this whole issue with DESTDIR, and X:/foo paths, is a
*separate topic* and really doesn't belong in THIS discussion at all.
The only reason it came up is because *I'm* the one testing Paolo's
patches, and *I* have a fistful of mingw-target cross compilers.  If I'd
tested only with sun-, hpux-, and irix- cross compilers, none of this
would ever have come up.

>>   2) Fix *every* package that records ANY directory information
>>      so that it is completely relocatable. Each binary or script, or
>>      tool that parses a text file containing directory information,
>>      should autodetect its installation location, and deduce relative
>>      paths to its resources from that.
> 
> Relocatability is one possibility, and one that I think we should
> expore eventually, because it is a conceptually good idea.

Well, there is a set of tools for it in gnulib, but it requires EVERY
library in your system to adopt it, really.  Plus the program component
is GPL, which limits acceptance -- especially if you want it to become
universal.

>> #2 is a gigantic job, and it's not libtool's task to do that for the
>> universe of software packages. Now, MAYBE, we might take a look at
>> relocation support for .la files -- but I fear that would open us up to
>> more security holes than you could shake a stick at.  Resolving all
>> paths into relative paths from whatever is recorded in libdir=''? Oh, boy...
> 
> Why would it be clear that unreliable automatisms need to come into play
> here?  Please, let's not confuse goals with flaws in conceived actual
> implementations.

I don't understand this statement.  But let it go for now; it's not
germane to "reviewing Paolo's sysroot patches".

>> Hence...I think the consensus is:
> 
> I don't think we need consensus on how people are going to use
> autotools.  They can (and will) do whatever they like.  What we need
> is consensus on which semantics we intend to (and can) support.

Which is another way of saying the same thing.  If we don't (or can't)
support X, then people won't do X (because they will try, and it won't
work, and they'll complain, and we'll say "we don't support that").

The discussion of semantics, when you start talking about how and
whether DESTDIR is supported, is a distraction from the sysroot support.
I'd rather drop all discussion of mingw since sysroot is useful and
needed for cross compilers that target other $hosts.

I wish I'd never mentioned DESTDIR in this thread.

> Just so that we don't misunderstand each other: I really intend for
> this patch series to be merged, and by that I mean merged hopefully
> soon after currently known issues are dealt with, even if we don't
> have full test coverage and even if we can't foresee all the indented
> and unintended semantics yet.  But the more we leave to later, the
> more work it is going to be then.
> 
> So I really hope that my comments don't discourage you from working on
> this.  You guys have done a great job so far, and I'm really trying to
> not delay review; it would be sad to see this not fly just because we
> talk past each other about some related questions.

I'm encouraged by the review. I'm discouraged by the rabbit trails
eating up scarce time and resources (and I consider tilting at the
longstanding DESTDIR+win32 windmill a rabbit trail).  It is what it is.
 If it is to be fixed, it's not part of *this* patch, and shouldn't
complicate consideration of *this* change.

>> Because if you then try to reconstruct the actual location on the disk,
>> on $build under sysroot, you get:
>>    /usr/i686-pc-mingw32/sysrootC:/MinGW/
>> and now, the word "sysrootC:", besides being a sibling, rather than a
>> child, of the ACTUAL sysroot returned by the compiler via
>> -print-sysroot, is unsplitable: how can libtool extract the "C:/MinGW"
>> part when it is geared towards splitting paths at path-delimiter characters?
> 
> What if we special-cased drive letters in the way that
>   DESTDIR=C:/foo/bar prefix=D:/baz/zork
> 
> gets mapped to
>   C:/foo/bar/D/baz/zork
> 

Well, I can easily do that as follows:
  make install prefix=/D/baz/zork DESTDIR=C:/foo/bar

but what does that BUY me?

It doesn't help me package for deployment on $host: I can't cd to
C:/foo/bar and do:
   tar cvf baz.tar d/baz/zork
because my end user can't unpack it "above" D:  So, I have to cd to
   cd C:/foo/bar/d/
   tar cvf baz.tar baz/zork
but that's what we ALREADY do, as I described before:

   configure --prefix=D:/baz/zork && make
   make install prefix=/anywhere/baz/zork
   cd /anywhere
   tar cvf baz.tar baz/zork

It doesn't help me deploy on the $build in sysroot, because unless I'm
canadian cross, my DESTDIR is likely to be a unix path anyway.  And my
unix $build has no mechanism to interpret -- as I continue developing --
dos-style paths like D:/baz/zork under $sysroot.  Unless we want to tie,
even more tightly, libtool-for-host=mingw-and-build=*nix to wine...and
recommend mapping /usr/$host/sys-root to a drive letter in your wine
setup.  But that only works for $build=x86-linux.

And if I say -LD:/baz/zork in a *nix-$build cross compile setup, gcc/ld
has to somehow figure out that when concatenating $sysroot and a path
like that, it needs to (a) put a '/' in there, and (b) get rid of the
':'.  But those tools also need to know that paths like
/my/sysroot/D/baz/zork are ALSO special, and when stripping the sysroot
if the first directory is a single letter, and $host=mingw, then /D/
should be turned back into D:/

Unless the user actually MEANT the /D/ directory at the root of some
unspecified drive E:.

Why are we considering this again?  And if we must, can we put it off to
another thread, and keep our focus on Paolo's actual patch for sysroot
support, and not hypothetical fixes for DESTDIR+mingw?

> This would need adjustments in Automake, Libtool, and an unknown number
> of user-written rules, I guess.

and ld, and gcc. and other non-gnu compilers and linkers. And any
compiled code that parses paths that contain this munged "prefix" needs
to special case absolute paths and make sure to map "/D/stuff" to
"D:/stuff".  And reverse.  (and god help you if you actually MEANT
"E:/D/stuff")

>> $ gcc --sysroot=/bob -print-sysroot
>> /bob
>>
>> So, you could easily create a test that uses the native system compiler
>> to "fake" a cross-toolchain "sysroot" test.
> 
> Exactly.  That's what I meant.
...
> These parts are what I don't know, because I have no experience in this
> area.  Is creating such a link farm feasible?  Presumably, some subtrees
> could just be created with one symlink to a directory?  Presumably the
> symlink-tree script from Automake/GCC could help?

Dunno. It's possible that all you really need from lib are elements in
that directory alone, and no subs.  For include, though, who knows how
ugly the #include chain could be, with sys/ and bits/ and sys/bits/ and
sys/bits/$arch and all.  (Oh, and multilib: what about lib64?)

> If such a test group proves to be very expensive, then I don't mind
> disabling it by default; but just the fact that some automated way of
> testing is fairly easily available, would be a big help.

Well, if it's an optional test, then it would be okay to rely on
non-standard (but common) tools, right?  Because symlink-tree is taking
over ten minutes to do my /usr/include, but lndir was less than 30 seconds.

> I don't think anybody argued that --with-sysroot should be the default.
> Right?

There was some confusion on that point right at first. We're all on the
same page now tho.

--
Chuck