[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Partial linking with _RELOCATABLES - Proposed enhancement (revised a

From: Marc Alff
Subject: Re: Partial linking with _RELOCATABLES - Proposed enhancement (revised and commented)
Date: Tue, 28 Mar 2006 20:47:20 -0700
User-agent: Mozilla Thunderbird 1.0.7 (X11/20060327)

Hi Alexandre and All,

Alexandre Duret-Lutz wrote:

I didn't know "ld -r".  How portable is it?
The command might be named differently per architecture/OS,
but I the concept exist on all platforms that I know of (called incremental linking, or partial linking).
"ld -r" as is can be found on Unixes (not only Linux).
It's very popular with embedded systems, and has been around for a while.

libtool already implements this feature, so I have to guess it's portable enough :)

Will your proposal allow the creation of *.a, *.so, and
binaries out of relocatable objects?  I'm wondering, because the
*.o should be compiled differently if they are meant to be part
of a shared library, and the _RELOCATABLES syntax doesn't
indicate where the object will be used.
You mean -fPIC and all these ? Thanks a lot for pointing that out.

My initial thought was to use only one syntax, and rely on someone to pass the correct CFLAGS to the compiler (with -fPIC and the like or not), but that is old school and pushes the platform
gory details back to the package maintainers.

Instead, there is a much better way :

Looking at how libtool works, it creates two explicitly different syntaxes
( *.o and *.a for static code, *.lo and *.la for code that can be shared),
and in fact injects the "-fPIC" etc compiling/linking options at the last minute, based on the platform, so that platform dependent options are pushed away from the maintainer
and into libtool.

I propose then to match this syntax, and leverage libtool to just do that :

noinst_RELOCATABLE = parts.o
parts_o_SOURCES = part1.c part2.c

will then compile the code to generate "plain" part1.o and part2.o,
and link them together in parts.o, which is suitable to be used later :
- in another *.o relocatable,
- in a static *.a library
- in an executable

noinst_RELOCATABLE = parts.lo
parts_lo_SOURCES = part1.c part2.c

will then compile the code with the magic compiling options to generate shared part1.lo and part2.lo,
and link them together in parts.lo, which is suitable to be used later :
- in another shared *.lo relocatable,
- in a shared *.la library
- in an executable (if it works, to be verified)

MA> The difference between "glued.o" and "libglued.a" is that a MA> "relocatable" object MA> contains all the code in one block, with internal link dependencies MA> already resolved,

Just to make sure I understand: would I get a similar result if
instead of gluing these objects I had compiled the concatenation
of all their sources in a single run of the compiler?
Yes, pretty much.

(Consider that question from the point of view of someone who is
used to split static libraries in as much objects as possible so
the linker picks only what it needs and not more.)
Before being burned as an Heretic by an angry mob, let me clarify a bit :

- I am not suggesting that everyone should stop using *.a or *.la, and link in this (by definition more bulky) way. If someone takes care to split code into small objects (ideally, up to one object per function, so I can link with cos() if I use it, but don't get sin() if I don't need it), to reduce the final binary
side to the minimum, by all means, continue to do so !

- However, there are cases where what the linker thinks is minimal and satisfies all the "obvious" dependencies is in fact too small and misses some code that should have been there (back to the auto-register-plugin example),
which is when partial linking can become useful (or, the only way).

- I view the _RELOCATABLES proposal as just another tool, it's still up to the package maintainer to use the best suited tool for the problem at hand (I just happen to have an unusual problem).


MA> The main concern is that, how the source code is organized into directories
MA> with recursive makefiles is an implementation choice,
MA> it should **not** impact what the final deliverable looks like (maybe I MA> still want ONE *.a or *.so or binary at the end,
MA> not expose a collection of *.a or *.so to my users).

Of your three points, this is the only one I do not understand.
If you use Libtool convenience libraries you'll have a
collection of *.la in your build tree for all the subparts of
your project, but then you'll gather all these in a big *.la
library which you will install.  So the final deliverable is
still one library, not a collection of libraries.
(Example in the "Libtool Convenience Libraries" section of the
Automake manual.)
Well, I just learned that libtool can help with a lot of problems that I had no idea were solved already. What I described was basically using *.o as convenience, and was based on previous experience on a very old project (as in 1992 on an embedded project, it was the only solution then). This point is gone then. Note that the issue of forcing to link code that is not referenced explicitly
still exist.

MA> ============ Part III : Proposed enhancement ============

MA> In short, can be written like this

MA> noinst_RELOCATABLES = glued.o
MA> glued_o_SOURCES = part-1.cpp part-2.cpp

MA> glued.o is a **notation**, not the actual name of a file :

MA> ..._RELOCATABLES = glued.o super-glued.obj
MA> glued_o_... = ...
MA> super-glued_obj_... = ...

Oops, thank you.
It might as well be grep -v "\.obj", see below.

MA> will work both on platforms that use .o or .obj for objects,
MA> and in both cases the generated makefile will contain names like :
MA> glued.$(OBJEXT) : ...
MA> super-glued.$(OBJEXT) : ...

Allowing both extensions in _RELOCATABLES (and hence in the
associated variable names) looks confusing to me.  In any way
the resulting objects may not be built as written, so the
extension we write here is just a convention.  I'd say let's
offer only syntax.
Agreed on removing *.obj.
Due to the earlier comment on shared objects/libraries,
there can still be two syntaxes *.o and *.lo, but for a different reason.
More on this at the end.

MA> You can add these if you like :
MA> glued_o_LDR = /my/own/ldr/program
MA> glued_o_LDRFLAGS = -please -make -section blah -start -at -address blah
MA> glued_o_LIBADD = more-part-X.$(OBJEXT) more-part-Y.$(OBJEXT)

(Independently of your patch I'm wondering if Automake should not try
to replace .o by .$(OBJEXT) in LDADD/LIBADD lines.)
Hey, that would be nice :)

MA> The generated Makefile uses LIBTOOL to actually do the linking work,
MA> so it should be portable : $LIBTOOL --mode=link xxx

Where xxx is "ld -r ..."?  I'd like to hear from the Libtool
folks about this.  Is libtool supposed to emulate "ld -r" or
something if it doesn't work?

My intuition is that if libtool knows how perform "ld -r"
portably then there is an easier way to go:
 - libtool's convenience libraries can be used to setup the
   kind of hierarchical layout you described, and they can be
   used to build static or shared libraries as well as when
   linking program
 - so I guess all you need is a libtool switch to ask
for a relocatable output.

Sorry, I should have expanded the syntax to avoid confusion.

Libtool today already supports this :

(1) $LIBTOOL --mode=link $(CC) -o parts.o part1.o part2.o

and this

(2) $LIBTOOL --mode=link $(CC) -o parts.lo part1.lo part2.lo

I have only tried (1) so far (with the proposed patch).
I will implement support for (2) in my next one.

From the libtool documentation, this is how it works :

LT> If the output-file ends in .la, then a libtool library is created, which must be built only from library objects (.lo files). LT> The -rpath option is required. In the current implementation, libtool libraries may not depend on other uninstalled libtool libraries (see Inter-library dependencies).

LT> If the output-file ends in .a, then a standard library is created using |ar| and possibly |ranlib|.

LT> If output-file ends in .o or .lo, then a reloadable object file is created from the input files (generally using ld -r). This method is often called partial linking.

LT> Otherwise, an executable program is created.

My understanding is that the $(CC) syntax is there for platforms that needs help from the compiler to do partial linking,
but libtool can implement calling directly ld -r when practical.

On my machine for example, the following command is executed from the libtool shell script :

/usr/x86_64-pc-linux-gnu/bin/ld -m elf_x86_64 -r -o lib_giop.o message_header.o giop.o request_header_12.o ... request_header_11.o

One important point is that :
- there is no explicit flag to tell libtool to produce relocatables
- instead, the behavior depends on the file extension used for the output file :
- *.a will be linked as static libraries
- *.la will be linked as shared libraries
- *.o and *.lo will be linked as relocatables

Actually, this is a good thing IMHO, as again it shields the package maintainer from the platform details (which flag to use, etc) and relies on a file name convention to clarify what the intent is.
The libtool implementation then takes care of the portability details.

Then you'd write something like

 libpart_la_SOURCES = foo.c bar.c
 libpart_la_LDFLAGS = -reloadable
 bin_PROGRAMS = whatever
 whatever_SOURCES = main.c
 whatever_LDADD =

Does it make any sense?

It does make sense and I wish it could be used as-is, but there is a gotcha : Based on the libtool command syntax, to use this feature of libtool (the script), one would have to write in automake :

 noinst_LTLIBRARIES = libpart.o <-- cough
 libpart_o_SOURCES = foo.c bar.c
 bin_PROGRAMS = whatever
 whatever_SOURCES = main.c
 whatever_LDADD = libpart.o


 noinst_LTLIBRARIES = libpart.lo <-- cough
 libpart_lo_SOURCES = foo.c bar.c
 bin_PROGRAMS = whatever
 whatever_SOURCES = main.c
 whatever_LDADD = libpart.lo

In both cases, this syntax is not supported today, so changes are needed
in automake to use relocatables in libtool, one way or another.

Instead of overloading the existing syntax for _LTLIBRARIES,
my proposal is to create a new primary _RELOCATABLES.

I think overloading existing primaries with a different file suffix (since it 
has to be) is a bad idea :

- having _LTLIBRARIES = libpart.o is ugly.
One would ask why a static relocatable object (*.o) is not specified like 
_LIBRARIES = libpart.o instead,
while shared relocatable objects are specified with _LTLIBRARIES = libpart.lo
- then, having logic for relocatables duplicated in both primaries is a bad 
- add other considerations to that : the current prefix for _LIBRARIES and 
don't necessarily apply for relocatables ...
- not to mention confusion, documentation, and code bloat/risk in the 
implementation of and the *.am involved.

Instead, the proposal is to match concepts :

1) For a static library (*.a), one would use _LIBRARIES, no changes.

2) For a shared library (*.la), one would use )LTLIBRARIES, no changes.

3) For a relocatable object ? libtool knows to deal with them, but automake does not facilitate that yet,
so the proposal is to have a dedicated primary : _RELOCATABLES

Since two flavors or relocatables can be used, the syntax comes in two flavors :

noinst_RELOCATABLES = libpart.o <-- implemented by my first patch
libpart_o_SOURCES = foo.c bar.c
bin_PROGRAMS = whatever
whatever_SOURCES = main.c
whatever_LDADD = libpart.o

(the generated code uses $(OBJEXT))


noinst_RELOCATABLES = libpart.lo <-- will be implemented by the next one
libpart_lo_SOURCES = foo.c bar.c
bin_PROGRAMS = whatever
whatever_SOURCES = main.c
whatever_LDADD = libpart.lo

(I need to look closely at the generated makefiles for *.lo today,
and the generated code will use $(OBJEXT) or whatever makes sense)


Ouch, long post.
I know it's a lot of details to digest (I am digesting more of libtool myself).

Thanks a lot Alexandre for your previous comments:
you can see by this post that they are taken into account.

While this project matures (Since I now have a work item for another patch),
everyone please post comments or questions if you have any :
I will be glad to correct missing parts sooner rather that later,
so reviews are most welcome.

Thanks all,
Marc Alff.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]