Re: shared versus archive library

[Paul Pluzhnikov]

"Greg" <greg.szeszko@tradingtechnologies.com> writes:

> I don't have a small example that I could post here.  The one example
> that I did try behaved in an opposite way of what my problem is.  But
> let me explain in a clearer way what the situation is.

Now that you've explained what you are doing, it's hard for me to
understand why you can't come up with a simple test.

I'll assume there is a reason (other than laziness), and will
provide my own.

> I have a library that multiple executables link with.  Some of the
> functions in the library invoke functions that the executables are
> supposed to provide. These are analogous to callbacks.  Some
> executables provide definitions of these callback functions and some do
> not.  The ones that don't, don't invoke the functions calling callbacks
> so the linker shouldn't care about the fact that the callbacks are not
> provided.  All object files (within the library and within executables)
> are compiled with -ggdb and -fPIC.  No other compiler flags (save for
> -I and -D) are used.  The library objects are collected into a static
> archive (.a) and shared library (.so).  The .so lib is compiled with
> -ggdb -fPIC -shared.
>
> Here is what I don't understand:
> When I link the executable with the static lib, .a, Everything links
> fine and the code runs.

This is because the code that references the missing callbacks
isn't even pulled out of the archive, as happens int Example 1 below
with bar.o. To understand how linker works with archive libraries,
you may wish to read this:
 http://webpages.charter.net/ppluzhnikov/linker.html

> But if I link the executable with the shared
> library, .so, the linker complains about unresolved symbols.

Naturally -- the libXXX.so is a single object, and you get all or
nothing from it [Example 2 below]. Since it *does* contain references
that the linker can't resolve, the linker is correct to complain.

> The
> symbols correspond to the missing callbacks that the executable doesn't
> really need. 

But the linker can't know that.

> If I force the linker to complete with the
> --allow-shlib-undefined flag, then the program links.

And it should also run, provided the unresolved references can be
resolved lazily [example 3].

> But when I try
> to run it, I get a message from the loader saying that these symbols
> (or at least the first one) are undefined.

That means that somehow you've made the unresolved symbols that
can't be lazy-resolved. One way this can happen is if you use the
symbols as data [example 4]. Another way is to mark the DSO as
non-lazy, or via LD_BIND_NOW environment [example 5].

> So it appears that when the
> shared library is built, the compiler doesn't optimize out the
> callbacks.

Compiler *can't* possibly optimize the callbacks out.
If it did, you'd be very unhappy, at least for executables that *do*
provide the callback and expect it to be called.

> Is there a way for me to instruct the linker to ignore the missing
> callbacks that it gets from the .so library?  Somehow I need to be able
> to do that the loader doesn't attempt to resolve these symbols.

Yes, there are multiple ways to resolve this problem. However,
we first need to know what your exact problem is. So modify one
of the examples below such that it matches your problem exactly,
and we'll tell you what the solution is.

Cheers,

=========================
All the examples will work with this source:

--- foo.c ---
void foo() { fooCallBack(); }

--- bar.c ---
void bar() { barCallBack(); }

--- main.c ---
void fooCallBack() { }
int main() { foo(); return 0; }

$ gcc -c -fPIC foo.c bar.c; gcc -c main.c # compile everything
$ ar ruv libfoobar.a foo.o bar.o
$ gcc -shared -o libfoobar.so foo.o bar.o

Example 1:

$ gcc main.o ./libfoobar.a

This links and runs with no problems. The bar.o is not pulled into
the a.out, because nobody references bar() [it's not in the "need"
list, as described in the linker.html]

Example 2:

$ gcc main.o ./libfoobar.so

Fails to link (as expected, see above):

  ./libfoobar.so: undefined reference to `barCallBack'
  collect2: ld returned 1 exit status

Example 3:

$ gcc main.o ./libfoobar.so -Wl,--allow-shlib-undefined

Links and *runs* fine.

Example 4: Here unresolved barCallBack() is bound to data, and data
can't be resolved lazily.

$ cat bar2.c 
extern void barCallBack();
void bar() { void (*p)(void) = barCallBack; p(); }

$ gcc -c -fPIC bar2.c; gcc -shared -o libfoobar2.so foo.o bar2.o
$ gcc -o a.out2 main.o ./libfoobar2.so -Wl,--allow-shlib-undefined 

Links fine, but doesn't run:

$ ./a.out2
./a.out2: error while loading shared libraries: ./libfoobar2.so: undefined 
symbol: barCallBack

Example 5: Make libfoobar.so non-lazy:

Either via LD_BIND_NOW environment variable:

$ env LD_BIND_NOW=1 ./a.out
./a.out: error while loading shared libraries: ./libfoobar.so: undefined 
symbol: barCallBack

Or by explicitly marking the DSO:

$ gcc -shared -o libfoobar.so -Wl,-znow foo.o bar.o
$ ./a.out
./a.out: error while loading shared libraries: ./libfoobar.so: undefined 
symbol: barCallBack


-- 
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.