Re: API version number policy

[Rafael Laboissiere]

* John W. Eaton <address@hidden> [2009-03-18 13:05]:

> I change it at pretty much every snapshot because I think there are
> usually changes that would break compatibility with previous versions
> of Octave.  I do this because it is a lot easier than looking
> carefully at the sometimes large diffs between snapshots and trying to
> decide whether any of the many changes is something that would cause
> trouble for someone somewhere if they were to try to use the newer
> libraries with an older .oct file compiled and linked with a previous
> version of Octave.

Thanks for the explanation.

If I understand the code correctly, there is a mechanism in defun.cc to
verify whether the .oct module loaded has the appropriate api-v* string.
I think that this can be replaced by a pure ld.so/SONAME-based technique.
I would volunteer to make the necessary changes in the code, if we decide
to make the switch.

Before looking at the details on how this would be implemented in Octave,
let me make a brief recall on shared libraries, SONAMEs, and all this
business.  Note that I will adopt a Linux/GNU-centric point-of-view, but
the ideas could be generalized to any system (BTW, this generalization is
what libtool does under the hoods).

When a C or C++ program is launched, the dynamic linker/loader ld.so
loads all the needed shared libraries.  Let's take this example:

$ ldd /usr/bin/octave-3.1.54 | grep libgfortran
        libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0x00007f127d741000)
        
"libgfortran.so.3" is what is called the SONAME of the library.  We have:

$ ls -l /usr/lib/libgfortran*
lrwxrwxrwx 1 root root     20 2009-02-02 17:31 /usr/lib/libgfortran.so.3 -> 
libgfortran.so.3.0.0
-rw-r--r-- 1 root root 895880 2009-01-02 12:16 /usr/lib/libgfortran.so.3.0.0

Now, shared libraries have typically three version numbers, like the
"3.0.0" above.  If a library is called libfoo.so.X.Y.Z, then X is the
major version number.  This number changes whenever the API/ABI undergoes
backward-incompatible changes.  Y is the minor number, which changes when
there are backward-compatible changes in the API, like additions of new
functions.  Finally, Z is the micro number and changes whenever a new
implementation of the library is released.

As regards ld.so, only the major number X counts, note the presence of
"3" in the SONAME libgfortran.so.3.  (Actually, this is not universally
true, it is only the common practice; nothing prevents developers of
creating a library with a SONAME like libbar.so.3.4-5.)

What happens in the case of .oct modules loaded by Octave?  The answer
is: pretty the same as with programs.  Let'us take this example:

$ ldd /usr/lib/octave/3.1.54/oct/x86_64-pc-linux-gnu/betainc.oct | grep 
liboctave
        liboctave.so => not found

The SONAME above is "liboctave.so".  In the example above, this library is
"not found" but it works in practice because octave sets an rpath.  We could
mimic this by doing:

$ LD_LIBRARY_PATH=/usr/lib/octave-3.1.54 \
  ldd /usr/lib/octave/3.0.4/oct/x86_64-pc-linux-gnu/betainc.oct | grep liboctave
        liboctave.so => /usr/lib/octave-3.1.54/liboctave.so (0x00007fccedbb2000)

When the betainc.oct module is loaded by dlopen, it goes through ld.so in
order to find the liboctave.so above.

Imagine now that Octave shipped with the following shared libraries:

    /usr/lib/liboctave.so.1 -> liboctave.so.1.2.3
    /usr/lib/liboctave.so.1.2.3
    /usr/lib/liboctinterp.so.4 -> liboctinterp.so.4.5.6
    /usr/lib/liboctinterp.so.4.5.6
    /usr/lib/libcruft.so.7 -> liboctinterp.so.7.8.9
    /usr/lib/libcruft.so.7.8.9
    
Note that, in Debian and probably other Linux distros, these files would
be placed in separate packages, names liboctave1, liboctinterp4 and
libcruft7.

In this case, we would have something like this:

$ ldd /usr/lib/octave/3.0.4/oct/x86_64-pc-linux-gnu/betainc.oct
        [...]
        liboctave.so.1 => /usr/lib/liboctave.so.1.2.3 (...)
        liboctinterp.so.4 => /usr/lib/liboctinterp.so.4.5.6 (...)
        libcruft.so.7 => /usr/lib/libcruft.so.7.8.9 (...)
        [...]

With the mechanisms, there is no need for worrying about api-v* number
checking, it is done automatically by ld.so.  In the case a given .oct
module was compiled against liboctave1 and a new version Octave is
installed that is linked against liboctave2, they both could work
together (well, I am not 100% sure about this, but it is principle
possible to do).

In practical terms, this means that the upstream authors of Octave would
have to take care of shared library versioning for liboctave,
liboctinterp, and libcruft, by detecting all the backward-compatible and
-incompatible changes in the API and ABI.  I do not know exactly how hard
this would be. In Debian, there is a nice utility called dpkg-gensymbols
that help maintainers of shared library packages to track some API
changes.

I would like to know your opinion about this.
 
-- 
Rafael