Re: Future plans for Autotools

[FOURNIER Yvan]

Hello,


Sorry for reacting a bit late to the January discussion about Autotool's future.


As a longtime user of the GNU Autotools for a large computational dynamics code 
(code_saturne.org), here are a few hopefully constructive remarks about our 
experience so far.


Sorry if my post is a bit long.


Most-liked aspects

-------------------------


- excellent cross-compilation support


- `configure --help` is very practical (CMake's equivalent is much messier, and 
one of the reasons we did not switch)


- "almost" no dependencies on the target system except for a shell and Make. 
This is especially useful on compute clusters which may have a minimal system.


- pretty good documentation


- it is very easy to find the configure options from a previous run at the top 
of the config.log and config.status files, and then copy/past them so as to 
generate an new build in a clean (empty) build directory. This is IMHO a strong 
point compared to CMake, whose caching aspects can lead to complex issues, and 
where regenerating a configuration command is not so easy.


Some Inconsistencies

-----------------------------


While configure/automale/libtool seem to be designed to work together, there 
are a few inconsistencies :


- autoconf can generate multiple configuration files (for example 
public/private header files), but automake assumes only one is used. I don't 
remeber the details, but had to work around this 15 years ago to re-implement 
the generation of a "light" external config.h in m4 and shell instead of using 
built-n features due to this.


- Though one of the initial advantages of the GNU autotools was that a shell 
and Make were sufficient on the target system, GNOME devs started breking that 
with pkg-config, and now it seems resorting to that has become ubiquitous in 
many autotools-based projects, so the dependency list has increased in 
practice. Tools like autopoint also add complexity, as they are available on 
some systems not on others.


- Our code is a mix of Fortran and C, with a bit of C++. Automake still deos 
not support Fortran 90-type module dependencies, so we have to manage manual 
dependencies in one of our Makefile.am's. More modern systems handle Fortran 
(not quite the latest fad) much better.


- When using Gettext, "make update-po" modifies code in the source tree based 
on a command in the build tree. When using separate source and build trees, 
this is somewhat surprising, so having an equivalent command (not based on the 
generated Makefiles) for the source tree would be more consistent as regards 
source/build separation.


Major issue

---------------


Worse, autoconf does not take advantage of libtool's ".am" files to find 
dependencies, while libtool does not allow bypassing these files. This means 
that:


- we need to manually search for dependencies anyways

- but when it encounters .la files with incorrect .la files, libtool adds the 
incorrect dependencies or options, with no workaround


So while .la files were probably a great idea in the early days of the GNU 
project, they are of limited use now (especially since tools not based on 
autotools do not provide them), and are a source of issues.


I have encountered several examples of this :


- .la files broken by Linux packaging: in the case of an MPI library built with 
various dependencies, some of which are loaded dynamically as plugins, and not 
installed by default with the minimal packages, the .la file lists some 
non-installed (and non-needed) libraries as dependencies.


- incorrect options: recently, trying to use the Cray compiler toolchain on an 
Arm64 machine, some low-level dependency related to back-end gcc libraries 
added a "-pthread" option in the link line, while the top-level compiler used a 
different syntax and did not accept that.

Although I have quite a bit of experience working around libtool's quirlks, I 
was simply not able to build my code using that toolchain on that machine. 
Total failure just before the finish line...


-I have had a similar issue generating shared libraries with with CUDA, though 
in that case I could work around this by duplicating and patching the generated 
libtool file, and adding a Makefile.am rule for .cu files. But I have never 
seen another project optionally using CUDA which bothered doing this rather 
than using another build system (ie. if there is a better solution, I did not 
find it).


Libtool has many nice aspects, such as handling of versioning, rpaths, and 
more, but its

insistence that it is always right, even in situations where it produces 
unusable commands can really be a showstopper.


I am seriously contemplating removing libtool (while keeping autoconf/automake) 
due to the blocking "no option to ignore .la files even when incorrect" issue, 
as I have no other workaround. This will probably require quite a bit of extra 
work, but is better than hitting a wall.


Comparison with other tools

-------------------------------------


Frequently installing dependencies with CMake, and even adding a call to CMake 
to detect configuration options in our Autotools scripts to detect a 
CMake-based library's configuration, I can attest that CMake is not always fun 
to work with either when things go wrong or less than perfect configuration 
files are used. So CMake can also be quite frustrating.


I have not looked at Meson much so far. In the short term, the fact that it 
requires ninja, which is not yet ubiquitous on HPC systems, makes things a bit 
less comfortable in our case, but the fact that is uses Python is an advantage 
(Python is so prevalent in scientific software that just about all users and 
developers in our community need to learn at least some rudiments anyways, 
which is not true of m4, Makefiles, or cmake). So it could become a serious 
contender (whereas the other Python-based contenders such as scons and Waf seem 
to have less momentum).


General impression

---------------------------


while we have a large investment (several thousand lines of local .m4, plus 
Makefile.am) in packaging with the GNU autotools, two opposite options which 
would make us happy would be to :


- either better integrate them, so as to remove the 
inconstencies/interoperability issues between packages


- or make them more modular, and better able to work independently. This is 
probably more of a documentation / example code issue than an actual 
development issue.

As an application software developer and maintainer, I fully appreciate the 
hard and painstaking work that goes into the autotools, and  know how difficult 
it can be to build a working and realistic roadmap and balance development and 
support efforts, and thank to autotools team for keeping these tools in a 
usable state.

Just hoping these comments are useful.

Best regards,

  Yvan Fournier







Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à 
l'intention exclusive des destinataires et les informations qui y figurent sont 
strictement confidentielles. Toute utilisation de ce Message non conforme à sa 
destination, toute diffusion ou toute publication totale ou partielle, est 
interdite sauf autorisation expresse.

Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le 
copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si 
vous avez reçu ce Message par erreur, merci de le supprimer de votre système, 
ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support 
que ce soit. Nous vous remercions également d'en avertir immédiatement 
l'expéditeur par retour du message.

Il est impossible de garantir que les communications par messagerie 
électronique arrivent en temps utile, sont sécurisées ou dénuées de toute 
erreur ou virus.
____________________________________________________

This message and any attachments (the 'Message') are intended solely for the 
addressees. The information contained in this Message is confidential. Any use 
of information contained in this Message not in accord with its purpose, any 
dissemination or disclosure, either whole or partial, is prohibited except 
formal approval.

If you are not the addressee, you may not copy, forward, disclose or use any 
part of it. If you have received this message in error, please delete it and 
all copies from your system and notify the sender immediately by return message.

E-mail communication cannot be guaranteed to be timely secure, error or 
virus-free.