[task #15686] Removing original software URLs from Maneage?

[Mohammad Akhlaghi]

Update of task #15686 (project reproduce):

                  Status:             In Progress => Ready For Test         

    _______________________________________________________

Follow-up Comment #11:

             !!!!!!!!!!!!!!!!!!!!!!!!!!
             !!!!! VERY IMPORTANT !!!!!
             !!!!!!!!!!!!!!!!!!!!!!!!!!

Implementing this task forced me to go a little low-level, and very soon
started a chain-reaction(!), in one way or another, merging into these tasks:
task #15399, task #15389, task #15272, task #15359 and task #15330! 

It effectively took my free time in the whole week, and resulted in an
almost-complete rewrite of the software building phase of Maneage, making it
SO MUCH more robust and easier to read/modify.

You can see the changes in Commit 603ba6d459e
<https://gitlab.com/maneage/project-dev/-/commit/603ba6d459e> on my
development branch. Just for the record, I am also putting the full commit
message at the end of this post.

I already checked the building of all the software (with the new
'--all-highlevel' option) in a 'slim' Debian-stable Docker image (while
testing task #15389!) and it worked nicely (after almost two days of fixing
issues due to the low-level change in each package!). But it would be great if
you could also try it out and look into it in the next two days. If a problem
isn't found or a suggestion isn't made by Saturday. I'll merge it into the
core Maneage branch.

In particular, Raul, it would be great if you could try this branch on your
macOS (don't forget the '--all-highlevel' option ;-)).


 IMPORTANT: Software recipes use GNU Make, many other improvements

POSSIBLE EFFECT ON YOUR PROJECT: The changes in this commit may only cause
conflicts to your project if you have changed the software building
Makefiles in your project's branch (e.g., 'basic.mk', 'high-level.mk' and
'python.mk'). If your project has only added analysis, it shouldn't affect
it, but still read through the points for using the newly added
features. In general, things have become much more clear and easier to
read/execute now ;-).

This is a large commit, involving a long series of corrections in a
differnt branch for an initial low-level change. Other changes naturally
came up as the low-level structure was improved.

 - Until now, Maneage needed the host to have a 'make' implementation which
   was necessary to build Lzip (Lzip is then used to uncompress the source
   of GNU Make). But this is not the case in the minimalist/slim versions
   of operating systems (for example used to build Docker images). After
   consulting Antonio Diaz Diaz (creator of Lzip), he kindly added the
   necessary functionality to Lzip and Lzip can now be built without Make.
   Hence we don't need this assumption any more. With this commit, Lzip and
   GNU Make are built without Make, allowing everything else to be safely
   built with our own custom version of GNU Make and not using the host's
   'make' at all.

 - Until recently (Commit 3d8aa595) GNU Make was built in
   'basic.mk'. Therefore 'basic.mk' was written in a way that it can be
   used with other 'make' implementations also (important shell commands
   starting with '&&' and ending in '\', it also didn't allow any
   comments!). Furthermore, to help in style uniformity, the rules in
   'high-level.mk' and 'python.mk' also followed a similar structure. But
   due to the point above, we can now guarantee that GNU Make is used from
   the very first Makefile, so this hard-to-read structure has been removed
   in the software build recipes.

 - Until now, the default backup servers where at some fixed URLs, on our
   own pages or on Gitlab. But recently we uploaded all the necessary
   software to Zenodo (https://doi.org/10.5281/zenodo.3883409) which is
   more suitable for this task (it promises longevity, has a fixed DOI,
   while allowing us to add new content, or new software tarball
   versions). With this commit, a small script has been written to extract
   the most recent Zenodo upload link from the Zenodo DOI and use it for
   downloading the software source codes.

 - Until now, we primarily used the webpage of each software as the primary
   source for downloading its tarball. But this caused many problems: 1)
   Some of them needed Javascript before the download, 2) Some URLs had a
   complex dependency on the version number, 3) some servers would be
   randomly down for maintenance and etc. So thanks to the point above, we
   now use the Zenodo server as the primary download location. However, if
   a user wants to use a custom software that is not (yet!) on Zenodo, the
   download script gives priority to a custom URL that the users can give
   as Make variables. If that variable is defined, then the script will use
   that URL before going onto Zenodo. We now have a special place for such
   URLs: 'reproduce/software/config/urls.conf'. The old URLs (which are a
   good documentation themselvs) are preserved here, but are commented by
   default.

 - The downloading and checksum verification step has been moved into a
   Make function called 'import-source' (defined in the 'build-rules.mk'
   and loaded in all software Makefiles). Having taken all the low-level
   steps there, I noticed that there is no more need for having the tarball
   as a separate target! So with this commit, a single rule is the only
   place that needs to be edited/added (greatly simplifying the software
   building Makefiles).

 - Following task #15272, A new option has been added to the './project'
   script called '--all-highlevel'. When this option is given, the contents
   of 'TARGETS.conf' are ignored and a script goes through all the software
   (by parsing the 'versions.conf' file) and sets all high-level software
   as targets to be built. I added and used this new option to confirm the
   extensive changes made in all the software building recipes!

 - Many of the software hadn't been tested for a long time! So after using
   the newly added '--all-highlevel', I found that some need to be
   updated. In general, with this commit 'libpaper' and 'pcre' were added
   as totally new software, and the versions of the following software was
   updated: 'flex', 'libtirpc', 'openblas' and 'lzip'. A 'run-parts.in'
   shell script was added in 'reproduce/software/shell/' which is installed
   with 'libpaper'.

 - Even though we intentionally add the necessary flags to add RPATH inside
   the built executable at compilation time, some software don't do it
   (different software on different operating systems!). Until now, we were
   simply checking if the operating system is macOS, but a more robust
   check is to see if 'patchelf' (used for manually updating RPATH in
   GNU/Linux systems and not installed on macOS) is present or not. Because
   of this, 'patchelf' has been put as a top-level prerequisite, right
   after Tar and is installed before anything else.

 - In 'versions.conf', GNU Libtool is recognized as 'libtool', but in
   'basic.mk', it was 'glibtool'! This caused many confusions and is
   corrected with this commit (in 'basic.mk', it is also 'libtool').

 - A new argument is added to the './project' script to allow easy loading
   of the project's shell and environment for fast/temporary testing of
   things in the same environment as the project. Before activating the
   project's shell, we completely remove all host environment variables to
   simulate the project's environment. It can be called with this command:
   './project shell'. Note that it doesn't yet have any prompt! all lines
   just start with a '$'. We should add one later.




    _______________________________________________________

Reply to this item at:

  <https://savannah.nongnu.org/task/?15686>

_______________________________________________
  Message sent via Savannah
  https://savannah.nongnu.org/