Re: maven build system - new insights

[Julien Lepiller]

Le 31 janvier 2019 11:00:42 GMT+01:00, Hartmut Goebel <address@hidden> a écrit :
>H Julien,
>
>as promised I've spend some time on a maven build system (as always
>when
>regarding maven it was much more time than expected). I did not yet
>create a build-system but want to share my insights so we can decide
>how
>to proceed further. If I'm wrong on some points, please feel free to
>correct this. Everything below is based on my investigations the last
>days.
>
>TL;DR: Let's use XMvn [11] for our maven build-system. What do you
>think?
>
>  * The only place where maven actually loads artifacts from is the
>    local repository, which is a single directory (default:
>    $HOME/.m2/repository). If an artifact is not found in the local
>    repository, it is downloaded from some "remote" repositories and
>    cached in the local repository. I have not found any way to change
>this behavior. See Analysis of repositories behavior below for details.
>  * The local repository path can be changed by passing
>    "-Dmaven.repo.local=..." to maven.
>  * The files in the repository need to be structured like this:
>o
>.../org/apache/maven/maven-plugin-descriptor/2.2.1/maven-plugin-descriptor-2.2.1.jar
>
>This implies a maven-build-system would need either to merge all
>artifacts into a "local repository" or to set up a lot of "mirrors"
>(one
>for each input, this will copy all artifacts into the "local
>repository"). Both is complicated.
>
>Thus I suggest to leverage Xmvn [11] for building (see Distro-Analysis
>below for details). Xmvn searches for configurations (and thus
>repository definitions) in e.g $XDG_DATA_DIRS plus it requires only ca.
>32 jar-files (compared to ca. 48 for maven). Thus we could not only
>solve the repository issue but also (hopefully) reduce the number of
>packages we need to bootstrap.
>
>*Distro-Analysis*
>
>Here is what other GNU/Linux distributions do:
>
>  * Debian stores the files in a system repository structured like
>    above, see [9]. (Thought this seem not to be done for all packages,
>    |e.g.|||[6] and [7] do not include ||these maven-repo files.)
>  * Debian provides a script "mvn-debian" [8] which seem to tweak maven
>    to use this system repository (by passing
>    -Dmaven.repo.local=$MAVEN_REPO").
>      o This is still a single directory. For guix this would require
>        all inputs to be merged into a single directory :-(
>  * Debian also has some build-helper "mh_make" [10] which does some
>    more magic I did not investigate by running some Debian-specific
>    java classes for collecting dependencies.
>
>  * Fedora uses xmvn [11] which "extends" maven to maintain a
>    system-wide repository of artifacts.
>  * There is also a "/Dependency-reduced version of XMvn" /[12],
>    incapable of using remote repositories. This could be helpful for
>    bootstrapping.
>      o A sort analysis says: 32 instead of 48 jar-files are required.
>        Thus we might be able to eliminate 1/3 of the dependence for
>        bootstrapping maven.
>  * The directory structure does not follow the repo-structure, but
>    lloks like this:
>      o /usr/share/java/maven-plugin-tools/maven-plugin-plugin.jar
>   /usr/share/maven-metadata/maven-plugin-tools-maven-plugin-plugin.xml
>       /usr/share/maven-poms/maven-plugin-tools/maven-plugin-plugin.pom
>  *
>
>  * xmvn searches for config-files, which are merged in
>      o $PWD - which would allow us to create one on the fly based on
>        all inputs
>      o $XDG_CONFIG_DIRS and $XDG_DATA_DIRS - which would allow to use
>        native search-path specification
>
>*Analysis of repositories behavior:*
>
>  * The local repository can consist of a single directory only, I did
>   not find any hint about how to specify several directories. Esp. the
>   "Configuring Maven Guide" [1] and [5] state a single directory only.
>    This makes sense somehow since the local repo is where downloaded
>    files are cached.
>  * "Remote" repositories
>      o can be specified in settings.xml (and require to pass a profile
>        name when running, e.g. mvn -Pmyprofile) or the pom.xml. Maybe
>        also in .mvn/{settings,extension).xml, I did not test this.
>      o repositories are processed *after* the central repository
>        (source: [2])
>      o Even if the repository is specified with file://... the
>        artifacts will be copied to the local repo (source: [2], also
>        observed). [3] does not show any option to suppress download.
>      o When using "--offline", all remote repositories are disabled,
>        even if specified with file://...
>  * Mirrors do not help
>      o Can be specified in settings.xml without requiring a profile
>      o Even if the mirror is specified with file://... the artifacts
>     will be copied to the local repo (observed, and somehow logically)
>  * Some more background about repositories can be found at [4].
>  * I also strace'd and ltrace'd several runs of maven to investigate
> whether there any other env-vars or directories are accessed. Nothing.
>
>[1] https://maven.apache.org/guides/mini/guide-configuring-maven.html
>[2] https://www.tutorialspoint.com/maven/maven_repositories.htm
>[3]
>https://maven.apache.org/ref/3.6.0/maven-settings/settings.html#class_releases
>[4]
>https://blog.packagecloud.io/eng/2017/03/09/how-does-a-maven-repository-work/
>[5] https://maven.apache.org/configure.html
>[6]
>https://packages.debian.org/stretch/all/libjsyntaxpane-java/filelist
>[7]
>https://packages.debian.org/stretch/all/libknopflerfish-osgi-framework-java/filelist
>[8] https://wiki.debian.org/Java/MavenBuilder
>[9]
>https://packages.debian.org/stretch/all/libmaven3-core-java/filelist
>[10] https://salsa.debian.org/java-team/maven-debian-helper
>[11] https://fedora-java.github.io/xmvn/
>[12] https://apps.fedoraproject.org/packages/xmvn-minimal/overview/

Thank you for this very detailed information! We've just had a session on the 
subject at the guix days in Brussels.

We found that we need more plugins to be able to build maven projects, and that 
requires some metadata in the jar file. This metadata is generated by 
maven-plugin-tools-generators, and maybe we can write a thin java wrapper 
around it, or use sxml to generate the file.

If we are going to use mvn and not xmvn, we could change the ant-build-system 
to generate the repository structure you describe and have a phase to build a 
union of these with symlinks. Hopefully, maven will follow the symlinks. It 
seems we don't need more info in the jar files themselves, but we need to 
install the pom.xml file alongside the jar itself. Additionnaly, we need 
"parent poms" because they contain properties and values used by our packages 
and maven will want to read those.

Xmvn looks very cool, but that's still 32 more packages to build properly and 
generate metadata for, so it's not clear if that's an advantage when we already 
have a maven package? I'm also a bit worried that it miqht not work properly, 
as it's an equivalent of 3.0.0, but we have a newer version of maven.

We will also need to override dependency and plugin versions for maven to find 
guix' versions, so we'll need a parser and generator for pom.xml files.

WDYT?