Re: [rb-general] Paper preprint: Reproducible genomics analysis pipelines with GNU Guix

[Catonano]

2018-05-13 7:07 GMT+02:00 Ricardo Wurmus <address@hidden>:

Catonano <address@hidden> writes:

> Ricardo, I don't understand the problem you're raising here (I didn't read
> the article yet, though)
>
> Would you mind to elaborate on that ?
>
> Why would you want to record the environment ?

I want to record the detected build environment so that I can restore it
at execution time.  Autoconf provides macros that probe the environment
and record the full path to detected tools.  For example, I’m looking
for Samtools, and the user may provide a particular variant of Samtools
at configure time. 

Thanks for clarifying !

Let me vent some thoughts on te issue !

Under Guix, the way to provide a specific version of the Samtools would be to run the configuration in an environment that offers a specific Samtools package, so that the configuration tool can pick that up

Under a traditional distro, it'd be to feed file paths to the configuration tool

 

So, how much of the traditional way of doing things do we want to support, in our pipelines ?

I record the full path to the executable at
configure time and embed that path in a configuration file that is read
when the pipeline is run.

This works fine for tools, but doesn’t work very well at all for modules
in language environments.  Take R for example.  I can detect and record
the location of the R and Rscript executables, but I cannot easily
record the location of build-time R packages (such as r-deseq2) in a way
that allows me to rebuild the environment at runtime.

Instead of writing an Autoconf macro that records the exact location of
each of the detected R packages and their dependencies I chose to solve
the problem in Guix by wrapping the pipeline executables in R_SITE_LIBS,
because I figured that on systems without Guix you aren’t likely to
install R packages into separate unique locations anyway — on most
systems R packages end up being installed to one and the same directory.

I think the desire to restore the configured environment at runtime is
valid and we do this all the time when we run binaries that have
embedded absolute paths (to libraries or other tools). 

I didn't mean to imply it's not valid

I was just trying to understand what are the concerns on the ground and the context

 

It’s just that
it gets pretty awkward to do this for things like R packages or Python
modules (or Guile modules for that matter).

The Guix workflow language solves this problem by depending on Guix for
software deployment.  For PiGx we picked Snakemake early on and it does
not have a software deployment solution (it expects to either run inside
a suitable environment that the user provides or to have access to
pre-built Singularity application bundles).  I don’t like to treat
pipelines like some sort of collection of scripts that must be invoked
in a suitable environment.  I like to see pipelines as big software
packages that should know about the environment they need, that can be
configured like regular tools, and thus only require the packager to
assemble the environment, not the end-user.

I understand your concern to consider pipelines as packages

But say, for example, that a pipeline gets distributed as a .deb package with dependencies to R (or Guile) modules

Or, say, that a pipeline is distributed with a bundled guix.scm file containing R modules (or Guile modules) as inputs

Would that break the idea of a pipeline as a package ?

I'm afraid that the idea of a pipeline as a package shouldn't be entrusted to the configuration tool, but rather to the package management tool

And the pipeline author shouldn't be assumed to work in isolation, confident that any package management environment will be able to rus their pipeline smoothly

The pipelines authors should be concerned with the collocation of their pipeline in the packaged graph, that shouldn't be a concern of the packager only

Maybe the sotware authors should provide dependency information in a standardized format (rdf ? ) and that should be leveraged by packagers in order to prepare .deb packages or guix.scm files

And if you are a developer and you want to test the software with a specific version of a dependency, then you should run the configuration tool in an environment where that version of the dependency is available, so that the configuration tool can pick that up

If you are on Guix, you will probably create that environment with the Guix environment tool

If you are on Debian or Fedora, you will have to rely on those distros development tools

On traditional distros, you can install packages in your user folder or in /opt or in other positions

And then, you can feed those to the configuration tool

On Guix, the conditions are different

The idea of pipelines as packages will be treated differently by the configuration tool under Guix and the configuration tool under Debian/Fedora

So, in my view a configuration tool should be quite dumb and assume that the package management is smarter

You object that implies the idea of the pipeline as a ugly hack

That is not necessarily so

It's just that I don't think that the pipelines authors can complete the issue in their configuration management

Guix introduces the idea of the whole dependencies stack and that can't be of concern to packagers only.
I don't think so

Maybe I'm too pessimistic, I don't know

Thanks for this discussion !