Re: Reproducible Project Environments

[zimoun]

Hi,

On Mon, 06 Dec 2021 at 09:18, Peter Polidoro <peter@polidoro.io> wrote:
> Guix is a wonderful way to distribute software, especially for 
> science. Thank you all for such a fantastic project and community!
>
> I am an engineer working with scientists at the Howard Hughes 
> Medical Institute, a science philanthropy based in the United 
> States funding researchers all over the world. I hope to help 
> convince the researchers I work with to all use Guix someday.

Cool!


> My question is, what files do you recommend placing within a 
> project directory to make it as straightforward as possible to 
> distribute that project to others to develop and use in a 
> reproducible environment?

>From my point of view, if we are speaking of a project gluing many thing
to produce analysis, then a Git repo with:

  - manifest.scm containing package set for the project
  - channels.scm containing the revision
  - all your files

If some package are not yet in Guix, or cannot be, they should be in a
custom channel (another Git repo).

All Git repos must be saved in Software Heritage.

Note that channels.scm describes all the channels (official Guix and all
custom channels) and each state of the channel are fixed by commit.

   guix time-machine -C channels.scm -- shell -m manifest.scm

will do a lot of job. :-)  If a Git repo disappears for some reasons,
then the command would fallback to SWH, more or less. ;-)


> What do you recommend when some of the project dependencies are 
> not yet Guix packages? Obviously it would be best to get those 
> dependencies into Guix as official packages eventually, but what 
> is the best way to handle those dependencies until then?

IMHO, custom channels.

    https://hpc.guix.info/channels/


> Say that in addition to some Guix packages, the project also needs 
> some Python packages that are available on pypi, but are not yet 
> Guix packages. Using something like direnv, an .envrc file in the 
> project directory can install the Guix package dependencies into a 
> local profile, then pip install the Python dependencies into a 
> project local virtualenv. Then the user just needs Guix and direnv 
> installed on their machine to get a reproducible environment when 
> they change into that directory. Specifying dependencies in an 
> .envrc file does not seem as elegant as using a pure Guix 
> solution, however.

>From a reproducibility point of view, this method is “dangerous” but
there is no guarantee that PyPI would be in the same state.

On a side note, there is no guarantee either for transparency.  For
instance,

    https://hpc.guix.info/blog/2021/09/whats-in-a-package/


> What is the best way to handle these dependencies without using 
> direnv and just Guix? Should I create Guix packages for the Python 
> dependencies in an external custom channel? If so, how do you 
> specify custom channels when using guix shell? Is there a method 
> that is as simple as direnv or does it involve a user separately 
> setting up a new profile to reference the custom channel before 
> running guix shell?

Yes, I would create a custom channel for these dependencies.  Starting
with “guix import pypi -r”.  Yeah, it can be a lot of work; if not
impossible for some stuff found in PyPI.  Well, it depends on your level
of transparency and the complexity of your stack.

As I wrote above,

    guix time-machine -C channels.m -- shell -m manifest.scm

appears to me the good practise.  Note that manifest.scm could contain
transformation or quickly define a variant using inherit.


> Do you ever create packages within a guix.scm file or a 
> manifest.scm file or perhaps in a separate channel.scm file in the 
> project directory to keep the project more self contained than 
> referencing an external custom channel? Then would you create some 
> project local profile that references this file to use when 
> running guix shell? Or is the recommended approach to always use a 
> custom channel before submitting the packages to be included into 
> Guix proper? Thanks!

To include packages to Guix proper, the package must comply with the
standards.

    https://guix.gnu.org/manual/en/guix.html#Packaging-Guidelines

If for some reasons, the package cannot be included to Guix proper, it
could be included to one of these channels,

    https://hpc.guix.info/channels/

which avoid too many package scattering. ;-)  And bonus, they often
provide substitutes.


Thanks for your message.


Cheers,
simon