Re: Conda environments and reproducibility

[Simon Tournier]

Hi Thibault,

On Tue, 29 Nov 2022 at 10:41, Thibault Lestang <t.lestang@imperial.ac.uk> wrote:

> I think the tweet above is about reproducing an enviroment after
> effectively freezing constitutive packages and their dependenies as you
> describe. They probably used something like
>
> conda env export
>
> Which outputs something similar to (trimmed)
>
> name: justnumpy
> channels:
>   - defaults
> dependencies:

[...]

>   - ncurses=6.3=h5eee18b_3
>   - numpy=1.23.4=py310hd5efca6_0
>   - numpy-base=1.23.4=py310h8e6c178_0
>   - ...

Do you list all the dependencies?  Other said, dependencies of
dependencies?  Is it only run-time dependencies?

Konrad pointed, (it = Conda)

                                         it claims that it cannot find a
     combination of package known to work together and available in the
     archive.

and from my understanding, I think it is because the solver (SAT or
else).  Well, for instance,

        Theorem 1 Checking whether a single package P can be installed, given a
        repository R, is NP-complete.

        https://www.mancoosi.org/edos/algorithmic/#toc15

Here (conda env export), you generated the Conda requirements using the
repository in the state R.  Then, later the repository becomes R’
(somehow it increases the number of combinations) and it does not matter
if the constraints are foo <= 1.23 or are foo=1.2.3 or are
foo=1.2.3=abcd456.

Maybe I am wrong, from my understanding, Conda builds the graph of
dependencies by resolving a combinatorial problem.  When you run,

    conda env create -f environment.yml

then Conda relies on a “dependency” solver documented here [1].  And,
IMHO, it is where it fails.  Well, if instead of ’conda env export’ you
run,

    conda list --explicit > spec-file.txt

then later and elsewhere,

    conda create --name myenv --file spec-file.txt

it should bypass the solver.  But the documentation [1] reads,

        Since the solver is not involved, the dependencies of the
        explicit package(s) are not processed at all. This can leave the
        environment in an inconsistent state, which can be fixed by
        running conda update --all, for example.

Done. :-)  Conda environments are hard, if not impossible, to reproduce
when time is flying.  It is by design, IMHO.


1: <https://conda.io/projects/conda/en/latest/dev-guide/deep-dives/solvers.html>
2: 
<https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html>

Cheers,
simon