bug#46569: python can't find internal pip modules in environment from manifest

[Jelle Licht]

Jelle Licht <jlicht@fsfe.org> writes:

> Python can not load all pip-related python modules when loaded from a
> manifest file, yet seems to work fine when loaded 'directly' as part of
> a `guix environment'-invocation.
>
>
> Provided that we have a file in `hi.py':
> --8<---------------cut here---------------start------------->8---
> import pip
> print("halfwaypoint")
> import pip._internal.index.package_finder
> print("I made it!")
> --8<---------------cut here---------------end--------------->8---
>
> and a `manifest.scm' with:
>
> --8<---------------cut here---------------start------------->8---
> (use-modules (gnu) (guix packages)
>            (gnu packages python)
>            (gnu packages python-xyz))
>
> (packages->manifest (list python python-pip))
> --8<---------------cut here---------------end--------------->8---
>
> I can get the expected output by running:
> `guix environment --pure --ad-hoc python python-pip -- python3 hi.py' =>
>> halfwaypoint
>> I made it!
>
> Yet, when I run the following:
> `guix environment --pure -m manifest.scm -- python3 hi.py' =>
>
> --8<---------------cut here---------------start------------->8---
> halfwaypoint
> Traceback (most recent call last):
>   File "hi.py", line 4, in <module>
>     import pip._internal.index.package_finder
>   File 
> "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/__init__.py",
>  line 40, in <module>
>     from pip._internal.cli.autocompletion import autocomplete
>   File 
> "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/autocompletion.py",
>  line 8, in <module>
>     from pip._internal.cli.main_parser import create_main_parser
>   File 
> "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/main_parser.py",
>  line 11, in <module>
>     from pip._internal.commands import (
>   File 
> "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/commands/__init__.py",
>  line 6, in <module>
>     from pip._internal.commands.completion import CompletionCommand
>   File 
> "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/commands/completion.py",
>  line 6, in <module>
>     from pip._internal.cli.base_command import Command
>   File 
> "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/base_command.py",
>  line 26, in <module>
>     from pip._internal.index import PackageFinder
> ImportError: cannot import name 'PackageFinder' from 'pip._internal.index' 
> (/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/index/__init__.py)
> --8<---------------cut here---------------end--------------->8---
>
> Why is there a difference in the first place? Shouldn't both approaches work?

Using diffoscope, I found the (practical) difference:

$GUIX_ENVIRONMENT/bin/pip3 is
│ │ -destination: 
/gnu/store/rz42ba0my9vrgbkjpkzr2drmnjk5ah50-python-3.8.2/bin/pip3
│ │ +destination: 
/gnu/store/d75bwzbla5ybhs0mdw80qy94mawnhhsw-python-pip-20.2.4/bin/pip3

If you make sure that `python-pip' precedes the `python' package in the
manifest, things work out fine. Likewise for the following --ad-hoc
example:

This fails:
guix environment --ad-hoc python-pip python -- python3 hi.py

This works:
guix environment --ad-hoc python python-pip -- python3 hi.py

So it seems the order in which guix sees packages when building profiles
determines how collisions are resolved, good to know :-).

Is the python-included pip package supposed to be broken like this, though?
 - Jelle