Re: PYTHONPATH woes

[Hartmut Goebel]

Hi,

thanks for raising the PYTHONPATH issue. Looks like we have a major problem here.

Am 20.02.2018 um 11:53 schrieb Ricardo Wurmus:

I don’t know why this happens.  I find it puzzling that in this
particular case the user’s profile contains an *older* version of
statsmodels (0.6.1).  The wrapper includes the correct version of
statsmodels (0.8.0) in the PYTHONPATH.  Here’s the backtrace:

You are addressing three issues here:
1) Why is the older package imported, where the newer one is expected to be first in path?
2) We are mixing Python2 and Python3 paths
3) Is the way we use PYTHONPATH in the wrapper the correct way?

re 1): This is extremely curious: Why is the older package imported, where the newer one is expected to be first in path?

To analyze this in detail, please make a copy of the resp. wrapper script and change the last command into:

exec -a "$0" python3 -m site

and run the changed wrapper. This should show all paths defined in the wrapper first.

If this does not give any insight, change it into

exec -a "$0" python3 -v -c "import statsmodels"

and then

exec -a "$0" python3 -v -c "import ribodiff.estimatedisp"

and try to find some insight there.

re 2): As soon as Python2 and Python3 are installed in the same profile, we put both site-packages for both versions into PYTHONPATH, which obviously is wrong.

$ PYTHONPATH= guix environment --ad-hoc address@hidden python
[…]
[guix] $ echo $PYTHONPATH
/gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python2.7/site-packages:/gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python3.5/site-packages:

A simple work-around would be to make the profile a (pseudo) virtual environment, which is a easy as creating a file "/gnu/store/…-profile/pyvenv.cfg". This will trigger a mechanism in site.py to insert /gnu/store/…-profile/lib/pythonX.Y/site-packages" into sys.path - for the current python version only!

Try it:

sudo touch $GUIX_ENVIRONMENT/pyvenv.cfg
$ PYTHONPATH= guix environment --ad-hoc address@hidden python
[…]
[guix] $ PYTHONPATH=/tmp/foo:/tmp/bar python3 -m site
[…]
    '/gnu/store/…-profile/lib/python3.5/site-packages',

re 3) When running

PYTHONPATH=/tmp/foo:/tmp/bar python -m site

on e.g. Debian, one can see that the order in sys.path is as follows:

- $PWD
- $PYTHONPATH elements
- built-in paths (e.g. /usr/lib64/python3.5)
- site-packages

The idea seems to be that PYTHONPATH can overwrite all packages, but site-package can not. This can be seen as if in the wrapper-scripts we are not using PYTHONPATH as indented: The user can not overwrite site-packages. We ought to think if this is what we want, as this is how Python works.

Depending on the result of the analysis for (1) and if we implement (2), we can investigate how to solve (3). One idea I already looked at this evening is to replace the wrappers by a minimal virtualenv.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | address@hidden               |
| www.crazy-compilers.com | compilers which you thought are impossible |