Anomalies in Python search-path (was: [PATCH] Update python-pip to 9.0.1)

[Hartmut Goebel]

Am 06.01.2017 um 19:41 schrieb Maxim Cournoyer:
> One thing which I discovered while testing pip on Guix was that
> depending on how you use Python on Guix the PYTHONPATH differs:

I analysed this problem and found set we have another serious problem
here. But good news: There is an easy solution for both problems.

Result of the analysis (as explained below): We must not extend
PYTHONPATH for adding the profile's site-packages directory.

Proposed solution: The standard library module "site" already contains a
place where we can hook in:

   # Prefixes for site-packages; add additional prefixes like /usr/local
here
    PREFIXES = [sys.prefix, sys.exec_prefix]

So we simply prepend $GUIX_ENVIRONMENT to this list:

    PREFIXES = [os.path.expandvar(GUIX_ENVIRONMENT), sys.prefix,
sys.exec_prefix]

Prepending to speed up searches, since within an environment almost all
site-packages will end in GUIX_ENVIRONMENT. There is no need to check if
$GUIX_ENVIRONMENT is set, since if it is not, it will be left unchanged
and the non-existing path will be skipped out automatically

This would solve both the problem Maxim described and the problem
described below, since the Guix site-packages behave exactly like any
other global site-packages directory.

When following this proposal, we *may* even remove some "wrapping" stuff
in the python-build-system. But this has to be investigated first.


Analysis (proof see below):

The python interpreter was two command line options, which are rarely
used, though:

-E ignores environment variables, esp. PYTHONPATH
-S Disable  the  import  of the module site and the site-dependent
manipulations of sys.path.

This means:
a) When passing -E, the Guix environment's site-packages are ignored
(together with PYTHONPATH), while they should not.
b) When passing -S, the Guix environment's site-packages should be
ignored, but is not (since it is specified in $PYTHONPATH, which is not
ignored)

Conclusion: We should must not use PYTHONPATH to specify the Guix's
environment's site-package directory.


Analysis details:

(guix)$ which python3
/gnu/store/zcnb…-profile/bin/python3


Without any options:

-> /home/USER/.local/lib/python3.5/site-packages and
   /gnu/store/zcnb…-profile/lib/python3.5/site-packages
   should be in sys.path

(base)$ python3 -c 'import sys ; print("\n".join(sys.path))'

/usr/lib64/python34.zip
/usr/lib64/python3.4
/usr/lib64/python3.4/plat-linux
/usr/lib64/python3.4/lib-dynload
/home/USER/.local/lib/python3.4/site-packages
/usr/lib64/python3.4/site-packages
/usr/lib/python3.4/site-packages

(guix)$ python3 -c 'import sys ; print("\n".join(sys.path))'

/gnu/store/zcnb…-profile/lib/python3.5/site-packages
/gnu/store/alk9…-python-3.5.2/lib/python35.zip
/gnu/store/alk9…-python-3.5.2/lib/python3.5
/gnu/store/alk9…-python-3.5.2/lib/python3.5/plat-linux
/gnu/store/alk9…-python-3.5.2/lib/python3.5/lib-dynload
/home/USER/.local/lib/python3.5/site-packages
/gnu/store/alk9…-python-3.5.2/lib/python3.5/site-packages


-E     Ignore environment variables  like  PYTHONPATH  and  PYTHONHOME
       that modify the behavior of the interpreter.

-> /home/USER/.local/lib/python3.5/site-packages and
   /gnu/store/zcnb…-profile/lib/python3.5/site-packages
   should be in sys.path

(base)$ python3 -E -c 'import sys ; print("\n".join(sys.path))'

/usr/lib64/python34.zip
/usr/lib64/python3.4
/usr/lib64/python3.4/plat-linux
/usr/lib64/python3.4/lib-dynload
/home/USER/.local/lib/python3.4/site-packages
/usr/lib64/python3.4/site-packages
/usr/lib/python3.4/site-packages

(guix)$ python3 -E -c 'import sys ; print("\n".join(sys.path))'

/gnu/store/alk9…-python-3.5.2/lib/python35.zip
/gnu/store/alk9…-python-3.5.2/lib/python3.5
/gnu/store/alk9…-python-3.5.2/lib/python3.5/plat-linux
/gnu/store/alk9…-python-3.5.2/lib/python3.5/lib-dynload
/home/USER/.local/lib/python3.5/site-packages
/gnu/store/alk9…-python-3.5.2/lib/python3.5/site-packages



-S = Disable  the  import  of the module site and the site-dependent
     manipulations of sys.path that it entails.

-> /home/USER/.local/lib/python3.5/site-packages and
   /gnu/store/zcnb…-profile/lib/python3.5/site-packages
   sould *not* be in sys.path

(base)$ python3 -S -c 'import sys ; print("\n".join(sys.path))'

/usr/lib64/python34.zip
/usr/lib64/python3.4/
/usr/lib64/python3.4/plat-linux
/usr/lib64/python3.4/lib-dynload


(guix)$ python3 -S -c 'import sys ; print("\n".join(sys.path))'

/gnu/store/zcnb…-profile/lib/python3.5/site-packages
/gnu/store/alk9…-python-3.5.2/lib/python35.zip
/gnu/store/alk9…-python-3.5.2/lib/python3.5/
/gnu/store/alk9…-python-3.5.2/lib/python3.5/plat-linux
/gnu/store/alk9…-python-3.5.2/lib/python3.5/lib-dynload

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | address@hidden               |
| www.crazy-compilers.com | compilers which you thought are impossible |