Re: PYTHONPATH woes

[Hartmut Goebel]

Am 23.02.2018 um 17:59 schrieb Pjotr Prins:

I may misunderstand how you are doing this, but I think it is going to
cause problems. Python makes assumptions about the environment and
virtualenv kinda redirects those. On top of that we have guix profiles
and people who need to redirect PYTHONPATH (temporarily). Adding
virtualenv into the mix will complicate things. Then there are people
using virtualenv on top of Guix ...

I have to admit that my description was a bit terse. I understand your concerns, but don't worry. This answer is a bit lengthy and you may read all of the mail as the techniques are a bit complex.

0) I'm proposing to replace the wrapper scripts by a private virtual environment.

1) This only effects the ways scripts are run, the package is still available as normal.

2) The venv-hack I posted a few days ago is proposing something different and is aiming to solve a different problem.

3) AFAIK virtualenvs can NOT be layered, they can only share the system site-packages. This needs to be investigated  further, but only effects the venv-hack I posted a few days ago.

Re 1):

This only effects the way *scripts* are run. Instead of using a wrapper script, this uses a private virtual env - for the script only! Any package is still available as normal.

Instead of using the python-executable of the profile, the script uses the one in its private virtual env. This will make the script being run in its own environment, which we also could name "profile".

The generated layout is as follows:

/gnu/store/…-my-app-0.1
+ lib/python3.5/site-packages/my_package/__init__.py  # unchanged
+ bin/my-app   # no wrapper! uses private venv's python
+ share/guix-venv/my-app-0.1
  +- pyvenv.cfg  # tells bin/python this is a virtual environment
  +- bin/python -> /gnu/store/…-python-3.5.3/bin/python
  +- lib/python3.5/site-packages/my-app.pth  # simulates PYTHONPATH

This comes down to something like

pyvenv-3.5 /gnu/store/…-my-app-0.1/share/guix-venv/my-app-0.1
/gnu/store/…-my-app-0.1/bin/pip install my-app-0.1.tar.gz
ln -s /gnu/store/…-my-app-0.1/share/guix-venv/my-app-0.1/bin/my-app \
      /gnu/store/…-my-app-0.1/bin/my-app
ln -s /gnu/store/…-my-app-0.1/share/guix-venv/my-app-0.1/lib/python3.5/site-packages/my-app \
      /gnu/store/…-my-app-0.1/lib/python3.5/site-packages/my-app

This solves several issues:

• - python-applications in one profile can use conflicting python-packages, since each application's dependencies are enclosed in this application's private environment
• no more problems since the scripts file-name is .my-app-real (see e.g. https://debbugs.gnu.org/cgi/bugreport.cgi?bug=26752)
• packages include references to their dependencies the gc can now pick up (only valid if the package includes a script, though)

Re 2) and 3):

I think the problem of mixing module versions has to be resolved
through profiles. That should just work(tm).

Mixing different package-versions for the same python version can never ever being solved by profile-means Python supports only a single package version in sys.path. Python has this pkg_resources.get_distribution()-stuff, but this never really took of and was superseded by virtual environments.

The problem of mixing interpreter versions can be resolved through
profiles. Though there is the danger that people mix them into one
profile.

Mixing versions of different Python version can by solved by the venv-hack I posted a few days ago. This basically makes the profile into a virtual environment. Thus there is no need for setting PYTHONPATH in the profile anymore (we can even remove the search-path) and packages of different Python versions will no longer be mixed.

For this I suggest we tell Python2 to only use PYTHONPATH2. That way
there is no interference. Python2 is being phased out (it is obsolete)
and upstream should consider such a solution too.

No other distribution is using something like PYTHONPATH2, neither does upstream. Upstream will never introduce such a heavy incompatible change.

Our problems are caused by misusing PYTHONPATH for collection the system/profile site-packages. We could of course implement search-paths GUIX-PYTHON-SITE-PACKAGE-2 and GUIX-PYTHON-SITE-PACKAGE-3 (instead of PYTHONPATH) and change "site.py" to honor them. I'll look into whether this or my venv-hack are a better solution, esp. regarding "stacked" virtual environments.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | address@hidden               |
| www.crazy-compilers.com | compilers which you thought are impossible |