[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Please review: documentation for python build system

From: Hartmut Goebel
Subject: Please review: documentation for python build system
Date: Sun, 13 Nov 2016 23:25:04 +0100
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0


we are about to merge the wip-python-build-system. I'd like to add some
commentary there explaining how the different install methods of Python
behave. This could save someone else the effort of anaylsing and testing
this again is some questions about the Python build system arise.

I ask you to review the text below. Since we want to merge soon, I'd ask
you to review soon. Thanks.

In Python there are different ways to install packages: distutils,
easy_install and pip.  All of these are sharing the file,
with distutils in Python 2.0. can be considered as a kind of
Makefile accepting targets (or commands) like "build" and "install".  As of
autumn 2016 the recommended way to install Python packages is using pip.

For both distutils and setuptools running "python install" is
the way
to install Python packages.  With distutils the "install" command basically
copies all packages into <prefix>/lib/pythonX.Y/site-packages.

Some time later "setuptools" have been established to enhance distutils.  To
use setuptools, the developer imports setuptools in  When
setuptools, the original "install" command gets overwritten by setuptools'
"install" command.

easy_install and pip are both command-line tools capable to search and
download the package source from PyPI (the Python Package Index).  Both of
them import setuptools and execute the "" file under their control.
Thus the "" behaves as if the developer had imported setuptools
within - even is still using only distutils.

Setuptools' "install" command (to be more precise: the "easy_install"
which is called by "install") will put the path of the currently installed
version of each package and it's dependencies (as declared in into
an "easy-install.pth" file.  In guix each packages gets it's own
"site-packages" directory and thus an "easy-install.pth" of it's own.  To
avoid conflicts this file gets renamed to <packagename>.pth in phase
rename-pth-file.  To ensure the .pth-file will be process, easy_install also
creates a basic "" in each "site-packages" directory. The file is the
same for all packages, thus there is no need to rename it.

The .pth-files contain the file-system paths (pointing to the store) of all
dependencies.  So the dependency is hidden in the .pth file but is not
in the file-system.  Now if packages A and B both required packages P,
but in
different versions, guix will not detect this when installing both A and
B to
a profile. (For details and example see

Now pip behaves a bit different: it always executes "" with the
"--single-version-externally-managed" set.  This makes setuptools' "install"
command to *not* run "easy_install" but the original "install" command and
thus no .pth-file (and no will be created. The "site-packages"
directory only contains the package and the related .egg-info directory.

This is exactly what we need for guix and this is what we mimic in the
install phase below.

As a draw back, the magic of the .pth file of linking to the other required
packages is gone and these packages have now to be declared as

Note: Importing setuptools also adds two sub-commands:
"install_egg_info" and
"install_scripts".  These sub-commands are executed even if
"--single-version-externally-managed" is set, thus the .egg-info
directory and
the scripts defined in entry-points will always be created.

Note: Even if the "easy-install.pth" is not longer created, we kept this
phase.  There still may be packages creating an "easy-install.pth" manually
for some good reason.

Hartmut Goebel

| Hartmut Goebel          | address@hidden               |
| | compilers which you thought are impossible |

reply via email to

[Prev in Thread] Current Thread [Next in Thread]