bug#38043: Incorrect Python byte-compiling for Python 3 and PyPy3

[Michał Górny]

Hello,

I've noticed that the logic in py-compile is built on assumptions from
Python 2 and does not fit Python 3 well.  Notably, there are two or
three bugs here:

1. .opt-2 (-OO) level is not compiled for py3.5+.

2. .opt-1 (-O) and .opt-2 (-OO) are wrongly skipped for PyPy3.


Firstly, the code in py-compile byte-compiles in two steps: without '-O' 
and with '-O'.  This is correct for Python <3.5 since '-O' and '-OO'
share the same file (.pyo).  However, since Python 3.5 they are written
into separate '.opt-1.pyc' and '.opt-2.pyc' files and therefore can
coexist.

It is therefore necessary to perform another byte-compilation step with
'-OO', so that all levels of optimization are installed and the module
can be correctly loaded from cache independently of Python options used.


Secondly, the code in '-O' block skips PyPy based on the existence of
sys.pypy_translation_info.  This is correct for PyPy2 that doesn't write
optimized caches.  However, PyPy3 (in particular versions based
on Python 3.5 and newer, I'm not sure about the older versions) do use
the same split naming scheme as CPython.

Therefore, the check here needs to be modified to apply only to Python 2
versions of PyPy.


To reproduce:

touch mytest.py
cat > configure.ac <<EOF
AC_INIT([test], [0])
AM_INIT_AUTOMAKE([foreign])
AM_PATH_PYTHON
AC_CONFIG_FILES([Makefile])
AC_OUTPUT
EOF
cat > Makefile.am <<EOF
pyexec_PYTHON = mytest.py
EOF
autoreconf -i
./configure PYTHON=...
make DESTDIR=... install


With Python 3.5+, the following files are installed:
/tmp/z1/usr/local/lib64/python3.5/site-packages/mytest.py
/tmp/z1/usr/local/lib64/python3.5/site-packages/__pycache__/mytest.cpython-35.opt-1.pyc
/tmp/z1/usr/local/lib64/python3.5/site-packages/__pycache__/mytest.cpython-35.pyc

What should additionally be installed is:
/tmp/z1/usr/local/lib64/python3.5/site-packages/__pycache__/mytest.cpython-35.opt-2.pyc


With PyPy3.5+, the following files are installed:
/tmp/z2/usr/local/lib/python3.6/site-packages/mytest.py
/tmp/z2/usr/local/lib/python3.6/site-packages/__pycache__/mytest.pypy3-72.pyc

What should additionally be installed is:
/tmp/z2/usr/local/lib/python3.6/site-packages/__pycache__/mytest.pypy3-72.opt-1.pyc
/tmp/z2/usr/local/lib/python3.6/site-packages/__pycache__/mytest.pypy3-72.opt-2.pyc


Please let me know if you need any more details.

-- 
Best regards,
Michał Górny

signature.asc
Description: This is a digitally signed message part