[bug#44656] [PATCH] Upgrade pypy3

[Maxim Cournoyer]

Hi Lars,

Lars-Dominik Braun <lars@6xq.net> writes:

> Hi,
>
> the first patch in this series upgrades pypy3 to 7.3.2 and the second
> adjusts several aspects of the package.
>
> Lars
>
>
>>From caf5e4bf9acd10f91de2e6a9c60185475144a1b9 Mon Sep 17 00:00:00 2001
> From: Lars-Dominik Braun <lars@6xq.net>
> Date: Sun, 15 Nov 2020 10:54:26 +0100
> Subject: [PATCH 2/2] gnu: pypy3: Various package fixes.
>
> * gnu/packages/python.scm (pypy3) [patches]: Add new patch.
> [inputs]: Remove bash-minimal and add nss-certs.
> [native-inputs]: Remove nss-certs here.

In Guix we leave the choice of the TLS certs to the users; meaning we
don't hard-code their location in packages and instead have the users
explicitly install them (and in the case of OpenSSL, set the necessary
environment variables), the same we do for icons and other 'choosable'
things; while convenient the above goes against this tradition.

> [arguments]: Use gdbm compat library, add 2to3 binary.
> [native-search-paths]: Add search path.
> * gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch: New file.
> * gnu/local.mk: Add it.
> ---
>  gnu/local.mk                                  |   1 +
>  .../patches/pypy3-7.3.1-ssl-paths.patch       |  41 +++++++
>  gnu/packages/python.scm                       | 109 ++++++++++++------
>  3 files changed, 116 insertions(+), 35 deletions(-)
>  create mode 100644 gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch
>
> diff --git a/gnu/local.mk b/gnu/local.mk
> index 91a3295e75..b644391a11 100644
> --- a/gnu/local.mk
> +++ b/gnu/local.mk
> @@ -1536,6 +1536,7 @@ dist_patch_DATA =                                       
>         \
>    %D%/packages/patches/python-unittest2-remove-argparse.patch        \
>    %D%/packages/patches/python-waitress-fix-tests.patch               \
>    %D%/packages/patches/pypy3-7.3.1-fix-tests.patch           \
> +  %D%/packages/patches/pypy3-7.3.1-ssl-paths.patch           \
>    %D%/packages/patches/qemu-build-info-manual.patch          \
>    %D%/packages/patches/qemu-glibc-2.27.patch                         \
>    %D%/packages/patches/qrcodegen-cpp-make-install.patch              \
> diff --git a/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch 
> b/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch
> new file mode 100644
> index 0000000000..d21133b4ae
> --- /dev/null
> +++ b/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch
> @@ -0,0 +1,41 @@
> +Fix default certificate search path, still allowing the user to override it
> +with environment variables.
> +
> +--- a/lib_pypy/_cffi_ssl/_stdssl/__init__.py
> ++++ b/lib_pypy/_cffi_ssl/_stdssl/__init__.py
> +@@ -1679,20 +1679,9 @@ def get_default_verify_paths():
> +      https://golang.org/src/crypto/x509/root_linux.go (for the files)
> +     '''
> +     certFiles = [
> +-        "/etc/ssl/certs/ca-certificates.crt",                # 
> Debian/Ubuntu/Gentoo etc.
> +-        "/etc/pki/tls/certs/ca-bundle.crt",                  # Fedora/RHEL 6
> +-        "/etc/ssl/ca-bundle.pem",                            # OpenSUSE
> +-        "/etc/pki/tls/cacert.pem",                           # OpenELEC
> +-        "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem", # CentOS/RHEL 7
> +-        "/etc/ssl/cert.pem",                                 # Alpine Linux
> +     ]
> +     certDirectories = [
> +-        "/etc/ssl/certs",               # SLES10/SLES11
> +-        "/system/etc/security/cacerts", # Android
> +-        "/usr/local/share/certs",       # FreeBSD
> +-        "/etc/pki/tls/certs",           # Fedora/RHEL
> +-        "/etc/openssl/certs",           # NetBSD
> +-        "/var/ssl/certs",               # AIX
> ++        "@GUIX_CERT_PATH@",
> +     ]
> +
> +     # optimization: reuse the values from a local varaible
> +@@ -1707,9 +1696,10 @@ def get_default_verify_paths():
> +     ofile = _cstr_decode_fs(lib.X509_get_default_cert_file())
> +     odir = _cstr_decode_fs(lib.X509_get_default_cert_dir())
> +
> +-    if os.path.exists(ofile) and os.path.exists(odir):
> +-        get_default_verify_paths.retval = (ofile_env, ofile, odir_env, odir)
> +-        return get_default_verify_paths.retval
> ++    if not os.path.exists(ofile):
> ++        ofile = None
> ++    if not os.path.exists(odir):
> ++        odir = None
> +
> +     # OpenSSL didn't supply the goods. Try some other options
> +     for f in certFiles:

As mentioned above, the choice of certs should not be hard coded the in
the package definition.  The correct behavior of honoring SSL_CERT_DIR
and SSL_CERT_FILE environment variables, and falling back to the system
provided location is sane, so the above patch is unwelcome, IMHO.

> diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
> index 8ef8ae2e1d..c0bd3335e3 100644
> --- a/gnu/packages/python.scm
> +++ b/gnu/packages/python.scm
> @@ -696,7 +696,8 @@ ease from the desktop to a microcontroller or embedded 
> system.")
>                (sha256
>                 (base32
>                  "03f1fdw6yk2mypa9pbmgk26r8y1hhmw801l6g36zry9zsvz7aqgx"))
> -              (patches (search-patches "pypy3-7.3.1-fix-tests.patch"))))
> +              (patches (search-patches "pypy3-7.3.1-fix-tests.patch"
> +                                       "pypy3-7.3.1-ssl-paths.patch"))))
>      (build-system gnu-build-system)
>      (native-inputs
>       `(("python-2" ,python-2)
> @@ -704,13 +705,13 @@ ease from the desktop to a microcontroller or embedded 
> system.")
>         ("tar" ,tar)                     ; Required for package.py
>         ("python2-pycparser" ,python2-pycparser)
>         ("python2-hypothesis" ,python2-hypothesis)
> -       ("nss-certs" ,nss-certs)         ; For ssl tests
>         ("gzip" ,gzip)))
>      (inputs
>       `(("libffi" ,libffi)
>         ("zlib" ,zlib)
>         ("ncurses" ,ncurses)
>         ("openssl" ,openssl)
> +       ("nss-certs" ,nss-certs)         ; For ssl module

This change is not needed without the TLS patching above.

>         ("expat" ,expat)
>         ("bzip2" ,bzip2)
>         ("sqlite" ,sqlite)
> @@ -718,10 +719,9 @@ ease from the desktop to a microcontroller or embedded 
> system.")
>         ("tcl" ,tcl)
>         ("tk" ,tk)
>         ("glibc" ,glibc)
> -       ("bash-minimal" ,bash-minimal)   ; Used as /bin/sh
>         ("xz" ,xz)))                     ; liblzma
>      (arguments
> -     `(#:tests? #f     ;FIXME: Disabled for now, there are many tests 
> failing.
> +     `(#:tests? #f ;FIXME: Disabled for now, there are many tests failing.
>         #:modules ((ice-9 ftw) (ice-9 match)
>                    (guix build utils) (guix build gnu-build-system))
>         #:phases (modify-phases %standard-phases
> @@ -750,6 +750,11 @@ ease from the desktop to a microcontroller or embedded 
> system.")
>                        (substitute* '("lib_pypy/_curses_build.py")
>                          ;; Find curses
>                          (("/usr/local") (assoc-ref inputs "ncurses")))
> +                      (substitute* '("lib_pypy/_dbm.py")
> +                        ;; Use gdbm compat library, so we don’t need to pull 
> in bdb
> +                        (("ctypes.util.find_library\\('db'\\)")
> +                         (string-append "'" (assoc-ref inputs "gdbm")
> +                                        "/lib/libgdbm_compat.so'")))

OK.

>                        (substitute* '("lib_pypy/_sqlite3_build.py")
>                          ;; Always use search paths
>                          (("sys\\.platform\\.startswith\\('freebsd'\\)") 
> "True")
> @@ -761,11 +766,18 @@ ease from the desktop to a microcontroller or embedded 
> system.")
>                                          "/lib/libsqlite3.so.0'")))
>                        (substitute* '("lib-python/3/subprocess.py")
>                          ;; Fix shell path
> -                        (("/bin/sh")
> -                         (string-append (assoc-ref inputs "bash-minimal") 
> "/bin/sh")))
> +                        (("/bin/sh") (which "sh")))

OK.

>                        (substitute* 
> '("lib-python/3/distutils/unixccompiler.py")
>                          ;; gcc-toolchain does not provide symlink cc -> gcc
>                          (("\"cc\"") "\"gcc\""))
> +                      (substitute* 
> '("lib_pypy/_cffi_ssl/_stdssl/__init__.py")
> +                        ;; Add nss-certs to default certificate search path,
> +                        ;; otherwise every packages has to specify nss-certs 
> and
> +                        ;; openssl as input to set the proper env variables.
> +                        ;; Depends on -ssl-paths.patch.
> +                        (("@GUIX_CERT_PATH@")
> +                         (string-append (assoc-ref inputs "nss-certs")
> +                                        "/etc/ssl/certs")))

Not every package; the required SSL environment variables it should be
set in the environment by the user or by via a native search path, and
the certs manually provided (installed) by the user.

>                        #t))
>                    (add-after
>                        'unpack 'set-source-file-times-to-1980
> @@ -785,7 +797,8 @@ ease from the desktop to a microcontroller or embedded 
> system.")
>                                  (string-append "--make-jobs="
>                                                 (number->string 
> (parallel-job-count)))
>                                  "-Ojit"
> -                                "targetpypystandalone"))
> +                                "targetpypystandalone"
> +                                "--allworkingmodules"))
>                        ;; Build c modules and package everything, so tests 
> work.
>                        (with-directory-excursion "pypy/tool/release"
>                          (unsetenv "PYTHONPATH") ; Do not use the system’s 
> python libs:
> @@ -793,7 +806,12 @@ ease from the desktop to a microcontroller or embedded 
> system.")
>                                          ; attribute 'IntFlag'
>                          (invoke "python2" "package.py"
>                                  "--archive-name" "pypy-dist"
> -                                "--builddir" (getcwd)))))
> +                                "--builddir" (getcwd))
> +                        ;; install pip and setuptools into the dist 
> directory.
> +                        ;; XXX: Breaks virtualenv, because it does not set +w
> +                        ;; on files copied from the store.
> +                        ;(invoke "pypy-dist/bin/pypy3" "-m" "ensurepip")
> +                        )))

Since the above attempt to bundle pip failed and is new, I'd just leave
it out.

>                    (replace 'check
>                      (lambda* (#:key tests? #:allow-other-keys)
>                        (if tests?
> @@ -811,32 +829,54 @@ ease from the desktop to a microcontroller or embedded 
> system.")
>                        #t))
>                    (replace 'install
>                      (lambda* (#:key inputs outputs #:allow-other-keys)
> -                      (with-directory-excursion "pypy/tool/release"
> -                        ;; Delete test data.
> -                        (for-each
> -                         (lambda (x)
> -                           (delete-file-recursively (string-append
> -                                                     
> "pypy-dist/lib-python/3/" x)))
> -                         '("tkinter/test"
> -                           "test"
> -                           "sqlite3/test"
> -                           "lib2to3/tests"
> -                           "idlelib/idle_test"
> -                           "distutils/tests"
> -                           "ctypes/test"
> -                           "unittest/test"))
> -                        ;; Patch shebang referencing python2
> -                        (substitute* '("pypy-dist/lib-python/3/cgi.py"
> -                                       
> "pypy-dist/lib-python/3/encodings/rot_13.py")
> -                          (("#!.+/bin/python")
> -                           (string-append "#!" (assoc-ref outputs "out") 
> "/bin/pypy3")))
> -                        (with-fluids ((%default-port-encoding "ISO-8859-1"))
> -                          (substitute* '("pypy-dist/lib_pypy/_md5.py"
> -                                         "pypy-dist/lib_pypy/_sha1.py")
> -                            (("#!.+/bin/python")
> -                             (string-append "#!" (assoc-ref outputs "out") 
> "/bin/pypy3"))))
> -                        (copy-recursively "pypy-dist" (assoc-ref outputs 
> "out")))
> -                      #t)))))
> +                      (let* ((out (assoc-ref outputs "out"))
> +                             (bin-pypy3 (string-append out "/bin/pypy3"))
> +                             (shebang-match-python "#!.+/bin/python")
> +                             (shebang-pypy3 (string-append "#!" bin-pypy3))
> +                             (dist-dir "pypy/tool/release/pypy-dist"))
> +                        (with-directory-excursion dist-dir
> +                          ;; Delete test data.
> +                          (for-each
> +                           (lambda (x)
> +                             (delete-file-recursively (string-append
> +                                                       "lib-python/3/" x)))
> +                           '("tkinter/test"
> +                             "test"
> +                             "sqlite3/test"
> +                             "lib2to3/tests"
> +                             "idlelib/idle_test"
> +                             "distutils/tests"
> +                             "ctypes/test"
> +                             "unittest/test"))
> +                          ;; Patch shebang referencing python2
> +                          (substitute* '("lib-python/3/cgi.py"
> +                                         "lib-python/3/encodings/rot_13.py")
> +                            ((shebang-match-python) shebang-pypy3))
> +                          (with-fluids ((%default-port-encoding 
> "ISO-8859-1"))
> +                                       (substitute* '("lib_pypy/_md5.py"
> +                                                      "lib_pypy/_sha1.py")
> +                                         ((shebang-match-python) 
> shebang-pypy3))))
> +                        (copy-recursively dist-dir out)
> +                        ;; Make sure pypy3 is callable as python/python3, so 
> we
> +                        ;; don’t have to patch every single package.
> +                        (symlink bin-pypy3 (string-append out "/bin/python"))
> +                        (symlink bin-pypy3 (string-append out 
> "/bin/python3"))
> +                        ;; 2to3 is missing from pypy3, create it.
> +                        (let ((2to3 (string-append out "/bin/2to3")))
> +                          (call-with-output-file 2to3
> +                            (lambda (port)
> +                              (format port "#!~a~%" (string-append out 
> "/bin/pypy3"))
> +                              (format port "
> +import sys
> +from lib2to3.main import main
> +
> +sys.exit(main('lib2to3.fixes'))")))
> +                          (chmod 2to3 #o755))
> +                        #t))))))

I'm unconvinced about the above symlinks; it seems preferable for users
to be able to unambiguously run both pypy3 and python3 (cpython) in the
same profile without conflicts.  The pypy3 package on Debian contains
the following, for example:

--8<---------------cut here---------------start------------->8---
$ apt-file show pypy3
pypy3: /usr/bin/pypy3
pypy3: /usr/bin/pypy3clean
pypy3: /usr/bin/pypy3compile
pypy3: /usr/lib/libpypy3-c.so
pypy3: /usr/lib/pypy3/bin/libpypy3-c.so
pypy3: /usr/lib/pypy3/bin/pypy3-c
pypy3: /usr/lib/pypy3/include/pypy_decl.h
pypy3: /usr/lib/pypy3/include/pypy_macros.h
pypy3: /usr/lib/pypy3/include/pypy_marshal_decl.h
pypy3: /usr/lib/pypy3/include/pypy_structmember_decl.h
pypy3: /usr/lib/pypy3/lib_pypy/_audioop_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: 
/usr/lib/pypy3/lib_pypy/_blake2/_blake2b_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: 
/usr/lib/pypy3/lib_pypy/_blake2/_blake2s_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_curses_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_decimal_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_gdbm_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_lzma_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_pwdgrp_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_pypy_openssl.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_resource_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_sha3/_sha3_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_sqlite3_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_syslog_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/share/doc/pypy3/TODO.Debian
pypy3: /usr/share/doc/pypy3/changelog.Debian.gz
pypy3: /usr/share/doc/pypy3/copyright
pypy3: /usr/share/lintian/overrides/pypy3
pypy3: /usr/share/man/man1/pypy3.1.gz
--8<---------------cut here---------------end--------------->8---

> +    (native-search-paths
> +     (list (search-path-specification
> +            (variable "PYTHONPATH")
> +            (files '("lib/pypy3.6/site-packages")))))

About this search path, how it is supposed to work?  The version is
wrong (3.6) and the package doesn't include this directory, and we do
not have a pypy build system or other pypy packages that would make use
of it.  Am I missing something?

Thank you!

Maxim