PEP 599: manylinux2014 PEP (#1121)

Updated manylinux baseline, to be considered in combination
with the (competing/complementary?) perennial manylinux
proposal in PEP 600.

Author: di

pep-0566.rst

@@ -2,7 +2,7 @@ PEP: 566
 Title: Metadata for Python Software Packages 2.1
 Version: $Revision$
 Last-Modified: $Date$
-Author: Dustin Ingram <di@di.codes>
+Author: Dustin Ingram <di@python.org>
 BDFL-Delegate: Daniel Holth
 Discussions-To: distutils-sig <distutils-sig at python.org>
 Status: Final

pep-0599.rst

@@ -0,0 +1,316 @@
+PEP: 599
+Title: The manylinux2014 Platform Tag
+Version: $Revision$
+Last-Modified: $Date$
+Author: Dustin Ingram <[email protected]>
+Sponsor: Paul Moore <[email protected]>
+BDFL-Delegate: Paul Moore <[email protected]>
+Discussions-To: https://discuss.python.org/t/the-next-manylinux-specification/
+Status: Draft
+Type: Informational
+Content-Type: text/x-rst
+Created: 29-April-2019
+Post-History: 29-April-2019
+
+
+Abstract
+========
+
+This PEP proposes the creation of a ``manylinux2014`` platform tag to
+succeed the ``manylinux2010`` tag introduced by :pep:`513`. It also
+proposes that PyPI and ``pip`` both be updated to support uploading,
+downloading, and installing ``manylinux2014`` distributions on
+compatible platforms.
+
+Rationale
+=========
+
+CentOS 6 is now the oldest supported CentOS release, and will receive
+maintenance updates through November 30th, 2020, [1]_ at which point
+it will reach end-of-life, and no further updates such as security
+patches will be made available. All wheels built under the
+``manylinux2010`` images will remain at obsolete versions after that
+point.
+
+Therefore, we propose the continuation of the existing manylinux
+standard, and that a new PEP 425-style [2]_ platform tag called
+``manylinux2014`` be derived from CentOS 7 and that the ``manylinux``
+toolchain, PyPI, and ``pip`` be updated to support it.
+
+Similar to how :pep:`571` and :pep:`513` drew allowed shared
+libraries and their symbol versions from CentOS 5.11 and CentOS 6,
+respectively, a ``manylinux2014`` platform tag will draw its libraries
+and symbol versions from CentOS 7, which will reach end-of-life on
+June 30th, 2024. [1]_
+
+The ``manylinuxYYYY`` pattern has a number of advantages that motivate
+continuing with the current status quo:
+
+- Well-defined Docker images with clearly specified compatible
+  libraries;
+- No need to survey for compatibility issues across multiple releases;
+- A single build image and ``auditwheel`` profile per architecture.
+
+There are also some disadvantages:
+
+- Requires drafting a new PEP for every new standard;
+- Requires adding the new platform tag to installers (e.g., ``pip``);
+- Installers are unable to install a platform tag which predates a
+  given release.
+
+There are also challenges which would exist for any proposal,
+including the time and effort it takes to define, prepare and release
+the Docker images and corresponding ``auditwheel`` profiles. These
+challenges were experienced in the long rollout period for
+``manylinux2010``, which took approximately 1 year from PEP acceptance
+to compatible build environment published. [3]_
+
+However, if this PEP can be an indicator, the process is now
+well-defined and easily repeatable, which should increase the timeline
+for rollout of a newer, updated platform tag.
+
+The ``manylinux2014`` policy
+============================
+
+The following criteria determine a ``linux`` wheel's eligibility for
+the ``manylinux2014`` tag:
+
+1. The wheel may only contain binary executables and shared objects
+   compiled for one of the following architectures supported by CentOS
+   7, or a CentOS 7 compatible base image (such as ubi7): [4]_ ::
+
+       x86_64
+       i686
+       aarch64
+       armhfp
+       ppc64
+       ppc64le
+       s390x
+
+   This list adds support for ARM (aarch64, armhfp) and PowerPC
+   (ppc64, ppc64le) architectures supported by the CentOS Alternative
+   Architecture Special Interest Group, as well as the IBM Z (s390x)
+   architecture. [5]_
+
+2. The wheel's binary executables or shared objects may not link
+   against externally-provided libraries except those in the following
+   list: ::
+
+       libgcc_s.so.1
+       libstdc++.so.6
+       libm.so.6
+       libdl.so.2
+       librt.so.1
+       libc.so.6
+       libnsl.so.1
+       libutil.so.1
+       libpthread.so.0
+       libresolv.so.2
+       libX11.so.6
+       libXext.so.6
+       libXrender.so.1
+       libICE.so.6
+       libSM.so.6
+       libGL.so.1
+       libgobject-2.0.so.0
+       libgthread-2.0.so.0
+       libglib-2.0.so.0
+
+   This list is identical to the externally-provided libraries
+   originally allowed for ``manylinux2010``, with one exception:
+   ``libcrypt.so.1`` was removed due to being deprecated in Fedora 30.
+   ``libpythonX.Y`` remains ineligible for inclusion for the same
+   reasons outlined in :pep:`513`.
+
+   On Debian-based systems, these libraries are provided by the
+   packages:
+
+   ============  =======================================================
+   Package       Libraries
+   ============  =======================================================
+   libc6         libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6,
+                 libpthread.so.0, libm.so.6, libutil.so.1, libnsl.so.1
+   libgcc1       libgcc_s.so.1
+   libgl1        libGL.so.1
+   libglib2.0-0  libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0
+   libice6       libICE.so.6
+   libsm6        libSM.so.6
+   libstdc++6    libstdc++.so.6
+   libx11-6      libX11.so.6
+   libxext6      libXext.so.6
+   libxrender1   libXrender.so.1
+   ============  =======================================================
+
+   On RPM-based systems, they are provided by these packages:
+
+   ============  =======================================================
+   Package       Libraries
+   ============  =======================================================
+   glib2         libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0
+   glibc         libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1,
+                 libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6
+   libICE        libICE.so.6
+   libX11        libX11.so.6
+   libXext:      libXext.so.6
+   libXrender    libXrender.so.1
+   libgcc:       libgcc_s.so.1
+   libstdc++     libstdc++.so.6
+   mesa          libGL.so.1
+   ============  =======================================================
+
+3. If the wheel contains binary executables or shared objects linked
+   against any allowed libraries that also export versioned symbols,
+   they may only depend on the following maximum versions::
+
+       GLIBC_2.17
+       CXXABI_1.3.7
+       GLIBCXX_3.4.19
+       GCC_4.8.5
+
+   As an example, ``manylinux2014`` wheels may include binary
+   artifacts that require ``glibc`` symbols at version ``GLIBC_2.12``,
+   because this an earlier version than the maximum of ``GLIBC_2.17``.
+4. If a wheel is built for any version of CPython 2 or CPython
+   versions 3.0 up to and including 3.2, it *must* include a CPython
+   ABI tag indicating its Unicode ABI.  A ``manylinux2014`` wheel
+   built against Python 2, then, must include either the ``cpy27mu``
+   tag indicating it was built against an interpreter with the UCS-4
+   ABI or the ``cpy27m`` tag indicating an interpreter with the UCS-2
+   ABI. [6]_ [7]_
+5. A wheel *must not* require the ``PyFPE_jbuf`` symbol.  This is
+   achieved by building it against a Python compiled *without* the
+   ``--with-fpectl`` ``configure`` flag.
+
+Compilation of Compliant Wheels
+===============================
+
+Like ``manylinux1``, the ``auditwheel`` tool adds ``manylinux2014``
+platform tags to ``linux`` wheels built by ``pip wheel`` or
+``bdist_wheel`` in a ``manylinux2014`` Docker container.
+
+Docker Images
+-------------
+
+A ``manylinux2014`` Docker image based on CentOS 7 x86_64 should be
+provided for building binary ``linux`` wheels that can reliably be
+converted to ``manylinux2014`` wheels. This image will come with a
+full compiler suite installed (``gcc``, ``g++``, and ``gfortran``
+4.8.5) as well as the latest releases of Python and ``pip``.
+
+Compatibility with kernels that lack ``vsyscall``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Similar to CentOS 6, CentOS 7 includes a version of ``glibc`` that
+depends on the ``vsyscall`` page. As described in :pep:`571`, this
+breaks the assumption that a Docker containers userland is compatible
+wtih its host's kernel. Similar to ``manylinux2010``, Docker images
+for ``manylinux2014`` require patching ``glibc`` to remove all
+dependencies on ``vsyscall`` in the version of ``glibc`` included with
+our Docker image.
+
+Auditwheel
+----------
+
+The ``auditwheel`` tool will also be updated to produce
+``manylinux2014`` wheels. [8]_ Its behavior and purpose will be
+otherwise unchanged from :pep:`513`.
+
+Platform Detection for Installers
+=================================
+
+Platforms may define a ``manylinux2014_compatible`` boolean attribute
+on the ``_manylinux`` module described in :pep:`513`. A platform is
+considered incompatible with ``manylinux2014`` if the attribute is
+``False``.
+
+If the ``_manylinux`` module is not found, or it does not have the
+attribute ``manylinux2014_compatible``, tools may fall back to
+checking for glibc. If the platform has glibc 2.17 or newer, it is
+assumed to be compatible unless the ``_manylinux`` module says
+otherwise.
+
+Specifically, the algorithm we propose is::
+
+    def is_manylinux2014_compatible():
+        # Only Linux, and only supported architectures
+        from distutils.util import get_platform
+
+        if get_platform() not in [
+            "linux-x86_64",
+            "linux-i686",
+            "linux-aarch64",
+            "linux-armhfp",
+            "linux-ppc64",
+            "linux-ppc64le",
+            "linux-s390x",
+        ]:
+            return False
+
+        # Check for presence of _manylinux module
+        try:
+            import _manylinux
+
+            return bool(_manylinux.manylinux2014_compatible)
+        except (ImportError, AttributeError):
+            # Fall through to heuristic check below
+            pass
+
+        # Check glibc version. CentOS 7 uses glibc 2.17.
+        # PEP 513 contains an implementation of this function.
+        return have_compatible_glibc(2, 17)
+
+Backwards compatibility with ``manylinux2010`` wheels
+=====================================================
+
+As explained in :pep:`513`, the specified symbol versions for
+``manylinux1`` allowed libraries constitute an *upper bound*.  The
+same is true for the symbol versions defined for ``manylinux2014`` in
+this PEP.  As a result, ``manylinux1`` and ``manylinux2010`` wheels
+are considered ``manylinux2014`` wheels.  A ``pip`` that recognizes
+the ``manylinux2014`` platform tag will thus install ``manylinux2010``
+wheels for ``manylinux2014`` platforms -- even when explicitly set --
+when no ``manylinux2014`` wheels are available.
+
+PyPI Support
+============
+
+PyPI should permit wheels containing the ``manylinux2014`` platform
+tag to be uploaded in the same way that it permits ``manylinux2010``.
+It should not attempt to verify the compatibility of ``manylinux2014``
+wheels.
+
+References
+==========
+
+.. [1] CentOS Product Specifications
+   (https://wiki.centos.org/About/Product)
+.. [2] PEP 425 -- Compatibility Tags for Built Distributions
+   (https://www.python.org/dev/peps/pep-0425/)
+.. [3] Tracking issue for manylinux2010 rollout
+   (https://github.com/pypa/manylinux/issues/179)
+.. [4] Red Hat Universal Base Image 7
+   (https://access.redhat.com/containers/?tab=overview#/registry.access.redhat.com/ubi7)
+   (https://wiki.centos.org/SpecialInterestGroup/AltArch)
+.. [5] The CentOS Alternative Architecture Special Interest Group
+   (https://wiki.centos.org/SpecialInterestGroup/AltArch)
+.. [6] PEP 3149
+   https://www.python.org/dev/peps/pep-3149/
+.. [7] SOABI support for Python 2.X and PyPy
+   https://github.com/pypa/pip/pull/3075
+.. [8] auditwheel
+   (https://github.com/pypa/auditwheel/)
+
+Copyright
+=========
+
+This document is placed in the public domain or under the
+CC0-1.0-Universal license, whichever is more permissive.
+
+..
+   Local Variables:
+   mode: indented-text
+   indent-tabs-mode: nil
+   sentence-end-double-space: t
+   fill-column: 70
+   coding: utf-8
+   End: