Merge pull request #31 from njsmith/pep-changes

PEP updates

Author: njsmith

Diff for: docker/build_scripts/manylinux1-check.py

@@ -6,11 +6,6 @@ def is_manylinux1_compatible():
     if get_platform() not in ["linux-x86_64", "linux-i686"]:
         return False
 
-    # "wide" Unicode mode is mandatory (always true on CPython 3.3+)
-    import sys
-    if sys.maxunicode <= 0xFFFF:
-        return False
-
     # Check for presence of _manylinux module
     try:
         import _manylinux

Diff for: pep-513.rst

@@ -126,8 +126,7 @@ For these reasons, to achieve broad portability, Python wheels
 
 To be eligible for the ``manylinux1`` platform tag, a Python wheel must
 therefore both (a) contain binary executables and compiled code that links
-*only* to libraries (other than the appropriate ``libpython`` library, which is
-always a permitted dependency consistent with the PEP 425 ABI tag) with SONAMEs
+*only* to libraries with SONAMEs
 included in the following list: ::
 
     libpanelw.so.5
@@ -153,11 +152,7 @@ included in the following list: ::
     libglib-2.0.so.0
 
 and, (b) work on a stock CentOS 5.11 [6]_ system that contains the system
-package manager's provided versions of these libraries. In addition,
-for wheels targeting CPython 3.2 and earlier (including all 2.x
-versions), there is an extra requirement that (c) the wheel be
-built against a version of CPython compiled with 4-byte unicode
-support (i.e. one where ``sys.maxunicode > 0xFFFF``).
+package manager's provided versions of these libraries.
 
 Because CentOS 5 is only available for x86_64 and i686 architectures,
 these are the only architectures currently supported by the ``manylinux1``
@@ -216,6 +211,81 @@ the list above to remove libraries that have turned out to be
 problematic or add libraries that have turned out to be safe.
 
 
+libpythonX.Y.so.1
+-----------------
+
+Note that ``libpythonX.Y.so.1`` is *not* on the list of libraries that
+a ``manylinux1`` extension is allowed to link to. Explicitly linking
+to ``libpythonX.Y.so.1`` is unnecessary in almost all cases: the way
+ELF linking works, extension modules that are loaded into the
+interpreter automatically get access to all of the interpreter's
+symbols, regardless of whether or not the extension itself is
+explicitly linked against libpython. Furthermore, explicit linking to
+libpython creates problems in the common configuration where Python is
+not built with ``--enable-shared``. In particular, on Debian and
+Ubuntu systems, ``apt install pythonX.Y`` does not even install
+``libpythonX.Y.so.1``, meaning that any wheel that *did* depend on
+``libpythonX.Y.so.1`` could fail to import.
+
+There is one situation where extensions that are linked in this way
+can fail to work: if a host program (e.g., ``apache2``) uses
+``dlopen()`` to load a module (e.g., ``mod_wsgi``) that embeds the
+CPython interpreter, and the host program does *not* pass the
+``RTLD_GLOBAL`` flag to ``dlopen()``, then the embedded CPython will
+be unable to load any extension modules that do not themselves link
+explicitly to ``libpythonX.Y.so.1``. Fortunately, ``apache2`` *does*
+set the ``RTLD_GLOBAL`` flag, as do all the other programs that
+embed-CPython-via-a-dlopened-plugin that we could locate, so this does
+not seem to be a serious problem in practice. The incompatibility with
+Debian/Ubuntu is more of an issue than the theoretical incompatibility
+with a rather obscure corner case.
+
+This is a rather complex and subtle issue that extends beyond
+the scope of ``manylinux1``; for more discussion see: [9]_, [10]_,
+[11]_.
+
+UCS-2 vs UCS-4 builds
+---------------------
+
+All versions of CPython 2.x, plus CPython 3.0-3.2 inclusive, can be
+built in two ABI-incompatible modes: builds using the
+``--enable-unicode=ucs2`` configure flag store Unicode data in UCS-2
+(or really UTF-16) format, while builds using the
+``--enable-unicode=ucs4`` configure flag store Unicode data in
+UCS-4. (CPython 3.3 and greater use a different storage method that
+always supports UCS-4.) If we want to make sure ``ucs2`` wheels don't
+get installed into ``ucs4`` CPythons and vice-versa, then something
+must be done.
+
+An earlier version of this PEP included a requirement that
+``manylinux1`` wheels targeting these older CPython versions should
+always use the ``ucs4`` ABI. But then, in between the PEP's initial
+acceptance and its implementation, ``pip`` and ``wheel`` gained
+first-class support for tracking and checking this aspect of ABI
+compatibility for the relevant CPython versions, which is a better
+solution. So we now allow the ``manylinux1`` platform tags to be used
+in combination with any ABI tag. However, to maintain compatibility it
+is crucial to ensure that all ``manylinux1`` wheels include a
+non-trivial abi tag. For example, a wheel built against a ``ucs4``
+CPython might have a name like::
+
+  PKG-VERSION-cp27-cp27mu-manylinux1_x86_64.whl
+                   ^^^^^^ Good!
+
+While a wheel built against the ``ucs2`` ABI might have a name like::
+
+  PKG-VERSION-cp27-cp27m-manylinux1_x86_64.whl
+                   ^^^^^ Okay!
+
+But you should never have a wheel with a name like::
+
+  PKG-VERSION-cp27-none-manylinux1_x86_64.whl
+                   ^^^^ BAD! Don't do this!
+
+We note for information that the ``ucs4`` ABI appears to be much more
+widespread among Linux CPython distributors.
+
+
 Compilation of Compliant Wheels
 ===============================
 
@@ -237,15 +307,15 @@ Docker Image
 
 The first tool is a Docker image based on CentOS 5.11, which is recommended as
 an easy to use self-contained build box for compiling ``manylinux1`` wheels
-[9]_. Compiling on a more recently-released linux distribution will generally
+[12]_. Compiling on a more recently-released linux distribution will generally
 introduce dependencies on too-new versioned symbols. The image comes with a
 full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as
 well as the latest releases of Python and ``pip``.
 
 Auditwheel
 ----------
 
-The second tool is a command line executable called ``auditwheel`` [10]_ that
+The second tool is a command line executable called ``auditwheel`` [13]_ that
 may aid in package maintainers in dealing with third-party external
 dependencies.
 
@@ -285,7 +355,7 @@ While we acknowledge many approaches for dealing with third-party library
 dependencies within ``manylinux1`` wheels, we recognize that the ``manylinux1``
 policy encourages bundling external dependencies, a practice
 which runs counter to  the package management policies of many linux
-distributions' system package  managers [11]_, [12]_. The primary purpose of
+distributions' system package  managers [14]_, [15]_. The primary purpose of
 this is cross-distro compatibility.  Furthermore, ``manylinux1`` wheels on PyPI
 occupy a different  niche than the Python packages available through the
 system package manager.
@@ -352,17 +422,14 @@ think otherwise.
 We know of four main sources of potential incompatibility that are
 likely to arise in practice:
 
-* "Narrow" unicode builds of Python 3.2 and earlier (including all
-  versions of Python 2)
 * Eventually, in the future, there may exist distributions that break
   compatibility with this profile (e.g., if one of the libraries in
   the profile changes its ABI in a backwards-incompatible way)
 * A linux distribution that is too old (e.g. RHEL 4)
 * A linux distribution that does not use ``glibc`` (e.g. Alpine Linux, which is
   based on musl ``libc``, or Android)
 
-Checking for unicode configuration compatibility is straightforward,
-but the other cases are more subtle. We propose a two-pronged
+To address these we propose a two-pronged
 approach. To handle potential future incompatibilities, we standardize
 a mechanism for a Python distributor to signal that a particular
 Python install definitely is or is not compatible with ``manylinux1``:
@@ -389,11 +456,6 @@ Specifically, the algorithm we propose is::
         if get_platform() not in ["linux-x86_64", "linux-i686"]:
             return False
 
-        # "wide" Unicode mode is mandatory (always true on CPython 3.3+)
-        import sys
-        if sys.maxunicode <= 0xFFFF:
-            return False
-
         # Check for presence of _manylinux module
         try:
             import _manylinux
@@ -522,13 +584,21 @@ References
    (https://groups.google.com/forum/#!topic/manylinux-discuss/-4l3rrjfr9U)
 .. [8] distutils-sig discussion
    (https://mail.python.org/pipermail/distutils-sig/2016-January/027997.html)
-.. [9] manylinux1 docker image
-   (https://quay.io/repository/manylinux/manylinux)
-.. [10] auditwheel tool
+.. [9] distutils-sig discussion
+   (https://mail.python.org/pipermail/distutils-sig/2016-February/028275.html)
+.. [10] github issue discussion
+   (https://github.com/pypa/manylinux/issues/30)
+.. [11] python bug tracker discussion
+   (https://bugs.python.org/issue21536)
+.. [12] manylinux1 docker images
+   (Source: https://github.com/pypa/manylinux;
+   x86-64: https://quay.io/repository/pypa/manylinux1_x86_64;
+   x86-32: https://quay.io/repository/pypa/manylinux1_i686)
+.. [13] auditwheel tool
    (https://pypi.python.org/pypi/auditwheel)
-.. [11] Fedora Bundled Software Policy
+.. [14] Fedora Bundled Software Policy
    (https://fedoraproject.org/wiki/Bundled_Software_policy)
-.. [12] Debian Policy Manual -- 4.13: Convenience copies of code
+.. [15] Debian Policy Manual -- 4.13: Convenience copies of code
     (https://www.debian.org/doc/debian-policy/ch-source.html#s-embeddedfiles)