diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 0c90dfc9aa7..aecadba5214 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -662,6 +662,8 @@ peps/pep-0780.rst @lysnikolaou peps/pep-0781.rst @methane peps/pep-0782.rst @vstinner # ... +peps/pep-0783.rst @hoodmane @ambv +# ... peps/pep-0789.rst @njsmith # ... peps/pep-0801.rst @warsaw diff --git a/peps/pep-0783.rst b/peps/pep-0783.rst new file mode 100644 index 00000000000..78fdbf24c80 --- /dev/null +++ b/peps/pep-0783.rst @@ -0,0 +1,221 @@ +PEP: 783 +Title: Emscripten Packaging +Author: Hood Chatham <roberthoodchatham at gmail.com> +Sponsor: Ćukasz Langa <lukasz at python.org> +Discussions-To: https://discuss.python.org/t/86862 +Status: Draft +Type: Standards Track +Topic: Packaging +Created: 28-Mar-2025 +Post-History: `02-Apr-2025 <https://discuss.python.org/t/86862>`__, + `18-Mar-2025 <https://discuss.python.org/t/84996>`__, + +Abstract +======== + +This PEP proposes a new platform tag series ``pyodide`` for binary Python package +distributions for the Pyodide Python runtime. + +`Emscripten <https://emscripten.org/>`__ is a complete open-source compiler +toolchain. It compiles C/C++ code into WebAssembly/JavaScript executables, for +use in JavaScript runtimes, including browsers and Node.js. The Rust language +also maintains an Emscripten target. :pep:`776` specifies Python's support for +Emscripten. + + +Motivation +========== + +Pyodide is a CPython distribution for use in the browser. A web browser is a +universal computing platform, available on Windows, macOS, Linux, and every +smartphone. Hundreds of thousands of students have learned Python through +Pyodide via projects like `Capytale +<https://web.archive.org/web/20241211090946/https://cfp.jupytercon.com/2023/talk/TJ9YEV/>`__ +and `PyodideU <https://stanford.edu/~cpiech/bio/papers/pyodideU.pdf>`__. Pyodide +is also increasingly being used by Python packages to provide interactive +documentation. + +Pyodide currently maintains ports of 255 different packages at the time of this +writing, including major scientific Python packages like NumPy, SciPy, pandas, +Polars, scikit-learn, OpenCV, PyArrow, and Pillow as well as general purpose +packages like aiohttp, Requests, Pydantic, cryptography, and orjson. + +About 60 packages are also testing against Pyodide in their CI, including NumPy, +pandas, awkward-cpp, scikit-image, statsmodels, PyArrow, Hypothesis, and PyO3. + +Python package projects cannot deploy binary distributions for Pyodide on PyPI. +Instead they must use other options like ``anaconda.org`` or ``jsdelivr.com``. +This creates friction both for package maintainers and for users. + + +Rationale +========= + +Emscripten uses a variant of musl libc. The Emscripten compiler makes no ABI +stability guarantees between versions. Many Emscripten updates are ABI +compatible by chance, and the Rust Emscripten target behaves as if the ABI were +stable with only `occasional negative consequences +<https://github.com/rust-lang/rust/issues/131467>`__. + +There are several linker flags that adjust the Emscripten ABI, so Python +packages built to run with Emscripten must make sure to match the ABI-sensitive +linker flags used to compile the interpreter to avoid load-time or run-time +errors. The Emscripten compiler continuously fixes bugs and adds support for new +web platform features. Thus, there is significant benefit to being able to +update the ABI. + +In order to balance the ABI stability needs of package maintainers with the ABI +flexibility to allow the platform to move forward, Pyodide plans to adopt a new +ABI for each feature release of Python. + +The Pyodide team also coordinates the ABI flags that Pyodide uses with the +Emscripten ABI that Rust supports in order to ensure that we have support for +the many popular Rust packages. Historically, most of the work for this has +been related to unwinding ABIs. See for instance `this Rust Major Change +Proposal <https://github.com/rust-lang/compiler-team/issues/801>`__. + +The ``pyodide`` platform tags only apply to Python interpreters compiled and +linked with the same version of Emscripten as Pyodide, with the same +ABI-sensitive flags. + + +Specification +============= + +The platform tags will take the form: + +.. code-block:: text + + pyodide_${YEAR}_${PATCH}_wasm32 + +Each one of these will be used with a specified Python version. For example, the +platform tag ``pyodide_2025_0`` will be used with Python 3.13. + +Emscripten Wheel ABI +-------------------- + +The specification of the ``pyodide_<abi>`` platform includes: + +* Which version of the Emscripten compiler is used +* What libraries are statically linked with the interpreter +* What stack unwinding ABI is to be used +* How the loader handles dependency lookup +* That libraries cannot use ``-pthread`` +* That libraries should be linked with ``-sWASM_BIGINT`` + +The ABI is selected by choosing the appropriate version of the Emscripten +compiler and passing appropriate compiler and linker flags. It is possible for +other people to build their own Python interpreter that is compatible with the +Pyodide ABI, it is not necessary to use the Pyodide distribution itself. + +THe Pyodide ABIs are fully specified in the `Pyodide Platform ABI +<https://pyodide.org/en/stable/development/abi.html>`__ documentation. + +The ``pyodide build`` tool knows how to create wheels that match the Pyodide +ABI. Unlike with manylinux wheels, there is no need for a Docker container to +build the ``pyodide_<abi>`` wheels. All that is needed is a Linux machine and +appropriate versions of Python, Node.js, and Emscripten. + +It is possible to validate a wheel by installing and importing it into the +Pyodide runtime. Because Pyodide can run in an environment with strong +sandboxing guarantees, doing this produces no security risks. + +Determining the ABI version +--------------------------- + +The Pyodide ABI version is stored in the ``PYODIDE_ABI_VERSION`` config variable +and can be determined via: + +.. code-block:: python + + pyodide_abi_version = sysconfig.get_config_var("PYODIDE_ABI_VERSION") + + +To generate the list of compatible tags, one can use the following code: + +.. code-block:: python + + from packaging.tags import cpython_tags, _generic_platforms + + def _emscripten_platforms() -> Iterator[str]: + pyodide_abi_version = sysconfig.get_config_var("PYODIDE_ABI_VERSION") + if pyodide_abi_version: + yield f"pyodide_{pyodide_abi_version}_wasm32" + yield from _generic_platforms() + + emscripten_tags = cpython_tags(platforms=_emscripten_platforms()) + +This code will be added to `pypa/packaging +<https://github.com/pypa/packaging/pull/804>`__. + + +Package Installers +------------------ + +Installers should use the ``_emscripten_platforms()`` function shown above to +determine which platforms are compatible with an Emscripten build of CPython. In +particular, the Pyodide ABI version is exposed via +``sysconfig.get_config_var("PYODIDE_ABI_VERSION")``. + +Package Indexes +--------------- + +Package indexes SHOULD accept any wheel whose platform tag matches +the regular expression ``pyodide_[0-9]+_[0-9]+_wasm32``. + + +Dependency Specifier Markers +---------------------------- + +According to :pep:`776#platform-identification`, in Emscripten Python +``sys.platform`` returns ``"emscripten"``. To check for the Emscripten platform in a +dependency specifier, one can use ``sys_platform == "emscripten"`` (or its +negation). + + +Trove Classifier +---------------- + +Packages that build and test Emscripten wheels can declare this by adding the +``Environment :: WebAssembly :: Emscripten`` classifier. PyPI already accepts uploads of +`packages with this classifier +<https://pypi.org/search/?q=&o=&c=Environment+%3A%3A+WebAssembly+%3A%3A+Emscripten>`__. + + +Backwards Compatibility +======================= + +There are no backwards compatibility concerns in this PEP. + + +Security Implications +===================== + +There are no security implications in this PEP. + +How to Teach This +================= + +For Pyodide users, we recommend the `Pyodide documentation on installing +packages <https://pyodide.org/en/stable/usage/loading-packages.html>`__. + +For package maintainers, we recommend the `Pyodide documentation on building and +testing packages +<https://pyodide.org/en/stable/development/building-and-testing-packages.html>`__. + +Reference Implementation +======================== + +For building packages, `pyodide build +<https://github.com/pyodide/pyodide-build>`__ and `cibuildwheel +<https://github.com/pypa/cibuildwheel/>`__. + +For installers to decide whether a wheel tag is compatible with a Pyodide +interpreter, `pypa/packaging#804 +<https://github.com/pypa/packaging/pull/804>`__. + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive.