No wasmtime-30.0.0-py3-none-any.whl on PyPI.

[cr1901]

Today, I got an error when trying to add wasmtime to a PDM project in a MinGW environment:

William@DESKTOP-3H1DSBV MINGW64 /tmp/pdm-test
$ pdm add wasmtime
Adding packages to default dependencies: wasmtime
  0:00:01 � Lock successful.
Changes are written to pyproject.toml.
Synchronizing working set with resolved packages: 2 to add, 0 to update, 0 to remove

  ✖ Install wasmtime 30.0.0 failed
  ✔ Install importlib-resources 6.5.2 successful
  ✖ Install wasmtime 30.0.0 failed

ERRORS:
add wasmtime failed:
Traceback (most recent call last):
  File "C:/msys64/mingw64/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:/Users/William/pipx/venvs/pdm/lib/python3.12/site-packages/pdm/installers/synchronizers.py", line 29, in install_candidate
    self.manager.install(can)
  File "C:/Users/William/pipx/venvs/pdm/lib/python3.12/site-packages/pdm/installers/manager.py", line 33, in install
    prepared.build(),
    ^^^^^^^^^^^^^^^^
  File "C:/Users/William/pipx/venvs/pdm/lib/python3.12/site-packages/pdm/models/candidates.py", line 401, in build
    self._obtain(allow_all=False)
  File "C:/Users/William/pipx/venvs/pdm/lib/python3.12/site-packages/pdm/models/candidates.py", line 442, in _obtain
    raise CandidateNotFound(
pdm.exceptions.CandidateNotFound: No candidate is found for `wasmtime` that matches the environment or hashes

  0:00:00 ✖ Some package operations failed. 1/1
See C:/Users/William/AppData/Local/pdm/pdm/Logs/pdm-install-jogfamkt.log for detailed debug log.
[InstallationError]: Some package operations failed.
WARNING: Add '-v' to see the detailed traceback

I've never seen this before, but according to the log, PDM can't find a compatible version, which would be none-any. I noticed on PyPI that the none-any wheel is missing for 30.0.0, but is present for 29.0.0. And indeed, installing 29.0.0 works! Is this intentional?

See #10 for prior art.

[alexcrichton]

I think this was a mistake from #268 where I didn't review that closely enough. You wouldn't happen to know the --plat-name for MinGW would you?

[cr1901]

Assuming you mean the setuptools argument, for Python 3.12 it's this:

$ python
Python 3.12.9 (main, Feb 13 2025, 09:16:40)  [GCC 14.2.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from sysconfig import get_platform
>>> get_platform()
'mingw_x86_64_msvcrt_gnu'
>>>

For Python 3.11 and below, it's mingw_x86_64. See here and here for more info.

[alexcrichton]

Ok I'll attempt to add those in #272

[alexcrichton]

Merging #272 to main with both names resulted in a failure to push to test.pypi.org because of what I presume is mingw_x86_64 being an unknown/invalid name. Removing that name resulted in a failure as well.

[cr1901]

because of what I presume is mingw_x86_64 being an unknown/invalid name

Most likely correct. I think none-any is the only wheel that'll be accepted to PyPI and downloaded by a MinGW Python distribution without a complaint on either end.

Normally the lack of none-any isn't a problem, since compiling from source using the sdist is a fallback. But there isn't an sdist here because there isn't really source to compile (didn't none-any come with "all the wasmtime blobs for supported OSes" because it can't predict on what system it'll be used?).

I think it's worth mentioning workarounds: pyenv and pdm py will download a Python interpreter whose platform matches the wheels uploaded. Compared to 2020 and before, I've had less of a reason to absolutely require a MinGW environment (more specifically, libraries that only compile with GCC and not MSVC). But I'd still rather wasmtime "just work" on MinGW because of inertia, current project setups, and just in case.

[alexcrichton]

Ok I can try that in #275. I really am clueless what the "right thing" to do here is. I'm clearly not equipped to review PRs like this for this project.

[cr1901]

I really am clueless what the "right thing" to do here

This isn't a direct answer, but I hope this helps with context. Basically, PyPI only meaningfully supports "all Linux distros" (manylinux), MacOS, and Windows using MSVC ABI. In my case, I've also used Python on NetBSD and MinGW/MSYS2. There probably won't be a manybsd effort, and PyPI's not interested in supporting MinGW packages.

The way you're supposed to use Python on such systems is "in tandem with the system package manager". Those will provide binary blobs/wheels for systems unsupported by PyPI. But the current practice of suggesting to use virtualenvs for all development complicates that (unless you allow using system deps in your venv, not always possible). So you may still need to use pip and friends anyway to download from PyPI.

If you use Python on any other system, when those Pythons' pip, pdm, pipx, whatever try to contact PyPI, PyPI will "I don't have binary blobs/wheels, here's some source code instead, try compiling it locally, into a local wheel". If you try to upload blobs/wheels compiled for these systems, PyPI will reject it (as you saw). One exception to "I don't have wheels" for these Pythons is "pure Python" packages, compatible with any OS and Python impl; those are none-any packages.

wasmtime doesn't provide a source tarball as part of it's package on PyPI. There's nothing to compile, wasmtime-py is a wrapper over a Rust blob, that's understandable. But PyPI doesn't like it when my MinGW Python asks for wasmtime, realizes there are no compatible wheels, and there's no source either! Having a none-any catch-all wheel which provides all the supported wasmtime blobs for all supported OSes makes PyPI happy; it sends that wheel to my MinGW Python, and AFAIK, after download, the none-any wasmtime-py wheel detects I'm running Windows. Again, AFAIK, wasmtime-py does not really care what Windows ABI is in use, so everything works happily. wasmtime also doesn't compile on BSD (yet!), so in practice, only MinGW Python is affected, but a similar approach would work.

You could also upload a source distribution with the same logic/all the supported wasmtime blobs that does the OS check during build time, or maybe even have the source shell out to build wasmtime using Maturin? I would have to look into it.

[alexcrichton]

Ok that all makes sense, thanks! Would you happen to know if there's a reference for the --plat-name argument in documentation somewhere? That'd be helpful for exploring what the various names are and such.

I figured the none-any was where we'd put Wasmtime's source code, and the only reason that's not done right now is that it's a bit of a pain to set up. I also generally don't know how to orchestrate that build process (although I do know how to build the C API...). If you're willing to help with that it would be much appreciated! I think it'd be reasonable to put the source (plus a precompiled MinGW binary) in the none-any wheel.

[cr1901]

Would you happen to know if there's a reference for the --plat-name argument in documentation somewhere?

I don't think it exists for bdist_wheel. See this thread. Additionally, I don't think a list of supported tags for PyPI are listed anywhere- from a widely-installed example, at least macosx, win32, win, musllinux, and manylinux are supported. For a while not even Linux wheels were accepted, so it's likely the list of PyPI supported tags is small.

Just so we're both aware (TIL!), in a different context outside of bdist_wheel, --plat-name is an MSVC Windows thing. It has caused several dreaded errors in the past for building MinGW packages, where distutils unconditionally thinks Windows means MSVC. This was fixed last year.

I figured the none-any was where we'd put Wasmtime's source code

I would personally put the source in an sdist, and "all the wasmtime blobs" in the none-any wheel. But please see disclaimer.

If you're willing to help with that it would be much appreciated!

Unfortunately, I've never uploaded a Python package with Rust code (or used Maturin, for that matter) so I'm not sure how helpful I'd be. What I know comes from involuntarily having to learn about this stuff over several years to prevent having multiple Python environments installed (currently failing at this :)...).

Disclaimer

Re: the bdist_wheel thread: I don't think the setuptools team wants you using bdist_wheel directly to set the platform tag. The feedback I've given re: sdist and none-any is to the best of my knowledge. In your position, I would probably open up a Discourse thread for further clarification on how to distribute wasmtime-py. I'd also be curious about how their advice differs from mine.

[alexcrichton]

This is all helpful, thanks! I don't have time myself to champion a thread on Discourse, but if you're willing to help I can frame the probably in non-Rust terms if that helps?

The crux is this package is built on the C API of Wasmtime itself, and this is all loaded into Python using ctypes. That chiefly means (in theory) we don't deal with Python-specific versions, so this is different from many other Python native extensions in that regard. The dynamic libraries we ship are intended to be loaded by any version of Python, not just one specific verison of Python.

The Wasmtime release process itself produces a set of dynamic libraries for a slew of platforms, and then the source code itself supports more platforms beyond this. Ideally what I'd want to do is:

• For users using Python on a platform with a precompiled binary, installing wasmtime-py would use that precompiled binary and wouldn't build any Rust code. Ideally only one precompiled binary would be downloaded as opposed to all of them (e.g. you're saying none-any would have all the blobs, but wouldn't that be a lot?)
• For users that don't have a precompiled binary, ideally they'd use CMake + Rust to compile the dynamic library from source during installation. This would boil down to running these commands in the Wasmtime source tree and then ensuring files are installed in the right place.

What I'm doing today I probably copied from somewhere not really knowing fully what I was doing. It's a rough approximation of this but as I'm discovering there's not a great way to have a wheel-per-binary since there's not a platform tag per target and then I also never got around to implementing the build-from-source path.

[cr1901]

The dynamic libraries we ship are intended to be loaded by any version of Python, not just one specific verison of Python.

For uploading to PyPI, that would probably be the py3 tag, which you already do :D!

(e.g. you're saying none-any would have all the blobs, but wouldn't that be a lot?)

Yea, probably. And I would probably go into the none-any approach head-on and realize that half-way through the implementation :). That being said, I think I misunderstood what none-any in 29.0.0 and below was doing; did none-any include only the Windows wasmtime blob to shut the MinGW Python up? I thought it already included all the blobs. But this conversation was nearly 5 years ago, and a lot has changed since then (for better or worse). It's fine if misunderstood; it happens.

[cr1901]

I don't have time myself to champion a thread on Discourse, but if you're willing to help I can frame the probably in non-Rust terms if that helps?

Took me a while to formulate what I wanted to say, but I posted a thread: https://discuss.python.org/t/best-practices-for-cross-compiling-wheels-with-setup-py/85224

[alexcrichton]

Oh dear sorry I forgot about that conversation :(

It's true that, today, it's not the case that all the dynamic libraries are bundled into one wheel. It's also correct that I think we were relying on MinGW as the "one platform" to use the fallback wheel which happens to have the right dynamic library for it (which is also a flawed assumption as many other platforms could use the fallback wheel).

I haven't actually looked into upload limits on PyPI and such, it may just be best to throw everything into one wheel and go from there. That would require some changes to the selection of which dynamic library to load but that wouldn't be the end of the world.

Thanks for opening up a thread though!

[cr1901]

Alright, opening the thread was definitely very productive. I encourage you to read it, but the short fix is:

• Add a stub pyproject.toml, that simply has:

  [build-system]
  requires = ["setuptools"]
  build-backend = "setuptools.build_meta"
  
  # Bonus!
  [tool.pdm]
  distribution = true

  Not actually meaningfully used; python -m build¹ will still handle setup.py alone correctly; this is for forward compatibility.

• Use python -m build¹ instead of calling setup.py directly. With the above pyproject.toml, this should "do the right thing" for everything except setting tags.

• Use the wheel tags command to set tag.

The above is the "well, at least you're no longer calling setup.py directly" fix, and there's a way to set the platform tag that's unlikely to break. A longer-term "don't call setuptools directly/easily set the platform" alternate would be to maybe transition the build-backend to maturin.

I may try this myself if you don't get to it, but I'm definitely distracted and trying to decrease my yak stack these past few weeks :D.

1. build is kinda broken on MinGW in ways that really don't matter. I may elect to use a different frontend for local stuff.

[alexcrichton]

Thanks! I'm not opposed to alternative means of managing this project's metadata, what's here is just what happened to be the first thing I got working. I don't fully grok most of that thread as it's got lots of terms/references to Python things I don't understand, but I can try to find time for things in the future for a change like this.

[cr1901]

Is there a Bytecode Alliance email address I can contact you at?

[alexcrichton]

I don't have a BA email but you can reach me at (EDIT: redacted) (I'll probably scrub this email from this comment once I get an email from you)

[cr1901]

Email sent, feel free to scrub the comment.