Implement building and publishing Android packages

[anderspitman]

No description provided.

[alexcrichton]

Would it make sense to also bundle in the changes to update this as well?

wasmtime-py/wasmtime/_ffi.py

Lines 11 to 31 in 5e89132

	if sys.platform == 'linux':
	libname = '_libwasmtime.so'
	elif sys.platform == 'win32':
	libname = '_wasmtime.dll'
	elif sys.platform == 'darwin':
	libname = '_libwasmtime.dylib'
	else:
	raise RuntimeError("unsupported platform `{}` for wasmtime".format(sys.platform))
	machine = platform.machine()
	if machine == 'AMD64':
	machine = 'x86_64'
	if machine == 'arm64' or machine == 'ARM64':
	machine = 'aarch64'
	if machine != 'x86_64' and machine != 'aarch64':
	raise RuntimeError("unsupported architecture for wasmtime: {}".format(machine))
	filename = Path(__file__).parent / (sys.platform + '-' + machine) / libname
	if not filename.exists():
	raise RuntimeError("precompiled wasmtime binary not found at `{}`".format(filename))
	dll = cdll.LoadLibrary(str(filename))

[anderspitman]

Yeah I think so. Working on that part now.

[anderspitman]

Looks like I'm getting some typing errors. @alexcrichton do you happen to know the right mypy incantation?

It's failing because the function doesn't exist. But the reason I'm running the function is to see if it exists then catch any errors, which I believe is the pythonic way to do this.

[alexcrichton]

For things like this I've done # type: ignore on the relevant line in the past, I'm not sure what the actual fix might be ...

I think you might also want to move the sys_platform calculation above where you test for sys.platform == 'android'?

[anderspitman]

@alexcrichton I believe this is ready for review. I decided to check for the existance of the function rather than ignoring mypy errors. Apparently it's not necessarily un-pythonic0.

[alexcrichton]

Can you try these wheels and confirm they work?

[anderspitman]

@alexcrichton I spent way to long trying to figure out how to get the briefcase config file to use test.pypi.org and finally gave up (@mhsmith any idea how to do this?), but downloading the wheel manually and using the absolute path seems to be working. Tested on the Android x86_64 emulator.

[mhsmith]

trying to figure out how to get the briefcase config file to use test.pypi.org and finally gave up (@mhsmith any idea how to do this?)

You can use Briefcase's requirement_installer_args option to pass pip's --extra-index-url option. To make it take effect, rerun briefcase create android.

[anderspitman]

Yeah I tried that one but it doesn't seem to be making it past Chaquopy:

> Task :app:generateDebugPythonRequirements FAILED
Chaquopy: Installing for arm64-v8a
Usage:
   /<app-path>/android/gradle/app/build/python/env/debug/bin/python -m pip install [options] <requirement specifier> [package-index-options] ...
   /<app-path>/android/gradle/app/build/python/env/debug/bin/python -m pip install [options] -r <requirements file> [package-index-options] ...
   /<app-path>/android/gradle/app/build/python/env/debug/bin/python -m pip install [options] [-e] <vcs project url> ...
   /<app-path>/android/gradle/app/build/python/env/debug/bin/python -m pip install [options] [-e] <local project path> ...
   /<app-path>/android/gradle/app/build/python/env/debug/bin/python -m pip install [options] <archive url/path> ...
no such option: --extra-index-url https://pypi.org/simple/
Chaquopy: Exit status 2

[mhsmith]

no such option: --extra-index-url https://pypi.org/simple/

Did you specify this as a list of two strings, as shown in the requirement_installer_args?

[anderspitman]

That was it, thanks. Is there any way to tell it to use test.pypi.org only for wasmtime? It seems to be pulling in other dependencies from there and breaking.

[anderspitman]

Also it doesn't look like it's grabbing the correct wheel:

Collecting wasmtime
  Downloading https://test-files.pythonhosted.org/packages/08/3c/4edd9c2e54071ecf120c839294e1ce7583f499b2c6a2fb9c7168fc98b8d6/wasmtime-32.0.0-py3-none-any.whl (6.0 MB)

[alexcrichton]

I'm alas no PyPI expert myself so I don't know how to use it :(

Not downloading the right wheel though is something I was worried about, I'm not sure how to fix and/or debug that though. I'm afraid to call this "mission complete" though until pip or similar can fetch it.

[mhsmith]

It looks like wasmtime-32.0.0-py3-none-any.whl is misnamed – it actually contains a Windows DLL.

The Android wheels are:

• wasmtime-32.0.0.dev360-py3-none-android_26_arm64_v8a.whl
• wasmtime-32.0.0.dev360-py3-none-android_26_x86_64.whl

Because of their filenames, there are a couple more steps necessary to get pip to download them:

• "dev" versions are pre-releases, so you need to either specify the version explicitly (wasmtime==32.0.0.dev360), or pass the --pre option to pip.

• Briefcase defaults to a minimum Android API level of 24, so you'll need to increase that before it will accept a wheel from API level 26.

I was able to install the wheel with the following Briefcase options, followed by rerunning briefcase create android:

build_gradle_extra_content = "android.defaultConfig.minSdkVersion 26"

requirement_installer_args = [
    "--extra-index-url", "https://test.pypi.org/simple",
    "--pre",
]

The first example in the README then ran successfully.

The only problem I can see with the wheel is the wasmtime-32.0.0.dev360.dist-info/WHEEL file:

Wheel-Version: 1.0
Generator: setuptools (80.3.1)
Root-Is-Purelib: true
Tag: py3-none-android_26_arm64_v8a

Root-Is-Purelib should be false, because the wheel is not pure-Python. When it's set to true, Chaquopy will assume the package doesn't need to be installed again for the second architecture, and you'll end up with an app that only contains the arm64 library, not the x86_64 one.

[mhsmith]

Is there any way to tell it to use test.pypi.org only for wasmtime? It seems to be pulling in other dependencies from there and breaking.

I think all indexes are the same priority, so the only workarounds I can think of are to avoid using --pre, or to pin the versions of all your requirements.

[anderspitman]

That's very helpful, thanks.

I managed to get Root-Id-Purelib: false with the following:

from setuptools.dist import Distribution

class BinaryDistribution(Distribution):
    def has_ext_modules(self):
        return True

setup(
    ...
    distclass=BinaryDistribution,
)

but then the wheel filename it spits out is wasmtime-32.0.0.dev360-cp313-cp313-android_26_x86_64.whl, which shouldn't be necessary because it's still py3-none compatibile. Currently trying to figure out the correct setuptools incantation to accomplish both. Python packaging really feels like an opaque game of wack-a-mole.

[anderspitman]

This is a shorter way:

setup(
    has_ext_modules=lambda: True,
)

Still no luck on the tags

[alexcrichton]

For py3-none-any there's some more discussion at #271 but the theory is that any platform which doesn't fit into PyPI's tags would have a DLL bundled in that image. Currently that's just MinGW which is why it's a Windows DLL. As for Root-Is-Purelib I had no idea nad had never seen that before... Agreed it'd be good to fix!

[anderspitman]

Ah maybe has_ext_modules is not the correct solution. I think there's a distinction between a python extension, which is a native program that uses the Python API (ie Python.h), and just using the CFFI, which is what wasmtime does. It makes sense that extensions would require a specific Python ABI.

So I need a way to tell it we're not pure python, without telling it we have extensions.

[anderspitman]

Looking at the setuptools source, it appears the only way to set Root-Is-Purelib false is by setting either has_ext_modules or has_c_libraries to true:

https://github.com/pypa/setuptools/blob/f37845bce6bb06ec25c24cf30210a485e945d21e/setuptools/command/bdist_wheel.py#L246C9-L248C10

Unfortunately setting either of those results in an ABI tag. Currently trying to figure out if that can be overriden separately.

[anderspitman]

Ok if I'm not missing anything, it looks like Root-Is-Purelib: false and py3-none are mutually exclusive:

https://github.com/pypa/setuptools/blob/f37845bce6bb06ec25c24cf30210a485e945d21e/setuptools/command/bdist_wheel.py#L335C9-L359C19

@mhsmith any ideas? Are you certain Root-Is-Purelib is supposed to imply no architecure-dependant code? I agree the name indicates so, but the upstream usage seems to be tightly coupled to the idea of a stable Python ABI specifically for extensions. I think the proper solution here would either be changes upstream (either rename Root-Is-Purelib or change the behavior) or changes to Chaquopy (don't assume Root-Is-Purelib means no architecure-dependant files).

[mhsmith]

Are you certain Root-Is-Purelib is supposed to imply no architecure-dependant code?

I believe so: see this section of the wheel spec.

However, it's possible setuptools can't produce a wheel with this combination of tags and attributes. In which case, you could edit the tag as a post-processing step, maybe by using the wheel tags command.

or changes to Chaquopy (don't assume Root-Is-Purelib means no architecure-dependant files).

Chaquopy could certainly be a bit smarter about this, so I've made some notes about that in chaquo/chaquopy#1374. However, the next version of Chaquopy probably won't be until October.