[wasm32-emscripten] Make EMSDK version part SOABI

[tiran]

Feature or enhancement

I propose to include the Emscripten SDK version in the SOABI for extensions and sysconfig.get_platform, which is used in wheel tag. I also propose to disable abi3.so for Emscripten.

Pitch

Typical compilers like gcc/clang and libcs like glibc provide a stable ABI. Modern VS and msvcrt also have a stable ABI. Code compiled with GCC 11 for glibc 2.34 generally works with code compiled with GCC 12 and glibc 2.35. The manylinux binary wheels make use of stable and common ABIs across Linux distros.

Emscripten does not provide a stable ABI for SIDE_MODULES (shared extensions) yet. That means Python packages must be compiled with exactly same version of EMSDK than the MAIN_MODULE, in our case the Python interpreter (python.wasm and python.js). An extension compiled with EMSDK 3.1.17 may or may not crash when it is loaded into a Python interpreter which is compiled with EMSDK 3.1.18.

If we include the EMSDK version in SOABI and wheel tag and disable abi3.so, then Python will only ever load extensions with compatible EMSDK version.

Implementation

• in configure.ac include __EMSCRIPTEN_major__, minor, and tiny in PLATFORM_TRIPLET
• in Python/dynload_shlib.c add #ifdef __EMSCRIPTEN__ around abi3.

tags

For technical reasons it is not possible to use a dot as version separator. We can either separate by _ or turn 3.1.18 into 30118.

• wasm32_emscripten_3_1_18
• wasm32_emscripten_30118

Previous discussion

emscripten-core/emscripten#15917

CC @hoodmane @rth @mdboom

[tiran]

Update: sysconfig.get_platform() already includes Emscripten version from os.uname():

>>> sysconfig.get_platform()
'emscripten-3.1.19-wasm32'

[rth]

That would be very nice to have indeed!

disable abi3.so

That's the default .so prefix it looks for if it can't find the .so with the exact expected tag, right?

. We can either separate by _ or turn 3.1.18 into 30118

Right now we use the former in Pyodide for the wheel name, and if possible it might be good to keep backward compatibility there. Also, it's more uniquely mapped to the original version. As 30119 could also be 3.11.8 instead of 3.1.18.

On some level, though I do wonder to what extent this system of .so tags applies in the browser. I imagine it arose in a Unix content that could have .so built for different platforms on the same filesystem. In the browser though that should never be the case, and the platform check would already be done when loading the wheel. Though it probably cannot hurt to have this additional check (e.g. if the files were loaded from some archive with .so for the incorrect platform) or if the wheel content is incorrect. That would certainly make the debugging easier.

[hoodmane]

In Pyodide we also sometimes end up with so files built for Linux or for a different emscripten version. Even a few users have had this problem. Detecting this with the file name would be less confusing than some cryptic dynamic linking error.

See igraph/python-igraph#560 (comment) for an example of an incorrectly tagged wheel.

[tiran]

Yeah, I also noticed the .so extension. Let's disable that one as well.

As 30119 could also be 3.11.8 instead of 3.1.18.

3.11.8 would be 31108

#  if __EMSCRIPTEN_minor__ < 10
#   define em_minor 0 __EMSCRIPTEN_minor__
#  else
#   define em_minor __EMSCRIPTEN_minor__
#  endif
#  if __EMSCRIPTEN_tiny__ < 10
#   define em_tiny 0 __EMSCRIPTEN_tiny__
#  else
#   define em_tiny __EMSCRIPTEN_tiny__
#  endif

wasm32-emscripten- __EMSCRIPTEN_major__ em_minor em_tiny

The extra white spaces are stripped away.

[rth]

3.11.8 would be 31108

OK, makes sense if there is a rule. My point is that normal users will not be able to determine from 31108 what version it corresponds to without doing some puzzle solving. So I would vote for the other schema, but it's not very important.

Reminds me of people who thought 051105 as a date format was really straightforward and me trying to figure it out 10 years later in some legacy database :)

[tiran]

You are right, @rth . Let's keep it simple.

[rth]

Also cc @DerThorsten @wolfv

[pmp-p]

i think it's hard to code all the wasm features that can be used in SOABI
why not just have

-mvp-
which will run as far as chrome 81 and has better compatibility on mobile ( think of a manylinux build )
examples here https://github.com/pygame-web/archives/tree/main/repo/pkg

test repl for import here : https://pygame-web.github.io/showroom/pythondev.html?-d

-emscripten-
and expect emscripten builds to always use the last features of latest wasm engines to get the max performances ( bigints/simd/neon/atomics etc .. ) ?

[erlend-aasland]

@brettcannon: with wasm32-unknown-emscripten being unsupported, I would assume we could close this issue as not-planned? Ditto for other emscripten issues.

[brettcannon]

@erlend-aasland probably, but I want to wait until #113632 lands and there's been an alpha in case someone gets inspired to pick it up.

[encukou]

Emscripten is now unsupported. I'll close this and the PR.
Like other Emscripten issues, it'll be listed on the GitHub project for anyone who wants to support the platform unffoicially or bring support back.